@adamw Now that I’m by a PC, bwlimit is Kbytes per second. Somewhere I also recall that rsync doesn’t necessarily limit at a constant speed:

“Rsync writes data over the socket in blocks, and this option both limits the size of the blocks that rsync writes, and tries to keep the average transfer rate at the requested limit. Some “burstiness” may be seen where rsync writes out a block of data and then sleeps to bring the average rate into compliance.”
https://www.cyberciti.biz/faq/how-to-set-keep-rsync-from-using-all-your-bandwidth-on-linux-unix/

That page also has other possible solutions.

Or as I mentioned, traffic shaping to make this low priority traffic.

@viragomann do you have layer one? (Link lights)

@rune-san said in Shutdown WAN when Reaching Data Cap?:

@Gblenn All of AT&T's DSL/Fixed Wireless/Fiber plans below 100Mbps plans have Data Caps. Same with Cox cable internet. I agree it's not common when looking across say, a large part of the US, but for those that are in that location, it's probably the only choice they've got.

Hmm, makes sense when they offer connections over a shared resource like FWA, but fiber (or cable)... seems like a way to force people to pay extra, just because they can. Or to force people to abandon costly (for the operator) DSL.

Globally though, quite unusual...

@oscar-pulgarin said in Problem routing:

When I do a traceroute from the ip 10.10.10.1 it follows the following path:
10.20.35.1
10.250.1.2

So I assume, that the Sophos is either the default gateway on the Forti or there is a static route in place on the later for 192.168.0.65.

Additionally you need two static routes on the Sophos for this to work.
One for 192.168.0.65 pointing to 10.250.1.2 and one for 10.10.10.0/24 pointing to the Fortinet.

AND you have to configure an IPSec phase 2 to connect 10.10.10.0/24 and 192.168.0.65, presuming none of the involved devices does masquerading the traffic.

@martincutts said in AWS S3 Sync Not Working:

Hi everyone,

I wonder if anyone has come across this issue before?

Background
We have well over 100 AWS accounts which we setup for customers who run our ERP software from within AWS. For a number of years we have been using a Vyatta instance for VPN connectivity when the customer has multiple branches, which is a lot cheaper than multiple AWS VPN's. The issue is that the old Vyatta doesn't support IKEv2, so I looked for a replacement.
Initially I looked at VyOS, which did seems to work OK apart from a few niggles, but then chose to adopt pfSense due to better OpenVPN support.

The Issue
For one new customer, I'd already setup a VyOS device which was working just fine, and then replaced it with a pfSense device which all seems to be working fine, however.

We have a script which backs up the PostgreSQL databases, and then uploads the backup files to an S3 bucket which has versioning enabled, so we can go back to various days in the event that historic data is required.

For some reason the 'aws s3 sync' command which uploads the files has stopped working. It was working just fine with a VyOS and Vyatta, and stopped working on the day I replaced it with a pfSense.
This is where it gets interesting as when you run the backup script manually (as the postgres) it runs just fine and the files upload, but when it runs via a daily cron job it doesn't upload :-(
Before you say it, this is not a pathing issue as the same script has been running OK for years on multiple systems.
The only change is the replacement of the VyOS with a pfSense. I know for a fact that if I replace the pfSense back with the VyOS it will start working again.

So my question is, what is causing the problem? I've been looking at this for days and getting nowhere!

There is nothing in the Firewall log to suggest it's being blocked, but surely if it was a FW issue it would be blocked when running manually?

At some point we will need to replace all the Vyatta instances (over 100) with pfSense devices, so need to know these are as solid as the Vyatta appliances they will be replacing or I need to go back to a VyOS?

Thanks

Hi there,

It sounds like you're encountering an issue with the 'aws s3 sync' command after replacing a VyOS device with a pfSense one in your ERP software environment on AWS. The script runs successfully when triggered manually but fails to upload files when executed via cron job. This behavior suggests a potential interaction issue between pfSense and your script execution environment. While firewall logs show no blocking activity, the change to pfSense might be affecting how cron jobs or script permissions are handled. You might want to check how cron jobs are configured on pfSense, ensure proper permissions are set for the script execution, and verify any networking or routing settings that could be impacting the AWS S3 connectivity. Considering your scale and the need for reliability across multiple instances, exploring configuration nuances between Vyatta, VyOS, and pfSense could provide insights into resolving this issue effectively.

I did not just try a DHCP request. Here is what I tried:

Disable / enable interface to force a DHCP request - Failed Reboot pfSense firewall - failed Reboot cable modem - failed One more reboot of pfSense - Sucess

So, I think that because the modem powers up before the ISP can provide a DHCP address, it is left in a state where my firewall will not get an address until the modem has been reset.

As I said, hopefully, I will be home next time this happens so I can determine what the problem is and put something in place to auto-correct the problem in the future.

But some of your thoughts on how to power cycle the various devices are helpful.

Thanks

@Gertjan said in How do I route outgoing email over WireGuard Tunnel?:

Of course I use have DANE available and set up :

I just noticed I had to recreate the TLSA records, something with Let's Encrypt must have changed. I hope I am good now for some time...

Figured it out. A cpl errors

Set static IP for 2nd Wan but also set the Gateway IP the same as the 2nd wan ip. Changed to the correct Gateway IP for the 2nd Wan and was able to connect.

Subnet mask range was incorrect on the device itself 192.168.0.20 was set to 255.255.255.0 when it should have been set to 255.255.0.0

The big thing was the wrong gateway IP. Setting it the same as the 2nd Wan IP is a definite no go situation. Not sure how that slipped by me but if it helps someone else I'll leave it up here.

@viragomann Thanks!

I did it the way someone said to on yootoob and we all know how anyone making videos there is always correct.. 🤣

@FCS001FCS Very helpful, thanks. Seems i'm on the right track which is encouraging. My assumption is that my situation is reversed and i may have to pick up more than the Netflix ASN (in this case) since they use AWS. I wonder if they authenticate on their own servers before handing over to AWS...

Thanks again.

So, I'm an idiot.

I never thought to just restart my modem. That fixed the problem without any configuration changes.

@QuietEnergy9215 No problem and glad you're back up and running.

Hi @viragomann

Thank you for your answer. This is what thougt to do at first. For whatever reason it doesn't work for me here. I ordered another /29 subnet and will check if it works with it. It didn't arrive yet.

@pfsense1921 said in HAproxy will not connect to remote server over IPSEC VPN:

Are you saying this works with OpenVPN Tunnel?

Yes, presumed you obey some setup steps.

At the remote site you will have to assign an interface to the respective OpenVPN instance and move over the firewall rule from the OpenVPN tab to it.

You have to ensure that there is no pass rule on the OpenVPN tab or even a floating rule applied to the forwarded traffic from the remote site!
This is necessary for the reply-to to work, so that pfSense can send the response packets back to the other site.

Just in case anyone else ends up in the same place as me..............

I managed to solve the issue and found there was no access to the internet on the PRINTERS VLAN, despite having the firewall rules allowing it!

Eventually I stumbled upon the Firewall NAT Outbound rules, which was populated with 2 rules per VLAN (had been done automatically, presumably when setting up the VLANs).

These rules did not exist for the PRINTERS VLAN. Added these rules manually and all sprung into life!

43fb5a5f-11d8-437e-b9a7-dabcee21984d-image.png

What I cannot understand is why these had not been created in the first place!! The outbound NAT mode was showing 'manual outbound NAT rule generation'.

492154ac-6783-4375-9f47-bc2b5809d6c9-image.png

Thought this would have been set to Auto??

Never mind, all working now but at least I've picked up some extra knowledge. Thanks to everyone for helping me sort this out, it is greatly appreciated.

Steve

@michmoor
Yes, I have a WAN gateway (ISP). For Internet B, I would need another gateway using the LAN interface. The WAN gateway would be Tier 1, the LAN gateway would be Tier 2. Right?

@viragomann Thanks. I think I have it running. I typed in "ntpq -pn" on my Linux Minecraft server (On Guest LAN) and it spewed out all of the NTP servers I have configured on my pfSense box (All clients/LAN's use 192.168.1.1 as NTP server). However, I did not use any firewall rules to allow this to happen. Is it just happenstance that it works, or am I supposed to add a firewall rule?