Was hoping to see some other responses, but clearing the rules and re-adding them in a specific order fixed my problem.

@chpalmer I tried. Unfortunately not.

@bhjitsense
Here is a screenshot of where you can access those settings. Go to the location shown for your Cellular gateway. Then click on the "Display Advanced" button at the bottom.

1b341c15-c6ba-4103-8ebf-b2d3dc33ab78-image.png

After expanding the advanced options, you will see the screen below.

dbac8604-8e10-4a8f-8a70-cd96184f6613-image.png

In order to do what you're requesting, it sounds like you have to change the lower and upper Packet Loss threshold to 100.
The Time Period is the time that the sampled data is averaged over. This is set to 1 minute (60000). Therefore, with that remaining with the default value, the 100 percent loss would have to occur for more than a minute since the 100 percent loss would be averaged in with the time before the failure and the time after the failure which had zero loss or close to it. At least that's what I would think. I could be wrong.

Alternatively, you could do what I did and simply Disable Gateway Monitoring all together for the Cellular link. In my case I don't care if my backup Cell link is down since there's not much I could do if that happened to go down at the exact same time my main link went down anyway. That's just a scenario I'm willing to accept. If I wanted anything more bullet proof, I would invest the money in a much more robust solution.

@viragomann that did the trick thank you so much!

A little help would be appreciated.

Learned quite a bit today ;-) and it works. Thanks for pfsense and the docs!

@rico said in Multi-WAN with same Gateway:

Nice....would you mind share?

-Rico

1- You must have router with a Multi-NAT option
2- You have to configure 1:1 Multi-NAT in your router with any IPs from your router's subnet
ex:
WAN IP-01 x.x.x.x = 192.168.1.11
WAN IP-02 x.x.x.x = 192.168.1.12
WAN IP-03 x.x.x.x = 192.168.1.13
3- If needed, Forward the ports for IPs we already made in step 2 and at the same time allow the ports from the router's interface in pfSense
4- Create a Virtual IPs for each IP we already made in the Multi-NAT configuration "Private IPs".
5- Create pfSense 1:1 NAT rules for each private IP to our desired Local Server IP
ex:
Private IP 192.168.1.11 = Local Server IP 172.16.10.11
Private IP 192.168.1.12 = Local Server IP 172.16.10.12
Private IP 192.168.1.13 = Local Server IP 172.16.10.13
6- Create a firewall LAN rule to pass the traffic and MUST choose the gateway for the interface we are working with.
7- check the firewall Rules for opened ports in the router's interface we allowed in step.
That's all

If your connected to an L2 be it a /22 or a /24 or even a /8 running on it, then yes your going to see arp entries for other devices on this L2 network.. This is how networking works..

The only way to not see other mac address for devices on the same L2 as you would be for the ISP to filter that on their network..

@johnpoz said in Add route to host:

Under system routing, when you add a gateway - bottom of page advanced. You can set that the gateway is not local to your wan subnet

Thank you, @johnpoz ! This was too simple. 🙄

OK, I finally found some time to dig into this. Writing this followup in case anyone runs into a similar issue in the future.

Some investigation with tcpdump showed that the remote host was seeing checksum errors on the TCP packets (yet ICMP was working fine).

Disabling hardware checksum offloading in pfSense resolved my issue (System -> Advanced -> Networking -> Hardware Checksum Offloading).

pfSense is running under XCPng 8.1.0 on an unknown motherboard using onboard Intel 82576 Gigabit NIC.
Not sure if the checksum error is creeping in due to XCP or NIC drivers, but in my case the network traffic is quite low, so I'll run with the hardware checksum disabled.

For my wireless setup, I altered the wifi access points to allow broadcast traffic from the pfsense box. In most cases, the wiresless drops broadcast packets from the LAN side.

I also increased the time for the ARP expiration.

Since doing the above my network seems a bit more stable, but I still don't feel like we've gotten to "root" cause.

i have this problem to
:(

@whosmatt Interesting, I did not know that gateway groups could span multiple devices. How do I do add a gateway from another Firewall to the group? Or do I just create a group with one Gateway on each firewall?