Usually the last visible rule created is what's allowed out. I say visible because the actual last rule is to block everything (built in / default rule).

So if you create a pass rule at the end of the list to allow, (your previous rules will block to LAN, WiFi, VPN) then you should be sweet.

I would use Split DNS there.

I fixed the problem by removing the virtual ip, then putting it back in.

You will need a routing protocol of some kind, there isn't likely to be any other way connecting to a third party could manage two separate tunnels which are up all the time.

The failover is much slower but you could maybe get away with only having one tunnel set to use a hostname for the remote gateway, and Dynamic DNS on the Fortigate side set to switch the hostname depending on which WAN is preferred at the time. That's assuming Fortigate supports that kind of function for Dynamic DNS.

@mohkhalifa said in Gateways drop:

Problem Solved by un-check "Flush all states when a gateway goes down" in the advanced setting.

That box is not checked on my pfSense. I think most of that is still set at the defaults. Never had a reason to change it.
At some point if all this doesn't get resolved, I'll just revert to pfSense 2.4.4 p3 till it's fixed.

@viragomann Thanks alot, this worked. So dumb of me :)

@Rico The video you pointed me to was very useful, and I now have MultiWAN set up, mostly working. The two weirdnesses I have are:

I have a group set with Tier 1 on my Fiber connection, and Tier 2 on my cable connection. My LAN uses this group. When I pull the cable on my Fiber it seems that TCP connections fail over, but things like ping (ICMP etc) out the failed over to WAN do not. I can't see where that is set :-(

Second issue is that I have my mail server in a subnet called DMZ, off it's own port from the router. It is set to have it's WAN traffic to only go to the cable connection - which works. What I cannot do is get it to make connections to the LAN. Even if I set up a rule for DMZ-net to LAN-net all I can't ping or ssh from DMZ. I really need/want just Bonjour and ping to be able to initiate DMZ->LAN but I can't even get DMZ-net -> LAN-net all to work, even though on the LAN I have LAN-NET - RFC1918 set and working, and I did try it as LAN-net -> DMZ-net and that works too.

Thank you
Cheers, Liam

Regarding the NordVPN issue, i use 2 NordVPN connections and one VPN connection from another provider at the same time on one pfSense gateway. I have checked the "Don't pull routes" and "Don't add/remove routes" as those would be problematic with the pfSense configuration. I have not experienced any DNS resolver issues. As it is now, i only have one internet connection.

The Netflix issue have been described in other posts using pfBlockerNG.

Hope it helps a little.

Right. Chances are the ISP was intermittently forwarding the traffic to you so you were therefore intermittently able to forward the traffic inward. Can only work on what you receive. Glad it looks like they found it.

@Jeonetgate Having this same issue as well on 2.4.4p3. No flush settings are enabled. Traffic is getting blocked by this default rule. ICMP traffic about 75% of it gets lost due to this. Do have two WANs configured.

I've switched over to AON NAT now, I didn't originally see all of the rules. Turned out that I didn't have an upstream gateway set. Once Set I see the rules automatically generated. I added my rule to disable NAT to create a routed network from my two servers.

I still see an issue however. Can anyone suggest a fix or maybe a better way to achieve what I'm trying to do? I'll try and detail what the issue is...

Screen Shot 2020-03-25 at 3.41.24 PM.png

So it seems like it's an L2 issue...

Here are the test details... Ping from Site1 Nested ESXi VM fails...

The path is: Request: "n-esx1" -> "p-esx1" -> "p-esx2" -> "n-esx7" Reply: "n-esx7" -> "p-esx2" -> dfGW.... where I need it to go back to "p-esx1"
n=Nested
p=Physical

[root@stevelab-n-esx1:/vmfs/volumes] ping 192.168.2.17
PING 192.168.2.17 (192.168.2.17): 56 data bytes

For reference...
Site1 WAN: 60:ac:a6
Site2 WAN: 7b:81:88
WAN GW: ff:fd:90

[root@stevelab-n-esx1:/vmfs/volumes] ping 192.168.2.17
PING 192.168.2.17 (192.168.2.17): 56 data bytes

Destination Nested ESXi sees the request and replies to the request.

[root@stevelab-n-esx7:/vmfs/volumes/3a3b5bc8-88bd4760] pktcap-uw --uplink vmnic0 --dir 2 --ip 192.168.1.11 -o - | tcpdump-uw -enr -
22:23:35.839598 00:0c:29:7b:81:9c > 00:0c:29:35:9b:0d, ethertype IPv4 (0x0800), length 98: 192.168.1.11 > 192.168.2.17: ICMP echo request, id 55660, seq 0, length 64
22:23:35.839758 00:0c:29:35:9b:0d > 00:0c:29:7b:81:9c, ethertype IPv4 (0x0800), length 98: 192.168.2.17 > 192.168.1.11: ICMP echo reply, id 55660, seq 0, length 64

Then the Physical host is sending the reply to the WAN GW. It doesn't send it back from where it came...

[root@stevelab-p-esx2:/vmfs/volumes] pktcap-uw --uplink vmnic0 --dir 2 --ip 192.168.1.11 -o - | tcpdump-uw -enr –
22:24:09.239994 00:0c:29:60:ac:a6 > 00:0c:29:7b:81:88, ethertype IPv4 (0x0800), length 98: 192.168.1.11 > 192.168.2.17: ICMP echo request, id 55660, seq 18, length 64
22:24:09.240589 00:0c:29:7b:81:88 > 00:08:e3:ff:fd:90, ethertype IPv4 (0x0800), length 98: 192.168.2.17 > 192.168.1.11: ICMP echo reply, id 55660, seq 18, length 64
22:24:10.241698 00:0c:29:60:ac:a6 > 00:0c:29:7b:81:88, ethertype IPv4 (0x0800), length 98: 192.168.1.11 > 192.168.2.17: ICMP echo request, id 55660, seq 19, length 64
22:24:10.242331 00:0c:29:7b:81:88 > 00:08:e3:ff:fd:90, ethertype IPv4 (0x0800), length 98: 192.168.2.17 > 192.168.1.11: ICMP echo reply, id 55660, seq 19, length 64

[root@stevelab-p-esx2:/vmfs/volumes] esxcli network ip neighbor list
Neighbor Mac Address Vmknic Expiry State Type
10.33.72.65 00:0c:29:60:ac:a6 vmk0 928 sec Unknown
10.33.72.64 00:0c:29:8c:15:73 vmk0 1196 sec Unknown
10.33.72.62 00:11:32:a6:9a:3f vmk0 1020 sec Unknown
10.33.75.253 00:08:e3:ff:fd:90 vmk0 1198 sec Unknown

Reply never arrives back at Site1 (Of course, because the packet went to the WAN GW of Site2.

[root@stevelab-p-esx1:~] pktcap-uw --uplink vmnic0 --dir 2 --ip 192.168.1.11 -o - | tcpdump-uw -enr -
22:24:24.270369 00:0c:29:60:ac:a6 > 00:0c:29:7b:81:88, ethertype IPv4 (0x0800), length 98: 192.168.1.11 > 192.168.2.17: ICMP echo request, id 55660, seq 33, length 64
22:24:25.271133 00:0c:29:60:ac:a6 > 00:0c:29:7b:81:88, ethertype IPv4 (0x0800), length 98: 192.168.1.11 > 192.168.2.17: ICMP echo request, id 55660, seq 34, length 64
22:24:26.271396 00:0c:29:60:ac:a6 > 00:0c:29:7b:81:88, ethertype IPv4 (0x0800), length 98: 192.168.1.11 > 192.168.2.17: ICMP echo request, id 55660, seq 35, length 64

Willing to try just about anything...

Thanks.

Well whatever was going on seems to be transient as things seem to be working now, although with the current situation extremely slow and laggy.

@Derelict Thanks bud, we just tested what you said and it worked beautifully. The process was a bit tricky to understand because they used Unifi on the opposite side but after letting go of preconceptions and just tried everything just popped into place.