As it happens, my question here kickstarted my brain and I got the problem solved: The "Default gateway IPv4" Setting was set to an no longer existing / working gateway-group. Well shit happens... Thank you!

We are now thinking it was the comcast router/modem that stopped routing. As soon as we disconnected another device that was connected to it, it went away.

@viragomann Hi thank you for answering. Yes you right, the screenshot doesn't show it, but it was working up to x.x.100.2, sorry about that. Finally I found the problem. R1 wasn't passing the traffic on to pfSense properly, only ICMP but no more. I changed the command for the static route from - ip route 192.168.20.0 255.255.255.0 g1/0 To - ip route 192.168.20.0 255.255.255.0 192.168.100.2 And it worked beautifully. I suppose that between routers there is no problem with that command but pfSense is in a VM and treated as end device. Not sure but that's the resolution in case anyone else has the same problem. Thank you for you response!

@pclausen said in 2 LB WANs and 2 LANs. Hosts on LAN1 can ping hosts on LAN2, but not vice versa: so I'm not sure what the advantage would be to have 3 gateway groups? For policy routing.. If your group is set as the default in the gateway section, you should not have to policy route it by placing gateway on the rule. But you could then policy route other traffic you wanted to use a specific gateway and not load balance, etc.

@chpalmer I think i am screwing up somewhere on the switches... sigh Thanks for your help~

After more testing I am beginning to suspect that PFSense is just straight up ignoring the state table when handling this traffic. This is the state table for 10.110.200.12 after performing a reset on the system state table and then re-running the test. States DCLINTRTG2550 tcp 198.199.98.246:54237 -> 10.110.200.12:8080 SYN_SENT:ESTABLISHED 2 / 2 120 B / 120 B DCLINTSTORJ200 tcp 198.199.98.246:54237 -> 10.110.200.12:8080 ESTABLISHED:SYN_SENT 2 / 2 120 B / 120 B DCLINTRTG2550 tcp 198.199.98.246:54240 -> 10.110.200.12:8080 SYN_SENT:ESTABLISHED 2 / 2 120 B / 120 B DCLINTSTORJ200 tcp 198.199.98.246:54240 -> 10.110.200.12:8080 ESTABLISHED:SYN_SENT 2 / 2 120 B / 120 B DCLINTRTG2550 tcp 198.199.98.246:54243 -> 10.110.200.12:8080 SYN_SENT:ESTABLISHED 2 / 2 120 B / 120 B DCLINTSTORJ200 tcp 198.199.98.246:54243 -> 10.110.200.12:8080 ESTABLISHED:SYN_SENT 2 / 2 120 B / 120 B If I'm not mistaken, a statefull firewall should be returning traffic out the interface it received it on if it is tracking the TCP state but PFSense does not appear to be doing that. Not sure if another rule somewhere is overriding that but all I have for rules outside of the policy based routes but outside of those rules my only other rules are permit any/any until I can get things working on this.

If you sniff the traffic on the Prod_Front interface, where the destination device is connected to and you can see the outgoing packets, PrepaidCardStatus but nothing comes back, so obviously the device does not respond.

Hello! Do you have ip address assigned on the vlans in the switch? I have a similar setup and had a similar problem, ssh timed out after 30 sec. I had missed to remove an ip address on the client vlan in the core switch. After removing that it worked fine.

Did you confirm with packet captures that things were taking the proper paths? And check the firewall logs? firewall states?

Have you tried to disable negate rules? See: https://docs.netgate.com/pfsense/en/latest/book/config/advanced-firewall-nat.html#disable-negate-rules I had a similar problem with multi-wan routing and it seems to be working as expected after I disabled this.

I just remembered that I didn't close the loop here. So, it turns out my wife's company uses some L2 VPN and due to a server misconfiguration, I was seeing the vpn client on her laptop misbehave. She raised a ticket with their IT and the rest is beyond our control. As far as the issue in my network, after turning off the ISP router's wifi and putting all our devices behind pfsense box, I'm not seeing those issues any more. Phew! The moment I was about to turn off the capture I saw the smoking gun. I was almost getting ready to call the device malicious and return it. Thanks for helping look into this issue guys. Much appreciated! Cheers!

@jimp said in Interface Groups vs LAGG: Multi-Wan DNS Streaming Service Problems: Don't select any outgoing interfaces, so the OS can decide on its own which egress path to use. OK, I am IMMENSELY grateful for the help...because I would have never thought "all" would be the correct choice. Based on the documentation: Outgoing Network Interfaces: Specific interface(s) to use for sourcing outbound queries. By default any interface may be used. Can be useful for selecting a specific WAN or local interface for VPN queries. outgoing-interface: <ip address or ip6 netblock> ****If none are given the default (all) is used.**** it would seem "all" would use every interface (including a VPN client which obviously I would NOT want to use generally). [image: 1587751277712-007478c1-d46d-4610-af57-be74654e2a31-image.png] Anyhow, with "all" selected there are NO "outgoing-interface" records in /var/unbound/unbound.conf dnsleaktest looks good (only primary wan dns being used) And there are NO DNS queries on the failover WAN. I would politely suggest a documentation change may be helpful.

Try to create a test condition that places a heavy load on the WAN-side connection (gateway group). The test should be stronger than your lower performing WAN connection and watch the graphs or log with Zabbix (for example)

There are lots of variables to dig though, but the first thing I would do is dedicate two NIC's to your VM. In other words, make sure the LAN interface has a dedicated NIC that is patched to your switch and isn't being shared with other VM's.

Recheck if your Siemens really need layer 2. With layer 3 you could run any default Site to Site VPN like OpenVPN in tun mode. -Rico