After more testing I am beginning to suspect that PFSense is just straight up ignoring the state table when handling this traffic. This is the state table for 10.110.200.12 after performing a reset on the system state table and then re-running the test.

States DCLINTRTG2550 tcp 198.199.98.246:54237 -> 10.110.200.12:8080 SYN_SENT:ESTABLISHED 2 / 2 120 B / 120 B DCLINTSTORJ200 tcp 198.199.98.246:54237 -> 10.110.200.12:8080 ESTABLISHED:SYN_SENT 2 / 2 120 B / 120 B DCLINTRTG2550 tcp 198.199.98.246:54240 -> 10.110.200.12:8080 SYN_SENT:ESTABLISHED 2 / 2 120 B / 120 B DCLINTSTORJ200 tcp 198.199.98.246:54240 -> 10.110.200.12:8080 ESTABLISHED:SYN_SENT 2 / 2 120 B / 120 B DCLINTRTG2550 tcp 198.199.98.246:54243 -> 10.110.200.12:8080 SYN_SENT:ESTABLISHED 2 / 2 120 B / 120 B DCLINTSTORJ200 tcp 198.199.98.246:54243 -> 10.110.200.12:8080 ESTABLISHED:SYN_SENT 2 / 2 120 B / 120 B

If I'm not mistaken, a statefull firewall should be returning traffic out the interface it received it on if it is tracking the TCP state but PFSense does not appear to be doing that. Not sure if another rule somewhere is overriding that but all I have for rules outside of the policy based routes but outside of those rules my only other rules are permit any/any until I can get things working on this.

If you sniff the traffic on the Prod_Front interface, where the destination device is connected to and you can see the outgoing packets, PrepaidCardStatus but nothing comes back, so obviously the device does not respond.

Hello!
Do you have ip address assigned on the vlans in the switch? I have a similar setup and had a similar problem, ssh timed out after 30 sec. I had missed to remove an ip address on the client vlan in the core switch. After removing that it worked fine.

Did you confirm with packet captures that things were taking the proper paths? And check the firewall logs? firewall states?

Have you tried to disable negate rules?
See: https://docs.netgate.com/pfsense/en/latest/book/config/advanced-firewall-nat.html#disable-negate-rules

I had a similar problem with multi-wan routing and it seems to be working as expected after I disabled this.

I just remembered that I didn't close the loop here.

So, it turns out my wife's company uses some L2 VPN and due to a server misconfiguration, I was seeing the vpn client on her laptop misbehave. She raised a ticket with their IT and the rest is beyond our control.
As far as the issue in my network, after turning off the ISP router's wifi and putting all our devices behind pfsense box, I'm not seeing those issues any more.

Phew! The moment I was about to turn off the capture I saw the smoking gun. I was almost getting ready to call the device malicious and return it.

Thanks for helping look into this issue guys. Much appreciated!

Cheers!

@jimp said in Interface Groups vs LAGG: Multi-Wan DNS Streaming Service Problems:

Don't select any outgoing interfaces, so the OS can decide on its own which egress path to use.

OK, I am IMMENSELY grateful for the help...because I would have never thought "all" would be the correct choice. Based on the documentation:

Outgoing Network Interfaces: Specific interface(s) to use for sourcing outbound queries. By default any interface may be used. Can be useful for selecting a specific WAN or local interface for VPN queries. outgoing-interface: <ip address or ip6 netblock> ****If none are given the default (all) is used.****

it would seem "all" would use every interface (including a VPN client which obviously I would NOT want to use generally).

007478c1-d46d-4610-af57-be74654e2a31-image.png

Anyhow, with "all" selected there are NO "outgoing-interface" records in /var/unbound/unbound.conf
dnsleaktest looks good (only primary wan dns being used)
And there are NO DNS queries on the failover WAN.
😂

I would politely suggest a documentation change may be helpful.

Try to create a test condition that places a heavy load on the WAN-side connection (gateway group).
The test should be stronger than your lower performing WAN connection and watch the graphs or log with Zabbix (for example)

There are lots of variables to dig though, but the first thing I would do is dedicate two NIC's to your VM. In other words, make sure the LAN interface has a dedicated NIC that is patched to your switch and isn't being shared with other VM's.

Recheck if your Siemens really need layer 2.
With layer 3 you could run any default Site to Site VPN like OpenVPN in tun mode.

-Rico

@t-np I'll second that...
I ve tried systat -ifstat 1
and I get correct stats like this
6652a2e9-830c-4eea-93ac-2a53c81f4506-image.png

But only this on dashboard
0d1fe276-9f8c-40af-9d72-3ed89c1dc3c5-image.png

I'm also on vlans

@viragomann
Ah, yes those devices do indeed have Windows firewalls on. I thought you meant like actual firewall hardware. My bad. :)

I'll check those too. Anyway, here are the subnet infos I promised.

WAN 1000baseT <full-duplex> 84.*********
LAN1 1000baseT <full-duplex,master> 10.0.0.1/28 (10.0.0.1 - 10.0.0.14 range / 10.0.0.10 - 10.0.0.14 for DHCP)
LAN2 none ......... 10.0.0.17/28 (10.0.0.17 - 10.0.0.30 / Fully allocated for DHCP)
LAN3 1000baseT <full-duplex> 10.0.0.33/28 (10.0.0.33 - 10.0.0.46 / Fully allocated for DHCP)
LAN4 1000baseT <full-duplex> 10.0.0.100/27 (10.0.0.97 - 10.0.0.126 / 10.0.0.101 - 10.0.0.126 for DHCP, 10.0.0.101 Static Leased for Access Point)

@Fehler21 said in Sharing WANs between 2 Pfsense Routers:

you have a dedicated interface/subnet for routing between the two pfsense via the Ubiquiti bridge.

Yes I Do.

Thanks for the info, I will give it a shot and report back.

Thanks!!!

Thanks mate, I'll give a go. Cheers.

Eoin

@Gertjan

yessssss and a pain in the a*%&§ when u ignored that uu knowed it

Covered in jimps Multi WAN hangout: https://www.netgate.com/resources/videos/multi-wan-on-pfsense-23.html

-Rico