@nmeth Of course I have now found the answer myself...

I did not have the "Skip rules when gateway is down" checkbox checked in the Advanced/Miscellaneous/Gateway Monitoring settings.

Information is at https://docs.netgate.com/pfsense/en/latest/config/advanced-misc.html#skip-rules-when-gateway-is-down

The new interface and routing through there resolved the slowness. Thanks for the help!

@viragomann

Site B Gateways
ec567ad8-ab56-434b-8ef3-5b696c41c567-image.png
I need to route via site1gw.

This is Site A Gateways
3b9632b9-bde1-4093-9e45-a1f45d336b57-image.png

Site A static route
56037d33-f0b5-4348-839e-0d24360d5ecb-image.png

Site B static route
ee1cbffe-7c37-4258-8a18-8141ca19d98c-image.png

@aiden21c it’s always the last place you look…

@keyser i dont use NtopNG

My current installed packages:
acme, cron, haproxy, openvpn, pfBlockerNG, snort, wireguard, zabbixagent

My current machine:
Supermicro Server 1HE
Intel(R) Xeon(R) D-2123IT
16GB DDR4
Intel SSD

2 different ISP WAN Interfaces
I have Failover Gateway Groups (trigger: Member down)

Update:
When my primary WAN is going down, the problem disapears.
If the primary is again enabled it gains 15ms...

@cyberzeus Ok so I made this testing with the same set up as before and then changed the following:

A rule on the main pfsense to block all ICMP on the TestVLAN (kill states required for it to "kick in"). Trigger Level set to Packet Loss State Killing on Gateway failure:
a. Kill states for all gateways which are down
b. Flush all states on gateway failure

Regardless of 3a or 3b, I see the exact same behaviour as before. When invoking the rule on the main pfsense, "Loss" starts to rise and soon after passing 20+, it switches over to WAN2.

Spikes now start to show up on the WAN2 graph and whatsmyip shows my correct LTE IP.
Toggling the rule off, and "Loss" goes down again and seconds after WAN indicates online, traffic shifts back and whatsmyip shows my fiber IP.

The only thing when using "Flush all states" (which affects LAN side states as well) is that the pfsense GUI appears to freeze for ~15 seconds before that session reengages. Using "Kill states" isn't noticed at all from a LAN to LAN perspective. This was of course true in my previous testing as well...

@chpalmer
Interesting. I’d assumed that being down line from the modem (only one connection to the service) connection of anything after that was like a switch distributing to as many devices as you want.
Being a numpty I took 5 years to realize I could turn off the Christmas lights on the old arris !
Still trying to find a way to force pfsense to take the address straight from the modem tho

@coreycoop If you are policy routing LAN1 to WAN1 and LAN2 to WAN2 then you need to bypass policy routing for LAN1 to LAN2 and LAN2 to LAN1.

Put a rule on LAN1 above your policy routing rule that passes the desired traffic from LAN1 to LAN2 without a gateway set.

Same for LAN2 to LAN1 on the LAN2 interface.

@viragomann Thanks. I did check that but its not clear to me on how to achieve what I'm looking for with those settings. That only seems to modify the monitoring probe thresholds but not a grace period to when the interface is monitored as OK. I want to wait say 10 minutes after the probe is good before the interface is used again for traffic.

@viragomann thanks you are right. I misunderstood that part. Only the machines on the LAN not the WAN side.

@irondog said in Routing OpenVPN to LAN:

DNS in my setup

please open another topic !
gonna be a lot of people to help u yout with dns issues

br NP

If the VPN user needs to access the office network as well as the 172 network, the tunnel needs to have both 172.16.0.0 /24 and 192.168.0.0 /24 as allowed IP's.
And under System > Routing > Static Routes in pfsense, you need to have both these IP's. The difference will be that 172 will use WAN2 as the Gateway and the 192 IP will have WAN1 as Gateway.

I actually have a somewhat similar setup at our cottage. I have a site to site wireguard tunnel set up and we use an LTE-router for failover in the cottage. So the pfsense router there has two WAN ports, one with a public IP, and one with a private IP from the LTE router.

To be able to access the LTE routers management interface, I have set a static route for 192.168.2.0 /24 towards WAN2 (the LTE router).

The only difference here would be that where I have a public IP, you have the office router in between. And having double NAT may present a problem in itself. You obviously need to do a port forward for the VPN tunnel towards pfsense...

@johnpoz
Yep, my WAN is 10gbps

@brewha12 both direct to their respective ISp provided modems

@gut733
OK, I realized that I am a total noob :D
I connected test client on default pfsense setup in client 1-3 subnet and there is everything ok. I can ping all allowed hosts in the network.

So it brings me to question, why can't I ping from LAN to any client in OPT1 subnet.

@galcorlo
You have two WANs from the same ISP?
What's the WANs connected to? To an ISP modem, cable, DSL?

What do the WAN settings show in Status > interfaces?

Apart from the PPPoE are there additional IPs assigned to the WAN interfaces?