I have the same behavior in a similar setup. Did you ever manage to find out what was causing this?

@malicair
9.9.9.9 is not responding to ping requests. So you cannot use this IP for monitoring. Use another one.

For instance 1.1.1.1.
Try to if you get a response on your PC.

ping 1.1.1.1

If it's okay use it for monitoring in the VPN gateway settings.

Danke für deine Antwort. Ich habe das jetzt anders geregelt. Über Lan und das geht.

@jpvonhemel Yes, some changes will trigger a restart. Some auto-update scripts as well.

@jc2it said in Asymmetric routing with multi WAN and OpenVPN:

Dec 8 14:38:25 php-fpm 50688 /rc.filter_configure_sync: Not installing NAT reflection rules for a port range > 500

@mrsunfire Do you have this message in your "Status/System Logs/System/General"

No.

Its not really that issue I think its actually closer to this https://forum.netgate.com/topic/152745/multi-wan-gateway-option-gets-ignored-in-firewall-rule I may try this later on. It does seem to be skipping the default gateway route in the firewall rule.

Well, YAHOO! I got my system working like I want it to.

I want to express sincere thanks to all who had the patience to point me in the right direction.

johnpoz, Next time you're in my area, get in touch with me. I'll take you out for a nice Buffalo steak!

Bart

@viragomann
OK, seems that I have full connectivity working now. :)

I created rules for both the WAN and LAN interfaces allowing the traffic for the 10.0.0.0/8 network. Initially I had a mistake in only allowing TCP, which showed up in the syslogs so changed that to ANY and now my clients are connecting.

After multiple days of chasing the configuration I'm quite happy that it's now working. THANKS MUCH!!

Now onto my next step of getting the NORDVPN working. (AFTER SAVING MY CONFIG!)

Cheers and have a great day!

@chpalmer thanks, yep. Both are "dumb" modems; the only purpose of the admin interface is for diagnostics. Funny, although the SB8200 is capable of bonding with updated firmware, the ISP refuses to apply it. So, I'm stuck with two separate GigE ports. Not a big deal since that service is only 600/50, but it highlights the asinine nature of DOCSIS.

I do actually have two separate ISPs (WOW and "Comcrap"). I live stream some classes and just wanted to make sure I had redundancy so I don't leave my students high-and-dry. But I've had nothing but trouble ever since adding the Xfinity service. Got the MB8611 for its 2.5GbE port since I had > 1Gbps with Comcast, but c'est la vie.

@scilek said in Load balancing does not utilize one of the gate ways.:

Double NATting? No, I could not do that.

No, triple-NAT. 😊
You have double already at this time. One time NAT happens on pfSense and one time at your ISP.

This seems similar to https://redmine.pfsense.org/issues/11733, which was closed for no reason (the issue was with ONT, not LAN link, so there was no reason fro web UI to stop responding).

This apparently affects all interfaces regardless of which gateway goes down.

Right now I have WAN as Tier 1 and WAN2 as Tier 2. When WAN2 (second ISP) has packet loss, I both lose Internet connectivity using WAN and web UI becomes unresponsive.
Not always, but often, which is especially annoying during video calls.

Here are the logs from the last time it happened:

Spoiler

Dec 3 07:35:44 nginx 2022/12/03 07:35:44 [crit] 39955#100173: *31629 SSL_write() failed (13: Permission denied) while processing HTTP/2 connection, client: 192.168.1.2, server: 0.0.0.0:443 Dec 3 07:35:11 php-fpm 30078 1.0.0.1|redacted|WAN2_DHCP|2.429ms|0.833ms|0.0%|online|none Dec 3 07:35:11 php-fpm 30078 /rc.openvpn: MONITOR: WAN2_DHCP is available now, adding to routing group MultiWAN Dec 3 07:35:10 check_reload_status 381 Reloading filter Dec 3 07:35:10 check_reload_status 381 Restarting OpenVPN tunnels/interfaces Dec 3 07:35:10 check_reload_status 381 Restarting IPsec tunnels Dec 3 07:35:10 check_reload_status 381 updating dyndns WAN2_DHCP Dec 3 07:35:10 rc.gateway_alarm 44475 >>> Gateway alarm: WAN2_DHCP (Addr:1.0.0.1 Alarm:0 RTT:2.444ms RTTsd:.829ms Loss:0%) Dec 3 07:35:00 sshguard 42588 Now monitoring attacks. Dec 3 07:35:00 sshguard 48246 Exiting on signal. Dec 3 07:34:34 php-fpm 30078 1.0.0.1|redacted|WAN2_DHCP|2.533ms|0.65ms|13%|down|highloss Dec 3 07:34:34 php-fpm 30078 /rc.openvpn: MONITOR: WAN2_DHCP has packet loss, omitting from routing group MultiWAN Dec 3 07:34:34 check_reload_status 381 Reloading filter Dec 3 07:34:34 php-fpm 62018 /rc.newwanip: rc.newwanip: on (IP address: redacted) (interface: WAN2[opt1]) (real interface: vtnet1). Dec 3 07:34:34 php-fpm 62018 /rc.newwanip: rc.newwanip: Info: starting on vtnet1. Dec 3 07:34:33 check_reload_status 381 Reloading filter Dec 3 07:34:33 check_reload_status 381 Restarting OpenVPN tunnels/interfaces Dec 3 07:34:33 check_reload_status 381 Restarting IPsec tunnels Dec 3 07:34:33 check_reload_status 381 updating dyndns WAN2_DHCP Dec 3 07:34:33 rc.gateway_alarm 41178 >>> Gateway alarm: WAN2_DHCP (Addr:1.0.0.1 Alarm:1 RTT:2.530ms RTTsd:.653ms Loss:11%) Dec 3 07:34:33 check_reload_status 381 rc.newwanip starting vtnet1

I didn't have this issue before Multi-WAN. Nginx error is especially concerning. That was me trying to refresh frozen page, but I was unable to do so.

@bob-dig said in Multicast traffic between LAN interfaces on different subnets:

I think you should solve it by putting all the devices in the same subnet. If you need a switch for that and maybe a wireless access point, both with vlan support, then get those. A firewall isn't a switch.

I agree with the last one. However, a switch cannot filter anything normally, but pfSense can, even on bridged interfaces sharing the same L2.

So there are specific circumstances, where a bridge may be the preferred solution.

@viragomann

Thank you for responding, I will proceed with duplicating the rules then.

so upon disabling and re enabling the WAN interface this is when i see the issue occur. the only action that can be taken it seems is to manually select the gateway removing it off the automatic option. restarting the gateway service nor reboot changes its behaviour.

Running on 2.6.0-RELEASE (amd64) wonder if anyone else is getting the same issue?

@rcoleman-netgate Thank you very much