@alvescaio can I help me?

@indiankid Have you looked at tnsr? I suspect between tnsr and multiple pf installs you'd be best situated.

I have even tried replacing hardware such as Hdds to 2 brand new SSD WD RED in Zfs "raid 1" and replaced Ram. Same thing happens when enabling the option........

I would verify with a packet capture that the traffic is crossing into the pfSense side properly.
What's the LAN interface VLAN in the that list? 4084? 4083? 4082?

In order for replies from those servers to go back over the tunnel you will need the states to be tagged reply-to unless the tunnel is the default route.
That usually means the tunnel must be a routed type and the states must be opened by firewall rules on the assigned interface.
So I'd start out by using OpenVPN and assigning the house side interface.

Steve

@ansarmehmood said in wan settings:

when i select staic on wan side it only ask for ip address and subnet mask

The gateway is requested as well, either in the initial setup in the console or when you configure the interface in the GUI.
In the GUI click "Add new gateway", then you're able to enter it.

The routed host IPs can be directly used in port forwarding rules as destination addresses.

@aziz-1 said in Static route over OpenVPN VPN:

The other way only requests

So that's the point where you should check the firewall on the destination device.

Do you get responses if you ping a LAN2 device from LAN1 pfSense?

To be sure, are both VPN endpoints the default gateway in their local networks?

@akuma1x simply blocking ip spoofing :)

I encountered another issue with dpinger. I changed one unimportant parameter in the WAN interface (CIDR length for the alias IPv4 address). Pressed save and apply and noticed that my IP switched over to the failover WAN2, then back to WAN1. I looked at the logs and found this:

php-fpm 89297 /interfaces.php: The command '/usr/local/bin/dpinger -S -r 0 -i WAN_DHCP -B x.x.x.x -p /var/run/dpinger_WAN_DHCP~x.x.x.x~y.y.y.y.pid -u /var/run/dpinger_WAN_DHCP~x.x.x.x~y.y.y.y.sock -C "/etc/rc.gateway_alarm" -d 1 -s 500 -l 2000 -t 30000 -A 1000 -D 500 -L 20 y.y.y.y.>/dev/null' returned exit code '1', the output was ''

It's weird that I saw changes for WAN3 that I haven't touched. I think the developers should have a look at the whole multi-WAN system. More logs:

Aug 4 19:02:27 php-fpm 73902 /rc.filter_configure_sync: Default gateway setting Interface WAN_DHCP Gateway as default. Aug 4 19:02:27 php-fpm 73902 /rc.filter_configure_sync: Gateway, switch to: WAN_DHCP Aug 4 19:02:25 upsmon 11497 Startup successful Aug 4 19:02:25 php-fpm 48340 /rc.start_packages: Starting service nut Aug 4 19:02:25 upsmon 85376 Signal 15: exiting Aug 4 19:02:25 php-fpm 48340 /rc.start_packages: Stopping service nut Aug 4 19:02:25 php-fpm 48340 /rc.start_packages: Restarting/Starting all packages. Aug 4 19:02:24 check_reload_status 483 Starting packages Aug 4 19:02:24 php-fpm 89297 /interfaces.php: Creating rrd update script Aug 4 19:02:24 check_reload_status 483 Reloading filter Aug 4 19:02:24 php-fpm 89297 /interfaces.php: Removing static route for monitor m.m.m.m and adding a new route through n.n.n.n [this is untouched WAN3] Aug 4 19:02:24 php-fpm 89297 /interfaces.php: Removing static route for monitor y.y.y.y and adding a new route through x1.x1.x1.x1 [this is WAN1] Aug 4 19:02:24 php-fpm 89297 /interfaces.php: Resyncing OpenVPN instances for interface WAN. Aug 4 19:02:23 php 77068 notify_monitor.php: Message sent to X OK Aug 4 19:02:22 php-fpm 202 /rc.dyndns.update: phpDynDNS (): (Success) IP Address Updated Successfully! Aug 4 19:02:22 php-fpm 202 /rc.dyndns.update: phpDynDNS: updating cache file /conf/dyndns_PreferWANcustom''0.cache: k.k.k.k [this is WAN2 IP] Aug 4 19:02:21 check_reload_status 483 updating dyndns wan Aug 4 19:02:19 dhcpleases 50889 Could not deliver signal HUP to process 71960: No such process. Aug 4 19:02:18 check_reload_status 483 Restarting IPsec tunnels Aug 4 19:02:18 php-fpm 775 /rc.filter_configure_sync: Gateway, switch to: WAN2_PPPOE Aug 4 19:02:18 php-fpm 89297 /interfaces.php: Default gateway setting Interface WAN2_PPPOE Gateway as default. Aug 4 19:02:18 php-fpm 89297 /interfaces.php: Gateway, switch to: WAN2_PPPOE Aug 4 19:02:17 check_reload_status 483 Reloading filter Aug 4 19:02:17 php-fpm 98601 /rc.newwanip: rc.newwanip: on (IP address: x.x.x.x) (interface: WAN[wan]) (real interface: igb0). Aug 4 19:02:17 php-fpm 98601 /rc.newwanip: rc.newwanip: Info: starting on igb0. Aug 4 19:02:16 check_reload_status 483 rc.newwanip starting igb0 Aug 4 19:02:16 php-fpm 89297 /interfaces.php: Error starting gateway monitor for WAN_DHCP Aug 4 19:02:16 php-fpm 89297 /interfaces.php: The command '/usr/local/bin/dpinger -S -r 0 -i WAN_DHCP -B x.x.x.x -p /var/run/dpinger_WAN_DHCP~x.x.x.x~y.y.y.y.pid -u /var/run/dpinger_WAN_DHCP~x.x.x.x~y.y.y.y.sock -C "/etc/rc.gateway_alarm" -d 1 -s 500 -l 2000 -t 30000 -A 1000 -D 500 -L 20 y.y.y.y.>/dev/null' returned exit code '1', the output was '' Aug 4 19:02:16 php-fpm 89297 /interfaces.php: Removing static route for monitor m.m.m.m and adding a new route through n.n.n.n [this is really weird - this is untouched WAN3]

@netblues said in Sticky connections not working with dual WAN:

@bijore8887 Yes, you are right about the documentation, my bad for that.
However working with smaller values of stickiness (or the default 0) the gateway is switced everytime all connections complete.

It would also be a bug if the gateway was switched every time all connections complete if stickiness was set to more than 0. Because it should be sticky for the number of seconds specified.

Do you have a high value in stickiness?

In my first comment I mentioned 1200.

Try it out.

No thanks. You have not bothered to read the pfSense documentation or the comments properly.

But at least others can now know that there could be a bug and use workarounds instead and not waste further time.

@justconfused
And to reply to my own post

Oh Shit - I am telnetting on the wrong bloody port.

Note that this is a demonstration of how writing things down can help think things through - even when you are being an utter moron.

I am certainly not going to admit just how long I have spent struggling with this (and a very similar related issue which is the reason for this test setup)

Sigh

@jeryd did you read the link provided by @viragomann

It even gives an example prefer wan over load balance

https://docs.netgate.com/pfsense/en/latest/multiwan/policy-route.html

example.jpg

@pete35 yes I found this information regarding architectural limitation later also.
and btw thanks for the answer, hope your links will be useful for other users.

@befree2 said in 5 WAN on 3 Interface. How?:

200 to 300 concurrent users that require internet

But your internet is only 200mbps total across all your connections, so lagg provides nothing other than a possible problem to deal with..

@viragomann Thank you for your suggestions.