Plugging/unplugging must trigger something that causes it to reset the connections on WAN2. What I found after extensive testing is that once a failover occurs and connections are established on WAN2, it will not break those connections and put them back on WAN1 unless forced to do so.

I documented my method for getting fail-back to work, maybe not ideal but the only way I could get it working reliably.

https://forum.netgate.com/topic/135614/failback-from-primary-wan-after-failover-to-secondary-wan

The cradlepoint works fine. Set it to IP passthrough, plugged it into an ethernet port, changed one routing rule to route my work computer to the wan_group instead for default gateway. I do no switch the default gateway on fail over.

It fails over and back seamlessly for my pc. It screws up Ooma for some reason, but that is OK.

So if anyone wants an LTE backup the cradlepoint makes it easy and they are quite cheap on ebay.

thanks
david

@jlittle988 Hi, I have done this configuration, following this tutorial..

https://www.netgate.com/docs/pfsense/vpn/openvpn/routing-internet-traffic-through-a-site-to-site-openvpn-connection-in-pfsense-2-1.html

what makes the magic is put this in Advanced Configuration -> Advanced -> redirect-gateway def1;

@cradulescu I have figureout how to solve this issue.
There is a bug on openBGPD. Event I do setup the neighbors the conf dose not update so I have to update it manually ( I know is not recommended) .

# This file was created by the package manager. Do not edit! AS 64500 fib-update yes listen on 0.0.0.0 router-id 192.168.0.1 network 192.168.0.1/24 neighbor 192.168.0.4 { remote-as 64501 descr "Kubernetes-Node01" } neighbor 192.168.0.8 { remote-as 64501 descr "Kubernetes-Node02" } #deny from any #deny to any

Resault Kubernetes

{"caller":"main.go:229","event":"serviceAnnounced","ip":"192.168.12.2","msg":"service has IP, announcing","pool":"default","protocol":"bgp","service":"default/elasticsearch","ts":"2018-09-16T14:37:20.876366531Z"}

Resault pfsense:

Neighbor AS MsgRcvd MsgSent OutQ Up/Down State/PrfRcvd Kubernetes-Node02 64501 337 327 0 02:42:09 1 Kubernetes-Node01 64501 337 327 0 02:42:09 1 OpenBGPD Neighbors BGP neighbor is 192.168.0.8, remote AS 64501 Description: Kubernetes-Node02 BGP version 4, remote router-id 192.168.0.8 BGP state = Established, up for 02:42:09 Last read 00:00:09, holdtime 90s, keepalive interval 30s Neighbor capabilities: Multiprotocol extensions: IPv4 unicast, IPv6 unicast 4-byte AS numbers Message statistics: Sent Received Opens 1 1 Notifications 0 0 Updates 1 11 Keepalives 325 325 Route Refresh 0 0 Total 327 337 Update statistics: Sent Received Updates 4 6 Withdraws 0 5 End-of-Rib 0 0 Local host: 192.168.0.1, Local port: 179 Remote host: 192.168.0.8, Remote port: 52807 BGP neighbor is 192.168.0.4, remote AS 64501 Description: Kubernetes-Node01 BGP version 4, remote router-id 192.168.0.4 BGP state = Established, up for 02:42:09 Last read 00:00:09, holdtime 90s, keepalive interval 30s Neighbor capabilities: Multiprotocol extensions: IPv4 unicast, IPv6 unicast 4-byte AS numbers Message statistics: Sent Received Opens 1 1 Notifications 0 0 Updates 1 11 Keepalives 325 325 Route Refresh 0 0 Total 327 337 Update statistics: Sent Received Updates 4 6 Withdraws 0 5 End-of-Rib 0 0 Local host: 192.168.0.1, Local port: 179 Remote host: 192.168.0.4, Remote port: 46850

Right. That is not how you do Multi-WAN. You would, instead, create a separate pfSense interface and put a gateway on each.

P.S. pfSense 2.3 is all but dead.

Do you have policy routing (gateways set on rules) enabled on your local network?

https://www.netgate.com/docs/pfsense/routing/bypassing-policy-routing.html

Thanks for your input.

I think i might have encountered some kind of software bug.
After i delete all phase 2 settings, redid them and rebooted the pfsense box it started working.

OpenVPN routing should be configured in the OpenVPN settings. Use the "Remote Network/s" box to enter the networks you want to route over the respective VPN.

If you want to route traffic over a OpenVPN client, assign an interface to the client instance. Interfaces > Assignments.
At "Available network ports" select the client instance (e.g. ovpnc1), hit Add, open the settings of the new interface, check Enable and set a proper name. No further configuration to make here!
If you have done that pfSense also add a virtual gateway to the vpn connection which can be used in firewall rules for policy routing or also for gateway monitoring.

But do not add static route to a vpn gateway! That's not recommended. As mentioned above, that is to be done in the OpenVPN settings.

Oh, and this option is only available in the dev 2.4.4...

Grrrrrrrrrrrrr.

Found the problem and it was of course my fault. I had a client specific override for my user account (which I knew about but never checked). In there, the 192.168.2.0/24 network was set as a "Remote Network" instead of a "Local Network." Deleted it form remote networks, added it to local networks and now all is working. I didn't realize there was an option for "Remote Networks" as that's not an option for the actual OpenVPN server itself.

Hi guys,

Thanks for your help, I guess it's because of a NAT thing, there is an extra layer between me and the host I cannot connect to.
I'm off to go move some cables around :)

Thx again

/tony

@viragomann Good to know. I will need to enable "skip rules" and put a "block" rule after. I do not want email services egressing anywhere except WAN_1_IP2

All working. I forgot to enable NAT Reflection (and I only had the bright idea of trying to go to https://virtualIP using my phone off WiFi. Thanks for your help guys!

Anyone?

@johnpoz
Dear John,

Thank you. My issue solved.

I got confused and made everyone confuse.

WAN- 203 Series
LAN - 192 Series (192.192. series will change in few days to 192.168.) Currently Windows DC is running in that 192.192 series. So will change in few days.

Please find the below link.
https://forum.netgate.com/topic/134674/how-to-configure-3-ip-s-internet-restriction/20

Big Thanks to you John. No words to express my gratitude.

Lokesh Kamath

Only 23days to find a problem.... I wish my bosses were as nice