Probably going to need a diagram of the pertinent pieces. Also sounds like that customer needs to beat that vendor with a wrench.

@jimp I created this rule on the WAN (attachment) but I do not know if it is enough. Port 161 was released in the output table and deletes it. [image: 1537387029627-snmp-blocking-resized.png]

I'm aware of sharing MACs being the expected behavior, and they were separate subnets. However the reason for all of this is because the WAN comes in via one fiber pair and eventually we wanted to do HA and have a segment of the switch be the WAN. I had the two interfaces separated in pfSense into two PVID ports on the internal switch, and they were plugged into the same PVID marked ports on the main switch in the rack, and I saw that MAC bouncing between two of the ports in that VLAN on the main switch. If that's tough to follow I can sketch out how it was configured since it's good now, but we're using a really convoluted solution for now.

Thanks, i found solutions

Plugging/unplugging must trigger something that causes it to reset the connections on WAN2. What I found after extensive testing is that once a failover occurs and connections are established on WAN2, it will not break those connections and put them back on WAN1 unless forced to do so. I documented my method for getting fail-back to work, maybe not ideal but the only way I could get it working reliably. https://forum.netgate.com/topic/135614/failback-from-primary-wan-after-failover-to-secondary-wan

The cradlepoint works fine. Set it to IP passthrough, plugged it into an ethernet port, changed one routing rule to route my work computer to the wan_group instead for default gateway. I do no switch the default gateway on fail over. It fails over and back seamlessly for my pc. It screws up Ooma for some reason, but that is OK. So if anyone wants an LTE backup the cradlepoint makes it easy and they are quite cheap on ebay. thanks david

@jlittle988 Hi, I have done this configuration, following this tutorial.. https://www.netgate.com/docs/pfsense/vpn/openvpn/routing-internet-traffic-through-a-site-to-site-openvpn-connection-in-pfsense-2-1.html what makes the magic is put this in Advanced Configuration -> Advanced -> redirect-gateway def1;

@cradulescu I have figureout how to solve this issue. There is a bug on openBGPD. Event I do setup the neighbors the conf dose not update so I have to update it manually ( I know is not recommended) . # This file was created by the package manager. Do not edit! AS 64500 fib-update yes listen on 0.0.0.0 router-id 192.168.0.1 network 192.168.0.1/24 neighbor 192.168.0.4 { remote-as 64501 descr "Kubernetes-Node01" } neighbor 192.168.0.8 { remote-as 64501 descr "Kubernetes-Node02" } #deny from any #deny to any Resault Kubernetes {"caller":"main.go:229","event":"serviceAnnounced","ip":"192.168.12.2","msg":"service has IP, announcing","pool":"default","protocol":"bgp","service":"default/elasticsearch","ts":"2018-09-16T14:37:20.876366531Z"} Resault pfsense: Neighbor AS MsgRcvd MsgSent OutQ Up/Down State/PrfRcvd Kubernetes-Node02 64501 337 327 0 02:42:09 1 Kubernetes-Node01 64501 337 327 0 02:42:09 1 OpenBGPD Neighbors BGP neighbor is 192.168.0.8, remote AS 64501 Description: Kubernetes-Node02 BGP version 4, remote router-id 192.168.0.8 BGP state = Established, up for 02:42:09 Last read 00:00:09, holdtime 90s, keepalive interval 30s Neighbor capabilities: Multiprotocol extensions: IPv4 unicast, IPv6 unicast 4-byte AS numbers Message statistics: Sent Received Opens 1 1 Notifications 0 0 Updates 1 11 Keepalives 325 325 Route Refresh 0 0 Total 327 337 Update statistics: Sent Received Updates 4 6 Withdraws 0 5 End-of-Rib 0 0 Local host: 192.168.0.1, Local port: 179 Remote host: 192.168.0.8, Remote port: 52807 BGP neighbor is 192.168.0.4, remote AS 64501 Description: Kubernetes-Node01 BGP version 4, remote router-id 192.168.0.4 BGP state = Established, up for 02:42:09 Last read 00:00:09, holdtime 90s, keepalive interval 30s Neighbor capabilities: Multiprotocol extensions: IPv4 unicast, IPv6 unicast 4-byte AS numbers Message statistics: Sent Received Opens 1 1 Notifications 0 0 Updates 1 11 Keepalives 325 325 Route Refresh 0 0 Total 327 337 Update statistics: Sent Received Updates 4 6 Withdraws 0 5 End-of-Rib 0 0 Local host: 192.168.0.1, Local port: 179 Remote host: 192.168.0.4, Remote port: 46850

Right. That is not how you do Multi-WAN. You would, instead, create a separate pfSense interface and put a gateway on each. P.S. pfSense 2.3 is all but dead.

Do you have policy routing (gateways set on rules) enabled on your local network? https://www.netgate.com/docs/pfsense/routing/bypassing-policy-routing.html

Thanks for your input. I think i might have encountered some kind of software bug. After i delete all phase 2 settings, redid them and rebooted the pfsense box it started working.

OpenVPN routing should be configured in the OpenVPN settings. Use the "Remote Network/s" box to enter the networks you want to route over the respective VPN. If you want to route traffic over a OpenVPN client, assign an interface to the client instance. Interfaces > Assignments. At "Available network ports" select the client instance (e.g. ovpnc1), hit Add, open the settings of the new interface, check Enable and set a proper name. No further configuration to make here! If you have done that pfSense also add a virtual gateway to the vpn connection which can be used in firewall rules for policy routing or also for gateway monitoring. But do not add static route to a vpn gateway! That's not recommended. As mentioned above, that is to be done in the OpenVPN settings.

Oh, and this option is only available in the dev 2.4.4...