@serbus Thanks John, those are really useful posts and I now have some things to experiment with.

M..

@cool_corona

What I wrote above is a wrong statement,
This is not true => You are right it looks like some built-in "kill state" procedure somewhere in background.
No proof = not true.
I can not indicate in the code where the flush would be executed.
Coincidence with something else... but still not a PRD so don't care.

The only explanation I can give is that in those three rules I force the traffic to use one of the three gateway groups, while in the rule for accessing the DMZ I use the default gateway.
But I still can't explain why.
The default gateway is set to "Automatic" and without the Load Balancing and Failover configuration it would be set towards the router.
Traffic to the DMZ should not go over the WAN.

If someone more experienced than me can explain it to me I would be grateful.

@st6, I understand, this should not be possible! but if it really happens to you I want you to stand guard and send your firewall lgos to be able to find the cause of the problem please.

@jpk_pfsense so you setup a route pointing to pfsense own interface?

The gateway would be the IP of where this 192.168.0 network is..

So maybe 192.168.201.2 is that device?

router1 (.1) -- 192.168.201 --- (.2) router2 -- 192.168.0

So the gateway and route would be setup on router 1, and don't forget the return traffic.. So router 2 default gateway needs to be the 192.168.201.1 or you need route on router 2 on how to get back to what your source network would be.

Also there should be no hosts on this 192.168.201 network, it should just be a transit network.

@viragomann said in Connect VPN over 4G if WAN fails, but not route any other traffic:

@keyser
The only traffic on the secondary would be the pings of gateway monitoring, while the primary is up. It's possible to disable it, but I'm not sure if the gateway group is still working then.

If you really want to shut down the 4G interface completely and start it up when the primary goes down, you might have to modify some scripts.

Okay, cool. That should not amount to much traffic.
I’ll see if I can get a 4G Sierra card and test it in my SG-2100

@makowner
Possibly the server sends packets out to the internal interface with its public source IP?
Sniff the traffic to check that out.

@darkcorner said in Failover doesn't work.:

I have configured a dual WAN with LoadBalance and Failover, but the failover doesn't work.
When I unplug the cable of a line, then the PING (in Win10 on 8.8.8.8 or on google.com) stops working until I reconnect it.

In summary these are the operations done:

WAN with static IP 4 DNS (2 from Google and 2 from Open DNS) 2 Gateways, associated with Google DNS and OpenDNS Monitor with OpenDNS (first and second DNS). I initially used those from Google, but yesterday with these, the Gateway on the optical fiber was oflline due to too many lost packets. Gateway group "LoadBalancer" with both at Tier1 2 Gateway groups for FailOver by inverting Tiers and with Trigger Level to "Member down" In LAN-DMZ1-DMZ2 I created 3 rules, on top of all other rules, with gateway load-balancer, failover1 and failover2 Default gateway IPv4 = Automatic

I did not use weights between the two lines so as not to further complicate things (also because I do not yet know the performance of the two lines when everything is up and running).

Yet as I said, when I unplug one of the two "WAN" cables, the PING on the PC stops working.
I checked everything both still using the guide and of course the official documentation.

The only thing is that in System/Advanced/Miscellaneous/Load Balancing
Load Balancing / Use sticky connections = OFF
But I have nowhere found a hint as to whether it is mandatory to activate it. In any case, it would be more about Load Balancing than Failover.

Another setting to look at also in System -> Advanced -> Miscellaneous is "State Killing on Gateway Failure". The information on that option reads:

"The monitoring process will flush all states when a gateway goes down if this box is checked."

My observation of WAN failover in pfSense by default settings, in which State Killing on Gateway Failure is NOT checked goes like this scenario:

Client1 is streaming from YouTube and is connected to Google's search engine Primary WAN gateway goes down After the Primary WAN goes down and the Secondary WAN picks up, Client1 establishes a connection to Spotify to stream music Client1's connections to YouTube and Google at first continue to try to use the Primary WAN, and thus don't perform. This situation persists until Client1's web browser decides the connections are lost and attempts to establish a new connection, which the router will utilize the Secondary WAN for. Client1's connection to Spotify works without issues because it established over the Secondary WAN initially.

This type of behavior is especially a problem if your primary WAN's failure mode isn't to go hard down, but is simply considered down by the router when in reality it's "up" but performing extremely poorly due to excessive latency or excessive packet loss. In this scenario, Client1's web browser with active connections to Google and YouTube over the Primary WAN won't try to establish new connections, because the connection via the Primary WAN is "up" it just works extremely poorly.

This can be redressed by using State Killing on Gateway Failure. Use this with caution. What it does, is when the Primary WAN fails, the monitor process clears ALL states in the router. This is equivalent to going to Diagnostics -> States -> Reset States -> checking the Checkbox and hitting Reset button. This breaks ALL existing connections and forces all client systems to re-establish all connections.

This can be good if, in a failover scenario, you want to immediately re-establish all connections over a Secondary WAN. However, if you have a primary WAN that has intermittent connection issues where it flips from up to down frequently, it can be extremely disruptive, because this feature will clear ALL states when a gateway goes down, NOT just the states associated to the gateway that went down. If you have primary WAN with intermittent connectivity issues, this could cause the router to 'thrash' between Primary and Secondary WAN. As a result, I think this feature is more useful for effecting a quicker failover to Secondary when the Primary WAN is hard down and not up with very bad latency / high packet loss. It can be used in a scenario with bad latency / packet loss causing primary to get marked down, but you have to be very cautious about how you tune the gateway monitoring to prevent thrashing.

Personally I'd like to see a feature where we could only kill states associated with a down gateway if possible, but not knowing the code in the networking stack, I don't know if that's actually possible or not.

In any case, bottom line: for your scenario, you are describing a hard down type event. Killing states on gateway failure should work well in your scenario and effect a faster and smoother transition to Secondary WAN.

@christian-loth said in Two gateways and policy routing:

Yes. I mean it's a private net, got enough of those. :-)
It's not a technical reason, just following a scheme of using private nets that are part of 10/8.

Better to use rather small subnets to prevent routing issues. For instance if you connect to a remote network using a site to site VPN.

@pilotryan2992 said in Multi WAN + Multi PPPoE IPs:

But i am still confused about the multiple IP / multiple PPPoE session. Is there any guide/documentation on how to do this ?

Interfaces, assignements, ppps, add
Select the same physical ethernet for all

After that, assign new interfaces to ppp's

and you end up with multiple wan ppp's interfaces with different ip's
(if your provider allows multiple logins that is)

It has been a long time and I totally forgot I posted this. I was searching for a solution when I came across my old post. I'm posting this as a future reference for myself or anyone who is looking for a similar solution. That said, this may not be the best solution, but it is the best one I have at this time.

In my most simplified example there are 3 routers (these routers are also pfSense firewalls).

R1, R2 and R3:

R1: CARP Primary at site A R2: CARP Secondary at site A R3: Standalone router at site B

A first hop redundancy protocol (FHRP) like CARP, VRRP, or HSRP allows clients to have a highly available default gateway. At site A we have a bunch of servers connected to a L2 switch whose default gateway is pointing to the CARP VIP for their associated VLAN.

Setting up a single VTI IPSec tunnel between site’s B physical WAN address and the WAN VIP at Site A, results in two main issues:

When CARP fails to the secondary making it the Master, OSPF failover involves services starting based on the CARP status, neighbor establishment, forming adjacencies, ect. Having a dynamic routing protocol like OSPF that has HA capabilities baked in and relying on CARP seems wrong. I’m not even sure bidirectional forwarding detection would have any benefit in this config. At least with FRR, OSPF advertises the physical address of the FW (as it should) and not the VIP. Managing R2 when it is functioning as the backup from site B results in an asymmetric routing issue. Basically, the client’s network traceroute path is not the same as the reverse path which is obvious since you would not want R2 default gateway pointing to R1. This makes pfSense management GUI unusably slow. Levering sloppy states looks like a poor work around to me.

To circumvent these issues I think the best approach is disabling IPsec sync between the clustered FW at site A and manually creating two VTI tunnels.

R3 -> R1 R3 -> R2

When setting up OSPF in this configuration you must change the cost of the R2 VTI interface on R3 to a number that is higher than R1 VTI interface. This cost determines the preferred router to receive the routes from. R1 in this case, since it is lower. If R1 dies and R2 takes over as CARP Master, OSPF independently identifies the failure as well and elects R2.

Since the VTI tunnels are simultaneously connected to R3 from R1 and R2, the VTI address of R1 and R2 can be used to manage the firewalls from site B. This eliminates the asymmetric routing issue, and allows the connecting client interface to be the same interface as the one you are logging into. ie., if you are on the LAN network you access the firewall using the LAN IP, but if you are at site B you access via the IPSec VTI IP.

This does create a DNS issue where you need to create a Split-Brain DNS configuration so that depending on your client address you resolve to a different address. Something like this:

Add-DnsServerClientSubnet -Name "SiteAMgmt" -IPv4Subnet "10.0.10.0/24" Add-DnsServerClientSubnet -Name "SiteALan" -IPv4Subnet "10.0.20.0/24" Add-DnsServerClientSubnet -Name "SiteADmz" -IPv4Subnet "10.0.30.0/24" Add-DnsServerZoneScope -ZoneName "Contoso.com" -Name "NotSiteA" #VTI transit IP defined Add-DnsServerResourceRecord -ZoneName "Contoso.com" -A -Name "R1" -IPv4Address "R1 VTI IPv4 address" -ZoneScope "NotSiteA" Add-DnsServerResourceRecord -ZoneName "Contoso.com" -A -Name "R2" -IPv4Address "R2 VTI IPv4 address" -ZoneScope "NotSiteA" Add-DnsServerQueryResolutionPolicy -Name "NotSiteAPolicy" -Action ALLOW -FQDN "eq,R1.Contoso.com,R2.Contoso.com" -ClientSubnet "NE,SiteAMgmt,SiteALan,SiteADmz" -ZoneScope "NotSiteA,1" -ZoneName "Contoso.com"

The default zone scope would contain the A record for R1 and R2 address on the management VLAN like normal.

Lastly, this configuration does create a dilemma. Say R1 does not crash but instead a DMZ interface or something goes down and triggers a CARP failover event to R2. Unlike VRRP that can run in a Active\Active config where half the VLAN\INTs on one router and the other half on the secondary, CARP does not appear to be able to do this. Subsequently, OSPF on R1 will continue to advertise routes even though it is the CARP backup. Another situation that could cause this same issue is just putting R1 into maintenance mode. To rectify this, I used the “CARP Status IP” in FRR’s Global setting in R1 to shutdown the FRR services when in backup status. However, I left the “CARP Status IP” configuration on R2 set to None.

Yes I read these before, but it isn't resolve the problem... As I mentioned LB working at the beginning, but stop after time or mostly when I restore config to another machine... Both are with same NIC's so I don't even need to mess with configuration after restoring :)
Everything other except Load Balancing work fine on my opinion...

Well.. after a LOT of inspection... finally managed to do it!

@trigg3r said in Move services to the public IPs of the second provider:

Probably the helpdesk service often has to deal with someone not doing his homework, so they probably insisted that my config wasn't ok (despite what I wrote during a whole week of emails ...)

The problem is that this behavior make you a lot of work and steals your time, when you're not really a network expert.

But after reading this thread probably someone gave up in front of your reputation and ... ta-da! ... this morning everything is working fine ( "A change has been made to the receptive antenna, so please check again if remote access is now possible.").

Nice to hear. Thx for feedback.

So what finally worked.

The Pfsense firewall has 7 interfaces, all bridged under 1 interface called BridgeLAN with BridgeLAN running the DHCP server. I wanted to let only 1 host out the internet without VPN because of GEOIP apps that requiring my country location.

viragomann gave me great direction any I tried his rule direction on every interfaces, what I found out that the BridgeLAN VPN rules superseded all other rule entrys because all the other interfaces are bridged inside the BridgeLAN Interfaces .

Like this

lan12.png

So I created the same rule to open that host above the VPN rule in the BridgeLAN rule and it worked as I wanted.

lan11.png

I know there is better ways to do this, and I will take any advice.

Thanks Again viragomann

@jtd said in Connect Two Private LANs from Different Companies Using Netgate 2100:

There's a lot of communications going on I noticed that had nothing to do with the needed application

:)
pfsense is more than pf
A dedicated firewall gives you visibility and monitoring
Well done

@viragomann Oh thank you so much.
That explains everything. For a moment there I thought I was going crazy :D

Thanks again for the help and clarification o/

@netblues Hi! Thank you for your reply. There's is no problem with my modem or its bridge mode. I did a standalone test using my Windows 7 PC, its able to connect quickly to the PPPoE connection.