Multi WAN Failover doesn't work

kevindd992002

We have somehow the same setup, VLANs and three modems from the same ISP.

Firstly, how do you exactly do the step1 you've mentioned above? My ISP's DNS servers are 202.69.165.10, 202.69.191.10, and 121.58.225.10.

Secondly, can I use 4.2.2.1, 4.2.2.2, and 4.2.2.3 for my three modems, respectively?

Thirdly, in the System->General Setup subtab, should I input two of my ISP's DNS addresses there or just tick "Allow DNS server list to be overridden by DHCP/PPP on WAN" ??

Lastly, do I need Manual Outbound NAT rule generation (Advanced Outbound NAT (AON)) ?? Or Automatic Outbound NAT will do? What is it anyway?

Thanks for your help in advance ;)

cheesyboofs

Hi,

By default all your outbound DNS will go out your default gateway, WAN1. When WAN1 goes down the DNS servers can no longer be resolved out of the default gateway as the router its self will not use the fail-over pools you have setup. So you need to force the secondary DNS server requests out of WAN2, to do this create a static route System->Static Routes that reads (in your case) Interface = WAN2 - Network = 202.69.191.10/32 - Gateway = what ever the gateway of you WAN2 interface is, not the ip address of you WAN2 interface in my example it is 10.10.0.1.
So when your WAN1 goes down your router won't be able to resolve the primary DNS anymore but it will still be able to resolve the secondary DNS because your router will stuff the requests out of WAN2.
I see no reason why not!
I choose to hard set the 2 ISP DNS and untick "Allow DNS server list to be overridden by DHCP/PPP on WAN"
No, Automatic outbound NAT rule generation should work fine.

kevindd992002

Thank you for the reply. Now I know what you mean ;) There are still some additional questions though.

What about static DNS routes for WAN3? In the General setup of pfsense, you can only input two DNS servers, but what if the case of WAN1 and WAN2 are down, you should force outgoing DNS requests out WAN3 right? How is it possible if you can only input two DNS server addresses? Is there a way to input more than two?

Thanks.

cheesyboofs

Is there a way to input more than two?

I don't think there is, I know you will be able to in pfsense v2.0.
Maybe the case you need a local LAN DNS server or try the TinyDNS package "pfSense version of TinyDNS which features failover host support " - I've yet to try.

kevindd992002

All right, the case of WAN1 and WAN2 being down is very seldom anyway. Thank you ;)

GruensFroeschli

You can download the config.xml and edit it directly.
Like this it's possible to add as many DNS servers as you want.

kevindd992002

Oh.. Where can I download config.xml?

GruensFroeschli

Diagnostics: Backup/restore

kevindd992002

I was able to download and restore again the file, no probs.

My 2nd DNS Server (202.69.191.10) and 3rd DNS Server (121.58.225.10) have already set static routes to WAN2 Gateway (192.168.2.1) and WAN3 Gateway (192.168.3.1), respectively.

With this setup, I'm expecting that if I traceroute 202.69.191.10 and 121.58.225.10 from one of my computers in the LAN side of pfsense, I will get a first hop of 192.168.2.1 and 192.168.3.1, respectively, right? Well, at least that's the case with my setup where it doesn't show pfsense's LAN gateway (192.168.1.1) as the first hop, I don't know why also.

But the results is not what I expected, sometimes the first hop of 202.69.191.10 is 192.168.2.1, sometimes it's 192.168.3.1, sometimes it's the gateway of WAN1 interface. Same goes with tracerouting 121.58.225.10.

I already tried setting pfsense to factory defaults and restarting from scratch, still no success. I checked the Diagnostics: Routes page and indeed the static routes were listed there.

So what could be the problem?

Thanks for your help ;)

GruensFroeschli

Use the texteditor of you choice and open it.
Look for this part:

<system><optimization>normal</optimization>
<schedulertype>priq</schedulertype>
<hostname>juhui</hostname>
<domain>blah.bl.ah</domain>
<username>notyou</username>
<password>nothingyoucansee</password>
<timezone>Europe/Zurich</timezone>
<time-update-interval><timeservers>0.pfsense.pool.ntp.org</timeservers>
−
<webgui><protocol>someprotocol</protocol>
<port>someport</port>
<certificate><private-key></private-key></certificate></webgui>
−
<ssh><authorizedkeys></authorizedkeys></ssh>
<maximumstates><shapertype><dnsserver>208.67.222.222</dnsserver>
<dnsserver>208.67.220.220</dnsserver>
<dnsallowoverride></dnsallowoverride></shapertype></maximumstates></time-update-interval></system>

copy/paste the red part and add your own DNS server IPs.

kevindd992002

Yup ;) Edited my last post because I was able to do that already, sorry, lol..

I hope you read my last post before this, I have a new problem regarding the setup. Thanks. That's Reply #14.

GruensFroeschli

@kevindd992002:

With this setup, I'm expecting that if I traceroute 202.69.191.10 and 121.58.225.10 from one of my computers in the LAN side of pfsense, I will get a first hop of 192.168.2.1 and 192.168.3.1, respectively, right? Well, at least that's the case with my setup where it doesn't show pfsense's LAN gateway (192.168.1.1) as the first hop, I don't know why also.

The static routes only apply to the pfSense.
If you do a traceroute from behind the pfSense, the connections will be handled how you defined it with your firewallrules on the LAN interface.
Since you have a loadbalancing pool as gateway, all connections will be balanced.

If you had a firewall rule on the LAN with as gateway default (*), only then would the connections be handled according to the routing table of the pfSense.

default = routingtable
anything else = you force it somewhere else
The loadbalancer is a special kind of policy routing.

If you want to check if your traffic goes out the correct gateway, do the traceroute on the pfSense itself.

kevindd992002

I thought so, thank you on that. I did a traceroute and everything went fine.

Now, I'm wondering why is it that when I do a traceroute, 192.168.1.1 is NOT ALWAYS the first hop?

I tried a different router and it shows the LAN gateway (usually 192.168.1.1) as the first hop but not with pfsense router.

kevindd992002

I followed all the suggestions above to make FAILOVER work but still sometimes it doesn't work.

How much time does it approximately take for pfsense to work when one or two of my modems are removed from their respective interfaces?

GruensFroeschli

How are you testing?
NEW connections should be on the new line as soon as it's detected as down.
What does the status –> loadbalancer page say about the different gateways when you do your test?
It should be in the range of a few seconds.

kevindd992002

What do you mean by "NEW connections should be on the new line as soon as it's detected as down." ??

I'm testing by removing the UTP cable that's plugged to WAN1, WAN2, and WAN3.

Within a few seconds, the status: load balancer page always give me the correct status. That is when I remove WAN2 cable, WAN1 and WAN3 will be the only one "Online". And so on and so forth.

This is how I set my load balancer pool:

Did I do it correctly?

How does failover work anyway? When one link is detected as offline, pfsense will disregard the "load balancer pool"? and revert to the failover pools?

Do I need to setup anything in the firewall rules in terms of failover mechanisms?

GruensFroeschli

Umm…
You either use loadbalancing OR failover. Never both at the same time for the same rule.

Please read up how the rules with pfSense work.
It's written quite often in this forum/faq/stickies.

If the loadbalancer updates its status correctly, then it's working as it should.

If you open a new connection, this connection will be accordig to the "gateway-field" of the firewall-rule (failover, loadbalancer, whatever) assigned to a gateway.
If you set as gateway the loadbalance pool, the connection will be assigned to one of the "online" gateways of the pool.
If you remove this link while traffic is going over it, you will have to reestablish the connection, to get on one of the still online gateways.
The connection wont "jump" to an online link.

You ask how the failover/loadbalance work together.
You either define a loadbalancer for a firewall rule (say all port 80, standard browse traffic), or a failoverpool (for port 443, https).
Loadbalance makes sense where you open lots of connections and it doesnt matter to the server if multiple requests come from different IPs.
However https doesnt like it if the same client has multiple sources. So it makes sense to force it to one WAN, and if this one is down you move to the next WAN.

kevindd992002

Yes, actually I have that same perception on how loadbalancer and failover works.

What I'm confused at is, for example, you did not set up any "firewall rule" that has a "failover gateway" on it. Say, you just have one firewall all in all, just Lan subnet, source: any, destination: any, and gateway: load balancer, and then I remove the WAN2 link, will pfsense automatically use the failover pools? Even though I don't have a rule setup with failover gateways?

Consider that yes I did refresh the website to make a NEW connection AFTER removing the WAN2 link.

GruensFroeschli

@kevindd992002:

What I'm confused at is, for example, you did not set up any "firewall rule" that has a "failover gateway" on it. Say, you just have one firewall all in all, just Lan subnet, source: any, destination: any, and gateway: load balancer, and then I remove the WAN2 link, will pfsense automatically use the failover pools? Even though I don't have a rule setup with failover gateways?

If you dont set up any rule with as gateway a "failover pool", you will never get the failover behavior.
With a single rule which has as gateway the loadbalancer, the traffic will always be balanced.

What the failover and loadbalancer have in common: if a link is marked as down, no new connections go to this link.
So if you were to only have 2 WANs, you would have no difference between a failover pool and a loadbalancing pool, if one of the WANs is down.

The only difference is that if all interfaces are up:

the loadbalancer pool uses all gateways
the failover pool uses the gateways in a top to down manner depending on if the gateway is up or not.

kevindd992002

Ahh.

So, for example, I did not create a failover, I only set up a load balancer pool. And I only have one firewall rule as I mentioned in my previous reply, everything will be "re-routed" properly when one or two of the links goes down?

So, technically, I don't need a failover pool if I don't care about HTTPs traffic having problems with load balancing, did I get that correctly?