Multi WAN Failover doesn't work

NOwlar

I have a Multi-WAN pfSense box (1.2.3-RC1):
1 WAN (wan) is connected through a Motorola Cable modem with DHCP assigned IP
2 WAN (opt) is connected through a DSL modem whith static IP

The Failover setup is from WAN1 to WAN2 and works perfectly when I unplug cables from modem, when I unplug the power cords.
But when the Cable modem looses the link to ISP the Failover doesn't work. The wan is marked offline in the Pool status, but it doens't help. I think the problem is the internal DHCP server in the Cable modem - then the link to ISP is broken the server become activated and assign IP and Gateway addresses out of the scope of my pfSense setup. In this case the default route with gateway 192.168.0.100 (the default ip for the cable modem, i think) is present in the routing table, and when I delete it everything works as before.

Who can I setu зеру pfSense box to work with such case?

GruensFroeschli

Which HowTo did you follow?
See this: http://doc.pfsense.org/index.php/MultiWanVersion1.2#Setting_up_DNS_for_Load_Balancing

(I just added this part to the wiki, since it seems to be missing in the 1.2 howto, it was there in the howto for previous versions).

NOwlar

I read this one too. I have a single Failover pool (from wan to opt1) and no loadbalncing pools.
The fact is that the Cable mode is supplied by the ISP and I have no right to configure it. I suppose it's configured to work in bridge mode (for USB only to restrict internet usage for one PC only). The ISP authentification is done by MAC control, so I spoof the modem USB MAC on pfsense WAN interface. Everything works fine. When the cable is disconnected or ISP have some internal troubles the failover works as planned, but when the calble modem looses the link to ISP it switches, I think, to the router mode and for some reason the failover doesn't work in this case even if the wan interface is marked as OFFLINE.

GruensFroeschli

Failover is (technically seen) the same as loadbalancing.

I'm talking about: "did you add static routes for your DNS-servers" ?

NOwlar

Sure. The DNS works.
I checked the connection state by ping to some well known (for me) IP addresses.

cheesyboofs

Not sure if this will help at all but here goes,

I have three cable modems so I am well aware how they can play up, not only that they are all from they same ISP. To make things even worse I'm using a vLAN set up meaning all three modems connect to one MAC address and as they are using DHCP and as we know three modems cant request three different IP's using the same MAC.

The two big things that I had to learn to make mine work correctly where,
1st as GruensFroeschli said set a static route forcing my ISP's secondary DNS server out of my WAN2 interface, so that when WAN1 went down I still had a valid route to an outside DNS server.

2nd use monitoring address of devices outside your environment, If your router thinks a route is still up it will continue to try and send data to it. Your modem might be semi up or flapping, try using say; 4.2.2.2 as one of your addresses work for me.

Look at my setup if it helps http://multi.cheesyboofs.co.uk/home.htm and I'll be happy to help if I can.

kevindd992002

@cheesyboofs

We have somehow the same setup, VLANs and three modems from the same ISP.

Firstly, how do you exactly do the step1 you've mentioned above? My ISP's DNS servers are 202.69.165.10, 202.69.191.10, and 121.58.225.10.

Secondly, can I use 4.2.2.1, 4.2.2.2, and 4.2.2.3 for my three modems, respectively?

Thirdly, in the System->General Setup subtab, should I input two of my ISP's DNS addresses there or just tick "Allow DNS server list to be overridden by DHCP/PPP on WAN" ??

Lastly, do I need Manual Outbound NAT rule generation (Advanced Outbound NAT (AON)) ?? Or Automatic Outbound NAT will do? What is it anyway?

Thanks for your help in advance ;)

cheesyboofs

Hi,

By default all your outbound DNS will go out your default gateway, WAN1. When WAN1 goes down the DNS servers can no longer be resolved out of the default gateway as the router its self will not use the fail-over pools you have setup. So you need to force the secondary DNS server requests out of WAN2, to do this create a static route System->Static Routes that reads (in your case) Interface = WAN2 - Network = 202.69.191.10/32 - Gateway = what ever the gateway of you WAN2 interface is, not the ip address of you WAN2 interface in my example it is 10.10.0.1.
So when your WAN1 goes down your router won't be able to resolve the primary DNS anymore but it will still be able to resolve the secondary DNS because your router will stuff the requests out of WAN2.
I see no reason why not!
I choose to hard set the 2 ISP DNS and untick "Allow DNS server list to be overridden by DHCP/PPP on WAN"
No, Automatic outbound NAT rule generation should work fine.

kevindd992002

Thank you for the reply. Now I know what you mean ;) There are still some additional questions though.

What about static DNS routes for WAN3? In the General setup of pfsense, you can only input two DNS servers, but what if the case of WAN1 and WAN2 are down, you should force outgoing DNS requests out WAN3 right? How is it possible if you can only input two DNS server addresses? Is there a way to input more than two?

Thanks.

cheesyboofs

Is there a way to input more than two?

I don't think there is, I know you will be able to in pfsense v2.0.
Maybe the case you need a local LAN DNS server or try the TinyDNS package "pfSense version of TinyDNS which features failover host support " - I've yet to try.

kevindd992002

All right, the case of WAN1 and WAN2 being down is very seldom anyway. Thank you ;)

GruensFroeschli

You can download the config.xml and edit it directly.
Like this it's possible to add as many DNS servers as you want.

kevindd992002

Oh.. Where can I download config.xml?

GruensFroeschli

Diagnostics: Backup/restore

kevindd992002

I was able to download and restore again the file, no probs.

My 2nd DNS Server (202.69.191.10) and 3rd DNS Server (121.58.225.10) have already set static routes to WAN2 Gateway (192.168.2.1) and WAN3 Gateway (192.168.3.1), respectively.

With this setup, I'm expecting that if I traceroute 202.69.191.10 and 121.58.225.10 from one of my computers in the LAN side of pfsense, I will get a first hop of 192.168.2.1 and 192.168.3.1, respectively, right? Well, at least that's the case with my setup where it doesn't show pfsense's LAN gateway (192.168.1.1) as the first hop, I don't know why also.

But the results is not what I expected, sometimes the first hop of 202.69.191.10 is 192.168.2.1, sometimes it's 192.168.3.1, sometimes it's the gateway of WAN1 interface. Same goes with tracerouting 121.58.225.10.

I already tried setting pfsense to factory defaults and restarting from scratch, still no success. I checked the Diagnostics: Routes page and indeed the static routes were listed there.

So what could be the problem?

Thanks for your help ;)

GruensFroeschli

Use the texteditor of you choice and open it.
Look for this part:

<system><optimization>normal</optimization>
<schedulertype>priq</schedulertype>
<hostname>juhui</hostname>
<domain>blah.bl.ah</domain>
<username>notyou</username>
<password>nothingyoucansee</password>
<timezone>Europe/Zurich</timezone>
<time-update-interval><timeservers>0.pfsense.pool.ntp.org</timeservers>
−
<webgui><protocol>someprotocol</protocol>
<port>someport</port>
<certificate><private-key></private-key></certificate></webgui>
−
<ssh><authorizedkeys></authorizedkeys></ssh>
<maximumstates><shapertype><dnsserver>208.67.222.222</dnsserver>
<dnsserver>208.67.220.220</dnsserver>
<dnsallowoverride></dnsallowoverride></shapertype></maximumstates></time-update-interval></system>

copy/paste the red part and add your own DNS server IPs.

kevindd992002

Yup ;) Edited my last post because I was able to do that already, sorry, lol..

I hope you read my last post before this, I have a new problem regarding the setup. Thanks. That's Reply #14.

GruensFroeschli

@kevindd992002:

With this setup, I'm expecting that if I traceroute 202.69.191.10 and 121.58.225.10 from one of my computers in the LAN side of pfsense, I will get a first hop of 192.168.2.1 and 192.168.3.1, respectively, right? Well, at least that's the case with my setup where it doesn't show pfsense's LAN gateway (192.168.1.1) as the first hop, I don't know why also.

The static routes only apply to the pfSense.
If you do a traceroute from behind the pfSense, the connections will be handled how you defined it with your firewallrules on the LAN interface.
Since you have a loadbalancing pool as gateway, all connections will be balanced.

If you had a firewall rule on the LAN with as gateway default (*), only then would the connections be handled according to the routing table of the pfSense.

default = routingtable
anything else = you force it somewhere else
The loadbalancer is a special kind of policy routing.

If you want to check if your traffic goes out the correct gateway, do the traceroute on the pfSense itself.

kevindd992002

I thought so, thank you on that. I did a traceroute and everything went fine.

Now, I'm wondering why is it that when I do a traceroute, 192.168.1.1 is NOT ALWAYS the first hop?

I tried a different router and it shows the LAN gateway (usually 192.168.1.1) as the first hop but not with pfsense router.

kevindd992002

I followed all the suggestions above to make FAILOVER work but still sometimes it doesn't work.

How much time does it approximately take for pfsense to work when one or two of my modems are removed from their respective interfaces?