2.0RC1 Multi-Wan: No default gateway/route change after link failure.

torsurfer

I'm also facing the same problem (2.0 RC1 snapshot on 5 April 2011).

I have put WAN (PPPoE connection) and OPT WAN (IP obtained from modem/router) in a Gateway group. If both WANs are up, the default route to the Internet is WAN (though I didn't select 'Default Gateway' for either of the WANs under System -> Routing) according to Diagnostics -> Routes.

But if WAN is down, the default route to the Internet is deleted from the routing table. Period. pfSense doesn't rewrite another default route using the OPT WAN interface. Hence, if I ping using the pfSense box itself, it says 'no trace route'. The only way for me to correct this is to go to the command line and issue 'route add default [IP of the modem/router]'.

Is this a bug that needs to be corrected or am I missing something?

ps: I need this to work so that 'fail-over' properly works for Squid on pfSense (where it's been configured to move its traffic via localhost).

Thanks.

eri--

Can you show me how many gateways you have?
It seems starnge to me that the code i have integrated for changing the default route is not working.

You should see "Default gateway down setting {$upgw} as default!" in the logs when this triggers.

torsurfer

Sure.

I have two gateways:

Grouped into:

This is the routing table before WAN is taken offline:

This is the routing table after WAN is taken offline:

The system log after WAN is taken offline:

So yeah, I'm curious why your latest codes didn't manage to kick-in to re-add a default route after the primary WAN is taken offline.

Let me know if you need further information.

Thanks for looking into this.

ps: Both my WANs uses the same gateway (ie. 219.93.218.177), it's just that the primary WAN is fibre and OPT WAN is ADSL.

eri--

can you get a copy of apinger.status under /tmp before and after this?

Also can you please explain to me why 192.168.2.1 is not on the routing table?

DarKcapricoRn

hello
I have almost the same problem as torsurfer:
2 Wans With Two ISP (SDSL and ADSL)
I would like to switch default gateway for openvpn can reconnect with the new gateway.
I did the update of 5 April 2011 and when I put down WAN 1, Default Gateway does not switch (though I have a Gateway Group "Internet").

thank you for interest because I try to find a way to remain connected Openvpn Although default gateway is not reachable.
Brice ;)

sot010174

I'm using almost the same setup on my test router.

Virtua is DHCP from ISP
WAN is PPPoe
Not the same gateway

However, I plan to deploy pfSense 2.0 on my production router which uses WAN as static and OPT as DHCP. Does that makes a difference?

Anyway here is my apinger before Virtua's gateway failure:

201.17.96.1|{Virtua's IP address}|VIRTUA|71|70|1302088482|8.179ms||none
200.222.117.78|{Wan's IP address}|WAN|71|70|1302088482|25.071ms||none

After disconnecting the modem cable and waiting for about 2 minutes:

201.17.96.1|{Virtua's IP address}|VIRTUA|383|293|1302088707|11.534ms|100.0%|down
200.222.117.78|{Wan's IP address}|WAN|383|336|1302088797|26.431ms|0.0%|none

System Log:

apinger: ALARM: VIRTUA(201.17.96.1) *** down ***
Apr 6 11:18:47 check_reload_status: reloading filter
Apr 6 11:18:48 php: : MONITOR: VIRTUA has high latency, removing from routing group
Apr 6 11:18:48 php: : MONITOR: VIRTUA has high latency, removing from routing group
Apr 6 11:18:48 php: : MONITOR: VIRTUA has high latency, removing from routing group

eri--

https://rcs.pfsense.org/projects/pfsense/repos/mainline/commits/19d9146637b36dd8cb3d42e300c44d4a8ccfe6d2
I committed the fix above because you guys do not seem to select the default checkbox.

But you could also try to edit the gateway that goes down and click the checkbox saying default.
After that redo the test and that should prove successful even without the commit above.

sot010174

My test router settings are attached to this post. Isn't like this how its supposed to be set?

Capture.PNG_thumb

eri--

Click edit on first gateway and show me the screenshot of that

sot010174

Here it is…

Capturar.PNG_thumb

eri--

Hrm, can you go to diagnostics->command and type under Php execute:
var_dump(return_gateways_array());
var_dump(return_gateways_status());

Paste the output here!

sot010174

Here it is:

array(2) {
["VIRTUA"]=>
array(10) {
["interface"]=>
string(3) "em2"
["gateway"]=>
string(11) "201.17.96.1"
["name"]=>
string(6) "VIRTUA"
["weight"]=>
string(1) "1"
["descr"]=>
string(32) "Interface VIRTUA Dynamic Gateway"
["defaultgw"]=>
string(0) ""
["dynamic"]=>
bool(true)
["monitor"]=>
string(11) "201.17.96.1"
["friendlyiface"]=>
string(4) "opt1"
["attribute"]=>
int(0)
}
["WAN"]=>
array(9) {
["interface"]=>
string(6) "pppoe0"
["gateway"]=>
string(14) "200.222.117.78"
["name"]=>
string(3) "WAN"
["weight"]=>
string(1) "1"
["descr"]=>
string(29) "Interface WAN Dynamic Gateway"
["dynamic"]=>
bool(true)
["monitor"]=>
string(14) "200.222.117.78"
["friendlyiface"]=>
string(3) "wan"
["attribute"]=>
int(1)
}
}
array(2) {
["201.17.96.1"]=>
array(7) {
["monitorip"]=>
string(11) "201.17.96.1"
["srcip"]=>
string(13) "{Virtua's IP Address}"
["name"]=>
string(6) "VIRTUA"
["lastcheck"]=>
string(31) "Wed, 06 Apr 2011 20:40:55 +0000"
["delay"]=>
string(8) "23.744ms"
["loss"]=>
string(4) "0.0%"
["status"]=>
string(4) "none"
}
["200.222.117.78"]=>
array(7) {
["monitorip"]=>
string(14) "200.222.117.78"
["srcip"]=>
string(12) "{Wan's IP}"
["name"]=>
string(3) "WAN"
["lastcheck"]=>
string(31) "Wed, 06 Apr 2011 20:40:55 +0000"
["delay"]=>
string(8) "25.985ms"
["loss"]=>
string(4) "0.0%"
["status"]=>
string(4) "none"
}
}

eri--

The latest snapshot should work on this.
Previous code might not work for your case.

sot010174

It did work indeed… Once. ???

I had both interfaces up, then as usual, I removed Virtua's cable and pfsense changed the route to WAN's gateway (YAY!!!). When Virtua was back online, pfsense recognized that and rewrote the rules back to virtua's gateway (Yay x2!).

However, the second time I tried it, it did rewrite the route to the WAN's gateway when Virtua failed but it didn't revert to Virtua when it was back online (Default gateway).

Oh well...

Here's the full log after restoring virtua's access and then removing the cable again:
Apr 6 22:32:51 kernel: arpresolve: can't allocate llinfo for 201.17.96.1
Apr 6 22:32:56 kernel: arpresolve: can't allocate llinfo for 201.17.96.1
Apr 6 22:33:05 kernel: arpresolve: can't allocate llinfo for 201.17.96.1
Apr 6 22:33:49 kernel: em2: link state changed to UP (Plugged cable back - 1st time)
Apr 6 22:33:49 check_reload_status: Linkup starting em2
Apr 6 22:33:49 php: : DEVD Ethernet attached event for opt1
Apr 6 22:33:49 php: : HOTPLUG: Configuring interface opt1
Apr 6 22:33:50 dhclient: PREINIT
Apr 6 22:33:50 dhclient[54413]: DHCPREQUEST on em2 to 255.255.255.255 port 67
Apr 6 22:33:50 dhclient[54413]: DHCPACK from 201.17.96.1
Apr 6 22:33:50 dhclient: REBOOT
Apr 6 22:33:50 dhclient: Starting add_new_address()
Apr 6 22:33:50 dhclient: ifconfig em2 inet 201.17.110.81 netmask 255.255.240.0 broadcast 201.17.111.255
Apr 6 22:33:50 dhclient: New IP Address (em2): 201.17.110.81
Apr 6 22:33:50 dhclient: New Subnet Mask (em2): 255.255.240.0
Apr 6 22:33:50 dhclient: New Broadcast Address (em2): 201.17.111.255
Apr 6 22:33:50 dhclient: New Routers (em2): 201.17.96.1
Apr 6 22:33:50 dhclient: Adding new routes to interface: em2
Apr 6 22:33:50 dhclient: /sbin/route add default 201.17.96.1
Apr 6 22:33:50 dhclient: Creating resolv.conf
Apr 6 22:33:50 check_reload_status: rc.newwanip starting em2
Apr 6 22:33:51 php: : rc.newwanip: Informational is starting em2.
Apr 6 22:33:51 dhclient[54413]: bound to 201.17.110.81 – renewal in 5400 seconds.
Apr 6 22:33:51 php: : rc.newwanip: on (IP address: 201.17.110.81) (interface: opt1) (real interface: em2).
Apr 6 22:33:51 php: : ROUTING: change default route to 201.17.96.1
Apr 6 22:33:51 apinger: alarm canceled: VIRTUA(201.17.96.1) *** down ***
Apr 6 22:33:51 apinger: Exiting on signal 15.
Apr 6 22:33:51 check_reload_status: reloading filter
Apr 6 22:33:52 php: : Gateways status could not be determined, considering all as up/active.
Apr 6 22:33:52 php: : Gateways status could not be determined, considering all as up/active.
Apr 6 22:33:52 php: : Gateways status could not be determined, considering all as up/active.
Apr 6 22:33:52 check_reload_status: reloading filter
Apr 6 22:33:53 apinger: Starting Alarm Pinger, apinger(13659)
Apr 6 22:33:53 php: : Gateways status could not be determined, considering all as up/active.
Apr 6 22:33:53 php: : Gateways status could not be determined, considering all as up/active.
Apr 6 22:33:53 php: : Gateways status could not be determined, considering all as up/active.
Apr 6 22:34:40 apinger: ALARM: VIRTUA(201.17.96.1) *** down *** - Second time it's down
Apr 6 22:34:50 check_reload_status: reloading filter
Apr 6 22:34:51 php: : Default gateway down setting WAN as default!
Apr 6 22:34:51 php: : MONITOR: VIRTUA is down, removing from routing group
Apr 6 22:34:51 php: : MONITOR: VIRTUA is down, removing from routing group
Apr 6 22:34:51 php: : MONITOR: VIRTUA is down, removing from routing group
Apr 6 22:35:07 dnsmasq[10400]: reading /etc/resolv.conf
Apr 6 22:35:07 dnsmasq[10400]: using nameserver 200.165.132.154#53
Apr 6 22:35:07 dnsmasq[10400]: using nameserver 200.149.55.142#53
Apr 6 22:35:07 dnsmasq[10400]: using nameserver 201.17.0.95#53
Apr 6 22:35:07 dnsmasq[10400]: using nameserver 201.17.0.94#53
Apr 6 22:35:13 apinger: alarm canceled: VIRTUA(201.17.96.1) *** down *** - UP again, but no route change :(
Apr 6 22:35:23 check_reload_status: reloading filter

Update 01 It seems if the GATEWAY fails, pfSense won't revert routes. However if the INTERFACE goes down, then it does rewrite. I'm going to test some more.

Update 02 That's it. if the gateway goes offline and the interface stays up, pfsense won't revert the routes BACK to the default GW (It does change the route to a backup however). Removing the cable causes the fix to work as intended.

torsurfer

Updated to snapshot 20110406-1323, and I'm thrilled to report it now works for me! ;D BIG thank you to Ermal for the hardwork. (Squid is 'cruising' along smoothly now for clients.)

This is what I've been doing before applying the Snapshot. I took Ermal's suggestion to edit the WAN gateway and 'ticked' the checkbox 'default gateway'. I didn't do this before, as I thought this was not needed if I wanted to load-balance the traffic. Anyhow…. I re-tested. Took down WAN, checked the routing table... Nope. No dice. Default route for OPT WAN was not written to the table. Pinging the Web from pfSense just returned 'no trace route'.

(Btw, in reply to Ermal's earlier question to me, I've no idea why 192.168.2.1 is not in the routing table. But I guess the routing table does have 192.168.2.0/24 (link #3). That ought to do it, I suppose. I've even manually keyed-in the IP in the OPT WAN gateway to replace 'dynamic', but it didn't change the routing table entry.)

After I applied the Snapshot though, everything just started to work. If WAN is down, a default route is written to the table based on OPT WAN. And when WAN is back up again, default route is re-written to the table based on WAN. I tested this four times, worked in all (yay!).

Think I have better luck than sot010174, in all my tests I just took down the gateway and default route gets properly written to the table (vice versa). I didn't need to take down the interface for this to work.

Good luck sot010174!

eri--

Good that it worked for you torsurfer.

sot010174 you skipped the interesting part of the log you posted at the end :)

sot010174

Hi ermal! It will work for me too eventually, I'm sure ! :D

That's the interesting bit. There aren't any additional entries on the log. I'll repeat the scenario and post everything here.

CAP01 This happens whenever I shutdown Virtua's gateway (notice that the interface stays up.)
CAP02 is the screenshot of the log panel before I restore Virtua's gateway.
CAP03 Virtua's gateway UP again. It recognizes that but doesn't restore the default route (Maintains Wan's route, previously set when Virtua's gw went down).

Can you please provide the command to activate the auto route changing mechanism if possible? Maybe if I force it to run even if it doesn't detect any changes it will restore the correct settings.

Oh, and another (bad) thing: I left the test router online and went to sleep. When I woke up this morning, no default route was set. I examined the system.log and saw that both ISPs had a busy night and there were times that both links went down. I'm going to conduct further testing and I'll report back.

Anyway, thanks a lot for looking into this issue.

CAP01.JPG_thumb

CAP02.JPG_thumb

CAP03.JPG_thumb

lnaimi

Then for a configuration loadbalance and failover is necessary to set the DefaultGateway?? I have 3 Gateway and I have not set any DefaultGateway, but I did not see how it behaves when a gateway is down.

eri--

lnaimi it is not neccessary to set it since pfSense will assume one by default, WAN.
Please do not hijack the thread since my comments are only related to the topic.

sot010174

Well an update:

I managed to test again (just once, I was in a hurry) the scenario which both gateways go down simultaneously.

1. This time, I simply removed both WAN and Virtua's cable. With the log cleared, I watched the results (2GWDown.log).

2. Cleared the log again and plugged in WAN's ethernet cable (WANUP.log). pfSense recognized the link up, restored the PPPOE link, but not the route (no default route in the table).

3. Cleared the log and plugged in Virtua's ethernet (VIRUPAFWAN.log).pfSense recognized the link up, got it's (same) IP configuration and WROTE Virtua's default route in the table.

I guess if I did as I did before (shutting down virtua's gateway and letting the interface stay up) I would end up with no route, as It happened previously. I think I even couldn't surf the web.

I tried to rename a zip file to .txt and attach to this message but I couldn't, so here's a link to it:

Logfiles:
http://www.mediafire.com/?xh4d08orgka6kyr