2.0RC1 Multi-Wan: No default gateway/route change after link failure.

sot010174

Hello again! :)

I've been noticing since I upgraded my pfsense router from BETA5 to RC1 that it no longer rewrites the default route after link failure. Whenever the default gateway goes offline, pfsense removes the default route and doesn't replace it (with the secondary one). When the link comes back online, it does rewrite back the default route.

The strange thing is: It used to do just that in Beta5. Why It doesn't work anymore? pfsense 1.2.3 release displays same behavior (no change).

Users can (oddly enough) access the internet normally (on the secondary link) if I don't enable squid. I cannot traceroute or ping anywhere though. The command returns "network unreacheable". NAT still works (which it didn't in this scenario on 1.2.3R).

Am I doing something stupid here?

Thank you in advance!

eri--

I committed this https://rcs.pfsense.org/projects/pfsense/repos/mainline/commits/1d60ed9bb4901c5b85813d0dab32630e8a135d41
Test new snapshots and see if it works correctly.

sot010174

Just upgraded and the issue persists… :(

But now pfsense won't delete the default route even if it's down.

Update 01: Just rebooted the router with the default link down. The default route was deleted with nothing on its place (no default gateway route). The secondary gateway lists as up (but not as "new" default route).

Update 02: After restoring the default link, pfsense rewrites the default route normally.

eri--

I do not think a snpashot with that commit has come out yet.
You should try tomorrow or a gitsync manually.

Please after testing give your system log here.

sot010174

Sorry about that. Quite embarassing. ;D

Anyway, I tried today's build, and still no changes :-.

Here's the log:

Scenario:
Removed cable from modem on Virtua Interface.
No changes on WAN (still online).

pr 5 22:05:32 php: /system_gateway_groups.php: ROUTING: change default route to {Virtua's gateway}
Apr 5 22:05:32 check_reload_status: reloading filter
Apr 5 22:05:32 apinger: Exiting on signal 15.
Apr 5 22:05:33 php: : Gateways status could not be determined, considering all as up/active.
Apr 5 22:05:33 php: : Gateways status could not be determined, considering all as up/active.
Apr 5 22:05:33 php: : Gateways status could not be determined, considering all as up/active.
Apr 5 22:05:33 apinger: Starting Alarm Pinger, apinger(23907)
Apr 5 22:06:32 apinger: ALARM: VIRTUA({Virtua's Gateway IP}) *** down ***
Apr 5 22:06:42 check_reload_status: reloading filter
Apr 5 22:06:42 php: : MONITOR: VIRTUA has high latency, removing from routing group
Apr 5 22:06:42 php: : MONITOR: VIRTUA has high latency, removing from routing group
Apr 5 22:06:42 php: : MONITOR: VIRTUA has high latency, removing from routing group

torsurfer

I'm also facing the same problem (2.0 RC1 snapshot on 5 April 2011).

I have put WAN (PPPoE connection) and OPT WAN (IP obtained from modem/router) in a Gateway group. If both WANs are up, the default route to the Internet is WAN (though I didn't select 'Default Gateway' for either of the WANs under System -> Routing) according to Diagnostics -> Routes.

But if WAN is down, the default route to the Internet is deleted from the routing table. Period. pfSense doesn't rewrite another default route using the OPT WAN interface. Hence, if I ping using the pfSense box itself, it says 'no trace route'. The only way for me to correct this is to go to the command line and issue 'route add default [IP of the modem/router]'.

Is this a bug that needs to be corrected or am I missing something?

ps: I need this to work so that 'fail-over' properly works for Squid on pfSense (where it's been configured to move its traffic via localhost).

Thanks.

eri--

Can you show me how many gateways you have?
It seems starnge to me that the code i have integrated for changing the default route is not working.

You should see "Default gateway down setting {$upgw} as default!" in the logs when this triggers.

torsurfer

Sure.

I have two gateways:

Grouped into:

This is the routing table before WAN is taken offline:

This is the routing table after WAN is taken offline:

The system log after WAN is taken offline:

So yeah, I'm curious why your latest codes didn't manage to kick-in to re-add a default route after the primary WAN is taken offline.

Let me know if you need further information.

Thanks for looking into this.

ps: Both my WANs uses the same gateway (ie. 219.93.218.177), it's just that the primary WAN is fibre and OPT WAN is ADSL.

eri--

can you get a copy of apinger.status under /tmp before and after this?

Also can you please explain to me why 192.168.2.1 is not on the routing table?

DarKcapricoRn

hello
I have almost the same problem as torsurfer:
2 Wans With Two ISP (SDSL and ADSL)
I would like to switch default gateway for openvpn can reconnect with the new gateway.
I did the update of 5 April 2011 and when I put down WAN 1, Default Gateway does not switch (though I have a Gateway Group "Internet").

thank you for interest because I try to find a way to remain connected Openvpn Although default gateway is not reachable.
Brice ;)

sot010174

I'm using almost the same setup on my test router.

Virtua is DHCP from ISP
WAN is PPPoe
Not the same gateway

However, I plan to deploy pfSense 2.0 on my production router which uses WAN as static and OPT as DHCP. Does that makes a difference?

Anyway here is my apinger before Virtua's gateway failure:

201.17.96.1|{Virtua's IP address}|VIRTUA|71|70|1302088482|8.179ms||none
200.222.117.78|{Wan's IP address}|WAN|71|70|1302088482|25.071ms||none

After disconnecting the modem cable and waiting for about 2 minutes:

201.17.96.1|{Virtua's IP address}|VIRTUA|383|293|1302088707|11.534ms|100.0%|down
200.222.117.78|{Wan's IP address}|WAN|383|336|1302088797|26.431ms|0.0%|none

System Log:

apinger: ALARM: VIRTUA(201.17.96.1) *** down ***
Apr 6 11:18:47 check_reload_status: reloading filter
Apr 6 11:18:48 php: : MONITOR: VIRTUA has high latency, removing from routing group
Apr 6 11:18:48 php: : MONITOR: VIRTUA has high latency, removing from routing group
Apr 6 11:18:48 php: : MONITOR: VIRTUA has high latency, removing from routing group

eri--

https://rcs.pfsense.org/projects/pfsense/repos/mainline/commits/19d9146637b36dd8cb3d42e300c44d4a8ccfe6d2
I committed the fix above because you guys do not seem to select the default checkbox.

But you could also try to edit the gateway that goes down and click the checkbox saying default.
After that redo the test and that should prove successful even without the commit above.

sot010174

My test router settings are attached to this post. Isn't like this how its supposed to be set?

Capture.PNG_thumb

eri--

Click edit on first gateway and show me the screenshot of that

sot010174

Here it is…

Capturar.PNG_thumb

eri--

Hrm, can you go to diagnostics->command and type under Php execute:
var_dump(return_gateways_array());
var_dump(return_gateways_status());

Paste the output here!

sot010174

Here it is:

array(2) {
["VIRTUA"]=>
array(10) {
["interface"]=>
string(3) "em2"
["gateway"]=>
string(11) "201.17.96.1"
["name"]=>
string(6) "VIRTUA"
["weight"]=>
string(1) "1"
["descr"]=>
string(32) "Interface VIRTUA Dynamic Gateway"
["defaultgw"]=>
string(0) ""
["dynamic"]=>
bool(true)
["monitor"]=>
string(11) "201.17.96.1"
["friendlyiface"]=>
string(4) "opt1"
["attribute"]=>
int(0)
}
["WAN"]=>
array(9) {
["interface"]=>
string(6) "pppoe0"
["gateway"]=>
string(14) "200.222.117.78"
["name"]=>
string(3) "WAN"
["weight"]=>
string(1) "1"
["descr"]=>
string(29) "Interface WAN Dynamic Gateway"
["dynamic"]=>
bool(true)
["monitor"]=>
string(14) "200.222.117.78"
["friendlyiface"]=>
string(3) "wan"
["attribute"]=>
int(1)
}
}
array(2) {
["201.17.96.1"]=>
array(7) {
["monitorip"]=>
string(11) "201.17.96.1"
["srcip"]=>
string(13) "{Virtua's IP Address}"
["name"]=>
string(6) "VIRTUA"
["lastcheck"]=>
string(31) "Wed, 06 Apr 2011 20:40:55 +0000"
["delay"]=>
string(8) "23.744ms"
["loss"]=>
string(4) "0.0%"
["status"]=>
string(4) "none"
}
["200.222.117.78"]=>
array(7) {
["monitorip"]=>
string(14) "200.222.117.78"
["srcip"]=>
string(12) "{Wan's IP}"
["name"]=>
string(3) "WAN"
["lastcheck"]=>
string(31) "Wed, 06 Apr 2011 20:40:55 +0000"
["delay"]=>
string(8) "25.985ms"
["loss"]=>
string(4) "0.0%"
["status"]=>
string(4) "none"
}
}

eri--

The latest snapshot should work on this.
Previous code might not work for your case.

sot010174

It did work indeed… Once. ???

I had both interfaces up, then as usual, I removed Virtua's cable and pfsense changed the route to WAN's gateway (YAY!!!). When Virtua was back online, pfsense recognized that and rewrote the rules back to virtua's gateway (Yay x2!).

However, the second time I tried it, it did rewrite the route to the WAN's gateway when Virtua failed but it didn't revert to Virtua when it was back online (Default gateway).

Oh well...

Here's the full log after restoring virtua's access and then removing the cable again:
Apr 6 22:32:51 kernel: arpresolve: can't allocate llinfo for 201.17.96.1
Apr 6 22:32:56 kernel: arpresolve: can't allocate llinfo for 201.17.96.1
Apr 6 22:33:05 kernel: arpresolve: can't allocate llinfo for 201.17.96.1
Apr 6 22:33:49 kernel: em2: link state changed to UP (Plugged cable back - 1st time)
Apr 6 22:33:49 check_reload_status: Linkup starting em2
Apr 6 22:33:49 php: : DEVD Ethernet attached event for opt1
Apr 6 22:33:49 php: : HOTPLUG: Configuring interface opt1
Apr 6 22:33:50 dhclient: PREINIT
Apr 6 22:33:50 dhclient[54413]: DHCPREQUEST on em2 to 255.255.255.255 port 67
Apr 6 22:33:50 dhclient[54413]: DHCPACK from 201.17.96.1
Apr 6 22:33:50 dhclient: REBOOT
Apr 6 22:33:50 dhclient: Starting add_new_address()
Apr 6 22:33:50 dhclient: ifconfig em2 inet 201.17.110.81 netmask 255.255.240.0 broadcast 201.17.111.255
Apr 6 22:33:50 dhclient: New IP Address (em2): 201.17.110.81
Apr 6 22:33:50 dhclient: New Subnet Mask (em2): 255.255.240.0
Apr 6 22:33:50 dhclient: New Broadcast Address (em2): 201.17.111.255
Apr 6 22:33:50 dhclient: New Routers (em2): 201.17.96.1
Apr 6 22:33:50 dhclient: Adding new routes to interface: em2
Apr 6 22:33:50 dhclient: /sbin/route add default 201.17.96.1
Apr 6 22:33:50 dhclient: Creating resolv.conf
Apr 6 22:33:50 check_reload_status: rc.newwanip starting em2
Apr 6 22:33:51 php: : rc.newwanip: Informational is starting em2.
Apr 6 22:33:51 dhclient[54413]: bound to 201.17.110.81 – renewal in 5400 seconds.
Apr 6 22:33:51 php: : rc.newwanip: on (IP address: 201.17.110.81) (interface: opt1) (real interface: em2).
Apr 6 22:33:51 php: : ROUTING: change default route to 201.17.96.1
Apr 6 22:33:51 apinger: alarm canceled: VIRTUA(201.17.96.1) *** down ***
Apr 6 22:33:51 apinger: Exiting on signal 15.
Apr 6 22:33:51 check_reload_status: reloading filter
Apr 6 22:33:52 php: : Gateways status could not be determined, considering all as up/active.
Apr 6 22:33:52 php: : Gateways status could not be determined, considering all as up/active.
Apr 6 22:33:52 php: : Gateways status could not be determined, considering all as up/active.
Apr 6 22:33:52 check_reload_status: reloading filter
Apr 6 22:33:53 apinger: Starting Alarm Pinger, apinger(13659)
Apr 6 22:33:53 php: : Gateways status could not be determined, considering all as up/active.
Apr 6 22:33:53 php: : Gateways status could not be determined, considering all as up/active.
Apr 6 22:33:53 php: : Gateways status could not be determined, considering all as up/active.
Apr 6 22:34:40 apinger: ALARM: VIRTUA(201.17.96.1) *** down *** - Second time it's down
Apr 6 22:34:50 check_reload_status: reloading filter
Apr 6 22:34:51 php: : Default gateway down setting WAN as default!
Apr 6 22:34:51 php: : MONITOR: VIRTUA is down, removing from routing group
Apr 6 22:34:51 php: : MONITOR: VIRTUA is down, removing from routing group
Apr 6 22:34:51 php: : MONITOR: VIRTUA is down, removing from routing group
Apr 6 22:35:07 dnsmasq[10400]: reading /etc/resolv.conf
Apr 6 22:35:07 dnsmasq[10400]: using nameserver 200.165.132.154#53
Apr 6 22:35:07 dnsmasq[10400]: using nameserver 200.149.55.142#53
Apr 6 22:35:07 dnsmasq[10400]: using nameserver 201.17.0.95#53
Apr 6 22:35:07 dnsmasq[10400]: using nameserver 201.17.0.94#53
Apr 6 22:35:13 apinger: alarm canceled: VIRTUA(201.17.96.1) *** down *** - UP again, but no route change :(
Apr 6 22:35:23 check_reload_status: reloading filter

Update 01 It seems if the GATEWAY fails, pfSense won't revert routes. However if the INTERFACE goes down, then it does rewrite. I'm going to test some more.

Update 02 That's it. if the gateway goes offline and the interface stays up, pfsense won't revert the routes BACK to the default GW (It does change the route to a backup however). Removing the cable causes the fix to work as intended.

torsurfer

Updated to snapshot 20110406-1323, and I'm thrilled to report it now works for me! ;D BIG thank you to Ermal for the hardwork. (Squid is 'cruising' along smoothly now for clients.)

This is what I've been doing before applying the Snapshot. I took Ermal's suggestion to edit the WAN gateway and 'ticked' the checkbox 'default gateway'. I didn't do this before, as I thought this was not needed if I wanted to load-balance the traffic. Anyhow…. I re-tested. Took down WAN, checked the routing table... Nope. No dice. Default route for OPT WAN was not written to the table. Pinging the Web from pfSense just returned 'no trace route'.

(Btw, in reply to Ermal's earlier question to me, I've no idea why 192.168.2.1 is not in the routing table. But I guess the routing table does have 192.168.2.0/24 (link #3). That ought to do it, I suppose. I've even manually keyed-in the IP in the OPT WAN gateway to replace 'dynamic', but it didn't change the routing table entry.)

After I applied the Snapshot though, everything just started to work. If WAN is down, a default route is written to the table based on OPT WAN. And when WAN is back up again, default route is re-written to the table based on WAN. I tested this four times, worked in all (yay!).

Think I have better luck than sot010174, in all my tests I just took down the gateway and default route gets properly written to the table (vice versa). I didn't need to take down the interface for this to work.

Good luck sot010174!