2.0RC1 Multi-Wan: No default gateway/route change after link failure.



  • Hello again!  :)

    I've been noticing since I upgraded my pfsense router from BETA5 to RC1 that it no longer rewrites the default route after link failure. Whenever the default gateway goes offline, pfsense removes the default route and doesn't replace it (with the secondary one). When the link comes back online, it does rewrite back the default route.

    The strange thing is: It used to do just that in Beta5. Why It doesn't work anymore? pfsense 1.2.3 release displays same behavior (no change).

    Users can (oddly enough) access the internet normally (on the secondary link) if I don't enable squid. I cannot traceroute or ping anywhere though. The command returns "network unreacheable". NAT still works (which it didn't in this scenario on 1.2.3R).

    Am I doing something stupid here?

    Thank you in advance!





  • Just upgraded and the issue persists… :(

    But now pfsense won't delete the default route even if it's down.

    Update 01: Just rebooted the router with the default link down. The default route was deleted with nothing on its place (no default gateway route). The secondary gateway lists as up (but not as "new" default route).

    Update 02: After restoring the default link, pfsense rewrites the default route normally.



  • I do not think a snpashot with that commit has come out yet.
    You should try tomorrow or a gitsync manually.

    Please after testing give your system log here.



  • Sorry about that. Quite embarassing. ;D

    Anyway, I tried today's build, and still no changes  :-.

    Here's the log:

    Scenario:
    Removed cable from modem on Virtua Interface.
    No changes on WAN (still online).

    pr 5 22:05:32 php: /system_gateway_groups.php: ROUTING: change default route to {Virtua's gateway}
    Apr 5 22:05:32 check_reload_status: reloading filter
    Apr 5 22:05:32 apinger: Exiting on signal 15.
    Apr 5 22:05:33 php: : Gateways status could not be determined, considering all as up/active.
    Apr 5 22:05:33 php: : Gateways status could not be determined, considering all as up/active.
    Apr 5 22:05:33 php: : Gateways status could not be determined, considering all as up/active.
    Apr 5 22:05:33 apinger: Starting Alarm Pinger, apinger(23907)
    Apr 5 22:06:32 apinger: ALARM: VIRTUA({Virtua's Gateway IP}) *** down ***
    Apr 5 22:06:42 check_reload_status: reloading filter
    Apr 5 22:06:42 php: : MONITOR: VIRTUA has high latency, removing from routing group
    Apr 5 22:06:42 php: : MONITOR: VIRTUA has high latency, removing from routing group
    Apr 5 22:06:42 php: : MONITOR: VIRTUA has high latency, removing from routing group



  • I'm also facing the same problem (2.0 RC1 snapshot on 5 April 2011).

    I have put WAN (PPPoE connection) and OPT WAN (IP obtained from modem/router) in a Gateway group. If both WANs are up, the default route to the Internet is WAN (though I didn't select 'Default Gateway' for either of the WANs under System -> Routing) according to Diagnostics -> Routes.

    But if WAN is down, the default route to the Internet is deleted from the routing table. Period. pfSense doesn't rewrite another default route using the OPT WAN interface. Hence, if I ping using the pfSense box itself, it says 'no trace route'. The only way for me to correct this is to go to the command line and issue 'route add default [IP of the modem/router]'.

    Is this a bug that needs to be corrected or am I missing something?

    ps: I need this to work so that 'fail-over' properly works for Squid on pfSense (where it's been configured to move its traffic via localhost).

    Thanks.



  • Can you show me how many gateways you have?
    It seems starnge to me that the code i have integrated for changing the default route is not working.

    You should see "Default gateway down setting {$upgw} as default!" in the logs when this triggers.



  • Sure.

    I have two gateways:

    Grouped into:

    This is the routing table before WAN is taken offline:

    This is the routing table after WAN is taken offline:

    The system log after WAN is taken offline:

    So yeah, I'm curious why your latest codes didn't manage to kick-in to re-add a default route after the primary WAN is taken offline.

    Let me know if you need further information.

    Thanks for looking into this.

    ps: Both my WANs uses the same gateway (ie. 219.93.218.177), it's just that the primary WAN is fibre and OPT WAN is ADSL.



  • can you get a copy of apinger.status under /tmp before and after this?

    Also can you please explain to me why 192.168.2.1 is not on the routing table?



  • hello
    I have almost the same problem as torsurfer:
    2 Wans With Two ISP (SDSL and ADSL)
    I would like to switch default gateway for openvpn can reconnect with the new gateway.
    I did the update of 5 April 2011 and when I put down WAN 1, Default Gateway does not switch (though I have a Gateway Group "Internet").

    thank you for interest because I try to find a way to remain connected Openvpn Although default gateway is not reachable.
    Brice  ;)



  • I'm using almost the same setup on my test router.

    Virtua is DHCP from ISP
    WAN is PPPoe
    Not the same gateway

    However, I plan to deploy pfSense 2.0 on my production router which uses WAN as static and OPT as DHCP. Does that makes a difference?

    Anyway here is my apinger before Virtua's gateway failure:

    201.17.96.1|{Virtua's IP address}|VIRTUA|71|70|1302088482|8.179ms||none
    200.222.117.78|{Wan's IP address}|WAN|71|70|1302088482|25.071ms||none

    After disconnecting the modem cable and waiting for about 2 minutes:

    201.17.96.1|{Virtua's IP address}|VIRTUA|383|293|1302088707|11.534ms|100.0%|down
    200.222.117.78|{Wan's IP address}|WAN|383|336|1302088797|26.431ms|0.0%|none

    System Log:

    apinger: ALARM: VIRTUA(201.17.96.1) *** down ***
    Apr 6 11:18:47 check_reload_status: reloading filter
    Apr 6 11:18:48 php: : MONITOR: VIRTUA has high latency, removing from routing group
    Apr 6 11:18:48 php: : MONITOR: VIRTUA has high latency, removing from routing group
    Apr 6 11:18:48 php: : MONITOR: VIRTUA has high latency, removing from routing group



  • https://rcs.pfsense.org/projects/pfsense/repos/mainline/commits/19d9146637b36dd8cb3d42e300c44d4a8ccfe6d2
    I committed the fix above because you guys do not seem to select the default checkbox.

    But you could also try to edit the gateway that goes down and click the checkbox saying default.
    After that redo the test and that should prove successful even without the commit above.



  • My test router settings are attached to this post. Isn't like this how its supposed to be set?




  • Click edit on first gateway and show me the screenshot of that



  • Here it is…




  • Hrm, can you go to diagnostics->command and type under Php execute:
    var_dump(return_gateways_array());
    var_dump(return_gateways_status());

    Paste the output here!



  • Here it is:

    array(2) {
      ["VIRTUA"]=>
      array(10) {
        ["interface"]=>
        string(3) "em2"
        ["gateway"]=>
        string(11) "201.17.96.1"
        ["name"]=>
        string(6) "VIRTUA"
        ["weight"]=>
        string(1) "1"
        ["descr"]=>
        string(32) "Interface VIRTUA Dynamic Gateway"
        ["defaultgw"]=>
        string(0) ""
        ["dynamic"]=>
        bool(true)
        ["monitor"]=>
        string(11) "201.17.96.1"
        ["friendlyiface"]=>
        string(4) "opt1"
        ["attribute"]=>
        int(0)
      }
      ["WAN"]=>
      array(9) {
        ["interface"]=>
        string(6) "pppoe0"
        ["gateway"]=>
        string(14) "200.222.117.78"
        ["name"]=>
        string(3) "WAN"
        ["weight"]=>
        string(1) "1"
        ["descr"]=>
        string(29) "Interface WAN Dynamic Gateway"
        ["dynamic"]=>
        bool(true)
        ["monitor"]=>
        string(14) "200.222.117.78"
        ["friendlyiface"]=>
        string(3) "wan"
        ["attribute"]=>
        int(1)
      }
    }
    array(2) {
      ["201.17.96.1"]=>
      array(7) {
        ["monitorip"]=>
        string(11) "201.17.96.1"
        ["srcip"]=>
        string(13) "{Virtua's IP Address}"
        ["name"]=>
        string(6) "VIRTUA"
        ["lastcheck"]=>
        string(31) "Wed, 06 Apr 2011 20:40:55 +0000"
        ["delay"]=>
        string(8) "23.744ms"
        ["loss"]=>
        string(4) "0.0%"
        ["status"]=>
        string(4) "none"
      }
      ["200.222.117.78"]=>
      array(7) {
        ["monitorip"]=>
        string(14) "200.222.117.78"
        ["srcip"]=>
        string(12) "{Wan's IP}"
        ["name"]=>
        string(3) "WAN"
        ["lastcheck"]=>
        string(31) "Wed, 06 Apr 2011 20:40:55 +0000"
        ["delay"]=>
        string(8) "25.985ms"
        ["loss"]=>
        string(4) "0.0%"
        ["status"]=>
        string(4) "none"
      }
    }



  • The latest snapshot should work on this.
    Previous code might not work for your case.



  • It did work indeed… Once. ???

    I had both interfaces up, then as usual, I removed Virtua's cable and pfsense changed the route to WAN's gateway (YAY!!!). When Virtua was back online, pfsense recognized that and rewrote the rules back to virtua's gateway (Yay x2!).

    However, the second time I tried it, it did rewrite the route to the WAN's gateway when Virtua failed but it didn't revert to Virtua when it was back online (Default gateway).

    Oh well...

    Here's the full log after restoring virtua's access and then removing the cable again:
    Apr 6 22:32:51 kernel: arpresolve: can't allocate llinfo for 201.17.96.1
    Apr 6 22:32:56 kernel: arpresolve: can't allocate llinfo for 201.17.96.1
    Apr 6 22:33:05 kernel: arpresolve: can't allocate llinfo for 201.17.96.1
    Apr 6 22:33:49 kernel: em2: link state changed to UP (Plugged cable back - 1st time)
    Apr 6 22:33:49 check_reload_status: Linkup starting em2
    Apr 6 22:33:49 php: : DEVD Ethernet attached event for opt1
    Apr 6 22:33:49 php: : HOTPLUG: Configuring interface opt1
    Apr 6 22:33:50 dhclient: PREINIT
    Apr 6 22:33:50 dhclient[54413]: DHCPREQUEST on em2 to 255.255.255.255 port 67
    Apr 6 22:33:50 dhclient[54413]: DHCPACK from 201.17.96.1
    Apr 6 22:33:50 dhclient: REBOOT
    Apr 6 22:33:50 dhclient: Starting add_new_address()
    Apr 6 22:33:50 dhclient: ifconfig em2 inet 201.17.110.81 netmask 255.255.240.0 broadcast 201.17.111.255
    Apr 6 22:33:50 dhclient: New IP Address (em2): 201.17.110.81
    Apr 6 22:33:50 dhclient: New Subnet Mask (em2): 255.255.240.0
    Apr 6 22:33:50 dhclient: New Broadcast Address (em2): 201.17.111.255
    Apr 6 22:33:50 dhclient: New Routers (em2): 201.17.96.1
    Apr 6 22:33:50 dhclient: Adding new routes to interface: em2
    Apr 6 22:33:50 dhclient: /sbin/route add default 201.17.96.1
    Apr 6 22:33:50 dhclient: Creating resolv.conf
    Apr 6 22:33:50 check_reload_status: rc.newwanip starting em2
    Apr 6 22:33:51 php: : rc.newwanip: Informational is starting em2.
    Apr 6 22:33:51 dhclient[54413]: bound to 201.17.110.81 – renewal in 5400 seconds.
    Apr 6 22:33:51 php: : rc.newwanip: on (IP address: 201.17.110.81) (interface: opt1) (real interface: em2).
    Apr 6 22:33:51 php: : ROUTING: change default route to 201.17.96.1
    Apr 6 22:33:51 apinger: alarm canceled: VIRTUA(201.17.96.1) *** down ***
    Apr 6 22:33:51 apinger: Exiting on signal 15.
    Apr 6 22:33:51 check_reload_status: reloading filter
    Apr 6 22:33:52 php: : Gateways status could not be determined, considering all as up/active.
    Apr 6 22:33:52 php: : Gateways status could not be determined, considering all as up/active.
    Apr 6 22:33:52 php: : Gateways status could not be determined, considering all as up/active.
    Apr 6 22:33:52 check_reload_status: reloading filter
    Apr 6 22:33:53 apinger: Starting Alarm Pinger, apinger(13659)
    Apr 6 22:33:53 php: : Gateways status could not be determined, considering all as up/active.
    Apr 6 22:33:53 php: : Gateways status could not be determined, considering all as up/active.
    Apr 6 22:33:53 php: : Gateways status could not be determined, considering all as up/active.
    Apr 6 22:34:40 apinger: ALARM: VIRTUA(201.17.96.1) *** down *** - Second time it's down
    Apr 6 22:34:50 check_reload_status: reloading filter
    Apr 6 22:34:51 php: : Default gateway down setting WAN as default!
    Apr 6 22:34:51 php: : MONITOR: VIRTUA is down, removing from routing group
    Apr 6 22:34:51 php: : MONITOR: VIRTUA is down, removing from routing group
    Apr 6 22:34:51 php: : MONITOR: VIRTUA is down, removing from routing group
    Apr 6 22:35:07 dnsmasq[10400]: reading /etc/resolv.conf
    Apr 6 22:35:07 dnsmasq[10400]: using nameserver 200.165.132.154#53
    Apr 6 22:35:07 dnsmasq[10400]: using nameserver 200.149.55.142#53
    Apr 6 22:35:07 dnsmasq[10400]: using nameserver 201.17.0.95#53
    Apr 6 22:35:07 dnsmasq[10400]: using nameserver 201.17.0.94#53
    Apr 6 22:35:13 apinger: alarm canceled: VIRTUA(201.17.96.1) *** down *** - UP again, but no route change  :(
    Apr 6 22:35:23 check_reload_status: reloading filter

    Update 01 It seems if the GATEWAY fails, pfSense won't revert routes. However if the INTERFACE goes down, then it does rewrite. I'm going to test some more.

    Update 02 That's it. if the gateway goes offline and the interface stays up, pfsense won't revert the routes BACK to the default GW (It does change the route to a backup however). Removing the cable causes the fix to work as intended.



  • Updated to snapshot 20110406-1323, and I'm thrilled to report it now works for me!  ;D BIG thank you to Ermal for the hardwork. (Squid is 'cruising' along smoothly now for clients.)

    This is what I've been doing before applying the Snapshot. I took Ermal's suggestion to edit the WAN gateway and 'ticked' the checkbox 'default gateway'. I didn't do this before, as I thought this was not needed if I wanted to load-balance the traffic. Anyhow…. I re-tested. Took down WAN, checked the routing table... Nope. No dice. Default route for OPT WAN was not written to the table. Pinging the Web from pfSense just returned 'no trace route'.

    (Btw, in reply to Ermal's earlier question to me, I've no idea why 192.168.2.1 is not in the routing table. But I guess the routing table does have 192.168.2.0/24 (link #3). That ought to do it, I suppose. I've even manually keyed-in the IP in the OPT WAN gateway to replace 'dynamic', but it didn't change the routing table entry.)

    After I applied the Snapshot though, everything just started to work. If WAN is down, a default route is written to the table based on OPT WAN. And when WAN is back up again, default route is re-written to the table based on WAN. I tested this four times, worked in all (yay!).

    Think I have better luck than sot010174, in all my tests I just took down the gateway and default route gets properly written to the table (vice versa). I didn't need to take down the interface for this to work.

    Good luck sot010174!



  • Good that it worked for you torsurfer.

    sot010174 you skipped the interesting part of the log you posted at the end :)



  • Hi ermal! It will work for me too eventually, I'm sure !  :D

    That's the interesting bit. There aren't any additional entries on the log. I'll repeat the scenario and post everything here.

    CAP01 This happens whenever I shutdown Virtua's gateway (notice that the interface stays up.)
    CAP02 is the screenshot of the log panel before I restore Virtua's gateway.
    CAP03 Virtua's gateway UP again. It recognizes that but doesn't restore the default route (Maintains Wan's route, previously set when Virtua's gw went down).

    Can you please provide the command to activate the auto route changing mechanism if possible? Maybe if I force it to run even if it doesn't detect any changes it will restore the correct settings.

    Oh, and another (bad) thing: I left the test router online and went to sleep. When I woke up this morning, no default route was set. I examined the system.log and saw that both ISPs had a busy night and there were times that both links went down. I'm going to conduct further testing and I'll report back.

    Anyway, thanks a lot for looking into this issue.








  • Then for a configuration loadbalance and failover is necessary to set the DefaultGateway?? I have 3 Gateway and I have not set any DefaultGateway, but I did not see how it behaves when a gateway is down.



  • lnaimi it is not neccessary to set it since pfSense will assume one by default, WAN.
    Please do not hijack the thread since my comments are only related to the topic.



  • Well an update:

    I managed to test again (just once, I was in a hurry) the scenario which both gateways go down simultaneously.

    1. This time, I simply removed both WAN and Virtua's cable. With the log cleared, I watched the results (2GWDown.log).

    2. Cleared the log again and plugged in WAN's ethernet cable (WANUP.log). pfSense recognized the link up, restored the PPPOE link, but not the route (no default route in the table).

    3. Cleared the log and plugged in Virtua's ethernet (VIRUPAFWAN.log).pfSense recognized the link up, got it's (same) IP configuration and WROTE Virtua's default route in the table.

    I guess if I did as I did before (shutting down virtua's gateway and letting the interface stay up) I would end up with no route, as It happened previously. I think I even couldn't surf the web.

    I tried to rename a zip file to .txt and attach to this message but I couldn't, so here's a link to it:

    Logfiles:
    http://www.mediafire.com/?xh4d08orgka6kyr



  • Thanks



  • I seem to have this exact same problem.  I am still on RC1 official build AMD64.  I have tried with a default route and without a default route checked.  netstat -rn just shows the default route regardless of what happens.  Both of my connections are ethernet to fiber feeds, so they don't go down very often.  I will follow this thread and upgrade to a build when it has been confirmed fixed.  If there is a build that this should work I will test it and report back.

    Thanks,



  • I have the same problem (2.0-RC1 (i386) built on Tue Apr 19 00:04:38 EDT 2011).
    If the gateway goes offline and the interface stays up, pfsense won't revert the routes BACK to the default GW.
    Is there a solution for this problem?

    Thank you.



  • Same problem on 2.0-RC1 (i386) built on Mon May 2 05:54:35 EDT 2011 .
    I'm likely to have an eye at source code a bit later.



  • After May, 18th again the same problem! :(

    It was necessary to be rolled away on the Snapshot "pfSense-Full-Update-2.0-RC1-i386-20110517-2328.tgz" - on it the route by default switches automatically:
    –--------
    May 20 2:36:45 PM pfsense php:: Default gateway down setting 2_GW as default!
    May 20 2:36:45 PM pfsense php:: MONITOR: 1_GW is down, removing from routing group

    On new (after May, 18th) Snapshots the first line in system.log missed!

    Sergu61



  • The change has been backed out since it caused issues and i have plans to put a knob under system->advanced to allow enabling it.

    Give me time to come to it.



  • @ermal:

    The change has been backed out since it caused issues and i have plans to put a knob under system->advanced to allow enabling it.

    Give me time to come to it.

    Please include a "timed refresh" to this feature. In my case I think it would solve all my problems, something like:

    Recheck default gateway route every ___ minutes. (and it would call the auto-change function or write the default gateway route if all gateways are down).



  • Hi Guys,
        Just curious what the status of this issue is…  Has the fix been incorporated into the latest snapshot yet??



  • In the file system_advanced_misc.php
    in line:
    …..
    />
    …..

    need to change:
    lb_use_sticky    ->      gw_switch_default.

    After that, everything starts to work!



  • I just fixed this.



  • Thank you!
    Now the system has become even more comfortable!



  • Awesome. I've been waiting for this fix as well.  How long does it usually take for the snapshot to get built?



  • I'm a little corrected. For me it works better.
    1. Restores the configured default gateway, if gateway is up.
    2. Bypasses until IPv6 gateways.
    3. Will don't touch routing without the need for.

    
            if (isset($config['system']['gw_switch_default'])) {
            /*
             * NOTE: The code below is meant to replace the default gateway when it goes down.
             *      This facilitates services running on pfSense itself and are not handled by a PBR to continue working.
             */
            $upgw = "";
            $dfltgwfound = false;
            foreach ($gateways_arr as $gwname => $gwsttng) {
                if (is_ipaddrv4($gateways_arr[$gwname]['gateway'])) {
                    if (isset($gwsttng['defaultgw'])) {
                            $dfltgwfound = true;
                            if (!stristr($gateways_status[$gwname]['status'], "down"))
                                    $upgw = $gwname;
                    }
                    /* Keep a record of the last up gateway */
                    if (empty($upgw) && !stristr($gateways_status[$gwname]['status'], "down"))
                            $upgw = $gwname;
                }
            }
            if ($dfltgwfound == false) {
                    $gwname = convert_friendly_interface_to_friendly_descr("wan");
                    if (!stristr($gateways_status[$gwname]['status'], "down"))
                            $upgw = $gwname;
            }
            if (!empty($upgw)) {
                    if ($gateways_arr[$upgw]['gateway'] == "dynamic")
                            $gateways_arr[$upgw]['gateway'] = get_interface_gateway($gateways_arr[$upgw]['friendlyiface']);
                    if (is_ipaddr($gateways_arr[$upgw]['gateway'])) {
                            exec("/usr/bin/netstat -rnf inet | /usr/bin/awk '/default/ {print $2}'", $currgwip);
                            if (!stristr($gateways_arr[$upgw]['gateway'], $currgwip[0])) {
                                log_error("Setting default gateway '{$upgw}' will replace (IP={$currgwip[0]})!");
                                mwexec("/sbin/route delete -inet default; /sbin/route add -inet default {$gateways_arr[$upgw]['gateway']}");
                            }
                    }
            }
            unset($upgw, $dfltgwfound, $currgwip, $gwname, $gwsttng);
            }
    
    

    Test is not easy, unfortunately. Frequent kernel panic when the twitching interfaces.



  • Hi guys. I'm having the same troubles dealing with failover gateway. I followed your discussion and now I can reach the correct behavior of switching to the secondary gateway if the default fails.
    As you wrote, I had to declare a default gateway; if I do NOT flag "Default Gateway" on any gateway, no gateway switch is done at all.
    So, if I flag "Default Gateway" on one of the gw, the switching is done correctly and the default route is changed. But here arises an issue: when the offline gateways returns online, the routing doesn't come back to this (default) gateway. Never more. Even after minutes and minutes.
    As consequence, if the secondary gateway goes down (also after many days), my network is definetly offline from internet access.

    It seems that pfSense can't switch away from an offline gateway if it is not the default gateway.
    Am I missing something? I really didn't understand if I can patch some php code or this issue is still open …

    Many thanks !!!!

    Note1: My WANs NICs always stay up
    Note2: My "Allow default gateway switching" is flagged on Advanced -> Miscellaneous settings
    Running 2.0-RC2 (amd64), built on Tue Jun 7 15:17:21 EDT 2011


Locked