2.0RC1 Multi-Wan: No default gateway/route change after link failure.

torsurfer

Updated to snapshot 20110406-1323, and I'm thrilled to report it now works for me! ;D BIG thank you to Ermal for the hardwork. (Squid is 'cruising' along smoothly now for clients.)

This is what I've been doing before applying the Snapshot. I took Ermal's suggestion to edit the WAN gateway and 'ticked' the checkbox 'default gateway'. I didn't do this before, as I thought this was not needed if I wanted to load-balance the traffic. Anyhow…. I re-tested. Took down WAN, checked the routing table... Nope. No dice. Default route for OPT WAN was not written to the table. Pinging the Web from pfSense just returned 'no trace route'.

(Btw, in reply to Ermal's earlier question to me, I've no idea why 192.168.2.1 is not in the routing table. But I guess the routing table does have 192.168.2.0/24 (link #3). That ought to do it, I suppose. I've even manually keyed-in the IP in the OPT WAN gateway to replace 'dynamic', but it didn't change the routing table entry.)

After I applied the Snapshot though, everything just started to work. If WAN is down, a default route is written to the table based on OPT WAN. And when WAN is back up again, default route is re-written to the table based on WAN. I tested this four times, worked in all (yay!).

Think I have better luck than sot010174, in all my tests I just took down the gateway and default route gets properly written to the table (vice versa). I didn't need to take down the interface for this to work.

Good luck sot010174!

eri--

Good that it worked for you torsurfer.

sot010174 you skipped the interesting part of the log you posted at the end :)

sot010174

Hi ermal! It will work for me too eventually, I'm sure ! :D

That's the interesting bit. There aren't any additional entries on the log. I'll repeat the scenario and post everything here.

CAP01 This happens whenever I shutdown Virtua's gateway (notice that the interface stays up.)
CAP02 is the screenshot of the log panel before I restore Virtua's gateway.
CAP03 Virtua's gateway UP again. It recognizes that but doesn't restore the default route (Maintains Wan's route, previously set when Virtua's gw went down).

Can you please provide the command to activate the auto route changing mechanism if possible? Maybe if I force it to run even if it doesn't detect any changes it will restore the correct settings.

Oh, and another (bad) thing: I left the test router online and went to sleep. When I woke up this morning, no default route was set. I examined the system.log and saw that both ISPs had a busy night and there were times that both links went down. I'm going to conduct further testing and I'll report back.

Anyway, thanks a lot for looking into this issue.

CAP01.JPG_thumb

CAP02.JPG_thumb

CAP03.JPG_thumb

lnaimi

Then for a configuration loadbalance and failover is necessary to set the DefaultGateway?? I have 3 Gateway and I have not set any DefaultGateway, but I did not see how it behaves when a gateway is down.

eri--

lnaimi it is not neccessary to set it since pfSense will assume one by default, WAN.
Please do not hijack the thread since my comments are only related to the topic.

sot010174

Well an update:

I managed to test again (just once, I was in a hurry) the scenario which both gateways go down simultaneously.

1. This time, I simply removed both WAN and Virtua's cable. With the log cleared, I watched the results (2GWDown.log).

2. Cleared the log again and plugged in WAN's ethernet cable (WANUP.log). pfSense recognized the link up, restored the PPPOE link, but not the route (no default route in the table).

3. Cleared the log and plugged in Virtua's ethernet (VIRUPAFWAN.log).pfSense recognized the link up, got it's (same) IP configuration and WROTE Virtua's default route in the table.

I guess if I did as I did before (shutting down virtua's gateway and letting the interface stay up) I would end up with no route, as It happened previously. I think I even couldn't surf the web.

I tried to rename a zip file to .txt and attach to this message but I couldn't, so here's a link to it:

Logfiles:
http://www.mediafire.com/?xh4d08orgka6kyr

lnaimi

Thanks

crzykidd

I seem to have this exact same problem. I am still on RC1 official build AMD64. I have tried with a default route and without a default route checked. netstat -rn just shows the default route regardless of what happens. Both of my connections are ethernet to fiber feeds, so they don't go down very often. I will follow this thread and upgrade to a build when it has been confirmed fixed. If there is a build that this should work I will test it and report back.

Thanks,

DimC

I have the same problem (2.0-RC1 (i386) built on Tue Apr 19 00:04:38 EDT 2011).
If the gateway goes offline and the interface stays up, pfsense won't revert the routes BACK to the default GW.
Is there a solution for this problem?

Thank you.

lp

Same problem on 2.0-RC1 (i386) built on Mon May 2 05:54:35 EDT 2011 .
I'm likely to have an eye at source code a bit later.

sergu61

After May, 18th again the same problem! :(

It was necessary to be rolled away on the Snapshot "pfSense-Full-Update-2.0-RC1-i386-20110517-2328.tgz" - on it the route by default switches automatically:
–--------
May 20 2:36:45 PM pfsense php:: Default gateway down setting 2_GW as default!
May 20 2:36:45 PM pfsense php:: MONITOR: 1_GW is down, removing from routing group

On new (after May, 18th) Snapshots the first line in system.log missed!

Sergu61

eri--

The change has been backed out since it caused issues and i have plans to put a knob under system->advanced to allow enabling it.

Give me time to come to it.

sot010174

@ermal:

The change has been backed out since it caused issues and i have plans to put a knob under system->advanced to allow enabling it.

Give me time to come to it.

Please include a "timed refresh" to this feature. In my case I think it would solve all my problems, something like:

Recheck default gateway route every ___ minutes. (and it would call the auto-change function or write the default gateway route if all gateways are down).

lhodgkins

Hi Guys,
Just curious what the status of this issue is… Has the fix been incorporated into the latest snapshot yet??

sergu61

In the file system_advanced_misc.php
in line:
…..
/>
…..

need to change:
lb_use_sticky -> gw_switch_default.

After that, everything starts to work!

eri--

I just fixed this.

sergu61

Thank you!
Now the system has become even more comfortable!

phospher

Awesome. I've been waiting for this fix as well. How long does it usually take for the snapshot to get built?

Michael Sh.

I'm a little corrected. For me it works better.
1. Restores the configured default gateway, if gateway is up.
2. Bypasses until IPv6 gateways.
3. Will don't touch routing without the need for.


        if (isset($config['system']['gw_switch_default'])) {
        /*
         * NOTE: The code below is meant to replace the default gateway when it goes down.
         *      This facilitates services running on pfSense itself and are not handled by a PBR to continue working.
         */
        $upgw = "";
        $dfltgwfound = false;
        foreach ($gateways_arr as $gwname => $gwsttng) {
            if (is_ipaddrv4($gateways_arr[$gwname]['gateway'])) {
                if (isset($gwsttng['defaultgw'])) {
                        $dfltgwfound = true;
                        if (!stristr($gateways_status[$gwname]['status'], "down"))
                                $upgw = $gwname;
                }
                /* Keep a record of the last up gateway */
                if (empty($upgw) && !stristr($gateways_status[$gwname]['status'], "down"))
                        $upgw = $gwname;
            }
        }
        if ($dfltgwfound == false) {
                $gwname = convert_friendly_interface_to_friendly_descr("wan");
                if (!stristr($gateways_status[$gwname]['status'], "down"))
                        $upgw = $gwname;
        }
        if (!empty($upgw)) {
                if ($gateways_arr[$upgw]['gateway'] == "dynamic")
                        $gateways_arr[$upgw]['gateway'] = get_interface_gateway($gateways_arr[$upgw]['friendlyiface']);
                if (is_ipaddr($gateways_arr[$upgw]['gateway'])) {
                        exec("/usr/bin/netstat -rnf inet | /usr/bin/awk '/default/ {print $2}'", $currgwip);
                        if (!stristr($gateways_arr[$upgw]['gateway'], $currgwip[0])) {
                            log_error("Setting default gateway '{$upgw}' will replace (IP={$currgwip[0]})!");
                            mwexec("/sbin/route delete -inet default; /sbin/route add -inet default {$gateways_arr[$upgw]['gateway']}");
                        }
                }
        }
        unset($upgw, $dfltgwfound, $currgwip, $gwname, $gwsttng);
        }

Test is not easy, unfortunately. Frequent kernel panic when the twitching interfaces.

gdavid

Hi guys. I'm having the same troubles dealing with failover gateway. I followed your discussion and now I can reach the correct behavior of switching to the secondary gateway if the default fails.
As you wrote, I had to declare a default gateway; if I do NOT flag "Default Gateway" on any gateway, no gateway switch is done at all.
So, if I flag "Default Gateway" on one of the gw, the switching is done correctly and the default route is changed. But here arises an issue: when the offline gateways returns online, the routing doesn't come back to this (default) gateway. Never more. Even after minutes and minutes.
As consequence, if the secondary gateway goes down (also after many days), my network is definetly offline from internet access.

It seems that pfSense can't switch away from an offline gateway if it is not the default gateway.
Am I missing something? I really didn't understand if I can patch some php code or this issue is still open …

Many thanks !!!!

Note1: My WANs NICs always stay up
Note2: My "Allow default gateway switching" is flagged on Advanced -> Miscellaneous settings
Running 2.0-RC2 (amd64), built on Tue Jun 7 15:17:21 EDT 2011