2.0.1: Multi ADSL with same ISP: better, but still some problems?



  • Hi,

    In 2.0, only one ADSL of my lines would start, because my ISP assigns the same remote IP for each PPPoE session.

    2.0.1 delivered good improvements  :D

    • Both ADSL lines now start

    • Dashboard now shows green status and correct IP for each ADSL link

    • Both ADSL lines can now receive packets from the internet

    But some problems remain  ???

    • The tiers set in "System / Routing / Gateway Groups" are not respected.  That is, the same gateway gets used for all outbound packets, regardless of the Gateway Group is set as the Gateway for an outbound firewall rule.

    • There is no failover to the second ADSL modem when one ADSL modem phone line is unplugged.

    • There is no failover to the 3G dongle when both ADSL modem phone lines are unplugged.

    I tried using default Gateway monitoring settings.  I also tried giving each Gateway its own Internet-visible IP address to ping. 
    I also read http://doc.pfsense.org/index.php/Multi-WAN_2.0

    Have I messed up a setting somewhere?  Or, is this a design limitation when using two ADSL lines with the same ISP?

    My setup is No-NAT, with blocks of public IPs routed from the ISP down all working links (depending on web control panel settings).

    • Interface 'ADSL_L1' = PPPoE

    • Interface 'ADSL_L4' = PPPoE

    • Interface '3G_DONGLE' = PPP over 3G

    Screenshot attached…

    Thanks for any pointers.



  • Rebel Alliance Developer Netgate

    Did you set a custom monitor IP for each WAN? By default it would try to ping the gateway which would not work at all if two interfaces had the same gateway.



  • Did you set a custom monitor IP for each WAN?

    Yes.  But the custom monitor IPs are on the Internet: so all the gateways can see them.  Is that OK?


  • Rebel Alliance Developer Netgate

    It should be adding a static route behind the scenes to ensure they go out the right way.

    If your gateway groups are set for 'packet loss or high latency' it should work, though I have not tried it myself.

    We've had several positive reports from other developers and users that it was working though.



  • Thanks, I'll try ticking those boxes tomorrow.

    But it's discouraging that even disabling the PPPoE interface ADSL_L4 doesn't cause traffic to fail over to ADSL_L1.

    Also, the VLANs for which ADSL_L1 is the Tier 1 gateway always send outbound traffic via ADSL_L4.  So I can't help thinking I've missed something fundamental - or there's still a basic problem where ADSL_L1 and ADSL_L4 have the same PPPoE endpoint IP.

    PS: One more thought.  Are Gateway Groups (Tier 1 / Tier 2 etc) known to work with No-NAT networks?

    Thanks again,

    • Martin


  • I also have the same problem i have three PPPoE, the Multi PPPoE only one WAN it`s working (Online), other WAN (gathering data).



  • Maybe just have a look in the routes to see if pfsense has added anything so certain gateways or monitors or whatever are only accessible through a certain interface. I remember setting up mine with 2 interfaces on the same ISP and it worked, but I had some trouble. I was using NAT. The failover took like a minute to kick in for me, but it was probably my settings not being tight enough.



  • Some good progress….

    The Gateway Groups are now working  :)

    That is, the voice traffic favours ADSL Line 1, and all other traffic favours ADSL Line 4.

    I will do more failover testing today, and do a write-up if I get it all working correctly.

    Many thanks!

    • Martin


  • Good stuff  8)

    Just made some tests, and failover is working pretty well.  It must have been some small config changes in my setup.

    I unplug one ADSL line, and services fail over to the other one.  :)

    I unplug both ADSL lines, and services fail over to the USB 3G cellular modem.  :)

    Observations remaining:-

    • There is disruption of 10 or 20 seconds during the changeover (e.g. when you unplug the primary ADSL line) despite low trigger settings on the gateways.

    • The DNS forwarding service can't be used (I guess it sends through the wrong interface as it won't know about gateway groups).  As a workaround, just make your DHCP server tell clients to use the correct name servers (internal or external).

    • If I have some streaming BBC audio running at the time of changeover, that session must be restarted.  I don't know whether this is an application issue, or a firewall session being killed.

    • So, it all works as well as could be expected, EXCEPT that it would be better if the pfSense simply spotted when a PPPoE session is up or down, even when the upstream ISP routers have the same IP for each PPPoE.

    I will post a writeup soon so that a known working example of No-NAT multi-WAN, multi-LAN is available for 2.0.1 for the case where the upstream ISP routers have the same IP.



  • Working,

    Thanks Martin.



  • All working OK now  :D

    So I have done a write-up with lots of screen shots, showing my working configuration.

    http://blog.martinshouse.com/2012/01/multi-wan-multi-lan-no-nat-routing-with.html

    Hope this helps  ;)



  • To follow up after more detailed testing…

    When one Gateway goes down, the failover happens quickly (10 seconds or so).  This is excellent work.

    Some subtle issues remain, however ;-)

    1. A simple continuous 'ping 8.8.8.8' from my Mac does not recover after failover.  But "while true; do sleep 1; ping -c 1 8.8.8.8; done" shows that the failover is really fast (only a few Pings lost).  ICMP Echo Request frames contain random ID fields which do not change on continuous pings, so those pings get dropped because the firewall associates it with a dead gateway.  Understandable, but not ideal.

    2.  Regardless of the setting under "System : Advanced : Miscellaneous : Gateway Monitoring", the firewall states for existing sessions do not seem to be maintained when failover occurs.

    So... For 1 & 2.... I think it would be better, if the firewall could maintain state across Gateway Groups even when one of the Gateways flaps down and up again.  This would make sense in my case, as all the WAN links go to the same ISP, and all the IP addresses are valid across all the WAN links.

    3.  If one Gateway goes down, and traffic fails over to another Gateway (according to the order of Tiers in the Gateway Group), then that's all very good.  But when the downed Gateway comes back up, then the outgoing traffic doesn't always swap back to the preferred Gateway in the Group.  This could cost money if the fallback Gateway is expensive to use.

    Keep up the great work.

    Best regards,

    • Martin


  • After further testing…  I am not convinced that the multi-WAN feature is really mature enough to be used in anger for policy-based routing and failover.

    Firstly, the Tier priorities (set in System / Routing / Gateway Groups) are not always respected.  My upstream VOIP traffic is consistently being sent up the Tier 2 link, when the Tier 1 link is (like the others) all showing Green with low latency (and the link down Alarm has not triggered in the syslog).  I wonder if it depends which ADSL link happened to come up first when pfSense booted?  Rebooting did not fix this.

    Secondly, the USB 3G failover link typically goes down after a day or two, and pfSense does try reconnecting for very long.  So unless you manually go to the Interfaces page and click 'connect', then the failover link will not be available.

    Are these known issues?

    Can we expect any relevant updates in 2.1?

    I would be happy to share my config files with one of the developers (and to run a 2.1 snapshot) if it would help in getting to the bottom of all this!

    The GUI config for multi-WAN is quite elegant.  The problem is simply that it does not appear to work consistently.

    Kind regards

    • Martin

  • Rebel Alliance Developer Netgate

    Multi-WAN works fine, and the tiers are always respected – with a normal Multi-WAN setup. If there are issues it's due to having multiple lines with the same gateway, which is a special case that only (sort of) works for PPPoE, so generalizing that it's a problem with Multi-WAN as a whole is not correct.

    I've seen 1-2 others have a similar problem with 3G but we've never been able to reproduce it. I don't have a 3G card/hardware here so I can't say for sure, but it may also vary by modem. The times I've heard of it happening, the modem fell off the USB bus and came back weird. That doesn't normally happen during a 3G disconnect.



  • Hi, Thanks for the quick reply.

    Ah - so Multi-WAN policy routing is confused, because I'm using the same ISP on both ADSL lines - hence the same next-hop IP address.  Fair enough!  Will this config work in 2.1?

    As regards 3G…  I am using a Huawei E367 USB dongle.  The USB device seems stable, but I suspect there can be some temporary disruption to the cellular network, causing the PPP session to drop.  Perhaps that's normal for cellular connections. But it would be nice if pfSense would try to re-connect every 5 or 10 minutes - because manually going to "Interfaces" and clicking "connect" always succeeds.

    Cheers

    • Martin

  • Rebel Alliance Developer Netgate

    Not sure about 2.1 and multi-pppoe, at this point it would probably be the same - not sure how that might be by release. We're primarily focusing on IPv6 there.


  • Rebel Alliance Developer Netgate

    For 3G please start a fresh thread with that in the topic - it's buried here and the right eyes won't see it - I don't use 3G so I don't have any more info there.



  • Great news about IPv6 :-)

    Many thanks


Log in to reply