Using second WAN speeds up first WAN ?????

  • I have a very strange problem. I believe it started after I installed pfSense but it could be a coincidence - unfortunately I cannot go back as the old dual WAN router is now dead.

    WAN1  80/20 connection. (PPPoE fibre from BT)
    WAN2  10/1 ADSL2 connection (PPPoE copper from Nildram)
    LAN    Simple setup - 6 PC's
    pfS      Running on Compaq SFF with three NIC's

    Usually use failover but have also tried load balancing
    No other packages running
    I am running a web server on one PC - with forwarding rules through both WANs. (Uses HFS but have also tried Apache)

    Everything works well in terms of routing etc.

    My daughter (ADSL2 12/1) in Australia regularly downloads from my server - until recently with no problems.
    She used to get 500KB/s+ through WAN1 or 90KB/s through WAN2 - both stable
    Now she usually gets 10-40KB/s (continuously variable) through WAN1 but still 90KB/s through WAN2
    If she starts DL on WAN1 (getting 10-40KB/s)
    If she then starts DL on WAN2 (getting 90KB/s)
    WAN1 DL then increases to higher speed - anywhere from 200KB/s to 500KB/s - but with stable speed
    If she then stops DL on WAN2
    After about 30 seconds WAN1 again drops down to variable 10-40KB/s

    This also shows if trying to use Skype - too slow to work on WAN1 unless she starts a DL on WAN2!
    She can Skype to many other people in UK (and globally) without any problem

    For other users (In UK) both WAN's work perfectly at high speed independently - DL and Skype

    Any thoughts / suggestions?  Any ideas on what to change or how to troubleshoot?


  • I have a theory:

    When connected to wan1 it seems some of the packets may be going out through wan2, as the routing via wan2 has lower latency. This creates a problem with packets out of sequence.

    When starting a dl on wan2 it fills the line (Only a relatively slow adsl2) to the latency increases

    Then traffic through wan1 ONLY returns through wan1 and packets are in order - and speed increases?

    Does that make sense?  I plan to try out the theory by disconnecting wan2 when next trying.  If it works how can I resolve the problem using pfSense?

  • How did you configured your gateways? Is there a default gateway configured? On LAN rules did you forced a gateway or a load balance?

  • WAN1 has default gateway selected.  I have tried with "allow default gateway switching" enabled and disabled - no difference.

    On Firewall-Lan-Rules I normally use "wan1failover" as gateway. (WAN1 on tier1, wan2 on tier2) But I have also tried WAN1 (Only) and load balanced between wan1 and wan2 (Both tier1 - tried with different and same weightings).
    Again - no difference between them all.

    I haven't tried "use sticky connections" as I believe they relate to connections for incoming only - and I further gather they may not work n 2.0.1, which I am using.

    Any help appreciated

  • Well, now I am totally puzzled. I tried with WAN2 disabled - hoping this would prove a point.

    See attached bandwidth graphs - read from right to left

    I disabled the WAN2 and she then tried dl  - it started slow but then increased and I thought it was ok. Then it dropped and again increased, repeating the "sawtooth" pattern 4 times. Then, unfortunately, it dropped and stayed low and variable.

    After running for some time I then re-enabled the second interface. Lo and behold - the speed increased immediately.  I then disabled the second interface - which reset the firewall so both stopped. Finally tried just WAN2 - again it ran more or less steadily - maxing out the interface.

    If it was just the "sawtooth" variation I could believe it was some sort of throttling - but that still doesn't explain why the speed increases when wan2 starts dl.

    Any suggestions on what I can try to further investigate?

  • Just a last appeal - please.

    Ignoring (for the moment) all the details, how is it possible that DL through WAN2 increase DL speed through WAN1?
    Removing WAN2 completely has no effect.
    The only way to get fast dl through WAN1 is to start a DL through WAN2.

    Any thoughts or suggestions as to how to troubleshoot would be much appreciated.

  • @hackin8:

    Ignoring (for the moment) all the details, how is it possible that DL through WAN2 increase DL speed through WAN1?
    Removing WAN2 completely has no effect.
    The only way to get fast dl through WAN1 is to start a DL through WAN2.

    OK, ignoring the details of your "problem" and the details required to make my suggestion happen, suppose the downloader's ISP has a web cache. Suppose the paths to WAN1 and WAN2 have significantly different round trip times. Downloading from WAN2 might prefill the cache so a download from WAN1 is handled from the cache rather than the real WAN1.


    Any thoughts or suggestions as to how to troubleshoot would be much appreciated.

    Comparing traceroute to WAN1 and WAN2 might be interesting. If I recall correctly, a major submarine cable connecting Western Australia to the rest of the world was broken a few months ago. That might have affected the route to WAN1 but not WAN2 (or vice versa)

  • Thanks for taking the time and making suggestions.

    Unfortunately first suggestion would only be true if downloading same thing from both WANs - whereas WAN1 speeds up regardless of what is DL on WAN2.  ANY DL on WAN2 will speed up WAN1.

    Second Idea is a good one - I have actually asked them to try MTR on both routes to compare - waiting for the result. I am hoping this might help - but not sure it would explain why WAN2 DL would immediately speed up WAN1 DL. (And WAN1 slows down again if WAN2 stops)

    Thanks again for suggestions - I had already run out of ideas / theories..

  • Thanks to all taking time to make suggestions.  Unfortunately still no further on.

    As I have no way of knowing whether the problem actually relates to pfS or something entirely different - any suggestions for a more general network forum that I could ask for suggestions of how to troubleshoot / find the root cause of the problem?

    Thanks again

  • Did you get traceroutes to both IPs? Care to post them? (blank out source IP if you wish)

    I suspect what you have reported is a problem in the sense of "unexplained behaviour" rather than a problem in the sense of something doing something it should not do.

  • Solved - I think!!

    It appears that the problem has been solved - although I will wait a few days before final confirmation.

    The solution - if it was the cause - was for the ISP in Australia to put a check mark (i.e. enable) in the "force MTU" option in the exchange.

    I have no idea how / why this would cause the strange problem - but so far it appears to have fixed it.

    I will mark the thread "Solved" once I have verified over a few days.