Dual WAN - LoadBalancing – Only one WAN is being utilized?



  • Just going to make this quick for now – I will go into detail upon request.

    But I have setup Pfsense, built my router out of an PIII-800Mhz, with 512MB of ram.  It has 4 NICS.  (One isn't being used)..

    Anyhow -- I have two incoming cable connections.  (Two seperate modems, both provisioned for 10/1 Service) --

    They both obtain full speed as before pFsense I had two SOHO routers and used one as a Master and one as a slave.  With the gateway determining which PC used which routers (WAN) input.

    A large reason for going with Pfsense and building the box was so I could Have both connections being utilized on one machine at a time.  IE: If I went above and beyond the 10Mb/1Mb barrier, the other connection would be there.  (Of course It wouldn't boost speeds above 10Mbps, but the overall bandwidth should be somewhere between 15-20Mbps) --

    For instance if I have 23 Connections going, should I not, via load balancing, be utilizing over a single Cable connection with Load balancing?

    As of now I have another computer with the Pfsense Status/Traffic monitor going and as I watch when I attempt to max one connection to see if load balancing kicks in.  It does not, it simply acts as if I had a single WAN cable modem connected, and never attempts to begin to use the other connection.  (This is with multiple connections going).

    I have setup a pool for loadbalancing, they say 'online' ,.. And the Pfsense boots up with both WAN and OPT1 showing (DHCP) and their respective DHCP addy's assigned from the ISP.

    Any ideas on how to get the load balancing to work ?

    What information might you guys need from me?  (Otherwise the connection is working superbly,. the new router box with pfsense as the OS).  So it works, I just need to figure out how to configure it to obtain load balancing, and such.

    If there is an all encompassing file that can be used to gather my sys/net info and supplied to you guys in this post, I will do whatever is asked. =)

    Also, just basically.. How does pfsense treat WAN load balancing?  Does it take a little from one wan and a little from the other.  Or does it wait until WAN1 is maxxed and then start pulling from the secondary broadband connection?

    *Just to note, I was a bit stumped on getting the WAN pools (gateways) configured per 'guide', as it was calling to setup static IP's for each WAN modem/routers respectively.  As if it were designed for DSL alone or some such.  Then I upgraded to 1.2.1 Beta and the options popped up where I could select my interfaces to create the Loadbalancing rule.

    I am not particularly concerened with Failover, as these are both Cable, and if one goes down, the other most 'likely' will as well.

    Thanks so much,



  • This might sound stupid but did you try starting a download and then disconecting the network cabel that is used then put it back.

    worked for me

    but there are a ton of reasons why it might not work. did you see this howto?
    http://doc.pfsense.org/index.php/Multi-Wan/Load-Balancing

    If you still have the same problem post your lan firewall rules. Will gave us some more insight into the problem



  • @Neofate:

    Also, just basically.. How does pfsense treat WAN load balancing?  Does it take a little from one wan and a little from the other.  Or does it wait until WAN1 is maxxed and then start pulling from the secondary broadband connection?

    Sessions are sticky, so once a sessions starts from any machine, that session stays on the same WAN, new sessions are allocated round-robin between the two wans.  So you can't make a single download go twice as fast - its only 1 session, and if you start 2 downloads, it is quite possible that they will both end up on the same connection, depending on what else happens in between.

    One way  to check is to run traceroute several times - you should see it using both connections, if not then I think you have a broken rule set - Check out against the howto as eddie4 says, and if you still have a problem post rules.



  • @eddie4:

    This might sound stupid but did you try starting a download and then disconecting the network cabel that is used then put it back.

    worked for me

    but there are a ton of reasons why it might not work. did you see this howto?
    http://doc.pfsense.org/index.php/Multi-Wan/Load-Balancing

    If you still have the same problem post your lan firewall rules. Will gave us some more insight into the problem

    Yes, have tried dropping the Network cable on the WAN1 – That drops all Internet connectivity.  IE: If I am pinging say, the dns server, 4.2.2.2 with a -t flag.

    It will show Destination not reachable 192.168.1.1  while WAN1 is unplugged.

    (I just unplugged again while I was downloading heavily, about 8 different ISO's) -- And I had to restart the router to get a network connection going again.

    I do think it has to do with 2 things..

    Obviously my Alias's/Firewall Rules -- The guide is a bit lacking though -- IE: in the Alias portion it is missing pictures and such for configuring things in the example.

    Not sure how to give you guys a 'copy' of my rules -- What is the easiest way to give you that output?

    Also, the example has two ADSL modems that apparently act as routers, set to static IP's in the modems themselves.

    This is not possible with cable, or my cable 'modems'.

    I have Dynamic IP's, and the modem is simple a transparent layer -- Its a modem, and that is all, has no configuration abilities, and the only GUI are the internal diagnostic (Read only) RF levels/SNR, at 192.168.100.1 -- Which is of no help.

    Now not all is lost,.. The IP's given to these modems are for all practical purposes 'static' -- They don't change unless the power goes out etc.  Once an IP is assigned to a modem it generally remains for months, if not years on end if you don't try to loose your IP.  Even then it often will pull the same IP from the CMTS's uBR Pool due to MAC association.

    It is just as if the Second Connection has no bearing on this config,.. basically operating as a single WAN router -- Though it does recognize WAN* - > rl0 -> xxx.xxx.xxx.xxx(DHCP) then OPT1(WAN2)* -->vr0-->xxx.xxx.xxx.xxx(DHCP) then LAN* -->vr1--> 192.168.1.1

    It is recognizing WAN2, and monitoring -- IE: with Traffic Graph open, if I unplug WAN2's connection, it will drop out to flat "0Kbps In and OUt" -- It always has a little bit of activity going on all interfaces, so that is promising that its picked up.

    Just a matter of proper configuration,.. I hope you guys can help me out on this.  With my 2 'regular' cable connections.

    Thanks a ton,



  • It's a bit easier with your setup as you simply can skip the part of the doc with the aliases for the modemroutersubnets. Simply create the pools using the interfaces from the dropdowns. The only thing you have to take care of is to have monitor IPs that respond to pings. Dynamic IPs for the WANs are no issues as they will be updated on change automatically.

    Btw, unplugging a WAN will kill all connections that have been running on that WAN but they can be reestablished through the other WAN again then. Loadbalancing-/Failoverpools is not like it will give you no interruptions of already established connections.



  • Thanks for the heads up – But what should I use to monitor?

    I mean should I just tracert out and pick something a hop or two out?

    Also,.. so basically go back to square one, kill these aliases/firewall rules.. and just load loadbalancing with both wans selected in pools, with the 'proper' IP monitored?

    I understand failover and loadbalancing will not keep interruption from occuring, naturally -- But, unplugging WAN1, should within a reasonable amount of time be routed to WAN2, and vice versa.

    As I inidicated, if I unplugged (WAN1) it will kill all connections, and when I plug WAN1 back in the connections sometimes resume, some time they don't.  (WAN2 never takes over, nor does it ever get accessed by anything but idle traffic from the router/CMTS --(ARP replies) etc.

    I guess I'm sort of asking, what do I do?  Do I put in X aliases? Or delete them all now?

    Do I put in any firewall rules? Or delete the ones I don't even know make a difference ?

    Of course Load balancing will have to be on,.. but which IP should be monitored.

    I feel this (these) ips are key in the way load balancing/failover works -- Can both WANs use the same Monitor IP?  (Is it the Edge router from my ISP? the Core? ... The modems IP?)

    Thanks,



  • Monitor addresses that are in your ISPs network - a router close to you, or perhaps their DNS server.  Don't pick a popular real workd site, as pfsense automatically routes all traffic to monitor addresses down the WAN it is set as the monitor for.

    For this reason you can't use the same address for both links.

    To find out if WAN 2 is working, use ping or traceroute diagnostic utility on the WAN2 network.



  • @Pootle:

    Monitor addresses that are in your ISPs network - a router close to you, or perhaps their DNS server.  Don't pick a popular real workd site, as pfsense automatically routes all traffic to monitor addresses down the WAN it is set as the monitor for.

    For this reason you can't use the same address for both links.

    To find out if WAN 2 is working, use ping or traceroute diagnostic utility on the WAN2 network.

    Ok that I can do .. I can pick 'monitors' in my network before it hits my ISP's backbone.  No problem.

    Mind telling me just what these monitors are doing?

    Both of these connections are off the same network – But each have their full bandwidth.  IE: I can hook a cable modem to one computer, and one to another.. and have 20Mbs/2Mbs going (in overall bandwidth) -- Aka: They don't share bandwidth.

    I went to the Diagnostics in pfsense -- and tested a ping to 4.2.2.2

    The WAN2 interface came back with 100% Packet loss -- it is obviously not getting out.

    WAN1 0 loss, LAN, 0 loss.

    The Ping utility has : "Note: Multi-wan is not supported form this utility currently."

    The Traceroute utility has no option to pick an interface.. .

    Though in the ping utility I could pick the interfaces (WAN/LAN/WAN2) -- And the results were as above.

    I guess I need a basic guide/rundown on how to get things just working on a Dual Wan setup -- Just the bare minimum,. even if Security is completely out the door.. Somewhere to start where I can say, ok, It is utilizing both WAN inputs, and Loadbalancing.

    Once I can get to that point, I can fickle with the 'rules' to enhance security .. But I need a 'working' basepoint.

    I apologize if this is asking too much.  -- Any help would be greatly appreciated.

    And, again, if there is any  way to give you guys the current configuration, to post in the forum, or even host on a webserver of mine I will be more than glad to do it.

    I know you are working with limited information.  But I'd just like to get it working at a fundamental level,.. Then, as mentioned, move up on securing it from there.

    (It is possible to run Multi-WAN with my DHCP cable modems right? ) -- I'm beginning to think it just doesn't support it unless it is a static IP.  (Which it is, but not in DHCP mode) -- I've had thoughts on changing it from DHCP to Static, and just entering the DHCP given IP's anyway -- But I haven't yet.

    As a matter of fact, I reset to factory defaults.. And reconfigured -- So it is pretty much at your basic, no rules, Loadbalancing turned on, with WAN/LAN/WAN2(OPT1) recognized at the console.  [Again, the Pfsense box (router), recognizes both WAN inputs , and displays their unique IP's on the monitor of the 800mhz box that has Pfsense running.

    Just to drop any doubt, both of these cable connections work just fine if taken out of this setup

    Thanks!



  • you have 2 times the same isp ??? not 2 diferend isp's ????

    failover and balingsing works only with 2 differend networks



  • @jeroen234:

    you have 2 times the same isp ??? not 2 diferend isp's ????

    failover and balingsing works only with 2 differend networks

    I am sure that 2 modems from  the same ISP can work, you just need to get 2 different monitor IP addresses.

    The monitor IP address is pinged every 5 seconds. If the reply comes in the link is UP and traffic is sent through.

    I have a cable modem that gives me a real ip address, but the cable modem itself has a private ip address. I use the cable modems private IP as a monitor and that works fine. However that took me lots of experimenting to find out that this works.

    @Neofate: what are the monitor IP address you use for the 2 WANs?



  • @sai:

    @jeroen234:

    you have 2 times the same isp ??? not 2 diferend isp's ????

    failover and balingsing works only with 2 differend networks

    I am sure that 2 modems from  the same ISP can work, you just need to get 2 different monitor IP addresses.

    The monitor IP address is pinged every 5 seconds. If the reply comes in the link is UP and traffic is sent through.

    I have a cable modem that gives me a real ip address, but the cable modem itself has a private ip address. I use the cable modems private IP as a monitor and that works fine. However that took me lots of experimenting to find out that this works.

    @Neofate: what are the monitor IP address you use for the 2 WANs?

    Sure I would think two Cable modems from the same ISP would work just fine.

    Each provides a completely seperate IP from the other.

    Here is the deal – Both interfaces, WAN and OPT1(WAN2) show 'up' in the status.

    WAN shows DNS servers in the status,.. OPT1(WAN2) does not.  No idea why.

    WAN2 would not ping out using the ping utility in pfsense.. Until:  I manually set the IP address to static for WAN2(OPT1).. Once I did that, it is now pinging out from WAN2 when using the ping tool in Pfsense.  (progress, I guess,.. but still not where I want to be).

    As for IP's.

    Each Cable modem has its own IP, and at bootup Pfsense at the console and in the web gui recognize both IP's, seperately.

    All cable modems have an internal ip of "192.168.100.1" -- This is the diagnostic page as well.  It is the ip you ping to determine if the Cable modem is online.

    Problem with that, is given that ALL cable modems have this imbedded,.. How can I use this, or should I use this 100.1 as a monitor?  How would it know which was which?

    IE: If I enter 192.168.100.1 into the webbrowser it will go to the WAN1's modem interface.

    I guess If I understood how this program was supposed to actually load balance, and pick between the two interfaces I would have more options to try.  For now I am at a loss.

    Please give me any ideas.

    An example of the Traffic Graphs in PFsense.

    WAN1 will who X IN, and Y Out -- While WAN2 will show X IN, but OUT is always Zero.

    This is a problem I am sure.

    I have 'rules' setup in the firewall that basically are set to pass, and 'any' interface for both WAN and WAN2, and LAN.  That to me would be opening EVERYthing up.

    I know it is atypical to use load balancing on two cable modems from the SAME ISP.  But, it should function.

    If I had two ADSL connections, they would both have internal modem diagnostics of 192.168.1.254 -- Not much different.

    Though with cable modems, there is no configuration of the cable modem itself.  It is read only memory for practicle purposes.  It has Firmware that it operates off of, and synchs with the Cable company and provides ME with an IP.

    For Example WAN1 = 68.113.90.164
    and WAN2 = 71.91.71.155

    LAN= 192.168.1.1

    The gateway Pfsense is picking up for Wan1 on the 68.113.90.164 IP is Gateway=68.113.88.1 -- The Gateway Pfsense is picking up for Wan2 on the 71.91.71.155 Interface is Gateway=71.91.68.1.

    So it is picking up Gateways and IP's for both cable modems.  (Each cable modem connected to its own computer runs seperately from another.. IE: If I max bandwidth on BOTH machines, they do not pull bandwidth from each other.. They are in no way connected.  Basically it would be identical if one of these cable modems was 20miles down the street in another house, and one was here.)

    I hope someone can provide some ideas..

    Thanks!



  • If the ISP allows you to ping its servers then it is better than pinging the modem (the modem might be up and pinging but have no connectivity - I use the modem as monitor because my ISP sometimes blocks pings).

    So find 2 server (DNS servers, web servers, anything on the ISPs network) that you can ping reliably and use one each for for each WAN interface. nmap is good for this kind of thing.



  • @sai:

    If the ISP allows you to ping its servers then it is better than pinging the modem (the modem might be up and pinging but have no connectivity - I use the modem as monitor because my ISP sometimes blocks pings).

    So find 2 server (DNS servers, web servers, anything on the ISPs network) that you can ping reliably and use one each for for each WAN interface. nmap is good for this kind of thing.

    I've used such monitors, and they are pingable, directly on the ISP's network,.. but only one WAN connection works.  No Load balancing.

    I just don't get it –-

    What are the bare essentials to getting a Dual Wan setup with Load-Balancing to work.  (Don't even care about Fail-Over).

    Are their particular rules, or some general setup steps I likely need to re-adjust?

    Again, both IP's are being recognized as (DHCP) on the console.

    The only way to get WAN2 to ping out from the Diagnostic utility is to make it static.

    Can I look at "states" and tell what my problem is?



  • screenshots of your setup would help.



  • @sai:

    screenshots of your setup would help.

    Ok thought there might be a method to output the general state of things all nice and neat.  But I'll get to work on screenshotting everything, and then converting to a JPG, and hosting.



  • **Edit:  Removed Most of the images as they are not what is currently configured, and are no longer needed. **
    ~~Ok – Here are all the screenshots I thought would be even remotely pertinent.  Anything you do not see here, assume it is left at 'default' out of the box.

    This should tell you everything you ever wanted to know about my setup -- I'm baffled.

    It is in no particular order:~~




    Thanks!

    P.S – If some of the pictures don't show, try reloading the page,.. They are all there.



  • @Neofate:

    Ok – Here are all the screenshots I thought would be even remotely pertinent.  Anything you do not see here, assume it is left at 'default' out of the box.

    This should tell you everything you ever wanted to know about my setup -- I'm baffled.

    It is in no particular order:
    --snip--

    I know its alot, but I tired to cover everything,.. If you can help me out please do.

    Thanks!

    P.S – If some of the pictures don't show, try reloading the page,.. They are all there.

    you need to change the gateway on youre lan rule to LB-WAN-WAN2
    to make youre loadbasing working



  • Well before looking back at this topic for responses I just went into every conceivable option in PFsense and configured it all from scratch.

    I have Load Balancing working.. Yay! – I'm not certain what exactly I did that caused it to work, because I did dozens upon dozens of changes.

    Though in response to the last suggestion,.. I did just that.

    First I created an Alias called "Modems"

    In which had two Host IP's attached:  (The IP of WAN1 Cable Modem, and IP of WAN2 Cable Modem)

    Though an alias alone is like a stored variable, does nothing.

    So I went into Firewall rules, and Under the LAN tab created a rule.. Here is where that Alias came in handy.

    Action was set to PASS
    Interface, obviously LAN

    Protocol : ANY

    Source: LAN Subnet

    Destination was inverted and selected type to : Single host or alias (So  I could point the LAN to the two IP's)
    Address: Modems  (Remember the Alias I created with both Public IPs)
    Gateway: Set to the LB-WAN-WAN2

    This is probably the most important change. But I can't say for sure if this did it.

    I also created rules under LAN for WAN1 and WAN2 --  Basically LAN to WAN1's Gateway, and LAN to WAN2's Gateway.

    Haven't checked failover yet,.. but will after this post.

    Now on to smaller problems.

    Port Forwarding.. Sheesh.  lol.

    I go into Firewall: NAT: Port Forward

    And create a new NAT for Utorrent.  I have Utorrent set to statically use port: 50498

    So I created two NAT Port Forwards, something like this:

    WAN - TCP/UDP - 50498 - 192.168.1.1 (68.113.90.164) - 50498
    WAN2 - TCP/UDP - 50498 - 192.168.1.1 (24.178.189.108) - 50498

    The port is still not forwarded in Utorrent.  So my config isn't right of course, but I think this is an issue that can be easily dealt with from you guys.

    Basically, forwarding in Firewall: NAT: Port Forward is the way to go right?

    From there I create rules.

    Say I am creating a Rule:

    Interface (I select WAN)  (correct?)

    External Address: I leave it at "Interface address"

    Protocol: I select TCP/UDP  (Give them both an opening)

    External port Range: From: Other -- 50498  To: Other: (Left empty)

    NAT IP: 192.168.1.1    -- (I really have no idea if this is correct.. It wants the IP addy of the "server" on which I want the ports mapped.. I could only come up with my General Gateway for my LAN.)  This right?

    Local port: Other: 50498

    Saved.

    I created the exact same rule again, only using WAN2.

    Do I need to create a LAN rule?

    What have I screwed up in that port forward that is causing it not to open up 50498 on my LAN?  I Want that port to be opened to all computers on the LAN.  I will configure Utorrent to use that port on them all.  Given they are Dynamically addressed (Prefferred to me) - I don't wish to setup each computer statically.

    Are there any other areas I need to be changing for the port to open up?

    This should be pretty basic,.. Port forward question.  I appreciate any answers.

    (Also, is there any command I can execute to give you guys a verbose output of my configuration?) -- So I don't have to spend 30mins taking screenshots and cropping them then uploading?  Something where you can scan and check this and that.  A sort of diagnostic system check/function log dump command I suppose.

    So I could say, ok I'm having this problem,.. Here is my config -- (Copy/Paste) the configuration info.. Or attach it if its extremely large, etc.

    Thanks again.. About to test failover, and will report back.

    PS:  The Diagnostic Utility still won't function on Ping when WAN2 interface is selected.    (It does say "Multi-Wan is not supported from this utility") -- But a few people told me to run the ping/traces from there to test Multi-Wan function.  Seems sort of contradictory.

    I've found that I can just go to www.whatismyip.com and now that load balancing is on it will switch out on a every other basis.  Round Robin I suppose.  (Between WAN1 and WAN2)

    Horray for Load balancing.  I was beginning to think that with my Dual Cable Modem setup, from the same ISP, that it somehow just wouldn't work with Pfsense.  I'm glad I proved that wrong.

    Again, thanks, and I anticipate your replies on the forwarding.



  • WAN - TCP/UDP - 50498 - 192.168.1.1 (68.113.90.164) - 50498
    WAN2 - TCP/UDP - 50498 - 192.168.1.1 (24.178.189.108) - 50498

    you need to replace 192.168.1.1 with the ipadres off the pc on youre lan where the ports are to be forwarded to
    also make sure the auto firewall checkbox is checkt with those 2 rules



  • Thank you –

    I was initially trying to avoid putting in a Dynamic IP in a NAT rule,.. but these MAC addy's request the same ip's,.. so it is no big deal.

    That worked, thank you.

    As for the 'check box' -- I did not see one,.. I created rules manually where they needed to be.

    I have two 10Mbps/1Mbps connections -- And downloading several Linux Distro's and 2 seasons of House,.. I am getting what I hoped for.. 1500-2100KB/s down total,.. and 175-250KB/s upload.. So the Dual WAN Load balancing is working even with the crazy environment of Bit Torrent.  (I know it was recommended to use one WAN for Torrents.. but I wanted to try initially anyway) --

    Also, I've limited my Upload to 175KB/s and Download to 1.5 Meg/s and with the balancing web browsing and other functions are running as fast as if No other processes/downloads/uploads were even occuring.

    Very pleased.

    Oh, and forgot to add -- Failover works.

    It doesn't automatically resume a connection already initiated, but the next one fails over.

    IE: If I went to a command prompt and put in Ping X.X.X.X -t

    And let it go, its pinging fine.. no loss.

    I unplug the Cable modem that has been assigned via Load Balancing (Showing activity) for that particular connection.  It will then show time-outs... and continue to do so.  However, if I stop it, and re-issue the command with that modem unplugged it will resume on the other modem.

    Can't ask for much more than that.

    I have alot to learn with this software, but I'm glad I've gotten 90% of the functionality I intended on working so far.

    Next is security.

    I have 4 NICS -- Only utilizing 3 of them.  2 WAN's, one LAN.

    Given I have an extra one already installed,.. is there any benefit of using it, perhaps to segment LAN from WAN.. more security? Anything beneficial?  Or am I pretty much where I need to be.

    Lastly,.. now SNMP and other such protocols can be monitored , whereas with the Linksys SOHO routers I was using didn't support it.

    Any good traffic monitoring software you guys recommend?  (Even stuff that needs to be purchased).

    I'd like to be able to monitor the packets/traffic from each individual computer, and not just the ENTIRE bandwidth of the LAN.  I know its possible,.. Some ideas would be great.

    Thanks,



  • @jeroen234:

    you have 2 times the same isp ??? not 2 diferend isp's ????

    failover and balingsing works only with 2 differend networks

    Just wanted to point out for anyone who might search in the future about Two connections running off the Same ISP network, Yet, have two seperate modems –-

    That, YES, it does work, in every conceivable way.

    Failover and Balancing works within the same ISP.  How do you think the Cable company is managing their own data traffic?  They are using the same methods failovers and load balancing among their OC-X lines, and even lines in the CMTS area.

    Don't mistake this with someone who has a SINGLE cable modem -- And wants dual wan connectivity.  You must have TWO seperate Public IP's, and furthermore devices/modems that are provisioned at X speed.

    IE: Two cable modems provisioned at 3Mbs/256Kbps --

    Or A cable modem provisioned at 5Mbps/512Kbps and a DSL modem at 3Mbps/384Kbps -- And on and on.

    As long as you get two seperate Public IP's from the ISP, that are each capable of independently pulling their own bandwidth it will work.

    I know it isn't typical to have Two Cable modems, off the same ISP,.. But for the price it is by far the cheapest manner to obtain a 2 MegaByte per second connection (Overall)  (20Mbps).  Basically 20Mbps for 80$.

    Without getting off topic too bad, of course there are reliability problems with residential lines, and running off the same infrastructure if one drops so will the other modem,.. But again, for the money it isn't a bad deal (If you utilize that kind of bandwidth).

    Of course a DSL connection and  cable connection would be much more reliable with Failover, as it isn't likely both would be out of service at the identical times, so you'd pretty much have a 100% uptime.

    I am relatively new to this software,. but with this Router I've built, I would feel somewhat confident in using the load balancing/failovers in a small business,.. It seems that stable.  And will only become more stable as time progresses.  I am very impressed with the software, and how new it is vs stability.

    Just to reiterate:  A Cable Modem and another Cable Modem from the Same ISP will work with Pfsense at full capacity on Load-Balancing and Failover.

    I would imagine a DSL modem, and an additional DSL modem from the Same ISP will work just as well. (Of course speeds varying on provisioning levels/line quality).


Log in to reply