Help with Load Blanancing WAN connections



  • I am running the x64 RC3 official release build.

    I have 2 ISPs, both 1Gig links.  I have no default gateway set.

    I have 1 gateway group with both ISPs set as Teir1.  I have 1 outbound rule on the LAN to push all traffic through the LoadBalanced GW group.

    All traffic goes out through whichever interface was last set as the Default Gateway.  I have never had traffic go out both links at once.

    We host a web server behind the PFSense firewall.  Have both ISP IPs set so we use Round Robin.  Inbound traffic is about even, but all outbound traffic goes out whatever the last default gateway was.  I am not sure why.  We currently have about 100mbit of traffic hitting the web server and splitting that outbound traffic to 50/50 per link would keep me from going over the 100mbit floor…

    What am I missing?



  • Ok more info.  So checking my external IP from behind the PFsense box it is load balancing new connections from behind the firewall.

    So it seems that when a connection is established to the web server, the reply from that request only goes out one interface.  I would think it would go out the same interface it came in, but it seems to go out what ever interface is set as the default interface.  Is there a setting or rule I have to add in order to have the response from an inbound request go out the same interface it came in on or at least load balance using the GW Group?

    Thanks,



  • You may want to try to create failover setup for your server.



  • You mean a redundant pair of web servers or is this something I have to do in pfsense?



  • Actually i meant to your firewall, i had a lousy set of words. Sorry for that



  • Can you give me a little more info on what you mean?



  • http://doc.pfsense.org/index.php/Multi-WAN_2.0#Failover

    On firewall->rules->Lan then default Lan do you have the gateway (down under advanced features) set to WAN or to your 'multiwan' you configured in routing groups?
    On that same area have any higher priority rules (above default lan) to force traffic down a particular gateway based on ip/port etc? (Not likely but while you are there might as well look)

    Btw you have either of the load balancing boxes checked on system-advanced-misc? (Sticky & gateway switching)

    Many places to keep an eye on.
    Bill



  • Bill,
    Thanks for the response.  I went through the guide you posted when I first went through this install.

    Currently both Sticky and Gateway switching are unchecked.

    Here is my lan rules.  Very simple and everything is using the GW Group.  I have attached a screen shot of the LAN settings.

    I think the key is in another rule somewhere.  The reason I think that is that if I make an outbound connectiong from behind the firewall it hops between the interfaces.  However web requests coming in to the firewall and heading to the web server, all the returned data from that goes out one Interface, and it doesn't seem to matter what interface it came in on.

    While there is no default GW set, the route table does show a default route and that is the Interface all that traffic goes to.

    Thanks,




  • crzykidd: Might try sticky & see if it acts more like you are expecting but wondering if you are expecting too much of the balancing..

    Generally responses to inbound should go back out same wan they came into. Imagine if you called your buddy Bob & your buddy Gary answered.. ;)  The remote end can get just as confused if it sends packets to one IP & another IP is sending data back.  So IOW if you have a remote user browse your site & grab a 100MB file to download expect that 100MB to go down 1 pipe not both. You don't really have much control over that if they are the one making the request other that trying to spread out requests in hopes that it is balanced.
    Bill



  • Exactly that is why I am confused.  Both WAN interfaces are using RRobin balancing for DNS.  In bound traffic is about 2.4mbit/sec on both circuits, however outbound is 80-140mbits on whatever circuit has the default route.  I have tried with and without sticky connections same results.

    I would expect that when a http request comes in on WAN1 that the response would go out WAN1 and vica versa.  I wouldn't think that a request to WAN2 would have the response packets go out WAN1.

    So do I need to do something different with my inbound rules?  Currently port 80 Rule is set on the WANInterfaces.  Screen shot attached.

    And thank you for all the responses.




  • Well RR is just going to help decide which connection they come in on and yeah logical for traffic to go back out on the same connection. Are you sure the traffic you are seeing go out is http traffic responding to the incoming requests & not something else?  Outbound originated traffic (such as an email being delivered to remote servers) won't be controlled by RR records, the router decides that based on your settings including the routes.
    Bill



  • Positive.  There is only one website running behind this.  It averages around 90mbit/sec.  The site is www.thetvdb.com.

    The pfsense box really just passes port 80 through the server.  The only exception to that is some very limited SSH managment traffic.

    I have confirmed RR is working correctly.  Not sure where to go with this now.  My thought was a connection is established on WAN1 the response would go out WAN1.  If that is not the case I will need to look at a 2 WAN and 2 LAN setup, adding another NIC to the web server, and using some advanced routing rules there to controll the load.  However this should be able to be done at the pfSense level I would think.

    Thanks,



  • Yeah thinkin you are overlooking something..  maybe post up what you've set for your load balancing that you haven't already just in case something someone else might pick up on.  You setup the Services-Load Balancing or traffic shaper setup? In your 1st post you said you have no default gateway but then you said traffic goes out whatever was last default gateway.. Maybe I'm reading that wrong. :D  Also, are your wan's thru the same ISP with the same gateway IP?
    Bill



  • Even though I have neither GW set as the "Default" when pfSense boots up, it grabs one of them and sets one of them as the default route.  Is there some custome route I need to add to have this work?

    ISP1 –------
                 
                  ------pfSense ----- Web Server
                  /
    ISP2 --------/

    Both are two seperate ISPs, 2 completly different IPs and gateways.  Half the inbound requests come in on ISP1 and half on ISP2

    The rules are extremely simple

    I have 2 gateways.  1 for ISP1 and the other ISP2

    I have 1 Gateway Group.  with both Gateways listed at Tier1, and trigger level at Packet Loss (I have tried setting it to Member Down as well)

    Under Firewall Rules: 
    Floating:
    I have a couple IP based rules to allow SSH from certain IPs.

    WANInterfaces:
    TCP * * 192.168.20.100 (Web server) Port=80 Gateway=* (I have tried using the Gateway Group, but then I get no traffic)  Queue=none Schedule=blank

    LAN:

    • LAN Net * * * GW=LoadBalancedWAN Queue=none Schedule=blank

    I have NAT Firewall rules built for each WAN to the web server as well as the few SSH ports I outlined above.

    For NAT: Outbound I have it set to automatic.

    I have the box unchecked for allowing dynamic change of default GW (have tried both ways)
    I have sticky connections unchecked (have tried both ways)

    Thanks,



  • I got essentially same here & works fine EXCEPT I have WAN set as default gateway not sure if that matters.
    Bill



  • Hmm really?  Maybe I should install a second FW from scratch and get it set the same and swap it in.

    This box has been upgraded from 1.2 to RC1 to RC3.  Could be something isn't correct between upgrades.

    Thanks for the help and ideas.. I will try a rebuild and see if it behaves differently, unless anyone else has some ideas :)



  • Sounds like a plan if you have the hardware & time to try. :)  And yeah hard to say if some stray setting is in there from previous versions.
    Bill


Locked