Malfunctioning Load Balancing Setup

[nitz]

Hello!

Sorry if it has been answered before but after a few searches I failed to find an answer.

I am playing with pfsense since my main objective was to combine both my WAN connections and perhaps make it faster getting most from both.

Here is how it looks like:

WAN1 = DHCP Cable Modem 20Mbps/1Mbps
WAN2 = PPPOE DSL Modem 15Mbps/1Mbps
LAN1 = STATIC 192.168.1.1 (Asus N56U connected to it in AP Mode)

So I made a gateway group with both WANs and set their priority the same, trigger level packet loss or high latency, I also set in the firewall rules to route everything to this new gateway group.

My problem is, the Internet surfing got pretty annoying since I setup these rules, sites sometimes will take forever to load or even time-out then suddenly works fine again and apparently only the WAN1 is being used (default).

I have tested downloading a huge file from usenet or download manager and it seems like it's not working.

The failover however, is. When I disconnect the cable connection by setting the Motorola Surfboard modem to standby it automatically switches to my DSL but it's not really load balancing them.

Can somebody please help me?

Thanks

[stephenw10]

It can only load balance on a per connection basis. If you download one large file (directly over http) it will only ever use a single connection. If you are using something that has multiple connections, like bit torrent, it will balance correctly.
The client at speedtest.net seems to able to correctly test over multiple connections.

Steve

[nitz]

I know it, but something is really wrong, I can't even browse the Internet properly since I set up the load balancing in pfsense.

Usenet uses up to 50 different connections to download the same file, I've also tried speedtest.net but the webside doesn't even load when using two WANs.

Pinging 8.8.8.8 with 32 bytes of data:
Request timed out.
Request timed out.
Request timed out.
Request timed out.

Ping statistics for 8.8.8.8:
    Packets: Sent = 4, Received = 0, Lost = 4 (100% loss),

C:\Users\nitz>ping 8.8.8.8 -t

Pinging 8.8.8.8 with 32 bytes of data:
Reply from 8.8.8.8: bytes=32 time=122ms TTL=51
Reply from 8.8.8.8: bytes=32 time=121ms TTL=51
Reply from 8.8.8.8: bytes=32 time=122ms TTL=51
Reply from 8.8.8.8: bytes=32 time=124ms TTL=51
Reply from 8.8.8.8: bytes=32 time=121ms TTL=51

Something is really really wrong in my setup, as long as I keep the load balancing to balance high ping/packet loss my connection gets crazy, can't even ping websites and it kinda works but it's troublesome to load anything, including these forums!

As soon as I changed to switch connections when "member down" it worked fine with one connection and switched to the other when I shut down the other one.

[stephenw10]

You have DNS servers on each WAN? Not that that should effect ping tests.

Some websites really have a problem with loadbalanced connections. Typically anything that requiers a login. You find yourself logging in repeatedly as the site sees your connection as coming from a different location.
Again does not effect ping.

Seems like it could be a routing problem. You don't have any conflicting subnets? Is pfSense handling authentication for both WANs?

Steve

[nitz]

Ok, about the DNS servers they're both provided by my ISPs, I haven't changed them.

I know some websites have issues with multi-wan setups but this is not the case, if I get it working then I'll probably add some exceptions or make it work with only torrents/usenet and so on.

pfSense is handling the DSL authentication fine I guess since either WAN works in failover mode.

Sorry but I'm kinda lost here, how can I have problems with conflicting subnets?

[stephenw10]

@nitz:

how can I have problems with conflicting subnets?

If you were still using your cable modem or dsl router for authentication and handing a private IP address to your pfSense WAN interface then there would be a good chance that your WAN and LAN would both be a 192.168.1.* address. This breaks routing. The same could happen if both WANs are in the same subnet.
However if pfSense is handling authentication via PPPoE it will have a public IP.

The other, in your case very remote, possibility is that both your WAN connections have the same gateway defined. This will break load balancing if they are not both using PPPoE.

Steve

[nitz]

Then it's not really a problem, my cable modem gives pfSense a public IP address just as the PPPOE connection.

I am really confused, trying it at home first so I can try pfSense for bigger things, I loved all the possibilities and the interface but I just can't get this load balancing working.

Here is the dashboard screenshot:

[stephenw10]

The fact that your screen shot is showing 'unable to check for updates' implies that system DNS may not be working. Can you ping from the pfSense console?

Other than that you may have misconfigured the firewall rule.  :-\

Steve

[nitz]

You've been very helpful so far, I probably messed something up!

Ok, the DNS are both pinging fine and I've figured out even with the load_balancing gateway set to failover mode if the WAN2 is connected it messes all my connectivity to outside the world so I gotta disable "WAN2(Velox)" so I have my Internet working fine.

The "pass" firewall rule is set for the load_balancing gateway group.

load_balancing gateway group is set as follows:

Gateway Priority: WAN1 = tier 1 WAN2 = tier 1

Trigger level: Packet Loss or High Latency.

There is not much else I have configured, only a few port forwardings and that's all!

[stephenw10]

Your DNS servers are being correctly assigned to each WAN? You definitely have at least one on each WAN?

Is there anything in the system logs when the connection becomes intermittent?

I assume you've read this already but just in case:
http://doc.pfsense.org/index.php/Multi-WAN_2.0

Steve

[nitz]

Ok, seems like it's working… Somewhat

Since the upstream of each connection is capped at 1024Kbps I think it might be using both but still it's far from what I expected since Speedtest usually manages to test it well with pfSense's wan balancing.

What I did change was the MTU on the dsl connection to 1492, that's VERY weird since MTU shouldn't impact the performance so much.

I am trying to figure out how it's working in real-world tests but I can't see the traffic graphic, whenever I click Traffic Graphic under the status menu it takes me back to the router's login page.

[stephenw10]

Well if the MTU on the connection is causing dropped packets it could dramatically effect speed.

How are your WANs connected?

Steve

[nitz]

WAN #1 is connected through a cable modem and has an external IP (DHCP), DNS provided by DHCP
WAN #2 is connected through a DSL modem and also has an external IP (DHCP), DNS provided by DHCP

My results are kinda mixed, while I could max out usenet at around 2.35MB/s which is pretty good considering I could only get about 1.8MB/s on cable-only connection.

Couldn't squeeze more from both connections, when I tried to download a torrent file with huge amount of seeders my usenet speed dropped and all I got get was about 2.4MB/s~2.5MB/s monitored at my own computer. I expected to get near 3.5MB/s at least.

Also tried doing it from two different computers downloading files from usenet and torrents, they shared the 2.5MB/s speed between them(around 1MB/s and 1.5MB/s on the other one) Also my Skype call was very laggy.

I noticed the WAN #2 is not really being used that much, on average of 600Kbps from the traffic graph.

I tested both on Speedtest individually and I got 15.62Mbps on the DSL connection and 18.92Mbps on the cable connection.

I know combining WANs might not be usual for torrents/usenet(I am mostly doing it for testing purposes) but my objective later on is to balance a server that has limited bandwidth and add a backup connection to it.

Talking about this subject, is it possible for pfSense to balance an UDP connection that's incoming? Let's say, I have a request for the port 1433 on WAN #1, is it possible to split this traffic between both WANs? Like redirect the traffic to my other WAN IP ?

[stephenw10]

Hmm, I agree something is not right. You should be able to max. out both connections.

In bound load balancing can be done, to share load between two internal web servers for example, but not to share WANs. For that you would need some sort of external proxy.

Steve

[nitz]

Ok, I got it fully working now.

I am not sure what was the problem but I did a config reset and started all over. Same problem until I set the MTU manually on the DSL connection.

Works like a charm, downloading at nearly 4MB/s from usenet.

It'll be very useful here since the kids do like to download a lot of stuff and I might even set up a mini itx firewall with load balancing and traffic shaping.

Thanks a lot  ;)

[stephenw10]

No problem, glad you got up and running.
Don't know how much help I was in the end.  ::)

Steve