Multi-Wan Issue (possible assymetric routing problem)?
I am using 1.2.3 (and have been for a few months). So far it has been rock solid and is exactly what my business has needed. We recently added an additional WAN connection to our existing set up. I added an opt interface (followed instruction in the Pfsense book) created a load balance pool and change my default gateway on my default lan rule. I now have an interesting problem. I did a speed test and my upload speed is now 10mb (the speed of the new wan) and my download speed is 4mb (the speed of my old wan). The first thing I noticed was the fact that it wasn't 14mb like I was expecting it to be but the second thing I noticed was that it obviously isn't using the same wans for upload and download which leads me to believe there is a routing problem. I check and rechecked my config but it seems correct. So now I have a few questions:
1: Do I have to enabling NAT on this device to make load balancing work?
- Natting is done by another device on our network (behind the pfsense). I can change this if it is required but I would like to know.
2: How do you change the default route?
- It would seem that no matter what I do it stays the original wan interface
Anyone have any ideas?
You may need to have NAT, it depends on how the device that does NAT is configured. Traffic that goes out a WAN must have one of your IPs on that ISP as its source or things won't work right. So if whatever is doing NAT isn't smart enough to NAT to the appropriate public IP depending on which WAN the traffic is leaving, you must do NAT on at least some traffic leaving from the edge firewall.
You can't change the default gateway in 1.2.3 (and there's generally no need to do so).
Thanks for the reply. I am going to copy my response from another forum:
Here is my topology:
<internet1>- 4mb <internet2>- 10bm
Keep in mind that all of my natting is done on the internal firewall. The pfsense box is only used for country blocking and for static routes to my two providers.
I set it to Gateway (which according to the pfsense book is the correct way). Server was for, well Web gardens and such. The load balancing pool shows up as green when I check services and the thing is when I do a tracert from a machine on the network, it does take the path of the wan that I want it to. I did the "ghetto" way of unequal load balancing by adding the faster wan's interface in there several times. It is really strange that upload works but download doesn't.
My default "any any" lan rule is pointing at "Wan Load Balance" (my pool name) for it's gateway and like I said, the only problem I have is download, not upload. That makes me believe that it is something with my wan rules coming back, like somewhere in my routing table, I should add the "Wan Load Balance" pool instead of my default gateway (which on 1.2.3 apparently cannot be changed, period).
I did some reading on asymmetrical routing and for the life of me I can't think of a reason A: Why you would want to do this and B: Why this is happening. Check out the pics:
One thing to consider is that a single TCP session will be tied to one WAN connection. You might try your results using the speed test on www.speakeasy.net/speedtest, it supports multiple connections and should yield different results. If you have access to an offsite location you could use iPerf (2 sessions) to do your own bandwidth testing which may be more consistent.
Have you experimented with the sticky connections setting (system\advanced)? If sticky connections is enabled successive connections to the same target IP address will be routed through the same WAN interface. Enabling/disable might improve your results. Is the bandwidth on your wan connections symmetrical, eg 4Mb down/4Mb up?
I feel that the load balancing support in pfSense 2.0 has been greatly improved over 1.2.3, it might be worth testing to see if you have improved performance.
When I view your screenshot of the gateway status I see that opt1 is listed 3 times, I'm not sure what's going on there. From your description I would expect WAN and OPT1, since you have 2 internet connections.
When you do load balancing with speedtest.net you'll end up with weird results because it recently changed to use multiple TCP sessions, but only one TCP connection at a time. With the type of load balancing setup you have, parts of the test will use each WAN, so it's normal to end up with results that either match only one or the other, or in some instances, that match neither (as part of the download test can use one WAN and part of it the other). Sounds like you're also expecting a single TCP connection to use the sum of your bandwidth, which is impossible as it must be tied to only one WAN.