pfSense dual WAN load-balancing where the LAN feeds the WAN on main router (a UDM-Pro)
-
Hi,
not quite sure what you want to end up with here. Load balancing, so that traffic is spread across both lines simultaneously or failover?
There are several approaches here. One would be the routing approach, where the pfSense becomes the default gateway for the UDM. The pfSense DOES need a public ip address on the outside for this, because the provider's router needs a gateway....
Another one would be a bridging approach where pfSense could just magically move the packets over to fibre 1 or 2.
The third one would be a brouting approach. Transparent to the outside, not needing a public ip address and default gateway with a private ip address on the inside for the UDM as main router. (This includes gateway pinging and monitoring to detected a faulted line, yes... I've done more nasty stuff than this with OpenBSD and IPSec ;-P )
UDM would then hold the two public ip addresses of fibre 1 and 2 and respond to them .
Unless the fibre connections are layer 2 and you control what is on the other end, then there could be a totally different setup here.
From the point of understanding I'd say routing is the simpler approach while brouting should be the best to go with here.
Cu
-
Hi,
Sorry for not being clear. So I have 2 Internet connections from my ISP. They are on different subnets.
So I have 2 fibers coming into my building, each terminate to separate Calix ONTs. The RJ45 output from each of those ONT's are then going to be connected to the 2 WAN interfaces I have created in pfSense.
In a perfect word, I'd love to do MLPPP, but my ISP does not offer that option currently. So I figured the next best thing would be to load balance the 2 incoming connections to give me twice the bandwidth a single connection can offer.
So to answer your first question, yes I want to spread the load evenly across both WAN connection coming from my provider.
So the UDM is going to be my primary gateway/router since it sucks at load-balancing dual WANs (in fact is only supports fail-over at this point). So all I want to pfSense configured for, is to load-balance the 2 incoming connections from my ISP, and pipe it through a single SFP+ 10Gbps connection to my UDM-Pro (it has SFP+ WAN and LAN interfaces).
The pfSense box I but together is really way overkill for this I think. It has a Xeon E5-2683 v3 CPU with 14 cores and 32GB of RAM. The motherboard is a Supermicro X10 series with dual Intel i210 NICs. I added a Chelsio T520-CR to give me the LAN SFP+ port I need to connect to the UDM-Pro.
My understanding is that I can't completely disable the Firewall features in pfSense since the load balancing requires it for the LAN Rule I setup to tie it to the MultiWAN gateway I created.
Here's my LAN rule I created:
And here's the gateway group:
Am I on the right track?
Thanks!
-
@pclausen said in pfSense dual WAN load-balancing where the LAN feeds the WAN on main router (a UDM-Pro):
In a perfect word, I'd love to do MLPPP, but my ISP does not offer that option currently. So I figured the next best thing would be to load balance the 2 incoming connections to give me twice the bandwidth a single connection can offer.
Yeah. But a rented dedicated Server can do that trick ;-P Has been done before, will be done again.
So far the configuration looks good. Do you have enough public IP addresses to spend two on the pfSense or will the UDM get private addresses and you will do port forwarding?
Cu
-
@Grimeton I do have a rented server with 20Gbit/s network speed, but this is for my home network. :)
Each connection from my ISP only has as single public IP associated with it. One of them is static, the other dynamic. I had to do this in order to get each WAN connection on different subnets as I do not currently have any intermediate NAT devices between the ONTs and pfSense. I'm not hosting anything locally, nor do I have a registered domain running on my home network.
In other words, I don't need any public IPs associated with anything on my network. I run OpenVPN between a host in my home network and the rented server.
So in pfSense, both WAN interfaces will be configured for DHCP and the provider will hand me Gateway information, etc.
On the LAN side, I'm still not sure if I should it up as static or dhcp. I leaning towards static and the following for the LAN interface on the pfSense side:
And the corresponding WAN configuration on the UDM-Pro side:
So given this, what I'm not clear on, is if the LAN interface on the pfSense side is considered an Internet connection that requires a Gateway to be associated with it?
-
So I went ahead and connected the LAN interface on pfSense to WAN2 on the UDM-Pro.
So right now this is what my network looks like:
The green line represent the interface the ISP ONT2 is currently connected to, namely WAN1 interface on the UDM-Pro.
The red line is where I want ISP ONT2 connected once everything is sorted out so that pfSense can load-balance both WAN connections.
If I unplug the green line my main 10.0.1.0/24 network looses Internet connectivity as it is unable to go through the pfSense router for some reason.
From the LAN behind the UDM-Pro, I can connect to the pfSense box fine on 10.0.2.1.
From a pfSense shell, I can ping the outside by ip or name as well as the LAN interface on 10.0.2.1, but I can't ping the WAN2 interface on the UDM-Pro on 10.0.2.2.
-
@pclausen said in pfSense dual WAN load-balancing where the LAN feeds the WAN on main router (a UDM-Pro):
So given this, what I'm not clear on, is if the LAN interface on the pfSense side is considered an Internet connection that requires a Gateway to be associated with it?
This has nothing to do with if the LAN-side is an internet connection or not. It clearly is not.
The thing you need to know here is what the UDM does on it's WAN side. If it's NATing everything that goes out over its WAN, then pfSense knows the way back to 10.0.2.2 as it is in the same subnet and everything is fine.
If you do NOT nat outgoing stuff on the UDM, then pfSense needs to know the way back to the networks that are behind the UDM. So you have to add the routes on pfSense.
Let's say behind the UDM is a 10.2.3.0/24 and the UDM does NOT nat, then on pfSense you'd go to System -> Routing and add a gateway there that points to the UDM (10.0.2.2).
Then you click on "Static Routes" and add a route to 10.2.3.0/24 via the UDM gateway you just created.
That's it.
-
@pclausen said in pfSense dual WAN load-balancing where the LAN feeds the WAN on main router (a UDM-Pro):
If I unplug the green line my main 10.0.1.0/24 network looses Internet connectivity as it is unable to go through the pfSense router for some reason.
Yeah the interface goes down and the UDM deletes the default gateway that went out via WAN1. No default gateway, no cookies.
From the LAN behind the UDM-Pro, I can connect to the pfSense box fine on 10.0.2.1.
Sure, why not? I guess even the UDM itself can ping 10.0.2.1 fine. (Same subnet)
From a pfSense shell, I can ping the outside by ip or name as well as the LAN interface on 10.0.2.1, but I can't ping the WAN2 interface on the UDM-Pro on 10.0.2.2.
Pinging 10.0.2.1 from pfSense is a bit pointless as it is the pfSense's own interface.
The UDM for sure has a default gateway deleted and no other gateways at hand so it doesn't know what to do here. So you either set the default gateway to pfSense (10.0.2.1), or you at least route to the ip of WAN2 via pfsense. So if WAN2 has 8.7.6.5 you'd add a route on the UDM to 8.7.6.5/32 via 10.0.2.1 and it should work.
The confusion here starts when you disconnect the UDM's WAN which erases the default gateway.
I BET, when UDM's WAN was connected and you pinged WAN2:IP then you thought the ping would flow through UDM-WAN2 -> pfSense-LAN -> pfSense WAN2 but in reality it went out UDM's WAN1 and came back in via WAN2.
Cu
-
Appreciate the help as I'm clearly an IP Routing novice!
I think I'm getting what you're saying, but to troubleshoot the issue further, I decided to remove the UDM from the equation altogether and connect a windows box directly to the LAN port on pfSense, and I can STILL not get to anything.
Quick review of my settings;
My LAN interface:
My gateways (I did add the one you recommended):
Gateway groups:
Firewall rules:
Details for the allow LAN to any rule:
On the host I have connected directly to the LAN port, I have these settings:
And the host can't get to anything on the Internet, but from a shell on pfsense I can. So I'm still missing something elementary to make this work I think.
Here's what the dashboard is showing for the Interfaces and Gateways:
-
Take this from the top:
- When both WANs are connected does it work as expected?
- If not, are the outgoing NAT rules still there?
- If not, can you login via SSH and check via tcpdump what happens to the packages that you send in on LAN? E.g. trace on WAN1 and/or WAN2 to see if the packets go out there.
- If they do, are they nated.
- If they don't, check that the rule you added for the gateways is AT THE TOP of the list or make sure no other rule interferes with it. Rules are processed top to bottom and it stops the moment one matches.
Cu
-
@Grimeton Thank you for your help!
It was indeed related to my Outbound NAT rules. For some reason that was set to "Disable Outbound NAT rule generation". Once I changed it to Automatic outbound NAT rule generation, the host was able to get out.
I then moved the pf Sense LAN connection back to the UDM-Pro, and it picked it up and was able to pass traffic.
-
@pclausen just found this thread (I recongize your username from the Unifi forums lol, you may recognize mine). How is this working for you so far? I am actually thinking about doing the same thing myself but not quite sure how port forwarding would work out. I'm by no means a network noob, but I haven't done a configuration like this before either, but it might be the perfect solution for me. I really want dual WAN with selective port forwarding so I can use a specific WAN for some machines (I have high down, but low upload speed, so having specific machines that upload a lot on their own WAN so I can upload my YT videos on my own WAN is great), is that functional in a setup like this?
I really don't want to give up the Unifi interface for my network, it's so nice, but I need more functionality, this could be the perfect blend for me if I can get it working right.
Thoughts?
-
@planedrop Yep, recognize you from the Unifi forums. Running pfSense in front of my UDMP has worked out great once I sorted out the Outbound NAT rules. For the last 30 days, I haven't had a need to touch anything in pfSsense, it just works.
So other than load balancing my 2 WANs, I don't do anything on pfSense, everything else is happening on the UDMP. I don't have any port forwarding in place right now as I don't really need it, but my VPN to my work machine on a corporate network (using Cisco Anyconnect) has been working flawlessly from my personal home workstation.
That said, I would imagine that in order to make port forwarding work properly, one would have to make entries on both the UDMP as well as in pfSense and I'd imagine pfSense will let you make port forwards sticky on one WAN or the other.
As I mentioned on the UniFi forum, once I get my dual symmetrical GigE WANs up and running, and be doing some benchmarks from machines behind the UDMP, as well as from a box hanging directly off the pfSense appliance.
