Help with Multiple WAN setup.

leimrod · Feb 16, 2007, 6:13 PM

Ok i'm trying to setup load balancing and failover of 3 inbound WAN connections. I have 4 identical 3COM 10/100 NICs in a PIII machine with 512MB of RAM

I've read the guide here (http://doc.pfsense.org/index.php/Multi-Wan/Load-Balancing) for setting up load balancing but it outlines how to do it with DHCP enabled. I'd prefer to have DHCP disabled if possible. Is there any guides to show me how to set up load balancing and failover without DHCP?

Has anyone got any ideas? If you's need any further information don't hesitate to ask.

sai · Feb 17, 2007, 5:51 AM

Without DHCP is easier. You just get rid of the internal routers shown in http://doc.pfsense.org/index.php/Image:LoadBalanceOview.png

(The internal routers are there because Loadbalancing needs static IP addresses and does not work with changing IP addresses that you get with DHCP)

hoba · Feb 18, 2007, 8:15 PM

It's just the same like when using dhcp. Just configure the interfaces static and set up the pools. Then add firewallrules to make use of the pools.

leimrod · Feb 19, 2007, 2:11 PM

ok bear with me for a minute. I found this pdf on the wiki: http://doc.pfsense.org/contrib/PFSENSE-LoadBalance-FailOver-V3.pdf

It seems to show how to set up load balancing of multiple WANs using 2 WAN connections.

Problem is I have 3 connections I need to load balance, will there be much difference to the setup using 3 inbound line? I'm guessing some things need to be changed with the failover settings. Could someone explain to me what would need to be changed.

Also in the "planned configuration" of this pdf, he has these settings:

LAN IP Address: 192.168.1.1
WAN IP Address: 190.165.30.30
WAN IP Gateway: 190.165.30.2
WAN Router IP: 190.165.30.2
Internet Static IP: ISP Assigned

OPT1 IP Address: 189.165.20.20
OPT1 IP Gateway: 189.165.20.2
OPT1 Router IP: 189.165.20.2
Internet Static IP: ISP Assigned

Now, my pfsense's IP address is 192.168.1.1 but my WAN IP address, WAN Address and WAN IP gateway are all 192.168.0.254? Is this normal? Should my WAN IP address be different to my Gateway?

Also, what do I need to change on my gateways for my incoming WAN connections?

All help appreciated.

Pootle · Feb 19, 2007, 4:23 PM

Leimrod, you need to treat that pdf file carefully, as it was not written to match the current version of load balancing.

If I understand you correctly from your original post, you want to avoid DHCP on the modem / routers you are using.

Do you still want your modem / routers to do NAT? If so then all you do is avoid using DHCP in the modem / router setup. On the NIC that connects to your modem router, pfsense and the modem / router need to have different IP addresses in the same subnet, and each NIC used for WAN access must be in a different subnet.

So with 3 of them you could use:

WAN1: Modem / router: 192.168.0.254, pfSense 192.168.0.2

WAN2: Modem router: 192.168.2.254, pfSense 192.168.2.2

WAN3: Modem router: 192.168.3.254, pfSense 192.168.3.2

pfSense should have gateway on each NIC set to 192.168.x.254 - the address of the corresponding modem / router

leimrod · Feb 19, 2007, 6:12 PM

Ok I understand that. I won't be using NAT, so i've ignored the NAT setup at the end of that PDF.

I have one problem, we have a wireless connection that has no router interface at its gateway (i.e. we connect to the aerial on the roof and that connects to the ISPs host Transceiver) So I can't change the gateway that we recieve. Will this be a problem

The gateway of the wireless connetion is in the form of:
89.16.71.1

Here is what I have setup from that PDF tutorial, all static connections:
WAN1: IP Address: 192.168.0.2 Gateway 192.168.0.254
WAN2: IP Address: 192.168.1.250 Gateway 192.168.1.254
WAN3: IP Address: 89.16.71.x Gateway 89.16.71.x

Would this work?

Also for my Monitor IPs, can I use any external IP's? i.e. could I use google.com for WAN1, hotmail.com for WAN2 and wikipedia.com for WAN3? Would this work?

I've also attached a screenshot of how my load balance section is set up. Other than just setting up this one line in the load balance section is there anything else I need to set up to enable load balancing? I'm not too concerned right now with a firewall or NAT, I just want to get the load balancing working first.

EDIT: Oh and one more question, in my status/load balancer section all 3 lines are showing as "online", but one of them is coloured green, the other 2 are yellow, what does this mean?

databeestje · Feb 19, 2007, 7:30 PM

Yes you can use direct links, I have 2, one is wireless to the neighbours. The other is my dhcp from the cable modem.

There is no preset limit to the amount of internet connections you can balance.

Your example should work fine.

You can use any external IP. It should be unique, e.g. not used on another interface. It should also be on of the first hops out from your router. In the case of you wireless this is already the case.

Just edit the LAN rule to use the load balancer pool as the gateway. This is listed on the bottom of the firewall rule edit page.

The colors change if the connection state changed within the last 5 minutes. e..g yellow.
Green means you're doing fine.

leimrod · Feb 20, 2007, 9:58 AM

Seriously though is that ALL I have to setup? One rule in the load balancing section? Why is there the option to add more rules if only one is needed?

Will this setup also work for failover, i.e. if I plugged out 2 cables would everybody on the LAN still be able to get their internet off the 3rd line?

Pootle · Feb 20, 2007, 10:50 AM

@leimrod:

Seriously though is that ALL I have to setup? One rule in the load balancing section?

I have 3 rules, 1 for full balance and 1 when each WAN has failed, so you may well need 4 in total

All as in the new wiki guide, note I am using 1.0.1-SNAPSHOT-02-14-2007

leimrod · Feb 20, 2007, 1:56 PM

Is there a default firewall enabled in PFsense? because i'm trying to test to see if my download speeds are load balanced by downloading some high speed linux torrents. But i'm getting an error that my router is behind a firewall. When I set my IP settings to connect directly to one of the WAN routers I don't get this error, I only get it when I set my gateway to the pfsense gateway.

Is there some firewall rules or nat settings I need to change to turn them off? I don't want any kind of blocking in place with this load balancer, at least not for the moment until i'm certain the load balancing is working.

Pootle · Feb 20, 2007, 2:12 PM

torrent isn't really a good way to test this because torrents get terminally confused if 1 client tries to use 2 different IP addresses - at least that's my experience with uTorrent.

I have set up forwarding on 1 of my WAN connections to forward to pfsense - see below - to run torrents. To test balance, run a couple of speed tests or something similar.

leimrod · Feb 20, 2007, 2:52 PM

see this is my main problem, I can't think of a way of testing the load balancing is actually working because i'd need 3 seperate downloads that would max out each connection.

the way you have it set up does that allow for torrents to be load balanced, or are you just setting up one line as your torrent line?

EDIT: Also is there any settings I need to change in the firewall section to ensure that NAT and the Firewall is turned off?

Pootle · Feb 20, 2007, 3:05 PM

@leimrod:

see this is my main problem, I can't think of a way of testing the load balancing is actually working because i'd need 3 seperate downloads that would max out each connection.

You don't need to max the connection for load balancing to kick in. I find just doing traceroute a few times usually picks up different WAN connections. Anyway what's so hard about running three browser sessions?

the way you have it set up does that allow for torrents to be load balanced, or are you just setting up one line as your torrent line?

I just use 1 connection for torrenting - it gets confused if I leave it load balanced.

Also is there any settings I need to change in the firewall section to ensure that NAT and the Firewall is turned off?

Wow, if you turn off nat and the firewall, I'm not sure you have anything left worth keeping, can't really help you there

hoba · Feb 20, 2007, 3:06 PM

To test the loadbalancer try tracerouting to different locations from a client behind pfSense like:
tracert google.com
tracert yahoo.com
tracert lycos.com
…

You should see the traffic taking different routes through your WANs. You also can visit a page with lot's of external images and open status>traffic graph for each wan in different windows. You should see traffic going through all WANs while browsing.

If you want to shut down NAT go to firewall>nat, outbound and enable advanced outbound NAT. Then delete all autocreated rules at the bottom, save and apply.

Btw, if you shut down the firewall you won't be able to use policybased routing or loadbalancing as it is handled by pf too. Create pass rules for all the traffic with appropriate pools and gateways instead.

leimrod · Feb 20, 2007, 3:43 PM

@hoba:

To test the loadbalancer try tracerouting to different locations from a client behind pfSense like:
tracert google.com
tracert yahoo.com
tracert lycos.com
…

You should see the traffic taking different routes through your WANs. You also can visit a page with lot's of external images and open status>traffic graph for each wan in different windows. You should see traffic going through all WANs while browsing.

If you want to shut down NAT go to firewall>nat, outbound and enable advanced outbound NAT. Then delete all autocreated rules at the bottom, save and apply.

Btw, if you shut down the firewall you won't be able to use policybased routing or loadbalancing as it is handled by pf too. Create pass rules for all the traffic with appropriate pools and gateways instead.

Thanks, that has given me plenty food for thought. I never thought to do a tracert, but it makes sense. Can I ask why it will use different routes if it doesn't have to? I thought it would only change over to the second connection once the first one was maxed out.

Also, do you have any links to tutorials on how to set up pass rules for the load balancer? What are the benefits of policybased routing and loadbalancing?

hoba · Feb 20, 2007, 4:21 PM

It distributes each new connections roundrobin to all poolmembers in loadbalancer mode. Loadbalancing won't work for some special applications like https for example (as you are hopping between IPs) so you want to use policybasedrouting for this. You should create a failoverpool and use this as gateway in your firewallrules for this kind of traffic.

http://doc.pfsense.org/index.php/Multi-Wan/Load-Balancing has quite some extensive information about multiwan setups.

databeestje · Feb 20, 2007, 5:29 PM

the default any LAN rule matches anything, including icmp/traceroute.
To make it even a better example, a traceroute from a LAN pc to each monitor IP should use the corresponding wan connection.

I balance with a 8/1 Dsl and a 8/8 fiber. I want all email traffic to use the fiber line because it has more bandwidth. So i make sure it uses that one, and fails over to the DSL if that one does not work. (ordering is important)

For all the webtraffic (which is mostly downstream) I match that with a rule that refers to my load balanced pool.

leimrod · Feb 21, 2007, 11:15 AM

@databeestje:

the default any LAN rule matches anything, including icmp/traceroute.
To make it even a better example, a traceroute from a LAN pc to each monitor IP should use the corresponding wan connection.

I balance with a 8/1 Dsl and a 8/8 fiber. I want all email traffic to use the fiber line because it has more bandwidth. So i make sure it uses that one, and fails over to the DSL if that one does not work. (ordering is important)

For all the webtraffic (which is mostly downstream) I match that with a rule that refers to my load balanced pool.

Could you post a screenshot of this setup?

Also databeestje I went through tutorial on the wiki, but it's for a DHCP setup, its hard for me to distinguish what settings are pertaining to DHCP and what ones I need for Static IPs. Some advice on what sections of it can be applied to a static IP setup would be appreciated.

hoba · Feb 21, 2007, 12:47 PM

Have a look at http://doc.pfsense.org/index.php/Multi-Wan/Load-Balancing

leimrod · Feb 21, 2007, 1:14 PM

@hoba:

Have a look at http://doc.pfsense.org/index.php/Multi-Wan/Load-Balancing

Hi hoba, yeha I did have a look at that link but its a tutorial for a DHCP setup which is not what i'm setting up. Would the load balancer and firewall rules be the same for static IPs as they would for DHCP?