[HOWTO] Multi WAN Traffic shaper with bandwidth limits per interface

deajan

The multi WAN traffic shaper wizard doesn't work with a lot of usecases, especially when having multiple WAN links with different download bandwidths.

In a discussion at https://forum.pfsense.org/index.php?topic=113586.0, Harvy66 suggested a way to use floating rules to assign download queues for outgoing traffic on WAN links. This is a quite nice solution to the problem, although a bit unintuitive to setup.

Here's a small tutorial on how to setup basic traffic shaping to limit upload and download speed per WAN link, and assign multiple priority queues.
All improvements to this tutorial are welcome.
Tested on pfSense 2.3 - 2.3.2-p1.

If too much traffic goes through your links, you'll encounter packet loss and bufferbloat.
You'll probably prefer having that packet loss on your pfSense than on your WAN modems in order to keep some degree of control and have some metrics.
Generally speaking, if you don't want to overload your WAN links and limit packet loss (on your WAN links only, pfSense will still suffer packet loss if too much traffic is sent), you should never use more than 95% of the available bandwidth of a WAN link.
This doesn't mean that you should use 9.5M of a 10M link, as 10M is probably more a commercial given value than a real one.
You'll have to measure the real speed by connecting a "clean" computer directly on your WAN modem / router and go to speedtest.dslreports.com or speedtest.net (or whatever you like).

In order to setup the traffic shaping, we'll use the following three hypotetical WAN links:

WAN1 = Fiber: 100M/10M (measured at 94Mb/9.8Mb)
WAN2 = ADSL: 18M/1M (measured at 15Mb/920Kb), some FEC errors on the line
WAN3 = SDSL: 4M/4M (measured at 3.9Mb/3.8Mb)

Go to Firewall > Traffic Shaper
Remove any traffic shaper queues if some are configured.
For every WAN interface listed in the Traffic Shaper:

Click on "Enable/disable discipline and its children"
Keep the HFSC scheduler as HFSC is the only scheduler allowing children queues without any errors in pfSense 2.3-2.3.2 so far. Also, mixing different schedulers isn't working yet on pfSense. So even if you don't need any special subqueues on WAN links, you'll still need them on the LAN interface later.
The bandwidth parameter to set here is 95% of the measured upload speed:
WAN1 = 9.8x0.95 = 9.3Mb
WAN2 = 920x0.9 = 828Kb (we use a lower multiplier because the line isn't stable)
WAN3 = 3.8x0.95 = 3.6Mb
Queue Limit and TBR Size are left empty unless you know exactly what you're doing
Click on Save

Configuring the bandwidth parameter here is sufficient to enforce the upload speed of pfSense to the WAN modems.

Now that the upload speed is enforced, we'll need to enforce the download speed.
Click on the LAN interface listed in the Traffic Shaper:

Click on "Enable/disable discipline and its children"
keep the HFSC scheduler
The bandwidth parameter to set here is the full bandwidth of your LAN. If your pfSense is connected to a gigabit switch, set it to 1000Mb. If not, set it to 100Mb.
Queue Limit and TBR Size are left empty unless you know exactly what you're doing
click on Save

We now need to divide the LAN queue into multiple subqueues, one for the LAN link, and one per WAN link.
Be sure to save the parent queue before creating a subqueue, as the pfSense interface is quite buggy here and let's you create a subqueue even when the parent queue isn't saved, resulting in the lost of both.

On the LAN queue, click on Add New Queue

Name the queue qLink
Click on "Enable/disable discipline and its children"
keep the HFSC scheduler
Set the priority: 4
Set this queue as "Default Queue"
Activate Codel Active Queue (in order to better deal with bufferfloat)
The bandwidth parameter to set here is the full LAN bandwidth minus all the download speeds of the WAN links combined. In our example, we'll have 10000 - 94 - 15 - 3.9 = 887.1 Mb
Set Min bandwidth for the queue (real time) m2 value to something like 20Mb to ensure that there's always some minimum bandwidth to access pfSense

Now create three other subqueues on the LAN queue

Name the queues qDownloadWANx where x is the WAN number
Click on "Enable/disable discipline and its children"
keep the HFSC scheduler
Set the priority: 4
Activate Random Early Detection and Codel Active Queue (in order to better deal with bufferfloat)
The bandwidth parameter to set here is 95% of the measured WAN download speed of each link
WAN1 = 940.95 = 89.3Mb
WAN2 = 150.9 = 13.5Mb
WAN3 = 3.8*0.95 = 3.61Mb
Set Max bandwidth for the queue (upper limit) m2 value to the same value as the bandwidth

If you only need to limit the bandwidth, we're done here.
If you also want to shape your traffic, you'll have to create some subqueues for every qDownloadWANx queue as following:

Name the queues qDownloadHighWANx and qDownloadLowWANx where x is the WAN number
Click on "Enable/disable discipline and its children"
keep the HFSC scheduler
Set the priority: 7 for high, 1 for low
Activate Random Early Detection and Codel Active Queue (in order to better deal with bufferfloat)
Set the bandwidth parameter as some percentage of the parent queue, example:
qDownloadHighWANx = 40%
qDownloadLowWANx = 60%

Note that the example image has more queues that the one described here for testing purposes.

We now have some queues ready to shape our traffic.
In order to let pfSense enforce the download speed of the WAN lines, our qDownload queues must be applied on the WAN links, even if they virtually exist on the LAN interface in the traffic shaper.
The way to achieve this is using matching floating rules, which basically will only add queues to the traffic.

Matching floating rules work from most generic to most precise rule, meaning that we should put the "catch all traffic" rules first, and then deal with more specific traffic rules.

Let's first create our "catch all" queue applying rules. In Firewall > Rules > Floating, create the following rule for each WAN link:

Action: Match
Interface: WANx where x is the WAN number
Direction: out (yes, it is outgoing direction !)
Address Familiy: IPv4 and IPv6
Protocol: Any
Gateway: default
Ackqueue / Queue: none / qDownloadWANx where x is the WAN number

Once applied, you may check on Status > Queues that all the traffic is affected to the corresponding queues.

Let's now create some more precise rules in order to increase DNS traffic priority and lower SMTP traffic priority. Create the following two rules for each WAN link:

Action: Match
Interface: WANx
Direction: out
Address Familiy: IPv4 and IPv6
Protocol: TCP/UDP
Destination Port Range: DNS
Gateway: default
Ackqueue / Queue: none / qDownloadHighWANx

and

Action: Match
Interface: WANx
Direction: out
Address Familiy: IPv4 and IPv6
Protocol: TCP
Destination Port Range: SMTP
Gateway: default
Ackqueue / Queue: none / qDownloadLowWANx

Those matching rules should be after the generic "catch all" rules in order to get executed last.

Rule order is important, from most generic to most precise.
Note that the example image has more download queues as described here for testing purposes.

You may check correct queue assignment in Status > Queues.

Note that you may also create subqueues with different priorities and bandwidth settings in the traffic shaper's WAN interfaces, and create floating rules for upload traffic.

reinhart47

We have 3 wans with different speeds and 4 vlans (representing different workgroups). The models I have found show multi Wan, single lan. How does multi lan change this? I assume it will complicate it.

deajan

If you're dealing with multi LAN, the problem would be that every LAN link is setup to fully use download link speed of the WANs, which means that multiple LANs would overload WAN download capacity.
What are you expecting from traffic shaper with your multi lan setup ?

Harvy66

Since you can't share bandwidth across multiple interfaces, you are left with three options.

Give each LAN segment a fixed amount of bandwidth for each of the WANs. No bandwidth shared, free bandwidth from one interface cannot be used by the others
Configure each LAN segment like the above multi-wan-single-lan shaping, but you won't gain a whole lot
Don't do any LAN shaping.

1 and 2 will have the same queue setups, just the amount of bandwidth assigned to the queues will be different.

travanx

Thank you for putting this together. I have 300/7 cable internet, speedtests varies between 300 and 400, and could not get the default wizard to work right when I input 300M as my download speed. Ended up using 1000M for LAN, and changing inside of qLink (687M) and qInternet (313M) from the default wizard settings.

Going step by step through your tutorial on a single WAN helped me understand how this gets setup along with how floating rules work.

deajan

Thanks, always nice to have some good feedback :)

pki

Hi.
I have set up the traffic shaper following this howto.

But, i still have problems:

i can not get the traffic into the right queue
i can not get the upload traffic to limit

Short to say, it's not working for me :(

About my setup. WAN (WAN1) is DSL 10/1, WAN2 is second internet provider 25/5 with PPPoE, WAN3 (opt2) is the same provider and connection as WAN2, second PPPoE channel with a second IP.

Attaching screenshots. On WAN2 i have permanent about 2Mb of traffic, but only a part is sorted into qDownloadWAN2. The graph on the main page shows two speedtests, i am getting 25/5 result (or above), not limiting :(

Can anyone help to catch where the problem is?

The worst problem is on the DSL connection, if overloaded it goes very lattency.

pfsense0.png_thumb

pfsense1.png_thumb

pfsense3.png_thumb

pfsense4.png_thumb

pfsense5.png_thumb

pfsense6.png_thumb

deajan

@pki incomplete data you gave here. No crystal ball available.

pki

I tried to give it all, what should i add??

What information are You missing?

deajan

@pki Detail the floating rules

pki

I have done the rules and traffic shaper again, from scratch. I think i missed the limit checkbox on the download shapers. Also the rules was not easy for me.

Can You explain:

how to do the rules for upload traffic should look? For example priority for VoIP.
how to catch the traffic by LAN ip if possible? For example to put whole traffic from VoIP server on highes priority? I have done this now by the ip of the external server, was not able to catch the traffic by LAN ip. I am setting some outgoing IP (Virtual IP) on the outgoing NAT.

Thx

allan34

thanks for this post on multiwan. It gave the inspiration to solve our main problem:

Multiwan per ip traffic shaping

Now - I am no FW expert, so please comment if you have better ideas.

The problem for us using the above approach is that the lan clients ip's are not visible to the floating rules, as this is the post NAT stage of the packet flow. i.e. they all have the same ip of the WAN interface of the fw. Only the dst port and ip are for matciing the rules to.

Policy based routing to the rescue:

Use floating rules, but instead assign queues based upon tags (which indicate the priority) and the WAN link:

First, the default rule remains unchanged, for each WAN link:

Action: Match
Interface: WANx where x is the WAN number
Direction: out (yes, it is outgoing direction !)
Address Familiy: IPv4 and IPv6
Protocol: Any
Gateway: default
Ackqueue / Queue: none / qDownloadLowWANx where x is the WAN number # Default to the lowest priority.

Now assign queues based on the "tag" of the packets, create rules for each of the wan links:

Action: Match
Interface: WANx
Direction: out
Address Familiy: IPv4 and IPv6
Protocol: TCP/UDP
Destination Port Range: any
- Tagged : qLow
Gateway: default
Ackqueue / Queue: none / qDownloadLowWANx
Action: Match
Interface: WANx
Direction: out
Address Familiy: IPv4 and IPv6
Protocol: TCP/UDP
Destination Port Range: any
- Tagged : qMedium
Gateway: default
Ackqueue / Queue: none / qDownloadMediumWANx
Action: Match
Interface: WANx
Direction: out
Address Familiy: IPv4 and IPv6
Protocol: TCP/UDP
Destination Port Range: any
- Tagged : qHigh
Gateway: default
Ackqueue / Queue: none / qDownloadHighWANx

Test this, and all traffic should go to the default download queue for each link.
i.e. verify using Status > Queues

To assign traffic to low, medium, high queues need to tag the packets earlier on as they enter firewall using LAN rules. Pretty much how you would do for a single WAN but instead of assigning a queue you tag the packets.

Lets assume we have aliases for our lan clients
highpri_hosts, mediumpri_hosts, lowpri_hosts

Create LAN rules to assign priorities based on source ip:

Action: Pass
Interface: LAN
Address Familiy: IPv4
Protocol: Any
Source - single host or alias: lowpri_hosts
Tag: qLow
Gateway: default
Ackqueue / Queue: none / none
Action: Pass
Interface: LAN
Address Familiy: IPv4
Protocol: Any
Source - single host or alias: mediumpri_hosts
Tag: qMedium
Gateway: default
Ackqueue / Queue: none / none
Action: Pass
Interface: LAN
Address Familiy: IPv4
Protocol: Any
Source - single host or alias: highpri_hosts
Tag: qHigh
Gateway: default
Ackqueue / Queue: none / none

Thanks - A

deajan

@pki:

Can You explain:

how to do the rules for upload traffic should look? For example priority for VoIP.

how to catch the traffic by LAN ip if possible? For example to put whole traffic from VoIP server on highes priority? I have done this now by the ip of the external server, was not able to catch the traffic by LAN ip. I am setting some outgoing IP (Virtual IP) on the outgoing NAT.

Outgoing rules can be set using the existing qInternetWANx queues as floating rules on interface WANx.
You may also set assign the WAN queues on your LAN interface which IMHO is easier.

Traffic by IP rules can be achieved using source parameter on all rules, just use some aliases for your VoIP servers.

deajan

@allan34 thanks for sharing :)

Ma_Fabulette

Hello.
I'm Frederique and, even if i've been reading your contributions for some time now, I'm a new member on this forum.
First of all, I would like to thank all of you for sharing your experience and tutorials. As always I'm amazed by the generosity ;) I am a recent user of Pfsense solutions and have actually only implemented "out of the box" configurations for the moment. We are now facing several challenges and one of them lead me to your discussion. I stumble upon your message while researching a solution to my client current situation and I would really care for experts advice on this matter.

The curret client architecture is the following

1 LAN which supports data + VoiP
3 WAN on 3 different ISP
1 inside server which needs to synchronize with a distant server. No VPN

Today each WAN is dedicated to 1 usage (Data /VoiP/Replication), 2 of these 3 links are underused and the client wants to use the maximum of the available bandwith. We would like to implement a PFSense configuration with load balancing on all 3 WANs. The problem is that we need to protect VoIP bandwith (in and out) and also leave available bandwitch for the daily ongoing replication of both distant servers. We still need to assign a particular gateway to VoiP and server Synchro (since there is no VPN implemented).

I was wondering if the traffic shaping you're presenting in your post could be implemented with load balancing in odrer to resolve our client's issue ?
I woud really appreciate you advise on this matter before modeling the solution in my lab.
Thanks in advance.

deajan

@Ma_Fabulette: The floating rules described in the post are only matching ones. So basically you could make failover rules on the LAN side using routing groups, as long as you don't specify any queues there.
You might also merge the LAN queues in one if all the WAN lines have the same download capacity, so you can use priority queues easily.

Ma_Fabulette

Thank You for your answer Dejean.

After testing, it seems then that I cannot limit bandwith from the WAN to avoid congestion without limiting drastically the gateway group total bandwith (since I need to shape traffic on the LAN interface)

It seems then that if I want to shape specific traffic I need to have it limited to a specific GTW and eventually create a group wuith the remaining GTW from the rest of the traffic.

deajan

@Ma_Fabulette What exactly are you trying to setup ? Could you make a schema and explain what you're trying to do ? Would make it easier to understand.

ndemou

Very well done how-to deajan. Thank you. Have you tested what happens when 2 LAN clients eventually end up downloading at full speed from the same WAN? Is the BW of that WAN shared evenly between the 2 or does one get to have a huge chunk and one starves? I'm using limiters to achieve fair sharing of BW on my LAN and I'm VERY SATISFIED[1] but I'm not sure if limiters and queues can be combined [2] and my health bar is low for the moment[3]

NOTES:
[1] I'm using limiters to based on foxale08's how-to found here from https://forum.pfsense.org/index.php?topic=63531.msg364520#msg364520 and an excellent explanation of limiters by reddit user drakontas https://www.reddit.com/r/PFSENSE/comments/3e67dk/flexible_vs_fixed_limiters_troubleshooting_with/

[2] This question came up before in the forums but it was on a more complex setup and there is no answer https://forum.pfsense.org/index.php?topic=88627.0

[3]I've spend dozens of weeks reading, experimenting and learning traffic shaping first on IPfire then (when I've hit its limits) on pfsense. I need some time to recover and my co-workers need a few weeks of NO-EXPERIMENTS-DURING-WORK-HOURS :-)

deajan

AFAIK, you'll depend on the bandwidth share algorithm of the HFSC scheduler. If you want totally fair bandwidth sharing, CODELQ / FAIRQ are good alternatives but I'm not sure they might be implemented toghether with HFSC as of new pfSense releases. And you'll have to stick with HFSC in order to have sub queues on LAN lines.

Maybe an explanation of a scheduler expert might fit better here than mine. @pfSense community: someone ? :)