[HOWTO] Multi WAN Traffic shaper with bandwidth limits per interface

deajan

Thanks, always nice to have some good feedback :)

pki

Hi.
I have set up the traffic shaper following this howto.

But, i still have problems:

i can not get the traffic into the right queue
i can not get the upload traffic to limit

Short to say, it's not working for me :(

About my setup. WAN (WAN1) is DSL 10/1, WAN2 is second internet provider 25/5 with PPPoE, WAN3 (opt2) is the same provider and connection as WAN2, second PPPoE channel with a second IP.

Attaching screenshots. On WAN2 i have permanent about 2Mb of traffic, but only a part is sorted into qDownloadWAN2. The graph on the main page shows two speedtests, i am getting 25/5 result (or above), not limiting :(

Can anyone help to catch where the problem is?

The worst problem is on the DSL connection, if overloaded it goes very lattency.

pfsense0.png_thumb

pfsense1.png_thumb

pfsense3.png_thumb

pfsense4.png_thumb

pfsense5.png_thumb

pfsense6.png_thumb

deajan

@pki incomplete data you gave here. No crystal ball available.

pki

I tried to give it all, what should i add??

What information are You missing?

deajan

@pki Detail the floating rules

pki

I have done the rules and traffic shaper again, from scratch. I think i missed the limit checkbox on the download shapers. Also the rules was not easy for me.

Can You explain:

how to do the rules for upload traffic should look? For example priority for VoIP.
how to catch the traffic by LAN ip if possible? For example to put whole traffic from VoIP server on highes priority? I have done this now by the ip of the external server, was not able to catch the traffic by LAN ip. I am setting some outgoing IP (Virtual IP) on the outgoing NAT.

Thx

allan34

thanks for this post on multiwan. It gave the inspiration to solve our main problem:

Multiwan per ip traffic shaping

Now - I am no FW expert, so please comment if you have better ideas.

The problem for us using the above approach is that the lan clients ip's are not visible to the floating rules, as this is the post NAT stage of the packet flow. i.e. they all have the same ip of the WAN interface of the fw. Only the dst port and ip are for matciing the rules to.

Policy based routing to the rescue:

Use floating rules, but instead assign queues based upon tags (which indicate the priority) and the WAN link:

First, the default rule remains unchanged, for each WAN link:

Action: Match
Interface: WANx where x is the WAN number
Direction: out (yes, it is outgoing direction !)
Address Familiy: IPv4 and IPv6
Protocol: Any
Gateway: default
Ackqueue / Queue: none / qDownloadLowWANx where x is the WAN number # Default to the lowest priority.

Now assign queues based on the "tag" of the packets, create rules for each of the wan links:

Action: Match
Interface: WANx
Direction: out
Address Familiy: IPv4 and IPv6
Protocol: TCP/UDP
Destination Port Range: any
- Tagged : qLow
Gateway: default
Ackqueue / Queue: none / qDownloadLowWANx
Action: Match
Interface: WANx
Direction: out
Address Familiy: IPv4 and IPv6
Protocol: TCP/UDP
Destination Port Range: any
- Tagged : qMedium
Gateway: default
Ackqueue / Queue: none / qDownloadMediumWANx
Action: Match
Interface: WANx
Direction: out
Address Familiy: IPv4 and IPv6
Protocol: TCP/UDP
Destination Port Range: any
- Tagged : qHigh
Gateway: default
Ackqueue / Queue: none / qDownloadHighWANx

Test this, and all traffic should go to the default download queue for each link.
i.e. verify using Status > Queues

To assign traffic to low, medium, high queues need to tag the packets earlier on as they enter firewall using LAN rules. Pretty much how you would do for a single WAN but instead of assigning a queue you tag the packets.

Lets assume we have aliases for our lan clients
highpri_hosts, mediumpri_hosts, lowpri_hosts

Create LAN rules to assign priorities based on source ip:

Action: Pass
Interface: LAN
Address Familiy: IPv4
Protocol: Any
Source - single host or alias: lowpri_hosts
Tag: qLow
Gateway: default
Ackqueue / Queue: none / none
Action: Pass
Interface: LAN
Address Familiy: IPv4
Protocol: Any
Source - single host or alias: mediumpri_hosts
Tag: qMedium
Gateway: default
Ackqueue / Queue: none / none
Action: Pass
Interface: LAN
Address Familiy: IPv4
Protocol: Any
Source - single host or alias: highpri_hosts
Tag: qHigh
Gateway: default
Ackqueue / Queue: none / none

Thanks - A

deajan

@pki:

Can You explain:

how to do the rules for upload traffic should look? For example priority for VoIP.

how to catch the traffic by LAN ip if possible? For example to put whole traffic from VoIP server on highes priority? I have done this now by the ip of the external server, was not able to catch the traffic by LAN ip. I am setting some outgoing IP (Virtual IP) on the outgoing NAT.

Outgoing rules can be set using the existing qInternetWANx queues as floating rules on interface WANx.
You may also set assign the WAN queues on your LAN interface which IMHO is easier.

Traffic by IP rules can be achieved using source parameter on all rules, just use some aliases for your VoIP servers.

deajan

@allan34 thanks for sharing :)

Ma_Fabulette

Hello.
I'm Frederique and, even if i've been reading your contributions for some time now, I'm a new member on this forum.
First of all, I would like to thank all of you for sharing your experience and tutorials. As always I'm amazed by the generosity ;) I am a recent user of Pfsense solutions and have actually only implemented "out of the box" configurations for the moment. We are now facing several challenges and one of them lead me to your discussion. I stumble upon your message while researching a solution to my client current situation and I would really care for experts advice on this matter.

The curret client architecture is the following

1 LAN which supports data + VoiP
3 WAN on 3 different ISP
1 inside server which needs to synchronize with a distant server. No VPN

Today each WAN is dedicated to 1 usage (Data /VoiP/Replication), 2 of these 3 links are underused and the client wants to use the maximum of the available bandwith. We would like to implement a PFSense configuration with load balancing on all 3 WANs. The problem is that we need to protect VoIP bandwith (in and out) and also leave available bandwitch for the daily ongoing replication of both distant servers. We still need to assign a particular gateway to VoiP and server Synchro (since there is no VPN implemented).

I was wondering if the traffic shaping you're presenting in your post could be implemented with load balancing in odrer to resolve our client's issue ?
I woud really appreciate you advise on this matter before modeling the solution in my lab.
Thanks in advance.

deajan

@Ma_Fabulette: The floating rules described in the post are only matching ones. So basically you could make failover rules on the LAN side using routing groups, as long as you don't specify any queues there.
You might also merge the LAN queues in one if all the WAN lines have the same download capacity, so you can use priority queues easily.

Ma_Fabulette

Thank You for your answer Dejean.

After testing, it seems then that I cannot limit bandwith from the WAN to avoid congestion without limiting drastically the gateway group total bandwith (since I need to shape traffic on the LAN interface)

It seems then that if I want to shape specific traffic I need to have it limited to a specific GTW and eventually create a group wuith the remaining GTW from the rest of the traffic.

deajan

@Ma_Fabulette What exactly are you trying to setup ? Could you make a schema and explain what you're trying to do ? Would make it easier to understand.

ndemou

Very well done how-to deajan. Thank you. Have you tested what happens when 2 LAN clients eventually end up downloading at full speed from the same WAN? Is the BW of that WAN shared evenly between the 2 or does one get to have a huge chunk and one starves? I'm using limiters to achieve fair sharing of BW on my LAN and I'm VERY SATISFIED[1] but I'm not sure if limiters and queues can be combined [2] and my health bar is low for the moment[3]

NOTES:
[1] I'm using limiters to based on foxale08's how-to found here from https://forum.pfsense.org/index.php?topic=63531.msg364520#msg364520 and an excellent explanation of limiters by reddit user drakontas https://www.reddit.com/r/PFSENSE/comments/3e67dk/flexible_vs_fixed_limiters_troubleshooting_with/

[2] This question came up before in the forums but it was on a more complex setup and there is no answer https://forum.pfsense.org/index.php?topic=88627.0

[3]I've spend dozens of weeks reading, experimenting and learning traffic shaping first on IPfire then (when I've hit its limits) on pfsense. I need some time to recover and my co-workers need a few weeks of NO-EXPERIMENTS-DURING-WORK-HOURS :-)

deajan

AFAIK, you'll depend on the bandwidth share algorithm of the HFSC scheduler. If you want totally fair bandwidth sharing, CODELQ / FAIRQ are good alternatives but I'm not sure they might be implemented toghether with HFSC as of new pfSense releases. And you'll have to stick with HFSC in order to have sub queues on LAN lines.

Maybe an explanation of a scheduler expert might fit better here than mine. @pfSense community: someone ? :)

tho

Hello,

I am trying to make my shaper working. I have only one WAN and one LAN (simple case :)), I would like to limit HTTP download and reserve bandwidth for VOIP, RDP and PCOIP. I followed approximately the howto, but it seems that download traffic is stuck in default download queue (except for voip, I don't understand why).

In the howto it is written

Action: Match

Interface: WANx where x is the WAN number

Direction: out (yes, it is outgoing direction !)

Address Familiy: IPv4 and IPv6

Protocol: Any

Gateway: default

Ackqueue / Queue: none / qDownloadLowWANx

Why for download the direction is out from the WAN ?

In my floating rules I set out on WAN interface for upload (and it seems to work) and out from LAN interface for download.

Another question : If a connection (for example HTTP) is established by a user and used to download, will TCP packets be queued in download or upload queue ?

So I'm quite lost about these traffic directions, and how I must write my floating rules to match traffic. You can find attached my floating rules and queues.

Thank in advance for you help.

queue_ts.png_thumb

traffic_shaper.png_thumb

deajan

@tho: I don't see any HTTP rules, so it goes to the default queue.
I've setup a full system for hotels where I used squid in order to limit http downloads too.

btw: Je viens de voir que tu as une règle "serveur tse", donc j'imagine pas me tromper en te parlant fr. Si tu veux j'ai écrit ma doc en FR à la base si ca peut t'aider, contacte moi par mail direct si tu veux :)

pollacomax

First of all, thanks a lot Deajan, the way and the time you take to write this post is to thank.
I have a problem on the upload, if i didnt misunderstood this shape limit the upload of the wans:

Go to Firewall > Traffic Shaper
Remove any traffic shaper queues if some are configured.
For every WAN interface listed in the Traffic Shaper:

Click on "Enable/disable discipline and its children"

Keep the HFSC scheduler as HFSC is the only scheduler allowing children queues without any errors in pfSense 2.3-2.3.2 so far. Also, mixing different schedulers isn't working yet on pfSense. So even if you don't need any special subqueues on WAN links, you'll still need them on the LAN interface later.

The bandwidth parameter to set here is 95% of the measured upload speed:
WAN1 = 9.8x0.95 = 9.3Mb
WAN2 = 920x0.9 = 828Kb (we use a lower multiplier because the line isn't stable)
WAN3 = 3.8x0.95 = 3.6Mb

Queue Limit and TBR Size are left empty unless you know exactly what you're doing

Click on Save

Configuring the bandwidth parameter here is sufficient to enforce the upload speed of pfSense to the WAN modems.

The others shapes works fine, the download are limit, but not the upload.
The only floating rules necessary are the download ones, right?

Does it have anything to do with the version of the pfsense?

Thanks in advance!

tho

@deajan:

@tho: I don't see any HTTP rules, so it goes to the default queue.
I've setup a full system for hotels where I used squid in order to limit http downloads too.

Thank you for replying, the first rule sould match with HTTP and send it to DownloadLow queue, not the default LAN queue qLink. Am I right ?

klou

@allen34

Do you think Policy-based routing would solve the issue of Multi-WAN/Multi-LAN?

Assuming that we have rules on each LAN interface tagging the traffic types, they can then be classified into outgoing queues on the WAN(s) side via floating rules.