[HOWTO] Multi WAN Traffic shaper with bandwidth limits per interface



  • The multi WAN traffic shaper wizard doesn't work with a lot of usecases, especially when having multiple WAN links with different download bandwidths.

    In a discussion at https://forum.pfsense.org/index.php?topic=113586.0, Harvy66 suggested a way to use floating rules to assign download queues for outgoing traffic on WAN links. This is a quite nice solution to the problem, although a bit unintuitive to setup.

    Here's a small tutorial on how to setup basic traffic shaping to limit upload and download speed per WAN link, and assign multiple priority queues.
    All improvements to this tutorial are welcome.
    Tested on pfSense 2.3 - 2.3.2-p1.

    If too much traffic goes through your links, you'll encounter packet loss and bufferbloat.
    You'll probably prefer having that packet loss on your pfSense than on your WAN modems in order to keep some degree of control and have some metrics.
    Generally speaking, if you don't want to overload your WAN links and limit packet loss (on your WAN links only, pfSense will still suffer packet loss if too much traffic is sent), you should never use more than 95% of the available bandwidth of a WAN link.
    This doesn't mean that you should use 9.5M of a 10M link, as 10M is probably more a commercial given value than a real one.
    You'll have to measure the real speed by connecting a "clean" computer directly on your WAN modem / router and go to speedtest.dslreports.com or speedtest.net (or whatever you like).

    In order to setup the traffic shaping, we'll use the following three hypotetical WAN links:

    WAN1 = Fiber: 100M/10M (measured at 94Mb/9.8Mb)
    WAN2 = ADSL: 18M/1M (measured at 15Mb/920Kb), some FEC errors on the line
    WAN3 = SDSL: 4M/4M (measured at 3.9Mb/3.8Mb)

    Go to Firewall > Traffic Shaper
    Remove any traffic shaper queues if some are configured.
    For every WAN interface listed in the Traffic Shaper:

    • Click on "Enable/disable discipline and its children"
    • Keep the HFSC scheduler as HFSC is the only scheduler allowing children queues without any errors in pfSense 2.3-2.3.2 so far. Also, mixing different schedulers isn't working yet on pfSense. So even if you don't need any special subqueues on WAN links, you'll still need them on the LAN interface later.
    • The bandwidth parameter to set here is 95% of the measured upload speed:
      WAN1 = 9.8x0.95 = 9.3Mb
      WAN2 = 920x0.9 = 828Kb (we use a lower multiplier because the line isn't stable)
      WAN3 = 3.8x0.95 = 3.6Mb
    • Queue Limit and TBR Size are left empty unless you know exactly what you're doing
    • Click on Save

    Configuring the bandwidth parameter here is sufficient to enforce the upload speed of pfSense to the WAN modems.

    Now that the upload speed is enforced, we'll need to enforce the download speed.
    Click on the LAN interface listed in the Traffic Shaper:

    • Click on "Enable/disable discipline and its children"
    • keep the HFSC scheduler
    • The bandwidth parameter to set here is the full bandwidth of your LAN. If your pfSense is connected to a gigabit switch, set it to 1000Mb. If not, set it to 100Mb.
    • Queue Limit and TBR Size are left empty unless you know exactly what you're doing
    • click on Save

    We now need to divide the LAN queue into multiple subqueues, one for the LAN link, and one per WAN link.
    Be sure to save the parent queue before creating a subqueue, as the pfSense interface is quite buggy here and let's you create a subqueue even when the parent queue isn't saved, resulting in the lost of both.

    On the LAN queue, click on Add New Queue

    • Name the queue qLink
    • Click on "Enable/disable discipline and its children"
    • keep the HFSC scheduler
    • Set the priority: 4
    • Set this queue as "Default Queue"
    • Activate Codel Active Queue (in order to better deal with bufferfloat)
    • The bandwidth parameter to set here is the full LAN bandwidth minus all the download speeds of the WAN links combined. In our example, we'll have 10000 - 94 - 15 - 3.9 = 887.1 Mb
    • Set Min bandwidth for the queue (real time) m2 value to something like 20Mb to ensure that there's always some minimum bandwidth to access pfSense

    Now create three other subqueues on the LAN queue

    • Name the queues qDownloadWANx where x is the WAN number
    • Click on "Enable/disable discipline and its children"
    • keep the HFSC scheduler
    • Set the priority: 4
    • Activate Random Early Detection and Codel Active Queue (in order to better deal with bufferfloat)
    • The bandwidth parameter to set here is 95% of the measured WAN download speed of each link
      WAN1 = 940.95 = 89.3Mb
      WAN2 = 15
      0.9 = 13.5Mb
      WAN3 = 3.8*0.95 = 3.61Mb
    • Set Max bandwidth for the queue (upper limit) m2 value to the same value as the bandwidth

    If you only need to limit the bandwidth, we're done here.
    If you also want to shape your traffic, you'll have to create some subqueues for every qDownloadWANx queue as following:

    • Name the queues qDownloadHighWANx and qDownloadLowWANx where x is the WAN number
    • Click on "Enable/disable discipline and its children"
    • keep the HFSC scheduler
    • Set the priority: 7 for high, 1 for low
    • Activate Random Early Detection and Codel Active Queue (in order to better deal with bufferfloat)
    • Set the bandwidth parameter as some percentage of the parent queue, example:
      qDownloadHighWANx = 40%
      qDownloadLowWANx = 60%


    Note that the example image has more queues that the one described here for testing purposes.

    We now have some queues ready to shape our traffic.
    In order to let pfSense enforce the download speed of the WAN lines, our qDownload queues must be applied on the WAN links, even if they virtually exist on the LAN interface in the traffic shaper.
    The way to achieve this is using matching floating rules, which basically will only add queues to the traffic.

    Matching floating rules work from most generic to most precise rule, meaning that we should put the "catch all traffic" rules first, and then deal with more specific traffic rules.

    Let's first create our "catch all" queue applying rules. In Firewall > Rules > Floating, create the following rule for each WAN link:

    • Action: Match
    • Interface: WANx where x is the WAN number
    • Direction: out (yes, it is outgoing direction !)
    • Address Familiy: IPv4 and IPv6
    • Protocol: Any
    • Gateway: default
    • Ackqueue / Queue: none / qDownloadWANx where x is the WAN number

    Once applied, you may check on Status > Queues that all the traffic is affected to the corresponding queues.

    Let's now create some more precise rules in order to increase DNS traffic priority and lower SMTP traffic priority. Create the following two rules for each WAN link:

    • Action: Match
    • Interface: WANx
    • Direction: out
    • Address Familiy: IPv4 and IPv6
    • Protocol: TCP/UDP
    • Destination Port Range: DNS
    • Gateway: default
    • Ackqueue / Queue: none / qDownloadHighWANx

    and

    • Action: Match
    • Interface: WANx
    • Direction: out
    • Address Familiy: IPv4 and IPv6
    • Protocol: TCP
    • Destination Port Range: SMTP
    • Gateway: default
    • Ackqueue / Queue: none / qDownloadLowWANx

    Those matching rules should be after the generic "catch all" rules in order to get executed last.


    Rule order is important, from most generic to most precise.
    Note that the example image has more download queues as described here for testing purposes.

    You may check correct queue assignment in Status > Queues.

    Note that you may also create subqueues with different priorities and bandwidth settings in the traffic shaper's WAN interfaces, and create floating rules for upload traffic.



  • We have 3 wans with different speeds and 4 vlans (representing different workgroups).  The models I have found show multi Wan, single lan.  How does multi lan change this?  I assume it will complicate it.



  • If you're dealing with multi LAN, the problem would be that every LAN link is setup to fully use download link speed of the WANs, which means that multiple LANs would overload WAN download capacity.
    What are you expecting from traffic shaper with your multi lan setup ?



  • Since you can't share bandwidth across multiple interfaces, you are left with three options.

    1. Give each LAN segment a fixed amount of bandwidth for each of the WANs. No bandwidth shared, free bandwidth from one interface cannot be used by the others
    2. Configure each LAN segment like the above multi-wan-single-lan shaping, but you won't gain a whole lot
    3. Don't do any LAN shaping.

    1 and 2 will have the same queue setups, just the amount of bandwidth assigned to the queues will be different.



  • Thank you for putting this together. I have 300/7 cable internet, speedtests varies between 300 and 400, and could not get the default wizard to work right when I input 300M as my download speed. Ended up using 1000M for LAN, and changing inside of qLink (687M) and qInternet (313M) from the default wizard settings.

    Going step by step through your tutorial on a single WAN helped me understand how this gets setup along with how floating rules work.



  • Thanks, always nice to have some good feedback :)



  • Hi.
    I have set up the traffic shaper following this howto.

    But, i still have problems:

    • i can not get the traffic into the right queue
    • i can not get the upload traffic to limit

    Short to say, it's not working for me :(

    About my setup. WAN (WAN1) is DSL 10/1, WAN2 is second internet provider 25/5 with PPPoE, WAN3 (opt2) is the same provider and connection as WAN2, second PPPoE channel with a second IP.

    Attaching screenshots. On WAN2 i have permanent about 2Mb of traffic, but only a part is sorted into qDownloadWAN2. The graph on the main page shows two speedtests, i am getting 25/5 result (or above), not limiting :(

    Can anyone help to catch where the problem is?

    The worst problem is on the DSL connection, if overloaded it goes very lattency.














  • @pki incomplete data you gave here. No crystal ball available.



  • I tried to give it all, what should i add??

    What information are You missing?



  • @pki Detail the floating rules



  • I have done the rules and traffic shaper again, from scratch. I think i missed the limit checkbox on the download shapers. Also the rules was not easy for me.

    Can You explain:

    • how to do the rules for upload traffic should look? For example priority for VoIP.
    • how to catch the traffic by LAN ip if possible? For example to put whole traffic from VoIP server on highes priority? I have done this now by the ip of the external server, was not able to catch the traffic by LAN ip. I am setting some outgoing IP (Virtual IP) on the outgoing NAT.

    Thx



  • thanks for this post on multiwan. It gave the inspiration to solve our main problem:

    Multiwan per ip traffic shaping

    Now - I am no FW expert, so please comment if you have better ideas.

    The problem for us using the above approach is that the lan clients ip's are not visible to the floating rules, as this is the post NAT stage of the packet flow. i.e. they all have the same ip of the WAN interface of the fw. Only the dst port and ip are for matciing the rules to.

    Policy based routing to the rescue:

    Use floating rules, but instead assign queues based upon tags (which indicate the priority) and the WAN link:

    First, the default rule remains unchanged, for each WAN link:

    • Action: Match
    • Interface: WANx where x is the WAN number
    • Direction: out (yes, it is outgoing direction !)
    • Address Familiy: IPv4 and IPv6
    • Protocol: Any
    • Gateway: default
    • Ackqueue / Queue: none / qDownloadLowWANx where x is the WAN number  # Default to the lowest priority.

    Now assign queues based on the "tag" of the packets, create rules for each of the wan links:

    • Action: Match

    • Interface: WANx

    • Direction: out

    • Address Familiy: IPv4 and IPv6

    • Protocol: TCP/UDP

    • Destination Port Range: any
      - Tagged : qLow

    • Gateway: default

    • Ackqueue / Queue: none / qDownloadLowWANx

    • Action: Match

    • Interface: WANx

    • Direction: out

    • Address Familiy: IPv4 and IPv6

    • Protocol: TCP/UDP

    • Destination Port Range: any
      - Tagged : qMedium

    • Gateway: default

    • Ackqueue / Queue: none / qDownloadMediumWANx

    • Action: Match

    • Interface: WANx

    • Direction: out

    • Address Familiy: IPv4 and IPv6

    • Protocol: TCP/UDP

    • Destination Port Range: any
      - Tagged : qHigh

    • Gateway: default

    • Ackqueue / Queue: none / qDownloadHighWANx

    Test this, and all traffic should go to the default download queue for each link.
    i.e. verify using Status > Queues

    To assign traffic to low, medium, high queues need to tag the packets earlier on as they enter firewall using LAN rules. Pretty much how you would do for a single WAN but instead of assigning a queue you tag the packets.

    Lets assume we have aliases for our lan clients
    highpri_hosts, mediumpri_hosts, lowpri_hosts

    Create LAN rules to assign priorities based on source ip:

    • Action: Pass

    • Interface: LAN

    • Address Familiy: IPv4

    • Protocol: Any

    • Source - single host or alias: lowpri_hosts

    • Tag: qLow

    • Gateway: default

    • Ackqueue / Queue: none / none

    • Action: Pass

    • Interface: LAN

    • Address Familiy: IPv4

    • Protocol: Any

    • Source - single host or alias: mediumpri_hosts

    • Tag: qMedium

    • Gateway: default

    • Ackqueue / Queue: none / none

    • Action: Pass

    • Interface: LAN

    • Address Familiy: IPv4

    • Protocol: Any

    • Source - single host or alias: highpri_hosts

    • Tag: qHigh

    • Gateway: default

    • Ackqueue / Queue: none / none

    Thanks - A



  • @pki:

    Can You explain:

    • how to do the rules for upload traffic should look? For example priority for VoIP.
    • how to catch the traffic by LAN ip if possible? For example to put whole traffic from VoIP server on highes priority? I have done this now by the ip of the external server, was not able to catch the traffic by LAN ip. I am setting some outgoing IP (Virtual IP) on the outgoing NAT.

    Outgoing rules can be set using the existing qInternetWANx queues as floating rules on interface WANx.
    You may also set assign the WAN queues on your LAN interface which IMHO is easier.

    Traffic by IP rules can be achieved using source parameter on all rules, just use some aliases for your VoIP servers.



  • @allan34 thanks for sharing :)



  • Hello.
    I'm Frederique and, even if i've been reading your contributions for some time now, I'm a new member on this forum.
    First of all, I would like to thank all of you for sharing your experience and tutorials. As always I'm amazed by the generosity ;) I am a recent user of Pfsense solutions and have actually only implemented "out of the box" configurations for the moment. We are now facing several challenges and one of them lead me to your discussion. I stumble upon your message while researching a solution to my client current situation and I would really care for experts advice on this matter.

    The curret client architecture is the following

    • 1 LAN which supports data + VoiP
    • 3 WAN on 3 different ISP
    • 1 inside server which needs to synchronize with a distant server. No VPN

    Today each WAN is dedicated to 1 usage (Data /VoiP/Replication), 2 of these 3 links are underused and the client wants to use the maximum of the available bandwith. We would like to implement a PFSense configuration with load balancing on all 3 WANs. The problem is that we need to protect VoIP bandwith (in and out) and also leave available bandwitch for the daily ongoing replication of both distant servers. We still need to assign a particular gateway to VoiP and server Synchro (since there is no VPN implemented).

    I was wondering if the traffic shaping you're presenting in your post could be implemented with load balancing in odrer to resolve our client's issue ?
    I woud really appreciate you advise on this matter before modeling the solution in my lab.
    Thanks in advance.



  • @Ma_Fabulette: The floating rules described in the post are only matching ones. So basically you could make failover rules on the LAN side using routing groups, as long as you don't specify any queues there.
    You might also merge the LAN queues in one if all the WAN lines have the same download capacity, so you can use priority queues easily.



  • Thank You for your answer Dejean.

    After testing, it seems then that I cannot limit bandwith from the WAN to avoid congestion without limiting drastically the gateway group total bandwith (since I need to shape traffic on the LAN interface)

    It seems then that if I want to shape specific traffic I need to have it limited to a specific GTW and eventually create a group wuith the remaining GTW from the rest of the traffic.



  • @Ma_Fabulette What exactly are you trying to setup ? Could you make a schema and explain what you're trying to do ? Would make it easier to understand.



  • Very well done how-to deajan. Thank you. Have you tested what happens when 2 LAN clients eventually end up downloading at full speed from the same WAN? Is the BW of that WAN shared evenly between the 2 or does one get to have a huge chunk and one starves? I'm using limiters to achieve fair sharing of BW on my LAN and I'm VERY SATISFIED[1] but I'm not sure if  limiters and queues can be combined [2] and my health bar is low for the moment[3]

    NOTES:
    [1] I'm using limiters to based on foxale08's how-to found here from https://forum.pfsense.org/index.php?topic=63531.msg364520#msg364520 and an excellent explanation of limiters by reddit user drakontas https://www.reddit.com/r/PFSENSE/comments/3e67dk/flexible_vs_fixed_limiters_troubleshooting_with/

    [2] This question came up before in the forums but it was on a more complex setup and there is no answer https://forum.pfsense.org/index.php?topic=88627.0

    [3]I've spend dozens of weeks reading, experimenting and learning traffic shaping first on IPfire then (when I've hit its limits) on pfsense. I need some time to recover and my co-workers need a few weeks of NO-EXPERIMENTS-DURING-WORK-HOURS :-)



  • AFAIK, you'll depend on the bandwidth share algorithm of the HFSC scheduler. If you want totally fair bandwidth sharing, CODELQ / FAIRQ are good alternatives but I'm not sure they might be implemented toghether with HFSC as of new pfSense releases. And you'll have to stick with HFSC in order to have sub queues on LAN lines.

    Maybe an explanation of a scheduler expert might fit better here than mine. @pfSense community: someone ? :)



  • Hello,

    I am trying to make my shaper working. I have only one WAN and one LAN (simple case :)), I would like to limit HTTP download and reserve bandwidth for VOIP, RDP and PCOIP. I followed approximately the howto, but it seems that download traffic is stuck in default download queue (except for voip, I don't understand why).

    In the howto it is written

    • Action: Match
    • Interface: WANx where x is the WAN number
    • Direction: out (yes, it is outgoing direction !)
    • Address Familiy: IPv4 and IPv6
    • Protocol: Any
    • Gateway: default
    • Ackqueue / Queue: none / qDownloadLowWANx

    Why for download the direction is out from the WAN ?

    In my floating rules I set out on WAN interface for upload (and it seems to work) and out from LAN interface for download.

    Another question : If a connection (for example HTTP) is established by a user and used to download, will TCP packets be queued in download or upload queue ?

    So I'm quite lost about these traffic directions, and how I must write my floating rules to match traffic. You can find attached my floating rules and queues.

    Thank in advance for you help.






  • @tho: I don't see any HTTP rules, so it goes to the default queue.
    I've setup a full system for hotels where I used squid in order to limit http downloads too.

    btw: Je viens de voir que tu as une règle "serveur tse", donc j'imagine pas me tromper en te parlant fr. Si tu veux j'ai écrit ma doc en FR à la base si ca peut t'aider, contacte moi par mail direct si tu veux :)



  • First of all, thanks a lot Deajan, the way and the time you take to write this post is to thank.
    I have a problem on the upload, if i didnt misunderstood this shape limit the upload of the wans:

    Go to Firewall > Traffic Shaper
    Remove any traffic shaper queues if some are configured.
    For every WAN interface listed in the Traffic Shaper:

    • Click on "Enable/disable discipline and its children"
    • Keep the HFSC scheduler as HFSC is the only scheduler allowing children queues without any errors in pfSense 2.3-2.3.2 so far. Also, mixing different schedulers isn't working yet on pfSense. So even if you don't need any special subqueues on WAN links, you'll still need them on the LAN interface later.
    • The bandwidth parameter to set here is 95% of the measured upload speed:
        WAN1 = 9.8x0.95 = 9.3Mb
        WAN2 = 920x0.9 = 828Kb (we use a lower multiplier because the line isn't stable)
        WAN3 = 3.8x0.95 = 3.6Mb
    • Queue Limit and TBR Size are left empty unless you know exactly what you're doing
    • Click on Save

    Configuring the bandwidth parameter here is sufficient to enforce the upload speed of pfSense to the WAN modems.

    The others shapes works fine, the download are limit, but not the upload.
    The only floating rules necessary are the download ones, right?

    Does it have anything to do with the version of the pfsense?

    Thanks in advance!



  • @deajan:

    @tho: I don't see any HTTP rules, so it goes to the default queue.
    I've setup a full system for hotels where I used squid in order to limit http downloads too.

    Thank you for replying, the first rule sould match with HTTP and send it to DownloadLow queue, not the default LAN queue qLink. Am I right ?



  • @allen34

    Do you think Policy-based routing would solve the issue of Multi-WAN/Multi-LAN?

    Assuming that we have rules on each LAN interface tagging the traffic types, they can then be classified into outgoing queues on the WAN(s) side via floating rules.



  • @deajan:

    @Ma_Fabulette: The floating rules described in the post are only matching ones. So basically you could make failover rules on the LAN side using routing groups, as long as you don't specify any queues there.
    You might also merge the LAN queues in one if all the WAN lines have the same download capacity, so you can use priority queues easily.

    How do you setup in LAN rules of firewall? Some screenshot might be help.

    Currently i'm using grouping my two WANs that i setup in system->gateway groups and name it as LoadBalancing. I use it in LAN rules as gateway.

    Thanks. Your configuration makes me want to learn this.



  • @klou:

    @allen34

    Do you think Policy-based routing would solve the issue of Multi-WAN/Multi-LAN?

    Assuming that we have rules on each LAN interface tagging the traffic types, they can then be classified into outgoing queues on the WAN(s) side via floating rules.

    Apologies for the late reply. Unfortunately have no experience of multiwan and multi lan.

    The approach outlined in my post works only because the WAN incoming traffic all ends up in the same queue on the same LAN interface.

    Maybe there are possibilities
    a) Assign multiple WANS to to each LAN
    i.e. 2 wans for LAN1 and 2 separate wans for LAN2

    b) Split each WAN into equal amounts for each LAN
    e.g. if you have  2x WAN links and 2x LAN, then split bw of each WAN in half and assign a half from each WAN to each LAN.

    But I do not see a way of balancing all LAN traffic across all WANS. This is because you will have separate queues for each LAN.

    In the end, buy more WAN links and divvy your users up across them. Pretty sure that is what sensible people do and why I believe not so many posts about this problem is found. Simply have a "WAN budget" per employee, so if you get 100 users you pay N $$, and if you have 200 users you pay 2x N $$. Unfortunately we are in a remote location and this is not possible so we try to squeeze as much as we can out of the 2x DSL lines we have and pay a small fortune for the privilege where others buy 10x the bandwidth at 1/4 the price.

    Cheers A



  • HI,
    i'm on version 2.3.4 and upload limit seems not to work( https://forum.pfsense.org/index.php?topic=145500.0).
    Also after create the qLink and the qDownloadWANX queues the tutorial say

    If you only need to limit the bandwidth, we're done here.

    to apply them i needed to create the rules on firewall/floating as described below, don't know if pfsense behavior changed or i misread the instruction ( to me is sounded like "If you only needed limiters you are done" ).



  • how create queue for upload? I tried creating queue on wan interface like on lan interface and apply it to floating rules, direction is IN but not working.. thanks


Log in to reply