Multi Wan 95% percentile bandwidth limiter



  • Hi,

    We have following setup and would like to best utilize our bandwidth with load balancing, fail-over and bandwidth limiting

    LAN - 1Gbps

    1. WAN 1 - 20Mbps
    2. WAN 2 - 10Mbps burstable to 100Mbps (monthly 95 percentile usage)

    How should we configure the limiters so that we can best utilize the bandwidth?



  • There is no one size fits all. Depends on what types of traffic you use and when you use them.



  • I have found no direct solution to shaping dual-wan.  If I use parent CBQ shapers, or perhaps HFSC, uplink bandwidth can be controlled to avoid buffer bloat at the modem without a limiter. This is because the CBQ parent is assigned by interface and a limiter would sum both Wans requiring substantial detuning.

    As for download most modems buffer bloat badly here too unless the Wan's are limited just below their provisioned rate. However, since the Wan's are summed together the best I've found is to set the limiter cap bandwidth about 70% of the sum of both modems.  Quite underutilized.  Anything higher and the load imbalance can allow one Wan to exceed the modem provision causing a seasaw affect between Wan's load with temporary spikes in latency as high as 300ms. I can find no way to first limit each Wan inbound separately and then carry them forward to a catch all load-share rule. A single Wan was so much easier to shape/limit.

    I hoped I could tag all packets inbound on Wan1 with "wan1" and all on Wan2 with "wan2" then process those tagged packets with two Lan rules that bandwidth limit followed by the load-balance rule not having a limiter associated with it. However it seems tagged packets don't migrate the Nat. They only work within a series of rules on the same adapter.

    Someone correct me if I'm wrong on this.



  • I'm seeing the same problem, i have 4 WAN interfaces and 3 of them suffer greatly from buffer bloat. as soon as a client tries to go full throtle on a connection the latency spikes to 400/600/900ms and pfSense marks the interface as down. I've been trying to find a way to limit those interfaces to just below their rated max (and since they are all asymetric connections i need to cap upload and download separately).
    Any ideas?



  • @Raiker:

    I'm seeing the same problem, i have 4 WAN interfaces and 3 of them suffer greatly from buffer bloat. as soon as a client tries to go full throtle on a connection the latency spikes to 400/600/900ms and pfSense marks the interface as down. I've been trying to find a way to limit those interfaces to just below their rated max (and since they are all asymetric connections i need to cap upload and download separately).
    Any ideas?

    Simplest way is just to select CoDell as the shaper and assign the bandwidth on each interface, done. Less than a minute to change all 4.



  • From what i've read the shapers/limiters only work on one way, and the other way requires a shaper on the LAN interface. That's where everything breaks down since LAN interface has the aggregate traffic of all 4 wan links with their wildly different upload/download capacities… I have just lowered the bandwith to a bit less than 90% on each wan link to see if it works, but at 95% the interfaces kept going offline due to high latency.



  • I don't have a multi-WAN in order to test this out, but you could try creating 4 queues in HFSC on your LAN, one for each WAN, then on your WAN, create an out-match-rule that assigns to the proper LAN queue? Then it's just a matter of setting the upperlimits on each of those HFSC queues, and of course mark each to use the CoDel sub-discipline.



  • @Harvy66:

    I don't have a multi-WAN in order to test this out, but you could try creating 4 queues in HFSC on your LAN, one for each WAN, then on your WAN, create an out-match-rule that assigns to the proper LAN queue? Then it's just a matter of setting the upperlimits on each of those HFSC queues, and of course mark each to use the CoDel sub-discipline.

    That's where i get lost. The assigning of rules via the firewall is something for which i haven't been able to find a good tutorial, also, you speak of HFSC first and then CoDelq. I admit i've invested less than 10 hours in reading the help and wikis but it should be easier to find this info. MultiWan setups are not that rare and what i'm after is basic interface limiting which, again, should be pretty common…



  • Just create a floating rule. Make sure it's the last in the list because floating are processed "last wins". Of course you will need to create a named queue for each interface. This means each WAN will have at least 1 queue and the LAN will have at least one queue for each WAN. In case it's not obvious, you also need a firewall rule per queue because each rule can only match to one queue, and you will have 4.

    I said HFSC for the LAN because CoDel cannot shape a queue, only an interface. You need HFSC if you want to shape a queue. On the WANs, shaping the interface is fine because you only have one queue, but the LAN will have 4, so you need HFSC.

    I'm not even sure if this will work because I don't have multiple WANs and I'm not about to create a whole bunch of VLANs to test it out.

    P.S. don't assign qACK, that was force of habit.

    ![Floating Example.png](/public/imported_attachments/1/Floating Example.png)
    ![Floating Example.png_thumb](/public/imported_attachments/1/Floating Example.png_thumb)



  • Harvy, that's so simple yet brilliant!  To get stability and performance back, from load-balance a couple weeks ago, I finally had to switch back to independent Wans and parce out a share of private subnets to one and the remainder to the other as best I could balance them.  Still doesn't fully utilize the bandwidth of each Wan.

    HFSC may be the best option for Lan queues but I get no bloat and good results on the Wans uplink with CBQ. So CBQ may work fine too on the Lan so long as RED items are unchecked like borrowing, sharing, etc.  I don't think you'd want any borrowing going on or packets could slip out the wrong Wan adapter and not return the internet, would it not?

    I may have to give this a go the next long weekend.
    Thanks again for your suggestion.



  • Let us know if it actually works. Have fun!



  • I figured I could try shaping the downlink, as Harvy suggested, even tho I'm not currently load-balancing the Wan's. Ran into a couple snags. I added a Lan queue with two child queues for Wan1 & Wan2, see attached. Then I addded a couple of floating Wan match out rules each to one of the Lan queues.

    Issues;
    PfSense warns one of the child queues should be default.  If not the Lan queue becomes default by default.  Every time a rule is modified it keeps warning me of this which is annoying. If a default is not selected the Status, Queues doesn't show info for the Lan adapter.  If a child queue is set to default then the Status, Queues works again but all matches are with the default queue and nothing on the other child queue.

    Did I miss something?

    Sample.txt



  • Has any of you guys ever managed to get this setup to work ?

    I have a test setup with 3x WAN ADSL 18M/1M.

    I've setup a CODEL queue on each WAN to limit upload. Works !
    I've setup a HFSC queue on LAN with 3 sub queues to limit download.
    pfSense then complains that there is no default queue and sets the parent queue as default.
    As soon as I setup the floating rules per WAN, direction out, with the sub queues, LAN machines no longer have internet.
    i've retried this multiple times without success.

    Btw, why must the floating rules on WAN be direction out when the queues are supposed to limit download (in) speed ?

    I'd be glad for any advice.



  • The floating rules match a state-pair. If you can at least match the state on the outgoing, then you also get the incoming. At least this has been my experience.

    If you're not sure which WAN interface your traffic is going to go out AT THE TIME THE LAN STATE IS CREATED, then you need to use a floating rule to capture at the time the WAN state is created.

    While you may have rules that will make sure traffic always get assigned to a queue, FreeBSD does not care. It wants a default queue no matter what. Keeps things safe and simple. Especially seeing that 99% of all issues are caused by the end user, you can't trust the end user to do anything right. Defensive programming.

    Not sure about losing the Internet. Can we see your floating rules, and your LAN queues?



  • Thanks Harvy66, I actually messed up my fist tests by using "pass" instead of "match" floating rules.

    I've done two full working test setups, with 2 fiber WANs, and with 3 ADSL WANs, including high priority ICMP queues to keep dpinger happy even on high latency lines.

    I'll probably write a howto shortly about this.
    Anyway, your idea is brilliant and yet simple.

    Thanks a lot.



  • @deajan:

    Thanks Harvy66, I actually messed up my fist tests by using "pass" instead of "match" floating rules.

    I've done two full working test setups, with 2 fiber WANs, and with 3 ADSL WANs, including high priority ICMP queues to keep dpinger happy even on high latency lines.

    I'll probably write a howto shortly about this.
    Anyway, your idea is brilliant and yet simple.

    Thanks a lot.

    Why prioritize pings?
    Usually, pings should be prioritized as standard traffic so that it's latency measurements correspond with regular traffic. If you prioritize pings the measured latency may go unchanged during link-saturation, limiting dpinger's diagnostic usefulness.



  • No need to add ICMP priority on SDSL lines, but those ADSL lines I have must be pretty bad and dpinger sat them offline like every 5 minutes, even when I increased alarm threshold.

    I actually have a probably worse problem:

    The 3 ADSL links I have a are 18Mb/1Mb links. I've limited them to 16Mb/850Kb using traffic shaper.
    When a computer is directly connected on those ADSL lines, I get 19Mb/900Kb on speedtest.net.
    LAN traffic is about 10Mb/600Kb, but still, I get really bad packet loss / high latency when keeping ICMP as standard traffic.

    I'm wondering what's the problem, but in the meantime, I've setup high ICMP priority to be able to keep the gateway online and be able to remotely connect.



  • @deajan:

    No need to add ICMP priority on SDSL lines, but those ADSL lines I have must be pretty bad and dpinger sat them offline like every 5 minutes, even when I increased alarm threshold.

    I actually have a probably worse problem:

    The 3 ADSL links I have a are 18Mb/1Mb links. I've limited them to 16Mb/850Kb using traffic shaper.
    When a computer is directly connected on those ADSL lines, I get 19Mb/900Kb on speedtest.net.
    LAN traffic is about 10Mb/600Kb, but still, I get really bad packet loss / high latency when keeping ICMP as standard traffic.

    I'm wondering what's the problem, but in the meantime, I've setup high ICMP priority to be able to keep the gateway online and be able to remotely connect.

    When you say you have packet loss, are you referring to actual packet loss or dpinger packet loss?

    Do these lines show packet loss at all times or only during link saturation?



  • Talking about dpinger packet loss. Gateway monitors IP are set to google DNS and opendns.
    Happens without link being saturated (max is about 8Mb/600Kb on 18Mb/1Mb links).



  • @deajan:

    Talking about dpinger packet loss. Gateway monitors IP are set to google DNS and opendns.
    Happens without link being saturated (max is about 8Mb/600Kb on 18Mb/1Mb links).

    Hmm… I would not expect prioritization to change anything if packets are randomly dropping regardless of link saturation.



  • Well it does ! Having ICMP traffic with high priority, I don't get packet loss on dpinger even if I probably have a lot of packet loss on the link itself (at least the link does not get shot every minutes or so by dpinger, unless it's completly down).
    Btw, lines themselves don't suffer packet loss.

    It's kind of a mystery for me here. I think of two solutions:
    1. Too much concurrent connections -> line quality drops ?
    2. pfSense related misconfig…

    I'll check this next days again, brain is off for today.
    Thanks anyway.



  • Multi-WAN single LAN HFSC names queue thing seems to be working? It would be cool if there was some way to get some basic statistics. Like all 3 links loaded(download and upload) and maintaining a relatively stable ping.



  • @deajan:

    Well it does ! Having ICMP traffic with high priority, I don't get packet loss on dpinger even if I probably have a lot of packet loss on the link itself (at least the link does not get shot every minutes or so by dpinger, unless it's completly down).
    Btw, lines themselves don't suffer packet loss.

    It's kind of a mystery for me here. I think of two solutions:
    1. Too much concurrent connections -> line quality drops ?
    2. pfSense related misconfig…

    I'll check this next days again, brain is off for today.
    Thanks anyway.

    So you have practically no ATM frame loss but you are constantly dropping IP packets regardless of link utilization?

    I doubt the line cares about the number of connections. Maybe packets per second is your problem?



  • As I am trying to digest this solution, does anyone have a set of screen shots to guide the setup?
    Also, does this solution achieve any improved load balancing across the available WANs or it is simply a method to maximize, with limits, the use of each WAN?
    Thanks,



  • I've wrote a quick tutorial from my multi WAN traffic shaper experience here: https://forum.pfsense.org/index.php?topic=120380
    Any improvements are welcome !

    And hey, thank you Harvy66 for your solution !

    @Nullity: There's still some serious packet loss going on. You thought of maybe too much packets. Is there a rule of thumb for the packet number / bandwidth ?


Log in to reply