This is how i used limiters to fairly share bandwidth among all devices on LAN
-
Hello,
This is something i always wanted to do, but never found clear documentation about: using Limiters to fairly share bandwidth among all devices on LAN. Meaning: if one device is downloading, it gets all the bandwidth, if two devices are downloading, they receive half of the bandwidth each, and so on. I recently managed to configure it, so this is what i learned and found:
The Limiters in pfSense can have two levels. As you can see in the UI, when you create a Limiter, this first level is called a pipe. If you go down on the page, you will find a "add new queue" button. Queues are the second level. If you create a Queue, you can clearly see how the Queue is nested under the Limiter (pipe), as it's graphically shown in the UI (on the top left), but probably only after you save the Queue. For both Pipes and Queues, you can use Masks to create them dynamically by source or destination IP. We will see this in detail below.
All Queues under a Limiter (pipe) fairly share the Limiter (pipe) bandwidth by default. This is what does the trick.
What you need to do to fairly share bandwidth among all devices on LAN is this:
- Create a DownloadLimiter and a UploadLimiter. Set the bandwidth of the limiters to around 95% of the real download/upload bandwith of the Internet connection. Hit Save.
- Edit the DownloadLimiter, and hit "add new queue". You can name it something like "DownloadQueue", and choose to Mask by "Destination Addresses" with IPv4 mask bits of 32, and IPv6 mask bits of 128. This means that a Queue is created for each IP of the Destination address in the rule the Queue is applied on (more on this later). Hit Save, and Save again on the DownloadLimiter.
- Edit the UploadLimiter, and hit "add new queue". You can name it something like "UploadQueue", and choose to Mask by "Source Addresses" with IPv4 mask bits of 32, and IPv6 mask bits of 128. This means that a Queue is created for each IP of the Source address in the rule the Queue is applied on (more on this later). Hit Save, and Save again on the UploadLimiter.
- Now go to the firewall Rules, click LAN, and you probably have one rule that is the one that allows for Internet access. You must make sure that the rule ONLY applies to traffic going to the Internet (obviously, you don't want to send traffic internal to your network thru the limiters). If you only have a simple one LAN and WAN configuration, it's probably enough for the rule to be something like "Action: Pass, Source: LAN Subnets, Destination: (invert match) LAN Subnets". Then hit "Show Advanced", go to "In / Out pipe", and choose "UploadQueue" for In, and "DownloadQueue" for Out. Hit Save.
NOTE: even if the UI says In / Out pipe, you can choose either Limiters (pipes) or Queues here. In our case we must choose the Queues here, not the Limiters (pipes). Also, pfSense chose to call them "In / Out" but they are really the "Forward / Reverse" pipes or queues. The first one applies to the traffic going in the same direction than the rule (the normal or Forward direction), and the second one applies to the response traffic that comes back from whatever the forward packets were going to. So, calling them "In / Out" was intended as a simplification, but it kind of hides their real function and can be quite confusing if you need to use floating rules, for reasons i won't explain here to avoid making things more complex.
The default Scheduler for the Limiter (pipe) is the one called "Worst-case Weighted fair Queueing". It's not the best of them, but it's a good idea to use it initially, because it allows you to go to "Diagnostics" -> "Limiter info" and watch how a UploadQueue and a DownloadQueue is created for each IP address that is actively uploading and/or downloading data to/from the Internet, and see how many packets, bytes and drops corresponds to each IP.
When you are sure everything is working fine, you can edit both the UploadLimiter and the DownloadLimiter and choose the FQ_CODEL Scheduler. This scheduler is great to combat buffer bloat, but sadly it won't show you almost any detail at all in "Diagnostics" -> "Limiter info". The Scheduler is like that, but it is working properly even if it doesn't shows the Queues by IP there. You can use something like speedtest.net in two computers at the same time, and both should get roughly half of the bandwidth (assuming nothing else is using bandwidth).
WARNING: some limiters won't even work with Queues on them. I once chosen one that resulted on pfSense to consider that the configuration was wrong/corrupt and it automatically loaded the last configuration and restarted.
If you only use pfSense to access the Internet, and you don't have any open ports. That's probably all the configuration you need to pull this off. One rule is enough, because you are initiating all connections. Even when you want to download something from the internet, you are always the one initiating the connection, and the download data is all response traffic for the connection you initiated. That's why a lone rule can handle upload and download. The download data automatically goes to your DownloadQueue, as configured in your rule. Of course, for an Upload it's even more straightforward: you initiated the connection and the upload data is coming from you, so that always goes to the UploadQueue.
Well, i think that's all. I hope this is heplful.
Regards -
@fsr Do you observe a tangible difference in behavior when you use masks on the child queues while having FQ_CODEL configured as the scheduler for your limiters? I'm not an expert, but my understanding is that FQ_CODEL should provide fairness with respect to bandwidth sharing among flows (FQ == fair queuing) without needing masks on your child queues to get one queue per host. But if you've gathered experimental evidence to suggest a significant difference, it would certainly be of interest.
-
@TheNarc Sorry, i have gathered no evidence, but i remember following this recipe quite some time ago: https://docs.netgate.com/pfsense/en/latest/recipes/codel-limiters.html , and it was not that good for my use case, to be honest, especially because we use OpenVPN for remote access to pfSense, and back in the day it wasn't that clear to me, but that recipe doesn't really do anything about connections initiated from the Internet. The recipe will be useful to keep latency low and prioritize small packets over big packets, however.
Also, any rules applied on WAN won't know the internal LAN addresses at all. For rules in the IN direction, they will only see Source = Some IP on the Internet, Dest = WAN adresss. For rules in the OUT direction, they will only see Source = WAN adresss, Dest = Some IP on the Internet. So, there is no way fq_codel could use the LAN IP at all on it's queuing.
So, the rules should be applied on LAN for fq_codel to have any chance to perform any kind of good enough fairness by LAN IP.
But i understand that fairly sharing the bandwidth among LAN IP is NOT really the purpose of this scheduler. According to: https://www.rfc-editor.org/rfc/rfc8290.html
++++++++++++++++++++
6.1. Fairness between Things Other Than FlowsIn some parts of the network, enforcing flow-level fairness may not
be desirable, or some other form of fairness may be more important.
Some examples of this include an ISP that may be more interested in
ensuring fairness between customers than between flows
++++++++++++++++++++And as i was already experimenting before with Limiters with Masks, i already had a Limiter implemented with Masked child Queues by the time i decided to switch the scheduler to fq_codel. But maybe the Masked Queues aren't really necessary when using fq_codel, or at least not critical to have, or it's even possible they are being ignored with the fq_codel scheduler (no way to see almost any detail in Diagnostics -> Limiter Info, so there doesn't seems to be any way to tell what it's really doing). I know that if i start two speedtest.net tests on two PCs at the same time to the same server, they get roughly 50% bandwidth each, however.
-
@fsr That's a good point about rules applied on the WAN interfaces not knowing about the LAN addresses, which I had not thought of. On the other hand, as you point out, fq_codel provides flow-level fairness, not host (LAN IP level) fairness. So I'm not sure how much that matters? But also because of that, it does seem that your point about achieving that level of fairness - between individual LAN hosts - is valid with respect to enabling masks on the limiter queues.
I'm definitely intrigued, and will do some testing of my own as well. I feel that every time I think I have something approaching a passable understanding of the traffic shaping nuances, I find I'm not quite there :)
-
So in my testing, switching my floating rules to apply the limiters on LAN interface traffic instead of WAN interface traffic completely eliminated my bufferbloat mitigation. I want from an A+ on waveform's test to a B or C. But it's interesting if you're seeing good bufferbloat mitigation with rules applied on your LAN interface.
Now the only other difference is that I was applying the limiters using floating rules with a Match action, because that's how I learned to do it and also because I couldn't have rule processing stop at that point since I have more complex policy routing rules in my preexisting LAN pass rules. But I still can't see an obvious reason why it would be substantively different from what you're doing.
-
@fsr said in This is how i used limiters to fairly share bandwidth among all devices on LAN:
rules applied on WAN won't know the internal LAN addresses at all
If I followed, this can be done for floating rules with tagging:
https://docs.netgate.com/pfsense/en/latest/firewall/floating-rules.html#marking-and-matching
so, tag=VOIP, match on tag=VOIPBut if what you have is working fine, don't break it. ;)
-
Last night i made some tests on traffic of the two WANs i have. WAN20 is a 20 Mbps dedicated connection, while WAN600 is a 600 Mbps symmetric, but not dedicated.
WAN20 has the limiters i described before (on LAN), while WAN600 has the limiters applied like the following recipe https://docs.netgate.com/pfsense/en/latest/recipes/codel-limiters.html
The test was two PCs performing a speedtest.net simultaneously against the same server.
WAN20 results
PC1
PC2
0.84% difference in download between PCs, and 4.79% difference in upload
Limiter info showed this (for the curious that at the same time can decipher that wall of numbers):
NOTE: The limiters have different speeds to be able to recognize them:
19005 Kbps for the Download Limiter on WAN20
19006 Kbps for the Upload Limiter on WAN20
570 Mbps for the Download Limiter on WAN600
571 Mbps for the Upload Limiter on WAN600TWO SIMULTANEOUS SPEEDTESTS (DOWNLOAD PHASE) Limiter Information Limiters: 00003: 19.006 Mbit/s 0 ms burst 0 q131075 50 sl. 0 flows (1 buckets) sched 65539 weight 0 lmax 0 pri 0 droptail sched 65539 type FIFO flags 0x0 0 buckets 0 active 00004: 19.005 Mbit/s 0 ms burst 0 q131076 50 sl. 0 flows (1 buckets) sched 65540 weight 0 lmax 0 pri 0 droptail sched 65540 type FIFO flags 0x0 0 buckets 0 active 00005: 570.000 Mbit/s 0 ms burst 0 q131077 50 sl. 0 flows (1 buckets) sched 65541 weight 0 lmax 0 pri 0 droptail sched 65541 type FIFO flags 0x0 0 buckets 0 active 00006: 571.000 Mbit/s 0 ms burst 0 q131078 50 sl. 0 flows (1 buckets) sched 65542 weight 0 lmax 0 pri 0 droptail sched 65542 type FIFO flags 0x0 0 buckets 0 active Schedulers: 00003: 19.006 Mbit/s 0 ms burst 0 q00003 50 sl. 0 flows (256 buckets) sched 4 weight 0 lmax 0 pri 0 droptail mask: 0x00 0x00000000/0x0000 -> 0xffffffff/0x0000 sched 3 type FQ_CODEL flags 0x0 0 buckets 1 active FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN Children flowsets: 4 BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp 0 ip 0.0.0.0/0 0.0.0.0/0 40 2495 0 0 0 00004: 19.005 Mbit/s 0 ms burst 0 q65540 50 sl. 0 flows (1 buckets) sched 4 weight 0 lmax 0 pri 0 droptail sched 4 type FQ_CODEL flags 0x0 0 buckets 1 active FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN Children flowsets: 3 0 ip 0.0.0.0/0 0.0.0.0/0 11472 13588082 33 48083 205 00005: 570.000 Mbit/s 0 ms burst 0 q65541 50 sl. 0 flows (1 buckets) sched 5 weight 0 lmax 0 pri 0 droptail sched 5 type FQ_CODEL flags 0x0 0 buckets 1 active FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN Children flowsets: 1 0 ip 0.0.0.0/0 0.0.0.0/0 2 184 0 0 0 00006: 571.000 Mbit/s 0 ms burst 0 q65542 50 sl. 0 flows (1 buckets) sched 6 weight 0 lmax 0 pri 0 droptail sched 6 type FQ_CODEL flags 0x0 0 buckets 0 active FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN Children flowsets: 2 Queues: q00001 50 sl. 0 flows (1 buckets) sched 5 weight 0 lmax 0 pri 0 droptail q00002 50 sl. 0 flows (1 buckets) sched 6 weight 0 lmax 0 pri 0 droptail q00003 50 sl. 0 flows (256 buckets) sched 4 weight 0 lmax 0 pri 0 droptail mask: 0x00 0x00000000/0x0000 -> 0xffffffff/0x0000 q00004 50 sl. 0 flows (256 buckets) sched 3 weight 0 lmax 0 pri 0 droptail mask: 0x00 0xffffffff/0x0000 -> 0x00000000/0x0000 -------------------------------- TWO SIMULTANEOUS SPEEDTESTS (UPLOAD PHASE) Limiter Information Limiters: 00003: 19.006 Mbit/s 0 ms burst 0 q131075 50 sl. 0 flows (1 buckets) sched 65539 weight 0 lmax 0 pri 0 droptail sched 65539 type FIFO flags 0x0 0 buckets 0 active 00004: 19.005 Mbit/s 0 ms burst 0 q131076 50 sl. 0 flows (1 buckets) sched 65540 weight 0 lmax 0 pri 0 droptail sched 65540 type FIFO flags 0x0 0 buckets 0 active 00005: 570.000 Mbit/s 0 ms burst 0 q131077 50 sl. 0 flows (1 buckets) sched 65541 weight 0 lmax 0 pri 0 droptail sched 65541 type FIFO flags 0x0 0 buckets 0 active 00006: 571.000 Mbit/s 0 ms burst 0 q131078 50 sl. 0 flows (1 buckets) sched 65542 weight 0 lmax 0 pri 0 droptail sched 65542 type FIFO flags 0x0 0 buckets 0 active Schedulers: 00003: 19.006 Mbit/s 0 ms burst 0 q00003 50 sl. 0 flows (256 buckets) sched 4 weight 0 lmax 0 pri 0 droptail mask: 0x00 0x00000000/0x0000 -> 0xffffffff/0x0000 sched 3 type FQ_CODEL flags 0x0 0 buckets 1 active FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN Children flowsets: 4 BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp 0 ip 0.0.0.0/0 0.0.0.0/0 14732 18775095 37 55500 1255 00004: 19.005 Mbit/s 0 ms burst 0 q65540 50 sl. 0 flows (1 buckets) sched 4 weight 0 lmax 0 pri 0 droptail sched 4 type FQ_CODEL flags 0x0 0 buckets 1 active FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN Children flowsets: 3 0 ip 0.0.0.0/0 0.0.0.0/0 745 49087 0 0 0 00005: 570.000 Mbit/s 0 ms burst 0 q65541 50 sl. 0 flows (1 buckets) sched 5 weight 0 lmax 0 pri 0 droptail sched 5 type FQ_CODEL flags 0x0 0 buckets 1 active FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN Children flowsets: 1 0 ip 0.0.0.0/0 0.0.0.0/0 43 4197 0 0 0 00006: 571.000 Mbit/s 0 ms burst 0 q65542 50 sl. 0 flows (1 buckets) sched 6 weight 0 lmax 0 pri 0 droptail sched 6 type FQ_CODEL flags 0x0 0 buckets 1 active FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN Children flowsets: 2 0 ip 0.0.0.0/0 0.0.0.0/0 48 13792 0 0 0 Queues: q00001 50 sl. 0 flows (1 buckets) sched 5 weight 0 lmax 0 pri 0 droptail q00002 50 sl. 0 flows (1 buckets) sched 6 weight 0 lmax 0 pri 0 droptail q00003 50 sl. 0 flows (256 buckets) sched 4 weight 0 lmax 0 pri 0 droptail mask: 0x00 0x00000000/0x0000 -> 0xffffffff/0x0000 q00004 50 sl. 0 flows (256 buckets) sched 3 weight 0 lmax 0 pri 0 droptail mask: 0x00 0xffffffff/0x0000 -> 0x00000000/0x0000WAN600 results
PC3
PC4
57% difference in download between PCs, and 26% difference in upload
It seems to me, like adding the Masked Queues to the LAN in the WAN20 test introduced "Host Fairness", as the percentages are quite a lot lower in that test.
I also have the Limiter info for the WAN600 test:
NOTE: The limiters have different speeds to be able to recognize them:
19005 Kbps for the Download Limiter on WAN20
19006 Kbps for the Upload Limiter on WAN20
570 Mbps for the Download Limiter on WAN600
571 Mbps for the Upload Limiter on WAN600SPEEDTEST FQ_CODEL EN WAN. CONEXION 600 Mbps TWO SIMULTANEOUS SPEEDTESTS (DOWNLOAD PHASE) Limiters: 00003: 19.006 Mbit/s 0 ms burst 0 q131075 50 sl. 0 flows (1 buckets) sched 65539 weight 0 lmax 0 pri 0 droptail sched 65539 type FIFO flags 0x0 0 buckets 0 active 00004: 19.005 Mbit/s 0 ms burst 0 q131076 50 sl. 0 flows (1 buckets) sched 65540 weight 0 lmax 0 pri 0 droptail sched 65540 type FIFO flags 0x0 0 buckets 0 active 00005: 570.000 Mbit/s 0 ms burst 0 q131077 50 sl. 0 flows (1 buckets) sched 65541 weight 0 lmax 0 pri 0 droptail sched 65541 type FIFO flags 0x0 0 buckets 0 active 00006: 571.000 Mbit/s 0 ms burst 0 q131078 50 sl. 0 flows (1 buckets) sched 65542 weight 0 lmax 0 pri 0 droptail sched 65542 type FIFO flags 0x0 0 buckets 0 active Schedulers: 00003: 19.006 Mbit/s 0 ms burst 0 q00003 50 sl. 0 flows (256 buckets) sched 4 weight 0 lmax 0 pri 0 droptail mask: 0x00 0x00000000/0x0000 -> 0xffffffff/0x0000 sched 3 type FQ_CODEL flags 0x0 0 buckets 1 active FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN Children flowsets: 4 BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp 0 ip 0.0.0.0/0 0.0.0.0/0 33 10054 0 0 0 00004: 19.005 Mbit/s 0 ms burst 0 q65540 50 sl. 0 flows (1 buckets) sched 4 weight 0 lmax 0 pri 0 droptail sched 4 type FQ_CODEL flags 0x0 0 buckets 1 active FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN Children flowsets: 3 0 ip 0.0.0.0/0 0.0.0.0/0 26 1996 0 0 0 00005: 570.000 Mbit/s 0 ms burst 0 q65541 50 sl. 0 flows (1 buckets) sched 5 weight 0 lmax 0 pri 0 droptail sched 5 type FQ_CODEL flags 0x0 0 buckets 1 active FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN Children flowsets: 1 0 ip 0.0.0.0/0 0.0.0.0/0 52664 78273224 295 438731 31 00006: 571.000 Mbit/s 0 ms burst 0 q65542 50 sl. 0 flows (1 buckets) sched 6 weight 0 lmax 0 pri 0 droptail sched 6 type FQ_CODEL flags 0x0 0 buckets 1 active FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN Children flowsets: 2 0 ip 0.0.0.0/0 0.0.0.0/0 10123 499009 0 0 0 Queues: q00001 50 sl. 0 flows (1 buckets) sched 5 weight 0 lmax 0 pri 0 droptail q00002 50 sl. 0 flows (1 buckets) sched 6 weight 0 lmax 0 pri 0 droptail q00003 50 sl. 0 flows (256 buckets) sched 4 weight 0 lmax 0 pri 0 droptail mask: 0x00 0x00000000/0x0000 -> 0xffffffff/0x0000 q00004 50 sl. 0 flows (256 buckets) sched 3 weight 0 lmax 0 pri 0 droptail mask: 0x00 0xffffffff/0x0000 -> 0x00000000/0x0000 --------------------------------------------------------------------------------------------- TWO SIMULTANEOUS SPEEDTESTS (UPLOAD PHASE) Limiters: 00003: 19.006 Mbit/s 0 ms burst 0 q131075 50 sl. 0 flows (1 buckets) sched 65539 weight 0 lmax 0 pri 0 droptail sched 65539 type FIFO flags 0x0 0 buckets 0 active 00004: 19.005 Mbit/s 0 ms burst 0 q131076 50 sl. 0 flows (1 buckets) sched 65540 weight 0 lmax 0 pri 0 droptail sched 65540 type FIFO flags 0x0 0 buckets 0 active 00005: 570.000 Mbit/s 0 ms burst 0 q131077 50 sl. 0 flows (1 buckets) sched 65541 weight 0 lmax 0 pri 0 droptail sched 65541 type FIFO flags 0x0 0 buckets 0 active 00006: 571.000 Mbit/s 0 ms burst 0 q131078 50 sl. 0 flows (1 buckets) sched 65542 weight 0 lmax 0 pri 0 droptail sched 65542 type FIFO flags 0x0 0 buckets 0 active Schedulers: 00003: 19.006 Mbit/s 0 ms burst 0 q00003 50 sl. 0 flows (256 buckets) sched 4 weight 0 lmax 0 pri 0 droptail mask: 0x00 0x00000000/0x0000 -> 0xffffffff/0x0000 sched 3 type FQ_CODEL flags 0x0 0 buckets 1 active FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN Children flowsets: 4 BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp 0 ip 0.0.0.0/0 0.0.0.0/0 39 18229 0 0 0 00004: 19.005 Mbit/s 0 ms burst 0 q65540 50 sl. 0 flows (1 buckets) sched 4 weight 0 lmax 0 pri 0 droptail sched 4 type FQ_CODEL flags 0x0 0 buckets 1 active FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN Children flowsets: 3 0 ip 0.0.0.0/0 0.0.0.0/0 25 2168 0 0 0 00005: 570.000 Mbit/s 0 ms burst 0 q65541 50 sl. 0 flows (1 buckets) sched 5 weight 0 lmax 0 pri 0 droptail sched 5 type FQ_CODEL flags 0x0 0 buckets 1 active FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN Children flowsets: 1 0 ip 0.0.0.0/0 0.0.0.0/0 6173 261323 0 0 0 00006: 571.000 Mbit/s 0 ms burst 0 q65542 50 sl. 0 flows (1 buckets) sched 6 weight 0 lmax 0 pri 0 droptail sched 6 type FQ_CODEL flags 0x0 0 buckets 1 active FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN Children flowsets: 2 0 ip 0.0.0.0/0 0.0.0.0/0 17678 26134501 0 0 0 Queues: q00001 50 sl. 0 flows (1 buckets) sched 5 weight 0 lmax 0 pri 0 droptail q00002 50 sl. 0 flows (1 buckets) sched 6 weight 0 lmax 0 pri 0 droptail q00003 50 sl. 0 flows (256 buckets) sched 4 weight 0 lmax 0 pri 0 droptail mask: 0x00 0x00000000/0x0000 -> 0xffffffff/0x0000 q00004 50 sl. 0 flows (256 buckets) sched 3 weight 0 lmax 0 pri 0 droptail mask: 0x00 0xffffffff/0x0000 -> 0x00000000/0x0000I will try to test moving the limiter on WAN600 to LAN if i have the chance, and if possible, because things certainly get complicated with many rules, adapters, policy routing, etc.
-
Those are some bufferbloat tests from https://www.waveform.com/tools/bufferbloat
Waveform test results for WAN20: https://www.waveform.com/tools/bufferbloat?test-id=317836bf-f9b1-4e10-a677-6a378ec596e2
++++++++++++++++++++++++++++++
Waveform test results for WAN600: https://www.waveform.com/tools/bufferbloat?test-id=185b5ddb-99fd-4380-bf62-2c918c79233f
-
@TheNarc said in This is how i used limiters to fairly share bandwidth among all devices on LAN:
So in my testing, switching my floating rules to apply the limiters on LAN interface traffic instead of WAN interface traffic completely eliminated my bufferbloat mitigation. I want from an A+ on waveform's test to a B or C. But it's interesting if you're seeing good bufferbloat mitigation with rules applied on your LAN interface.
Now the only other difference is that I was applying the limiters using floating rules with a Match action, because that's how I learned to do it and also because I couldn't have rule processing stop at that point since I have more complex policy routing rules in my preexisting LAN pass rules. But I still can't see an obvious reason why it would be substantively different from what you're doing.
That's curious. It should work the same if all of the packets are really going thru the limiter. In fact, i have two WAN adapters, and one of them have the limiters applied to LAN, and the other has the limiters applied on WAN (the netgate recipe liked above).
In fact, i have several LAN adapters, but every one of them has a rule similar to "Action: Pass, Source: LAN Subnet, Destination: (invert match) LAN Subnet, In Pipe: UploadQueue, Out Pipe: DownloadQueue"
There should be no problem to use Floating Rules with a Match Action, but they can get tricky, and the rule must match traffic going to the Internet only, and if you add queues with masks on source and destination addresses, it's critical that the rule matches traffic with LAN addresses as source or destination, according to the case.
If you have only outgoing traffic (no open ports for connections originating from the internet), one LAN adapter and one WAN adapter, then a floating rule like "Action: MATCH, Adapter: LAN, Source: LAN Subnet, Destination: (invert match) LAN Subnet, In Pipe: UploadQueue, Out Pipe: DownloadQueue" should match all traffic going to the internet, and frequently other stuff like multicast. I don't use multicast, so it can be ignored in my case. Or you could create a "NotTheInternet" alias, and use it as destination (with "invert match", of course, and the alias could include networks like all the private network space, and other stuff that won't go to the internet). You can fine-tune by looking at Status -> System Logs -> Firewall, if you enabled logging in you rules (i always do).
If you used masks and you want to check that the queues are being applied to the correct addresses, you can temporarily select the default scheduler ("Worst-case Weighted fair Queueing"), then you can see in Diagnostics -> Limiter Info the exact IP addresses of all the queues that are active.
Things can get complicated, so better to have a saved configuration before starting modifications.
In my case, this is used in an office, so there are many different devices trying to get/send some data simultaneously, several different protocols simultaneously, and it's important that if several people are downloading (or other kind of heavy use), every one of them get a fair share of the internet connection. But for other kind of uses, maybe there is no reason to complicate things with masks, and fq_codel by it's own results to be fair enough.
@SteveITS said in This is how i used limiters to fairly share bandwidth among all devices on LAN:
@fsr said in This is how i used limiters to fairly share bandwidth among all devices on LAN:
rules applied on WAN won't know the internal LAN addresses at all
If I followed, this can be done for floating rules with tagging:
https://docs.netgate.com/pfsense/en/latest/firewall/floating-rules.html#marking-and-matching
so, tag=VOIP, match on tag=VOIPBut if what you have is working fine, don't break it. ;)
Thanks, i use tags for connections originating from the internet. I use a tag to mark which packets come from a specific WAN adapter, so that then i can match the packets on LAN that have that tag, so i can apply the correct limiters to that traffic.
I was just pointing to the fact that a rule on WAN doesn't knows anything about the LAN address the packet came from (NAT has already taken place), so fq_codel couldn't possibly apply any kind of host fairness, even if it was designed for that, which it doesn't.
-
@fsr Thanks for the detailed write-up. I decided to take another shot at it and I seem to have things working well with floating match rules applied to the LAN interfaces now, using limiter sub-queues with masks. I'm a bit confused because I honestly don't know how this configuration is different than my first attempt, but the bufferbloat tests look good.
And also to your point, the host-based fairness is probably a bit overkill for my small home network anyway, but conceptually it does seem to make sense. And I figure if it works, why not? So unless someone more knowledgeable stops by and has some compelling reason fir it being a bad idea, I'm going to leave it.
Also a neat trick temporarily changing the scheduler algorithm to make the queuing behavior more visible in Diagnostics > Limiter.
One other note that may be of interest: I also have an explicit delay of 1ms set on my limiters per the workaround suggested in this bug: https://redmine.pfsense.org/issues/11192 and I believe that it did make a difference. Though obviously there is normal test-to-test variation as well so it can be difficult to be certain except when changes make a very dramatic and obvious impact.
-
@TheNarc You're welcome.
It's great that it works now. Maybe some parameter was reset to defaults when switching from fq_codel to "Worst-case Weighted fair Queueing" (WF2Q+) and back to fq_codel?
I will add some screen captures of my current limiter and queue configuration here. One picture is worth a thousand words.I didn't know about that bug, but it doesn't seems to be happening in my case, because the Gateway Quality delay average graph is between 3 and 10 ms, even when the 20 Mbps link traffic graph is maxed out for like 30 minutes, while several people are using meet, zoom, and the like. And no user complains (that's the most sensitive indicator there is
). In my case pfSense is running as a VM on Hyper-V in Windows Server, and the pfSense is showing hn network adapters.These are all the limiters i have configured right now:
- WanDLCodel and WanULCodel and it's Queues are configured like in the netgate recipe linked above, so i will not show them. Their Queues are applied to the 600 Mbps link on the WAN side, as indicated in the recipe.
- ShareLimitByDestIP and ShareLimitbySourceIP and it's Queues are configured like in the first post i made in this topic. I will show a capture of it's configurations next. Their Queues are applied to the 20 Mbps link on the LAN side. I will post screen captures of them.
If the names are confusing, "ShareQueueByDestIP" applies the mask by Destination Addresses. As the rule is applied on LAN that means it's the Download limiter queue (it's probably better to name it that way).
Likewise, "ShareQueueBySourceIP" applies the mask by Source Addresses. As the rule is applied on LAN that means it's the Upload limiter queue (it's probably better to name it that way).
======================================
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
======================================
-
NOTE: just tried to apply the same kind of limiter on LAN for my second internet connection (thus, using a non-default gateway), and i found that there is a bug in pfSense CE 2.8 that prevents doing this. It will be fixed in 2.9. For rules using the default gateway there is no problem. It only affects non-default gateways:
https://forum.netgate.com/topic/197993/limiter-source-mask-now-after-nat-when-using-gateway-groups-2-8-change/15
https://redmine.pfsense.org/issues/15770
https://github.com/pfsense/FreeBSD-src/commit/7ec06143964a949ebf6885ac120fdf839ad29eab
So, my second internet connection will continue to use the netgate recipe on WAN at least until pfSense 2.9 is out.
-
Hello everyone.
I'm trying to do the same thing: per-LAN-IP sharing of Internet bandwidth. I built the same Download and Upload CoDel limiters with nested queues with 32-bit masks.
I'm not using any limiters on WAN interface - only on LAN and my buffebloat score improved from B-C to A+
But there is no equal bandwidth sharing per-LAN-IP : if two PCs are downloading it is 30-60% not 50-50. I tried removing masking from my queues - same 30-60 problem.
I expected FQ_CoDel scheduler to equly devide the available bandwidth.
Anybody has any suggestions ? -
S SteveITS referenced this topic
-
@fsr It wasn't clear to me from the redmine bug if this may work, but have you tried using a floating "Match" rule on your LAN interface to apply the limiter, and a separate "Pass" LAN Interface rule to policy route to the non-default gateway? May make no difference, but just a thought.
-
@strannik Not sure . . . you may want to post screen shots of your limiter and queue configurations. Have you double checked that you have the proper source versus destination masking for your upload versus download queues? For applying the limiter on the LAN, it should be masking on source for upload and destination for download.
-
@TheNarc - Thanks for reply. Here are my screenshots.
Download Limiter (part-2)
and finally - the nested Queue with 32-bit masking:
I also tried using just Pipes with per-IP masking without nesting Queues - CoDel does not work at all.
I was able to get rid of bad bufferbloat (150-300 ms) with this nested configuration but still it is not "fairly" sharing the assigned bandwidth.*** My total WAN bandwidth is approx. 850 Mb/s down and 650-700 Mb/s up. I'm trying to create separate limers per-department so each department could use no more that 100 Mb/s total - depending on how many users are downloading at once: if 1 user needs bandwidth - he has 100%. If 3 users need bandwidth - each gets approx. 100/3=33% of that 100Mb/s department limit.
-
@strannik Thanks for the update. I definitely don't see anything wrong with how you have the limiters and queues configured, and Tail Drop is definitely what you want for the queue management algorithm on both the limiter and the child queue.
I'll see whether I can do some more of my own testing to determine whether I seem to get per-host fairness, as I'm configured very much the same. Although I'm also only running it on a small home network. The curious thing is that even without the LAN-side application of the limiters and the masks on the child queues, I would expect things to be generally fair. That is to say, if you only apply limiters on the WAN interface, it should enforce per-flow fairness, based on my understanding. So if host A and host B both run a speed test that makes a connection to a single remote host, they would each be initiating one flow, and FQ_CODEL would split the available bandwidth between them fairly (assuming no other network activity). I think the host-based fairness would come into play more if one host is establishing far more flows than another; say, host A is torrenting with connections to who knows how many remote hosts and host B is just trying to download something from a single remote host.
I realize none of that really helps to answer why you're seeing the behavior that you are, more just to express my confusion, and give others reading this thread a chance to point out anything I may have egregiously wrong there.
If I think of any other suggestions, I'll update, but unfortunately I'm not sure what else to suggest at the moment. I trust that both the hosts you're using for your testing are wired, since wireless would throw all kinds of wrenches into attempting to assess bandwidth sharing fairness. I suppose if you have a 3rd host it may be interesting to add it to the mix as well. For example, is it always one host that receives an unfair share, or does the unequal distribution seem to move from host to host seemingly at random?
-
@TheNarc Thank you for reply. I keep testing and I just discovered that Upload FQ_CoDel is dividing almost perfectly 50-50 unlike download.
-- Also I found that having CoDel setup on WAN interface has almost NO effect on LAN per-IP fairness. I disabled my floating limiter rule on WAN and nothing changed. My bufferbloat score is still great (under 10 ms in all modes)
-- Also I discovered that changing the scheduler from Default to FQ_CoDel and back does NOT work. I had to create a brand new limiter from scratch for CoDel to kick in even if I reboot the pfSense. That is definitly a BUG in pfSense 2.7.2
Now I'm testing v2.8.1 in VBox before I upgrade my production firewalls. Downtime is very expensive :) -
@strannik That's very interesting. I wonder if that implies that even though the limiters are being applied using a rule on the LAN interface, it's still pre-NAT? Because, for download, if the destination always just looks like the WAN IP, then masking on destination is going to have no effect because everything will still go in one queue. And if it's pre-NAT in both directions, then from the upload perspective, you're going to have the LAN hosts as sources, so masking on source will get you one queue per host and therefore the fairness that you're seeing. What about keeping the upload rule on LAN masking by source, and moving the download rule to WAN, also masking on source? Does that make any sense, or am I full of it?
The issue you found switching between schedulers sounds like a pain. I've been running 2.8.1 since it was released since my stakes are much lower with a home network, but I haven't tried going back and forth on the scheduler settings. I can try that, but I'm not sure how to definitively tell whether it's working or not. Did you just find that after setting it to something other than FQ_CODEL and then back that you still had bad bufferbloat?
-
@TheNarc After changing the scheduler and applying it I always go to Diag --> Limiter Info to verify that "scheduler" has changed. Here is where I discovered that it did not - all my pipes and queues kept their old configuration. I rebooted the whole pfSense - same thing. Then I created a new limiters from scratch and it did work.
To bad I can not experiment on production firewalls so I created a Lab setup in Virtual Box to be able to reboot more often. I'm testing every possible way but still I have not found the exact step-by-step procedure that works. Perhaps this FQ_CoDel needs some more development :)