This is how i used limiters to fairly share bandwidth among all devices on LAN
-
@TheNarc - Thanks for reply. Here are my screenshots.
Download Limiter (part-2)
and finally - the nested Queue with 32-bit masking:
I also tried using just Pipes with per-IP masking without nesting Queues - CoDel does not work at all.
I was able to get rid of bad bufferbloat (150-300 ms) with this nested configuration but still it is not "fairly" sharing the assigned bandwidth.*** My total WAN bandwidth is approx. 850 Mb/s down and 650-700 Mb/s up. I'm trying to create separate limers per-department so each department could use no more that 100 Mb/s total - depending on how many users are downloading at once: if 1 user needs bandwidth - he has 100%. If 3 users need bandwidth - each gets approx. 100/3=33% of that 100Mb/s department limit.
-
@strannik Thanks for the update. I definitely don't see anything wrong with how you have the limiters and queues configured, and Tail Drop is definitely what you want for the queue management algorithm on both the limiter and the child queue.
I'll see whether I can do some more of my own testing to determine whether I seem to get per-host fairness, as I'm configured very much the same. Although I'm also only running it on a small home network. The curious thing is that even without the LAN-side application of the limiters and the masks on the child queues, I would expect things to be generally fair. That is to say, if you only apply limiters on the WAN interface, it should enforce per-flow fairness, based on my understanding. So if host A and host B both run a speed test that makes a connection to a single remote host, they would each be initiating one flow, and FQ_CODEL would split the available bandwidth between them fairly (assuming no other network activity). I think the host-based fairness would come into play more if one host is establishing far more flows than another; say, host A is torrenting with connections to who knows how many remote hosts and host B is just trying to download something from a single remote host.
I realize none of that really helps to answer why you're seeing the behavior that you are, more just to express my confusion, and give others reading this thread a chance to point out anything I may have egregiously wrong there.
If I think of any other suggestions, I'll update, but unfortunately I'm not sure what else to suggest at the moment. I trust that both the hosts you're using for your testing are wired, since wireless would throw all kinds of wrenches into attempting to assess bandwidth sharing fairness. I suppose if you have a 3rd host it may be interesting to add it to the mix as well. For example, is it always one host that receives an unfair share, or does the unequal distribution seem to move from host to host seemingly at random?
-
@TheNarc Thank you for reply. I keep testing and I just discovered that Upload FQ_CoDel is dividing almost perfectly 50-50 unlike download.
-- Also I found that having CoDel setup on WAN interface has almost NO effect on LAN per-IP fairness. I disabled my floating limiter rule on WAN and nothing changed. My bufferbloat score is still great (under 10 ms in all modes)
-- Also I discovered that changing the scheduler from Default to FQ_CoDel and back does NOT work. I had to create a brand new limiter from scratch for CoDel to kick in even if I reboot the pfSense. That is definitly a BUG in pfSense 2.7.2
Now I'm testing v2.8.1 in VBox before I upgrade my production firewalls. Downtime is very expensive :) -
@strannik That's very interesting. I wonder if that implies that even though the limiters are being applied using a rule on the LAN interface, it's still pre-NAT? Because, for download, if the destination always just looks like the WAN IP, then masking on destination is going to have no effect because everything will still go in one queue. And if it's pre-NAT in both directions, then from the upload perspective, you're going to have the LAN hosts as sources, so masking on source will get you one queue per host and therefore the fairness that you're seeing. What about keeping the upload rule on LAN masking by source, and moving the download rule to WAN, also masking on source? Does that make any sense, or am I full of it?
The issue you found switching between schedulers sounds like a pain. I've been running 2.8.1 since it was released since my stakes are much lower with a home network, but I haven't tried going back and forth on the scheduler settings. I can try that, but I'm not sure how to definitively tell whether it's working or not. Did you just find that after setting it to something other than FQ_CODEL and then back that you still had bad bufferbloat?
-
@TheNarc After changing the scheduler and applying it I always go to Diag --> Limiter Info to verify that "scheduler" has changed. Here is where I discovered that it did not - all my pipes and queues kept their old configuration. I rebooted the whole pfSense - same thing. Then I created a new limiters from scratch and it did work.
To bad I can not experiment on production firewalls so I created a Lab setup in Virtual Box to be able to reboot more often. I'm testing every possible way but still I have not found the exact step-by-step procedure that works. Perhaps this FQ_CoDel needs some more development :)
-
Thanks for the idea, but i can't do that, as i'm applying the limiters on the rules of the LAN interface (not in floating rules), because i have multi-WAN with failover, and the only way i found to apply different limiters to different internet links on the LAN side, is to apply the limiters on the same rule that chooses the gateway, and enable [System] [Advanced] [Miscellaneous] [Skip rules when gateway is down]. So, the failover works by using two separate rules for each link on the LAN interface: a rule allowing traffic to go to the internet with the fast gateway and the limiters for the fast link, followed by another rule allowing traffic to go to the internet with the default (slow) gateway and the limiters for the slow link.
So, if the fast gateway is up, the first rule applies. If the gateway is down, the first rule is skipped, and the second rule applies. Each rule applies the correct limiters for each link speed.I will have to wait for the next version to come out. Meanwhile, applying the limiter on WAN is good enough for the 600 Mbps link, while the slow 20 Mbps link has it's limiter on LAN.
The bug is that the limiters/queues will get the pre-NAT address on rules that set a non-default gateway. So if you use masking, instead of seeing a queue for each LAN address, you will see there is a queue for your WAN address (if you are using a scheduler that shows this, for example the default scheduler "Worst-case Weighted fair Queueing").
-
@strannik I have readed the messages, and i agree with TheNarc. Your limiter/queue configuration looks ok. Is your rule to assign traffic to queues like the one i wrote on the first message?
I don't see that strange behaviour about changing schedulers with 2.8.1. You should upgrade your test environment to the latest version to avoid those problems.
Using the default "Worst-case Weighted fair Queueing" scheduler is a good way to check that the masking is working ok, and the download/upload behaviour, and fairness (i readed that the default scheduler is designed for "host fairness", but it will do nothing to keep latency low, and that's why schedulers like fq_codel exist which will keep latency low, but won't provide "host fairness", only "flow fainess").
In my tests with speedtest.net, i have seen fair sharing by IP, when running two simultaneous tests against the same server, with low or no traffic from other hosts. That is expected to result in the same amount of connections from each IP. And it showed a clear difference between using masks, and not using them. But in real life use, there could be many reasons why two simultaneous downloads or uploads could get different speeds. The remote server could be the limiting factor, or one of your PCs is opening more connections than the other.
Also, if i understand this correcty, adding queues with masks to the fq_codel limiter will pass packets in a round-robin fashion from each of your active hosts to the fq_codel scheduler, but that only guarantees that no single host will be able to saturate the scheduler by opening a lot of connections (flows) at the same time. The queues with masks will add some "host fairness" to a scheduler that only cares about "flow fairnes", but it's likely that the result won't have a perfect "host fairness".
To get an idea of how little fq_codel cares about "host fairness", consider this: the first thing fq_codel does to assign flows to it's queues is to hash together 5 properties of each packet it receives: source and destination IP addresses, source and destination port numbers, and IP protocol number. Each flow is identified like this. Once this hashing is performed, the scheduler won't know about any of the original properties, only that each flow is a different flow. 50 different flows could be one flow from 50 of your hosts, or 50 flows from one host. No way for it to tell the difference. Host with more flows will get more of the bandwidth. But adding the queues with masks, they will pass packets in a round-robin fashion from each of your active hosts to the fq_codel scheduler, making sure that each active host can send one packet to the scheduler in turn before the hashing begins. That will add some host fairness to it, but when fq_codel has to decide which packets to drop and which ones to prioritize, it will only act according to the information it has, to do what it does: keep latency low, which probably won't result in 100% perfect host fairness. But in most cases it's more important to be able to keep using the link for real-time communications while heavy downloads are performed, than for each of your hosts to get perfect fairness about their downloads.
If you only care about host fairness and not latency, then the default scheduler would probably be a better choice, but the latency will likely suffer a lot.If you want to read more about fq_codel, this is a very interesting source: https://datatracker.ietf.org/doc/html/draft-ietf-aqm-fq-codel
-
@fsr I think it was a bad idea on my part anyway :) However, I would have thought that you could apply limiters using floating match rules on the LAN interface and then still do your policy routing to apply gateways using normal LAN pas rules. At least, that's what I do in my home setup; I basically have "virtual" multi-WAN because I have VPN client connections, so I policy route to either their tun interface gateways or my true WAN interface gateway. That seems to work as expected for me, but I also know it's clearly not identical to your setup.
I think it may also be possible to use altq shaping in addition to the dummynet limiter shaping in order to achieve per-host fairness. But obviously that would add even more complexity and overhead, so I'm not sure it would be worth it . . .
-
@fsr Thank you for detailed explanation. I updated to v2.8.1 and found no difference. Looks like FQ_CoDell can not provide perfect per-IP fairness.
I will create multiple groups of IPs (like 5 users each) and also 2 limiters for each group. I will assign like 100 Mbit/s to each group. Child queues with CoDell shedulers will take care of latency and also do some fairness (as accurate as possible) .
That way - no single user can saturate the whole bandwidth. -
@strannik Why would you want to do something like that? Too complex, you would be underutilizing you bandwidth, and it won't be fair anyways. I'm pretty sure that the configuration presented on the first message up there would manage bandwidth, fairness and latency much better than that. With this, it would be impossible for one user to "hijack" all of the bandwidth. Of course, if other users are not using much of the bandwidth at all, it can look like one user has hijacked the link, but in reality he's just using the bandwidth that no one else is using anyways. You want that to happen. Why would you pay for a 800 Mbps link to download at 100 Mbps if you could be using almost the 100% of it? The whole point of traffic shaping is to be able to share bandwidth between all users in the most efficient way, fairly, and for latency-critical protocols to continue to work well even when the link is under heavy load. This seems to accomplish this task quite well.
In a "live" network it's hard to see if the bandwidth is being fairly shared, because the hosts are likely not even attempting to download at the same rate (using different amount of connections/flows, using different protocols, etc), they're downloading from different servers with different bandwidth and different amount of clients connected to them, etc.
That said, i have seen several times in the traffic graph than two PCs from LAN were downloading at nearly the same rate. And bandwidth tests off-hours when the link is not utilized should show good "host fairness" with this configuration, even if not mathematically perfect.
Just check that the per-IP queues are being created in the way you think they are in the limiter diagnostics (the default scheduler shows this), check the firewall logs to see if the traffic is hitting your rules correctly. Test off-hours when you can control which PC or PCs are downloading/uploading, etc.
It's odd that you continue to see problems when switching schedulers in 2.8.1, even after restarting. I didn't found such issues. At most you could need to reset the state table. You could try to delete all limiters, restart and create them from scratch, to see if that fixes the odd behaviour.
Or if you continue to find such problems, you could resort to keep one set of FQ_Codel limiters and another set of limiters with the default Scheduler (for testing). Then you only need to modify the Queues configured on the rules to apply one or the other.