HFSC shaper dropping packets before bandwidth reached
-
I'm having some issues getting a shaper working on one of my firewalls. I've used HFSC on pfsense for several years on several systems. I'm no expert at the black magic involved, but at times I think I understand it. I have a 60Mbit down by 5Mbit up cable connection at this site. I'm trying to setup queuing so that I can give priority to a facebook live streaming computer on Sundays during our services. The bitrate on the stream is around 2000Kbps So, I have the shaper setup to give 60% to this when it is needing it.
My issue seems to be that no matter what I set my parent queue bandwidth to be, I'm getting packets queued in my queues and drops, and I'm only reaching about half of my 5Mbit upload speed on my connection (According to the bandwidth readings on the root queues in the GUI traffic shaper. I'm wondering if there is some sort of bug in the latest versions of pfsense as I'm also not getting accurate traffic graphs on the main page of the firewall either? I've tried setting the qLimit queue up to a 9Mbit upper limit and I still can't push more than 2-3Mbits up without queuing and dropping packets.
Here is my basic configruation:
pfSense version 2.3.3-RELEASE-p1 (i386)
WAN - 100 Mbps (Root)
–>qLimit - 5Mbps (Limit bandwidth for child queues to 5Mbps, no linkshare or realtime set.)
--> -->qInternet (Bandwidth 10%) - this has some child queues, but none have realtime set.
--> -->qStreaming (Bandwidth 60%, Linkshare m2 60%)
--> -->qVPN (Bandwidth 10%)
--> -->qVOIP (Bandwith 180KBit/s, Realtime m2 of 180Kb)
--> -->qGuests (Bandwidth 10%)I'm sorry I don't have any pictures of what the action looks like when the shapers are in action. This site is about 30 Min from me and hard for me to get there to test everything. What I experienced though is that even though pfTop shows my qStreaming to have 3000Kb BW available to it, it starts queuing and then dropping packets when it gets around 350-400K in the B/S column.
Another issue I seem to be having on the newer version of pfSense is that pftop and the Queues monitoring in the gui seem to "reset" themselves often zeroing out the stats. I don't know if this is something just happening on my machines or if others are experiencing this too.
I'm thinking about trying to roll back to an earlier version of PF that is more stable. If anyone agrees and has recommendations, I'm open to hearing them.
I appreciate any help and insights anyone can give me. Or maybe just another set of eyes will see something I overlooked.
-
I see your queue sizes for some off the other queues are only 50. I'd recommend checking "Codel Active Queue" under a queues "Scheduler options" section. Averages are very deceiving, networks are bursty.
YouTube likes to send me an average of about 5Mb/s video streams, by bursting about 250KiB (170 1500byte packets) at 1Gb/s 2-3x per second. Percentiles are much more useful.
-
I see your queue sizes for some off the other queues are only 50. I'd recommend checking "Codel Active Queue" under a queues "Scheduler options" section. Averages are very deceiving, networks are bursty.
YouTube likes to send me an average of about 5Mb/s video streams, by bursting about 250KiB (170 1500byte packets) at 1Gb/s 2-3x per second. Percentiles are much more useful.
Thanks for the info. I hadn't checked the Codel Active Queue because the documentation is so sketchy on all those options that I figured I best leave them alone. Should I enable that on every queue, parent and child? Is there somewhere I can read up on what exactly "Codel Active Queue" does?
What do you mean by "percentiles are more useful"? Are you meaning looking at bandwidth used as an aggregate over time like under the monitoring RRD graphs?
-
With HFSC, only leaf queues can have traffic, so no point checking parent queues.
Codel is a dynamic AQM that has a goal of not having any packet spending more than some target(10ms default I think) in the queue, and uses an interval (100ms default I think) where it will drop a packet and wait another interval before dropping another packet, with non-linear reducing intervals. 10ms is saying it does not want more than 10ms of bufferbloat and 100ms is because assuming a 100ms RTT(ping), it will take that long before the sender gets the signal to back-off.
Codel does a few things
- Has a large queue that allows transient bursts to be readily absorbed
- Uses Head-drop instead of tail-drop, which penalizes old traffic, not new traffic, while also giving quicker signalling to the sender
- Ramps up the drop rate, trying to stay as small as possible. Since it drops 1 packet at a time instead of bursts like a full FIFO buffer, it allows for better overall utilization.
It also has a secondary affect of affecting "bandwidth hogs" more than small light flows. This not only helps stabilize the bandwidth, but it keeps latency low, minimizes packetloss, and to some degree helps distribute bandwidth.
This is one of the first of its kind. There are newer better ones, like fq_Codel or Cake.
Yes, like the aggregate over time, but at much smaller time-slices, which do not exist.
-
I'm at the remote site this afternoon and I'm testing different settings for the traffic shaper. I cannot find one that is consistently good. I'm trying to check using the DSLReports buffer bloat test tool.
What I'm seeing is that the leaf queue that I have qStreaming is dropping and queueing packets at around 400Kbps bandwidth when the queue has 2880K available to it. Sometimes I get all A's on the DSL reports test, and then I test again and I'm getting C's and F's. What would cause a queue to prematurely queue packets before it gets to its bandwidth limit?
I Enabled "Codel Active Queue" on all of my queues and I thought it made a difference. But now I'm back to getting F's on the test, and dropping all kinds of packets when my queue length never goes above zero. I thought I used to understand hfsc traffic shaping on pfsense, but nothing makes sense anymore. I'm not sure if I'm dealing with bugs or misconfigurations or both.
Are there any other testing tools than the ones I'm using? I used to rely on pftop and the web gui to see what is happening with the queues, but that doesn't seem to work any more.
Should I be assigning Codel Active to all Queues, or just those I need low delay on? Should it be applied to a leaf queue that has child queues?
Another fun fact I'll throw in there, I'm using a LAN rule to "mark" packets in order to catch them on the WAN side of things for shaping. Could that be causing any issues?
-
I noticed something when I was comparing pfTop with the web GUI interface. It looks like if I multiply the B/S column value by 8 I get close to the number showing in the GUI. So, if I'm not off base here I would say the GUI is displaying the traffic incorrctly as Kbits/s when it is really showing the value of KBytes per second. So, that probably accounts for some of the reason that I thought my queues were dropping packets before they were full. Is this a bug?
-
- Does qLimit have its upper limit set?
- Are you using RealTime at all? Because it pulls directly from the root queue and IGNORES upper limits
- Simple thing to try is to set your WAN interface to be limited to your 4.8Mb/s that qLimit has. This way there is zero question if some traffic is being missed and bypassing qLimit.
- Don't forget to shape LAN. While downloading tends to have much less bufferbloat, if you're not shaping your downloads, it could be adding to your overall bloat.
-
- Does qLimit have its upper limit set?
Yes, I have qLimit set at 4800 bits per second.
- Are you using RealTime at all? Because it pulls directly from the root queue and IGNORES upper limits
Yes, I have my qVOIP setup to have 180Kb/second realtime. However, there was no VOIP traffic on the network at the time of my testing.
- Simple thing to try is to set your WAN interface to be limited to your 4.8Mb/s that qLimit has. This way there is zero question if some traffic is being missed and bypassing qLimit.
I will try this suggestion and see what it does for me. I have set them both the same in the past, but I don't know what other settings I was using in combination with them at the time.
- Don't forget to shape LAN. While downloading tends to have much less bufferbloat, if you're not shaping your downloads, it could be adding to your overall bloat.
I guess this is a possiblity, I don't shape my LAN traffic for two reasons: One is that lots of people say it makes no difference to shape traffic that's already delivered to the interface (I guess unless you wanted to artifically limit some traffic). The other is that I have multiple LAN interfaces and from what I've figured out the queues don't talk to each other. (Though I could be wrong on this) At the time of my testing my 60Mbit per second download speed was not tapped more than 50% so I don't think that the downloads are a contributing factor.
So, I'm wondering what you think about the difference between pfTop and the Monitoring > Queues gui for traffic reporting? Which can I trust? Is there any way to get a more instantaneous refresh on these so I can see what's happening better in real time?
Thanks for your help with this.
-
I noticed something when I was comparing pfTop with the web GUI interface. It looks like if I multiply the B/S column value by 8 I get close to the number showing in the GUI. So, if I'm not off base here I would say the GUI is displaying the traffic incorrctly as Kbits/s when it is really showing the value of KBytes per second. So, that probably accounts for some of the reason that I thought my queues were dropping packets before they were full. Is this a bug?
The GUI has always been screwy. pftop is the best source. Did pftop ever show dropped packets?
-
The GUI has always been screwy. pftop is the best source. Did pftop ever show dropped packets?
Yes, pfTop shows dropped packets, however I hardly ever see the queue length change from zero even when it is dropping packets, and it seems to drop the packets at just a fraction of the bandwidth I have designated for the queue.
-
The GUI has always been screwy. pftop is the best source. Did pftop ever show dropped packets?
Yes, pfTop shows dropped packets, however I hardly ever see the queue length change from zero even when it is dropping packets, and it seems to drop the packets at just a fraction of the bandwidth I have designated for the queue.
Have you double-checked that your WAN link isn't having bandwidth problems?