Playing with fq_codel in 2.4
-
Here is some additional information from the NBN in terms of how they deal with their policy enforcer on uploads (ignore downloads as that is fine).
*******The PBS defines the length of a burst of Layer 2 traffic (either in bytes or milliseconds as set out below) that may be received at ingress to the NBN Co Network for a burst of traffic that pushes the average Information Rate above the configured bandwidth profile for a PIR traffic class. Traffic in excess of the PBS will be discarded by the NBN Co Network. The PBS is set by NBN Co for each PIR specification, and cannot be modified.
The PBS is used by the policing functions of the NBN Co Network at ingress to the NBN Co Network to determine whether a stream of ingress data complies with the subscribed PIR. Customer is responsible for ensuring that all ingress traffic is shaped to comply with the PIR/PBS as specified for the required traffic class and interface, before presentation to the UNI-D or NNI as relevant.
It goes on to define the PBS as:
Downstream at the NNI: 10ms
Specific PBS setting in Bytes is dependent on the TC-4 PIR (bandwidth profile) selected
Upstream at the UNI-D: 40,000 bytes
The AVC TC-4 PBS is set by NBN Co and cannot be modified by Customer******* -
Apologies in advance if you've already provided this information - I read back through but may have missed it - but are you saying that with the pfSense machine you can never get above ~300Mbps regardless of whether you use limiters at all? Or are you saying that you've tried different limiter settings but never get above that speed? In other words, through your testing have you isolated this apparent speed ceiling specifically to when you use limiters?
Another interesting test (maybe, I'm not sure whether it would really provide useful information) might be to set up a download limiter. I know that you don't need one, and it would only be temporary. But may be interesting to know whether setting up a download limit with the same parameters as your upload limiter would "over-limit" your download in the same manner.
I can say that I haven't seen anything like this and I use a roughly identical shaping configuration, but also with dramatically different bandwidth limits. I only have 10Mbps upstream, so it's nowhere near the same throughput.
One other thought, and this is really off the wall and should make no difference, but it's also trivial to try: maybe specify the bandwidth in different units. For example, 380,000 Kbps.
-
@TheNarc said in Playing with fq_codel in 2.4:
Apologies in advance if you've already provided this information - I read back through but may have missed it - but are you saying that with the pfSense machine you can never get above ~300Mbps regardless of whether you use limiters at all? Or are you saying that you've tried different limiter settings but never get above that speed? In other words, through your testing have you isolated this apparent speed ceiling specifically to when you use limiters?
Yes, I am able to get over 360+mbps when I put a switch between pfsense and the NBN connection (where the switch is doing the shaping). I don't own that switch and reluctant to buy one to fix this when hopefully we can resolve this within pfsense.
Another interesting test (maybe, I'm not sure whether it would really provide useful information) might be to set up a download limiter. I know that you don't need one, and it would only be temporary. But may be interesting to know whether setting up a download limit with the same parameters as your upload limiter would "over-limit" your download in the same manner.
I've tried this - no difference in results.
One other thought, and this is really off the wall and should make no difference, but it's also trivial to try: maybe specify the bandwidth in different units. For example, 380,000 Kbps.
I'll give this a shot and revert.
-
@Larrikin Sorry about that, somehow I missed that the pfSense box was still in the loop when you were using the Ubiquity switch. Only other thought I have offhand is whether you've tried ALTQ shaping instead of limiters? Again, I see nothing wrong with what you're doing now, but if it simply won't work you can try ALTQ as a point of comparison.
I can also say that on my upload limiter, I have a much lower limit value (1024 as opposed to 10240) based on information from the following sources:
https://www.bufferbloat.net/projects/codel/wiki/Best_practices_for_benchmarking_Codel_and_FQ_Codel/
Also based on those sources I have my upload limiter quantum set at 300, but then you have a much higher upload than the 100Mbps for which the setting of 300 is recommended.
-
@TheNarc said in Playing with fq_codel in 2.4:
@Larrikin Sorry about that, somehow I missed that the pfSense box was still in the loop when you were using the Ubiquity switch. Only other thought I have offhand is whether you've tried ALTQ shaping instead of limiters? Again, I see nothing wrong with what you're doing now, but if it simply won't work you can try ALTQ as a point of comparison.
I'm unfamiliar with ALTQ. How do I go about setting that up? I can't find anything in the GUI.
-
@Larrikin Sorry, it's not referred to as such in the GUI. It's set up on the "By Interface" tab from the "Firewall > Traffic Shaper" page. But typically it's easiest to set it up using the wizard (far right tab, Wizards, on the same page). It's been a while since I've set it up myself, but there's good information in the pfSense book:
Must head to bed here but good luck! Will check back in.
-
@TheNarc said in Playing with fq_codel in 2.4:
@Larrikin Sorry about that, somehow I missed that the pfSense box was still in the loop when you were using the Ubiquity switch. Only other thought I have offhand is whether you've tried ALTQ shaping instead of limiters? Again, I see nothing wrong with what you're doing now, but if it simply won't work you can try ALTQ as a point of comparison.
Boom - you are right. Using ALTQ and switching off Explicit Network Congestion on qLink has fixed the problem. I've disabled the limiters and use this shaping and it works! Thank you good sir.
-
For those who want to know how exactly I got it working, you can find the instructions here:
https://whirlpool.net.au/wiki/pfsense_traffic_shaping
A big thanks to @TheNarc for pointing me in this direction.
Also a big thanks to all the others who contributed to helping me troubleshoot. I am most grateful you took your own personal time to help me.
-
I have a bit of a problem with my setup, and was hoping to gain some insight into why this may be happening. I will say that I'm no traffic shaping expert, but I have played around with it a bit with varying success. The biggest issue for me is trying to solve bufferbloat and have a better zoom experience.
I followed the instructions to get it up and running based on the google hangout post and slideshare instructions here: https://www.slideshare.net/NetgateUSA/pfsense-244-short-topic-miscellany-pfsense-hangout-august-2018
In my particular example, when running a speedtest from dslreports, I'm seeing 350Mbps down and about 22Mbps up. I have since used the speeds when configuring pfsense to 340/20. I have also set a 1700 Queue Length (5 * 340 [speed]). My internal network consists of about 11 different networks; 5 of them using VLAN's, the other are using dedicated NIC's in my pfsense box.
This does improve my bufferbloat score tremendously from a "D" to an "A". The problem I have are icmp packets dropping. When running a speedtest and pinging something (google.com, 8.8.8.8, Comcast's DNS Servers), I never get a response back while running the speedtest. After the test is over, I start seeing responses again. When somebody is on a Zoom conference, I'm seeing about 2% packet loss on the WAN connection.
I came across this bug:
https://redmine.pfsense.org/issues/9024
This sounds similar to what I've been seeing on my end. I was wondering if anybody else has had a similar experience, and if there's a better way to control this than setting a rule up on every interface.
-
While reviewing that bug, there was a comment made: “ If a separate floating match rule is created for ICMP, then packets will not be dropped.”
Have you tried that?
-
@fabrizior said in Playing with fq_codel in 2.4:
While reviewing that bug, there was a comment made: “ If a separate floating match rule is created for ICMP, then packets will not be dropped.”
Have you tried that?
Yup. I first tried to create a regular rule on every interface but was still having problems. I then thought hey, ping is ICMP so let's change this floating rule from "all protocols" to just TCP/UDP. I did that and it fixed my packet loss.
Will this reported bug affect anything else or is this fq_codel setup good to use in production?
Thank You!
-
Define Production...
:)Been working here just fine. entire family’s online experiences are vastly improved.
Though, as Karma would have it, these errors started at 3am last night:
There were error(s) loading the rules: /tmp/rules.debug:218: no routing address with matching address family found. - The line in question reads [218]: pass out quick on { em0 } $GWWAN_DHCP6 inet6 from any to any tracker 1595024432 keep state dnqueue( 2,1) label "USER_RULE: CoDel Limiters IPv6"
@ 2020-09-01 03:18:18There were error(s) loading the rules: /tmp/rules.debug:218: no routing address with matching address family found. - The line in question reads [218]: pass out quick on { em0 } $GWWAN_DHCP6 inet6 from any to any tracker 1595024432 keep state dnqueue( 2,1) label "USER_RULE: CoDel Limiters IPv6"
@ 2020-09-01 03:18:40 -
@fabrizior said in Playing with fq_codel in 2.4:
Define Production...
:)Been working here just fine. entire family’s online experiences are vastly improved.
Good to hear. I haven't had any major issues per say, but have had some things happen in the past couple of months that have made me look more at quality vs quantity. Figuring out my Comcast inactive wifi connection was actually active and the same channel as my home AP. I've also had some quality issues with Zoom and Teams that may or may not be related to Comcast. I'm hoping that traffic shaping improves this. I honestly don't think it will because I'm only using about 5/2 and the connection is a 340/20. But hey, maybe it will make things better.
Though, as Karma would have it, these errors started at 3am last night:
Sorry to jinx you. ;)
-
@qwerty123
ha! thanks.Was having a lot of jitter and semi-frequent high percentage of dropped packets earlier this summer.
Turned out that the squirrels were munching on the 75-ft cable from the house to the pole and it was full of water! When the Comcast tech unscrewed it at the service demarcation box on the wall, it ran like a garden hose for over a minute! He laughed and replaced it. service has been solid ever since. -
I’d suggest running tests from a hardwired host when you experience issues. wifi adds too many variables to account for if you suspect router issues.
How’d your xfinitywifi re-activate? thats odd.
will go check my XB4 out of curiosity. Didn’t think we had access to those settings when in bridge mode. -
@fabrizior said in Playing with fq_codel in 2.4:
I’d suggest running tests from a hardwired host when you experience issues. wifi adds too many variables to account for if you suspect router issues.
I agree. When I run these tests, I do it from my hardwired workstation. Too many variables over wifi.
How’d your xfinitywifi re-activate? thats odd.
will go check my XB4 out of curiosity. Didn’t think we had access to those settings when in bridge mode.This is interesting and extremely frustrating. I have an XB6. When I set it up, I configured it to make wifi 'inactive' through the web UI, then I placed it in bridge mode to use it with pfsense. I discovered a wifi issue a few months ago and decided to change wifi channels. Instead of using the Ubiquiti scan tool, this time I used inSSIDer. When I scanned, I discovered a really hot channel without a SSID and using about 5 or 6 bssid's. I was able to match the bssid's close enough to the mac address of the wifi card in the xb6. When I brought my laptop down to the modem, I discovered an extremely high signal coming off of the xb6.
I asked around and it turns out that even if you have your modem in bridged mode with wifi being 'inactive', they still activate wifi for their personal hotspot. If you disable that, it's STILL active because of their security system. Rumor has it that the security system Xfinity sells uses wifi. So the Xfinity modems have wifi utilizing a channel that can't be disabled because of this.
I don't have the security system and just want to use the modem in bridge mode with wifi totally disabled. Looks like that's not an option unless you use your own modem. I've been down that road before and whenever there's a problem, Comcast always blames your user-owned modem. Plus, I wanted to get rid of my data cap so I pay for the modem rental plus no cap.
-
@fabrizior said in Playing with fq_codel in 2.4:
There were error(s) loading the rules: /tmp/rules.debug:218: no routing address with matching address family found. - The line in question reads [218]: pass out quick on { em0 } $GWWAN_DHCP6 inet6 from any to any tracker 1595024432 keep state dnqueue( 2,1) label "USER_RULE: CoDel Limiters IPv6"
@ 2020-09-01 03:18:18This :
/tmp/rules.debugis just a file : the one that gets build after 'something' was changed in the firewall rule set. It could be an alias that points to a new IP or network, pfBlockerNG that retrieved new info, or some sheduled firewall rule.
What is on, around this line "218" ?
Check the frequency of this line in the main log file :
Typically, one, two three times a day is normal.
Many more ? Could be ok, just find out the 'why' part.To make a long story short : disable this :
( DNS Resolver settings) -
@qwerty123 Where did you place the floating rule, above or below the WAN-Out FQ-ColDel rule?
-
@superweasel said in Playing with fq_codel in 2.4:
@qwerty123 Where did you place the floating rule, above or below the WAN-Out FQ-ColDel rule?
I ended up making the rule just for tcp/udp. All the guides say to use "ALL" for the protocols. In most situations, the ISP's mostly going to route tcp/udp and icmp. Since ICMP was the issue, I just changed the protocol "TCP/UDP" instead of "ALL".
-
Another question, since everybody was so helpful and helped me out with my last problem. I ended up rebuilding my pfsense box this week and had put the same settings in the traffic limiter as my old config had.
When I rebuilt my box and ran a speedtest, I see that my download speed is about half of what I have. When running a speedtest without the limiter set, I have about 350/25. My WAN Down Queue is set to 340 Mbit/s and WAN Up Queue is 20 Mbit/s. When running a speedtest on a wired computer, I get about 175/20.
I know the limiter will drop throughput but I'm surprised that it was this much. And I'm almost positive that before, I was seeing about 320/20.
Does anybody happen to have a similar experience, and any hints on what to look for? Thanks!
