Playing with fq_codel in 2.4

chrcoluk

I’ve noticed all of the fq_codel config examples have inbound and outbound queues.

It’s been my experience that shaping or prioritizing inbound WAN traffic, on a home/office router, is ineffective and degrades network performance.

Quick testing on DSLreports, I get much better results when I disable the inbound fw rules and only shape the outbound WAN traffic:

P4 2.4Ghz 2 core w/HT
Spectrum 30down 15up

No shaping - normal throughout & C+ buffer bloat
IN & OUT shaping - reduced download throughput (50%) & A+ buffer bloat
OUT only shaping - normal or slightly improved overall throughput & A+ buffer bloat

Has anyone seen download performance increased with the inbound shaper activated?

Btw, I’m new to the forum, thanks to everyone who’s posted, great discussion.

Well given for it to work you have to set the inbound pipe lower than your line capacity, yes download speeds will be a bit slower to match the pipe size.

Shaping isnt about max possible throughput but maintaining fairness across different network applications and QoS.

Some applications will by their design completely swamp a line for downloading and act as a sort of ddos.

chrcoluk

@tman222:

I think might be the exception here - I actually get (seemingly) slightly better performance when I have fq_codel applied to both upload and download on my fiber connection. That being said, the only way I have tested this to date has been through speed tests, and in particular the DSL Reports speed test to get an idea of bufferbloat. With fq_codel applied to the download side as well, my download during the test is a bit more stable, comes in slightly higher (bandwidth) and has lower average latency during the test. In fact, I have found the best performance so far for me has been by using a 940/940 limit on a gigabit FTTH connection, with a little bit more aggressive target of 3ms and interval of 60ms – the fq_codel defaults are 5ms and 100ms. This does limit the download and upload speed to about 915-920Mbit on my connection, but I'm willing to take a 3 - 3.5% hit on bandwidth to have average lower average latency and connection stability. One could argue that a on a gigabit Fiber connection all this really doesn't matter anyway since there are very few cases where the bandwidth is maxed out anyway, and that's probably true. But nonetheless I do like having these settings to ensure stability.

you not the exception :)

robnitro

I'm running fq_codel along with HSFC classes.
It helped my bufferbloat go from A to A+ even when I put the max speed in HSFC and the limiter!
Before with just hsfc I had to run 2 or 3 mbit lower for up and down to get just A.
I also have 2 other limiters set lower, for guest clients unknown and known.

In shellcmd (also on filter change)
ipfw sched 1 config pipe 1 type fq_codel target 5 noecn quantum 300 limit 1000 interval 50 && ipfw sched 2 config pipe 2 type fq_codel target 5 noecn quantum 300 limit 1000 interval 50 && ipfw sched 3 config pipe 3 type fq_codel target 5 noecn quantum 300 limit 1000 interval 50 && ipfw sched 4 config pipe 4 type fq_codel target 5 noecn quantum 300 limit 1000 interval 50 && ipfw sched 5 config pipe 5 type fq_codel target 5 noecn quantum 300 limit 1000 interval 50 && ipfw sched 6 config pipe 6 type fq_codel target 5 noecn quantum 300 limit 1000 interval 50

Harvy66

@tman222:

I think might be the exception here - I actually get (seemingly) slightly better performance when I have fq_codel applied to both upload and download on my fiber connection. That being said, the only way I have tested this to date has been through speed tests, and in particular the DSL Reports speed test to get an idea of bufferbloat. With fq_codel applied to the download side as well, my download during the test is a bit more stable, comes in slightly higher (bandwidth) and has lower average latency during the test. In fact, I have found the best performance so far for me has been by using a 940/940 limit on a gigabit FTTH connection, with a little bit more aggressive target of 3ms and interval of 60ms – the fq_codel defaults are 5ms and 100ms. This does limit the download and upload speed to about 915-920Mbit on my connection, but I'm willing to take a 3 - 3.5% hit on bandwidth to have average lower average latency and connection stability. One could argue that a on a gigabit Fiber connection all this really doesn't matter anyway since there are very few cases where the bandwidth is maxed out anyway, and that's probably true. But nonetheless I do like having these settings to ensure stability.

I see gigabit maxed out all of the time. Youtube, Netflix, and Hulu microburst their ~250KiB chunks at 1Gb/s. Packet sniff these TCP connections and I see back-to-back 1500 byte frames for about 2ms at a time. That's for steady state. I technically only have a 150Mb connection, but it's a 1Gb link that is policed to 150Mb. If I keep jumping around the video timelines, I can keep the video stream in a perma-buffering state, which attempts to send at full 1Gb/s. I can see 1Gb/s for a bout the first 100ms or so before the policer starts ramping up. That could represent a 100ms burst in latency if it was not for my ISP's AQM plus my HFSC shaping.

dennypage

https://github.com/pfsense/pfsense/pull/3941

And happiness ensued…

:)

gsmornot

@dennypage:

https://github.com/pfsense/pfsense/pull/3941

And happiness ensued…

:)

Fired up about it. Looking forward to this addition.

strangegopher

;D ;D ;D ;D

tman222

Great news!!!

I wonder if it's a good idea to clear the current fq_codel config before upgrading to 2.4.4 and rebuild it using the GUI after the upgrade? Otherwise, maybe things might not upgrade correctly? Does anyone have any thoughts on that?

Thanks in advance.

matt_

@gsmornot:

@dennypage:

https://github.com/pfsense/pfsense/pull/3941

And happiness ensued…

:)

Fired up about it. Looking forward to this addition.

Me too! I submitted that PR after working on it this week, actually got the idea from reading this thread. I'm testing it on my own right now; I have multi-LAN and multi-WAN, and I needed something to be able to classify traffic coming from WAN A / WAN B out LAN A/B and WAN B out the same LAN paths. Dummynet works awesome for this, and I've gone from a Bufferbloat score of C to A with the patch. If you want to load the patch, you just have to make a queue under the limiter, and assign it with a floating rule on the WAN interface (out direction).

Gripes about Traffic Shaper right now that the Limiter PR seems to help out with:

Not enough parameter configuration for FQ_CODEL on the GUI (default FQ_CODEL target is 5000 on my install)
Couldn't get traffic shaper (altq) to bind to LAGG interfaces
Couldn't get traffic shaper (altq) to work with a multi-LAN setup for download classification
PIE support

For those of you interested, here's a SS: https://i.imgur.com/N36gpXF.png

If anyone has any suggestions for changes or notices any problems, I'd be happy to factor them into my fork (and therefore the PR). Full disclosure, I am by no means an expert on dummynet/ipfw. I just have a lot of free time on my hands…

dennypage

Really appreciate the work Matt!

Harvy66

@matt, why is the interval so large? The interval should be roughly equal to your upper typical RTT. 100,000ms is a pretty big RTT.

matt_

@Harvy66:

@matt, why is the interval so large? The interval should be roughly equal to your upper typical RTT. 100,000ms is a pretty big RTT.

Not sure. I'm loading these params in from sysctl upon save, so it's set to whatever is default. I did think that was odd, also. Maybe I have the units wrong and```
target


EDIT:

You're exactly right on this!  I make a sysctl call to load in defaults but it seems the units used on the sysctls are definitely not milliseconds!  Looks like microseconds to me.  I'm fixing the PR.  Also, I've got a patch for 2.4.3 as well on my repository which will get this change, too.

EDIT 2:

All fixed!  And, here is the applicable DIFF file for the repository should anyone want to load the patch into their 2.4.3 install.  I run pfSense at home, on the stable release, so I'm using this branch and copying changes to the PR (which is ahead in terms of code).  Definitely don't use the PR's diff on a stable 2.4.3 install, since they've got some wonky PHP 7 stuff going on that causes some really weird behavior in the whole shaper module.  This diff doesn't have those changes (main reason for two heads)

https://github.com/pfsense/pfsense/compare/RELENG_2_4_3…mattund:RELENG_2_4_3.diff

If they accept this PR after reviewing it -- and I can almost guarantee they'll have critique -- I'll turn my attention to Limiter Info and work on a PR for that, too.  IMHO we desperately need a better status page for dummynet if FQ_CODEL/FQ_PIE are gonna be in use more.

grandrivers

won't lie I am super excited about this !!!!!!!

w0w

Applied this diff https://github.com/pfsense/pfsense/compare/RELENG_2_4_3…mattund:RELENG_2_4_3.diff
Adding new queue by pressing button on limiter settings page opens new page and you can add new queue save and apply it, but it does not appear in the list and I don't see it anywhere. But it can be only in my case, just because I have some April version of 2.4.4, before PHP7 preparations were made.

matt_

That is really strange. Do you still have your old shaper.inc, or could revert back to it? Perhaps what commit ID you were at for your 2.4.4 install? I can try to factor the changes in appropriately for you.

Although, just a guess, if it was pre-PHP7 compliance it might be more likely that the 2.4.4 code has a bug somewhere. Can't be sure, though.

w0w

I will do some tests later this week, but if this bug only applicable to my 20 April snapshot then it's not a big problem. I just waiting they fix all broken php7 ready code, thats why I am sitting on this april version. I want to say thank you for the great work you have done!

w0w

So I have found what was the problem — I have been applied this patch on already configured limiters and queues. Now I removed all queues, leaved limiters and applied patch again and then recreated queues — all went OK.
It looks like we have problems with parsing old config? Tried also latest 2.4.4 and behavior is the same.

BTW, That's what I'm trying to achieve https://forum.pfsense.org/index.php?topic=63531.0

matt_

That may require more testing from me, then, if it is ignoring or otherwise not parsing those limiters once the patch is applied. I'll have to take a look some time

w0w

Config.xml parts with ipfw configuration
Original, before applying patch:


 <dnshaper><queue><name>Upload</name>
			<number>1</number>

			 <bandwidth><bw>265576</bw>

					<bwscale>Kb</bwscale>
					<bwsched>none</bwsched></bandwidth> 
			<enabled>on</enabled>

			<mask>none</mask>
			<maskbits></maskbits>
			<maskbitsv6></maskbitsv6>
			<delay>0</delay>
			 <queue><name>UploadQueue</name>
				<number>1</number>

				<enabled>on</enabled>

				<mask>srcaddress</mask>
				<maskbits>32</maskbits>
				<maskbitsv6>128</maskbitsv6></queue></queue> 
		 <queue><name>Download</name>
			<number>2</number>

			 <bandwidth><bw>275576</bw>

					<bwscale>Kb</bwscale>
					<bwsched>none</bwsched></bandwidth> 
			<enabled>on</enabled>

			<mask>none</mask>
			<maskbits></maskbits>
			<maskbitsv6></maskbitsv6>
			<delay>0</delay>
			 <queue><name>DownloadQueue</name>
				<number>2</number>

				<enabled>on</enabled>

				<mask>dstaddress</mask>
				<maskbits>32</maskbits>
				<maskbitsv6>128</maskbitsv6></queue></queue></dnshaper>

Original after applying patch:

 <dnshaper><queue><name>Upload</name>
			<number>1</number>

			 <bandwidth><bw>265576</bw>

					<bwscale>Kb</bwscale>
					<bwsched>none</bwsched></bandwidth> 
			<enabled>on</enabled>

			<mask>none</mask>
			<maskbits></maskbits>
			<maskbitsv6></maskbitsv6>
			<delay>0</delay>
			 <queue><name>UploadQueue</name>
				<number>1</number>

				<enabled>on</enabled>

				<mask>srcaddress</mask>
				<maskbits>32</maskbits>
				<maskbitsv6>128</maskbitsv6></queue> 
			<sched>fq_codel</sched>
			<param_fq_codel_target>5</param_fq_codel_target>
			<param_fq_codel_interval>100</param_fq_codel_interval>
			<param_fq_codel_quantum>1514</param_fq_codel_quantum>
			<param_fq_codel_limit>10240</param_fq_codel_limit>
			<param_fq_codel_flows>1024</param_fq_codel_flows>
			<aqm>codel</aqm>
			<param_codel_target>5</param_codel_target>
			<param_codel_interval>100</param_codel_interval>
			<ecn>on</ecn></queue> 
		 <queue><name>Download</name>
			<number>2</number>

			 <bandwidth><bw>275576</bw>

					<bwscale>Kb</bwscale>
					<bwsched>none</bwsched></bandwidth> 
			<enabled>on</enabled>

			<mask>none</mask>
			<maskbits></maskbits>
			<maskbitsv6></maskbitsv6>
			<delay>0</delay>
			 <queue><name>DownloadQueue</name>
				<number>2</number>

				<enabled>on</enabled>

				<mask>dstaddress</mask>
				<maskbits>32</maskbits>
				<maskbitsv6>128</maskbitsv6></queue> 
			<sched>fq_codel</sched>
			<param_fq_codel_target>5</param_fq_codel_target>
			<param_fq_codel_interval>100</param_fq_codel_interval>
			<param_fq_codel_quantum>1514</param_fq_codel_quantum>
			<param_fq_codel_limit>10240</param_fq_codel_limit>
			<param_fq_codel_flows>1024</param_fq_codel_flows>
			<aqm>codel</aqm>
			<param_codel_target>5</param_codel_target>
			<param_codel_interval>100</param_codel_interval>
			<ecn>on</ecn></queue></dnshaper>

New fully working:

	 <dnshaper><queue><name>Upload</name>
			<number>1</number>

			 <bandwidth><bw>265576</bw>

					<bwscale>Kb</bwscale>
					<bwsched>none</bwsched></bandwidth> 
			<enabled>on</enabled>

			<mask>none</mask>
			<maskbits></maskbits>
			<maskbitsv6></maskbitsv6>
			<delay>0</delay>
			<sched>fq_codel</sched>
			<param_fq_codel_target>5</param_fq_codel_target>
			<param_fq_codel_interval>100</param_fq_codel_interval>
			<param_fq_codel_quantum>1514</param_fq_codel_quantum>
			<param_fq_codel_limit>10240</param_fq_codel_limit>
			<param_fq_codel_flows>1024</param_fq_codel_flows>
			<aqm>codel</aqm>
			<param_codel_target>5</param_codel_target>
			<param_codel_interval>100</param_codel_interval>
			<ecn>on</ecn>
			 <queue><name>Upload_Q</name>
				<number>1</number>

				<enabled>on</enabled>

				<mask>srcaddress</mask>
				<maskbits>32</maskbits>
				<maskbitsv6>128</maskbitsv6>
				<aqm>pie</aqm>

				<param_pie_target>15</param_pie_target>
				<param_pie_tupdate>15</param_pie_tupdate>
				<param_pie_alpha>125</param_pie_alpha>
				<param_pie_beta>1250</param_pie_beta>
				<param_pie_max_burst>150000</param_pie_max_burst>
				<param_pie_max_ecnth>99</param_pie_max_ecnth></queue></queue> 
		 <queue><name>Download</name>
			<number>2</number>

			 <bandwidth><bw>275576</bw>

					<bwscale>Kb</bwscale>
					<bwsched>none</bwsched></bandwidth> 
			<enabled>on</enabled>

			<mask>none</mask>
			<maskbits></maskbits>
			<maskbitsv6></maskbitsv6>
			<delay>0</delay>
			<sched>fq_codel</sched>
			<param_fq_codel_target>5</param_fq_codel_target>
			<param_fq_codel_interval>100</param_fq_codel_interval>
			<param_fq_codel_quantum>1514</param_fq_codel_quantum>
			<param_fq_codel_limit>10240</param_fq_codel_limit>
			<param_fq_codel_flows>1024</param_fq_codel_flows>
			<aqm>codel</aqm>
			<param_codel_target>5</param_codel_target>
			<param_codel_interval>100</param_codel_interval>
			<ecn>on</ecn>
			 <queue><name>Download_Q</name>
				<number>2</number>

				<enabled>on</enabled>

				<mask>dstaddress</mask>
				<maskbits>32</maskbits>
				<maskbitsv6>128</maskbitsv6>
				<aqm>pie</aqm>

				<param_pie_target>15</param_pie_target>
				<param_pie_tupdate>15</param_pie_tupdate>
				<param_pie_alpha>125</param_pie_alpha>
				<param_pie_beta>1250</param_pie_beta>
				<param_pie_max_burst>150000</param_pie_max_burst>
				<param_pie_max_ecnth>99</param_pie_max_ecnth></queue></queue></dnshaper>

strangegopher

any progress on fixing traceroute?