Playing with fq_codel in 2.4

csutcliff

@robnitro Thanks for the suggestion, It has seemingly improved my cable connection responsiveness slightly but doesn't make a difference to the upload problem on vdsl.

lightsy

I wanted to give a brief update - I actually am here about the issues with traceroutes showing the default destination addresses for every hop, caused by the Floating Firewall rule.

I fixed this on my side, initially, by only having the rule shape TCP/UDP - but some other clients do traceroute via UDP so I had this same issue. Instead, there's a firewall option to limit/scope it down to particular TCP flags - you can set the ones you wish, but I ultimately ended up targeting it to "Any" protocol, and "Any flags" - this fixed it for me. :)

EDIT: My pfSense randomly crashed... and... it rebooted... and... somehow now it isn't working.
Sigh. Guess I'm back to applying this to a non-floating rule. :(

robnitro

Hi, what rule did you use for non floating?
When I do tracert from windows, it shows only 2 hops. Router and then the destination.

From openwrt dumb access point, all hops show fine like From the router I get all of the hops. Floating rule instant match allow any IPV4 ICMP trace - and another rule icmp any at top of list. So that means it should skip using the codel limiters right? I just dont understand why my desktop client gets bad tracert info?

lightsy

@robnitro

i actually just reverted to using a simple TCP/UDP floating rule, setting up the non-floating was a huge pain (I had it working before) but can't remember how.

I'm going to be creating exemptions / rules that exempt specific clients in the future - now that it's working fine (on my primary use case, Windows!)

Here is the exact rule I have (that is, somehow, letting pfSense UDP traceroute work - I am tired of messing with it / don't care). Anything that is missing means it's default.

Action: Pass
Interface: WAN
Direction: Out
Addr Family: IPv4
Protocol: TCP/UDP
Gateway: WAN_DHCP (my gateway selected)
In/Out Pipe: WANUpQ (my name) & WANDownQ (my name)

Here are the rules I have:
Firewall Rules > Traffic Shaper > "Limiters"

"CODEL_QMDown"
Limiter: Enabled
Bandwidth: 320 Mbit
Queue Management Algorithm: CoDel
Scheduler: FQ_CODEL

Subqueue, "WANDownQ"
Queue algorithm: CoDel

"CODEL_QMUp"
Limiter: Enabled
Bandwidth: 340Mbit (yes, I have higher upload than download)
Queue management algorithm: CoDel
Scheduler: FQ_CODEL

Subqueue, "WANUpQ"
Queue algorithm: CoDel

I know these are basically defaults - I'm not sure why these work? I simply re-created everything, and now everything works great... it's quite odd. I'm curious if there is an issue with pfSense and having the rules edited, or large changes made to the queues (I hit Save/Apply, particularly, after every single change / creation - no going back and editing for me!)

Quite odd. I've used FQ_CoDel in other devices/implementations and have never run into these nuances. It's working great, now, though - but the traceroute/etc was bothering me horribly. With these rules (no floating rule for 'any') it seems to be working great... for... whatever reason.

maverick_slo

Hi all!
Configured limiters and now my traceroute always shows resolved IP address instead of actual hops.
I`ve read here this is common but just to be sure, is this supposed to be like this or is it a bug?

Thanks!

Veldkornet

Hi guys,

I’m also having an issue with my upload speed... So I have a 400Mbit down / 40 Mbit up connection, and I've followed the guide here to enable the limiters, filling in the 400 and 40 as my down and up speeds.

However when I do speedtests, while my bufferbloat is now gone... I no can no longer reach my maximum speeds.
Down I may get around 360Mbit max which I can live with butup is just pathetic, maybe between 2-8Mbit at most....

Is this known? Anything I can do to improve this? I played with the Queue Length value which helped a bit, but in the end didn't really have too much of an impact....

Limiters:
00001: 400.000 Mbit/s    0 ms burst 0 
q131073 10000 sl. 0 flows (1 buckets) sched 65537 weight 0 lmax 0 pri 0  AQM CoDel target 5ms interval 100ms ECN
 sched 65537 type FIFO flags 0x0 0 buckets 0 active
00002:  40.000 Mbit/s    0 ms burst 0 
q131074 1000 sl. 0 flows (1 buckets) sched 65538 weight 0 lmax 0 pri 0  AQM CoDel target 5ms interval 100ms ECN
 sched 65538 type FIFO flags 0x0 0 buckets 0 active


Schedulers:
00001: 400.000 Mbit/s    0 ms burst 0 
q65537  50 sl. 0 flows (1 buckets) sched 1 weight 0 lmax 0 pri 0 droptail
 sched 1 type FQ_CODEL flags 0x0 0 buckets 0 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 1 
00002:  40.000 Mbit/s    0 ms burst 0 
q65538  50 sl. 0 flows (1 buckets) sched 2 weight 0 lmax 0 pri 0 droptail
 sched 2 type FQ_CODEL flags 0x0 0 buckets 0 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 2 


Queues:
q00001  50 sl. 0 flows (1 buckets) sched 1 weight 0 lmax 0 pri 0  AQM CoDel target 5ms interval 100ms ECN
q00002  50 sl. 0 flows (1 buckets) sched 2 weight 0 lmax 0 pri 0  AQM CoDel target 5ms interval 100ms ECN

Disabled:

Enabled:

xciter327

You are supposed to set the limiters lower than your actual connection speed. I set mine to 95% of tested(not ISP advertised) speed.

Also please use the guide posted a little while earlier. The one from the hangouts session is outdated and does not cover a bug workaround.

https://forum.netgate.com/post/807490

P.S - I am also on Ziggo NL 240/24 connection.

Veldkornet

@xciter327 Even at 95% of 400/40 which is 380/38, the upload speed is nowhere near that. The download at least comes in the vicinity.

csutcliff

@Veldkornet I don't suppose it's a PPPoE connection? I had a similar problem.

maverick_slo

Hi how did you resolve pppoe problem?
Thanks!

csutcliff

@maverick_slo I didn't. I am only able to use the Download shaper for my PPPoE WAN. It works perfectly on my other WAN which is DHCP.

Veldkornet

@csutcliff Nope, mine is just a DHCPv4 connection

xciter327

@Veldkornet said in Playing with fq_codel in 2.4:

@xciter327 Even at 95% of 400/40 which is 380/38, the upload speed is nowhere near that. The download at least comes in the vicinity.

Hm. That should not be. Did You follow the guide I linked? Also what hardware is the pfsense running on? Limiters add extra CPU usage. Take a look with "htop -d 1" (need to install it first "pkg install htop") and see if You are peaking the CPU while doing the test.

Veldkornet

@xciter327 ah! I didn't see the guide. I've now re-created everything as per the guide... although it didn't change the results...

I have a PCEngines APU2. I just had a look in "top" while doing the tests. Didn't really see anything climb very high at all. Even then, if it can handle the download, it should be able to manage the upload which is 10% of the download.

If I watch the test, the upload starts strong and climbs to around 20Mbit quickly, for 2 seconds or so, but then drops down to around 4/5 for the remainder of the test where it eventually finishes.

xciter327

Could You perhaps post some picture of your firewall rules and limiter config? I am shaping my Ziggo connection on a Zotac NUC, which theoretically should be less powerful than the APU2. Also make sure You clear the states/reset the firewall when applying the limiters. Keep an eye in the system log for any log messages when You apply the limiters as well.

Veldkornet

I can post screenshots instead of the below.. but it will take lots of space. I just checked everything though, and except for the speed limits, to me it looks the same as in the post. Might as well just be a copy paste. Looking at the floating rule screenshot though, I see that the WAN-In FQ-CoDel queue is pretty small considering all the tests I was doing... normal?

Also, see how the upload just dies off:

FQ_CODEL_OUT

Name: FQ_CODEL_OUT
Bandwidth: 38 Mbit/s
Mask: None
Queue Management Algorithm: Tail Drop
Scheduler: FQ_CODEL
target: 5
interval: 100
quantum: 300
limit: 10240
flows: 20480

fq_codel_out_q

Name: fq_codel_out_q
Mask: None
Queue Management Algorithm: Tail Drop

FQ_CODEL_IN

Name: FQ_CODEL_IN
Bandwidth: 380 Mbit/s
Mask: None
Queue Management Algorithm: Tail Drop
Scheduler: FQ_CODEL
target: 5
interval: 100
quantum: 300
limit: 10240
flows: 20480

fq_codel_in_q

Name: fq_codel_in_q
Mask: None
Queue Management Algorithm: Tail Drop

Firewall Rules - Floating:

policy routing traceroute workaround

Action: Pass
Quick: Tick Apply the action immediately on match.
Interface: WAN
Direction: out
Address Family: IPv4
Protocol: ICMP
ICMP subtypes: Traceroute
Source: any
Destination: any
Description: policy routing traceroute workaround

limiter drop echo-reply under load workaround

Action: Pass
Quick: Tick Apply the action immediately on match.
Interface: WAN
Direction: any
Address Family: IPv4
Protocol: ICMP
ICMP subtypes: Echo reply, Echo Request
Source: any
Destination: any
Description: limiter drop echo-reply under load workaround

WAN-In FQ-CoDel queue

Action: Match
Interface: WAN
Direction: in
Address Family: IPv4
Protocol: Any
Source: any
Destination: any
Description: WAN-In FQ-CoDel queue
Gateway: Default
In / Out pipe: fq_codel_in_q / fq_codel_out_q

WAN-Out FQ-CoDel queue

Action: Match
Interface: WAN
Direction: out
Address Family: IPv4
Protocol: Any
Source: any
Destination: any
Description: WAN-Out FQ-CoDel queue
Gateway: WAN_DHCP
In / Out pipe: fq_codel_out_q / fq_codel_in_q

xciter327

Looks good to me. Mine at home is the same, with lower speeds tough. When You do the dslreports test, You can open up htop(prefer it because it's easier to deal with multiple cores) in one window and "ipfw sched show" in another to see if the limiters are actually matching traffic. Anything else on this box(like squid or snort)?

Veldkornet

@xciter327 said in Playing with fq_codel in 2.4:

Looks good to me. Mine at home is the same, with lower speeds tough. When You do the dslreports test, You can open up htop(prefer it because it's easier to deal with multiple cores) in one window and "ipfw sched show" in another to see if the limiters are actually matching traffic. Anything else on this box(like squid or snort)?

Well I have Suricata, no squid. Although I turned Suricata off and it made no difference.

Download maxes the CPU, but upload doesn't seem to do much...
Download:

00001:  38.000 Mbit/s    0 ms burst 0
q65537  50 sl. 0 flows (1 buckets) sched 1 weight 0 lmax 0 pri 0 droptail
 sched 1 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 100ms quantum 300 limit 10240 flows 20480 NoECN
   Children flowsets: 1
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
  0 ip           0.0.0.0/0             0.0.0.0/0     2478   112953 488 26508   0
00002: 380.000 Mbit/s    0 ms burst 0
q65538  50 sl. 0 flows (1 buckets) sched 2 weight 0 lmax 0 pri 0 droptail
 sched 2 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 100ms quantum 300 limit 10240 flows 20480 NoECN
   Children flowsets: 2
  0 ip           0.0.0.0/0             0.0.0.0/0     33141 49232048 193 287300  12

Upload:

00001:  38.000 Mbit/s    0 ms burst 0
q65537  50 sl. 0 flows (1 buckets) sched 1 weight 0 lmax 0 pri 0 droptail
 sched 1 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 100ms quantum 300 limit 10240 flows 20480 NoECN
   Children flowsets: 1
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
  0 ip           0.0.0.0/0             0.0.0.0/0       31    45860  0    0   0
00002: 380.000 Mbit/s    0 ms burst 0
q65538  50 sl. 0 flows (1 buckets) sched 2 weight 0 lmax 0 pri 0 droptail
 sched 2 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 100ms quantum 300 limit 10240 flows 20480 NoECN
   Children flowsets: 2
  0 ip           0.0.0.0/0             0.0.0.0/0       16      664  0    0   0

xciter327

Hm, I don't think You will have a great experience with all those thing's You've loaded on that little box. I read somewhere the APU2 is good for ~400Mpbs without Suricata/other heavy software.

I would try disabling all add-on(haProxy, Openvpn, pfblocker, Suricata, snmp, fancy unbound settings, tftp-server etc.) and try vanilla pfsense with just the limiters(via floating rules) and a simple "Allow all" on the LAN side. If your CPU is peaked(like You have on the download test), then better run without limiters.

Veldkornet

Oh? The CPU load on it is almost non-existant usually. This is the first time I've seen it go so high now with the traffic shaping. Even so, I can understand that the load may be an issue for download and I'm okay with that.

It's the upload speed that's annoying me...... which is not doing much on the CPU side.