Playing with fq_codel in 2.4

xciter327

You are supposed to set the limiters lower than your actual connection speed. I set mine to 95% of tested(not ISP advertised) speed.

Also please use the guide posted a little while earlier. The one from the hangouts session is outdated and does not cover a bug workaround.

https://forum.netgate.com/post/807490

P.S - I am also on Ziggo NL 240/24 connection.

Veldkornet

@xciter327 Even at 95% of 400/40 which is 380/38, the upload speed is nowhere near that. The download at least comes in the vicinity.

csutcliff

@Veldkornet I don't suppose it's a PPPoE connection? I had a similar problem.

maverick_slo

Hi how did you resolve pppoe problem?
Thanks!

csutcliff

@maverick_slo I didn't. I am only able to use the Download shaper for my PPPoE WAN. It works perfectly on my other WAN which is DHCP.

Veldkornet

@csutcliff Nope, mine is just a DHCPv4 connection

xciter327

@Veldkornet said in Playing with fq_codel in 2.4:

@xciter327 Even at 95% of 400/40 which is 380/38, the upload speed is nowhere near that. The download at least comes in the vicinity.

Hm. That should not be. Did You follow the guide I linked? Also what hardware is the pfsense running on? Limiters add extra CPU usage. Take a look with "htop -d 1" (need to install it first "pkg install htop") and see if You are peaking the CPU while doing the test.

Veldkornet

@xciter327 ah! I didn't see the guide. I've now re-created everything as per the guide... although it didn't change the results...

I have a PCEngines APU2. I just had a look in "top" while doing the tests. Didn't really see anything climb very high at all. Even then, if it can handle the download, it should be able to manage the upload which is 10% of the download.

If I watch the test, the upload starts strong and climbs to around 20Mbit quickly, for 2 seconds or so, but then drops down to around 4/5 for the remainder of the test where it eventually finishes.

xciter327

Could You perhaps post some picture of your firewall rules and limiter config? I am shaping my Ziggo connection on a Zotac NUC, which theoretically should be less powerful than the APU2. Also make sure You clear the states/reset the firewall when applying the limiters. Keep an eye in the system log for any log messages when You apply the limiters as well.

Veldkornet

I can post screenshots instead of the below.. but it will take lots of space. I just checked everything though, and except for the speed limits, to me it looks the same as in the post. Might as well just be a copy paste. Looking at the floating rule screenshot though, I see that the WAN-In FQ-CoDel queue is pretty small considering all the tests I was doing... normal?

Also, see how the upload just dies off:

FQ_CODEL_OUT

Name: FQ_CODEL_OUT
Bandwidth: 38 Mbit/s
Mask: None
Queue Management Algorithm: Tail Drop
Scheduler: FQ_CODEL
target: 5
interval: 100
quantum: 300
limit: 10240
flows: 20480

fq_codel_out_q

Name: fq_codel_out_q
Mask: None
Queue Management Algorithm: Tail Drop

FQ_CODEL_IN

Name: FQ_CODEL_IN
Bandwidth: 380 Mbit/s
Mask: None
Queue Management Algorithm: Tail Drop
Scheduler: FQ_CODEL
target: 5
interval: 100
quantum: 300
limit: 10240
flows: 20480

fq_codel_in_q

Name: fq_codel_in_q
Mask: None
Queue Management Algorithm: Tail Drop

Firewall Rules - Floating:

policy routing traceroute workaround

Action: Pass
Quick: Tick Apply the action immediately on match.
Interface: WAN
Direction: out
Address Family: IPv4
Protocol: ICMP
ICMP subtypes: Traceroute
Source: any
Destination: any
Description: policy routing traceroute workaround

limiter drop echo-reply under load workaround

Action: Pass
Quick: Tick Apply the action immediately on match.
Interface: WAN
Direction: any
Address Family: IPv4
Protocol: ICMP
ICMP subtypes: Echo reply, Echo Request
Source: any
Destination: any
Description: limiter drop echo-reply under load workaround

WAN-In FQ-CoDel queue

Action: Match
Interface: WAN
Direction: in
Address Family: IPv4
Protocol: Any
Source: any
Destination: any
Description: WAN-In FQ-CoDel queue
Gateway: Default
In / Out pipe: fq_codel_in_q / fq_codel_out_q

WAN-Out FQ-CoDel queue

Action: Match
Interface: WAN
Direction: out
Address Family: IPv4
Protocol: Any
Source: any
Destination: any
Description: WAN-Out FQ-CoDel queue
Gateway: WAN_DHCP
In / Out pipe: fq_codel_out_q / fq_codel_in_q

xciter327

Looks good to me. Mine at home is the same, with lower speeds tough. When You do the dslreports test, You can open up htop(prefer it because it's easier to deal with multiple cores) in one window and "ipfw sched show" in another to see if the limiters are actually matching traffic. Anything else on this box(like squid or snort)?

Veldkornet

@xciter327 said in Playing with fq_codel in 2.4:

Looks good to me. Mine at home is the same, with lower speeds tough. When You do the dslreports test, You can open up htop(prefer it because it's easier to deal with multiple cores) in one window and "ipfw sched show" in another to see if the limiters are actually matching traffic. Anything else on this box(like squid or snort)?

Well I have Suricata, no squid. Although I turned Suricata off and it made no difference.

Download maxes the CPU, but upload doesn't seem to do much...
Download:

00001:  38.000 Mbit/s    0 ms burst 0
q65537  50 sl. 0 flows (1 buckets) sched 1 weight 0 lmax 0 pri 0 droptail
 sched 1 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 100ms quantum 300 limit 10240 flows 20480 NoECN
   Children flowsets: 1
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
  0 ip           0.0.0.0/0             0.0.0.0/0     2478   112953 488 26508   0
00002: 380.000 Mbit/s    0 ms burst 0
q65538  50 sl. 0 flows (1 buckets) sched 2 weight 0 lmax 0 pri 0 droptail
 sched 2 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 100ms quantum 300 limit 10240 flows 20480 NoECN
   Children flowsets: 2
  0 ip           0.0.0.0/0             0.0.0.0/0     33141 49232048 193 287300  12

Upload:

00001:  38.000 Mbit/s    0 ms burst 0
q65537  50 sl. 0 flows (1 buckets) sched 1 weight 0 lmax 0 pri 0 droptail
 sched 1 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 100ms quantum 300 limit 10240 flows 20480 NoECN
   Children flowsets: 1
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
  0 ip           0.0.0.0/0             0.0.0.0/0       31    45860  0    0   0
00002: 380.000 Mbit/s    0 ms burst 0
q65538  50 sl. 0 flows (1 buckets) sched 2 weight 0 lmax 0 pri 0 droptail
 sched 2 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 100ms quantum 300 limit 10240 flows 20480 NoECN
   Children flowsets: 2
  0 ip           0.0.0.0/0             0.0.0.0/0       16      664  0    0   0

xciter327

Hm, I don't think You will have a great experience with all those thing's You've loaded on that little box. I read somewhere the APU2 is good for ~400Mpbs without Suricata/other heavy software.

I would try disabling all add-on(haProxy, Openvpn, pfblocker, Suricata, snmp, fancy unbound settings, tftp-server etc.) and try vanilla pfsense with just the limiters(via floating rules) and a simple "Allow all" on the LAN side. If your CPU is peaked(like You have on the download test), then better run without limiters.

Veldkornet

Oh? The CPU load on it is almost non-existant usually. This is the first time I've seen it go so high now with the traffic shaping. Even so, I can understand that the load may be an issue for download and I'm okay with that.

It's the upload speed that's annoying me...... which is not doing much on the CPU side.

xciter327

Well If I look at the picture, the upload is mostly pegged by Surricata. So that would be the first thing I disable. Also note that not all packages work normally with limiters. There used to be issues with Suricata/Squid and limiters, I don't know if it was ever fixed.

Veldkornet

Okay, I disabled all of the packages, ran the test, upload good.
Then I enabled each one and tested until the upload speed decreased.

So it seems I have discovered the culprit... I have an OpenVPN Client setup to PIA, this is what's causing the bad upload speeds. I have an alias with an IP range defined, and basically everything within that alias should go over the VPN. And then I have an interface & gateway which is bound to the OpenVPN client; with a Firewall rule which says anything to that alias should use the VPN gateway.

Is there anything special that I would need to do with my rules in this situation? The traffic itself isn’t going over the VPN.

xciter327

My guess would be to place it at the very top of the floating rule set. I believe pf does "match most specific" unless "quick match" is selected.

uptownVagrant

@xciter327 said in Playing with fq_codel in 2.4:

My guess would be to place it at the very top of the floating rule set. I believe pf does "match most specific" unless "quick match" is selected.

pf is last match wins unless quick is enabled. Quick is enabled by default on interface rules but has to be selected on floating rules

xciter327

Good to hear. I did not know about the default quick on interface rules. That explains some things.

w0w

I've rebuilt my network a bit, but I am not sure when this was happened — one PC behind pfSense always get the A or A+ rating and the other get the D or even lower grade rating on dslreports, both are on the same LAN. FQ_CODEL configuration is similar to the others in this thread except only that it is applied on the LAN side (LAN to any). When I disable limiters, then both PCs get A or A+ rating. Does anyone have an idea, what is going wrong?