Playing with fq_codel in 2.4

[A Former User]

Hi @tman222 thanks for the reply.

I am running pfSense virtualised on a Proxmox host.
I am using vtnet as my interfaces, they report themselves as 10Gb interfaces.

The only plugins I had active at the time were avahci and openvpn-exporter.

I'm unsure how to answer your question about queue sizes, I have included the output of a couple of commands in the hope they might capture the answer you're after?

hw.vtnet.rx_process_limit: 512
hw.vtnet.mq_max_pairs: 8
hw.vtnet.mq_disable: 0
hw.vtnet.lro_disable: 0
hw.vtnet.tso_disable: 0
hw.vtnet.csum_disable: 0
dev.vtnet.1.txq0.rescheduled: 0
dev.vtnet.1.txq0.tso: 0
dev.vtnet.1.txq0.csum: 0
dev.vtnet.1.txq0.omcasts: 2765830
dev.vtnet.1.txq0.obytes: 785586398380
dev.vtnet.1.txq0.opackets: 708668034
dev.vtnet.1.rxq0.rescheduled: 0
dev.vtnet.1.rxq0.csum_failed: 0
dev.vtnet.1.rxq0.csum: 198931285
dev.vtnet.1.rxq0.ierrors: 0
dev.vtnet.1.rxq0.iqdrops: 0
dev.vtnet.1.rxq0.ibytes: 116215930004
dev.vtnet.1.rxq0.ipackets: 482511639
dev.vtnet.1.tx_task_rescheduled: 0
dev.vtnet.1.tx_tso_offloaded: 0
dev.vtnet.1.tx_csum_offloaded: 0
dev.vtnet.1.tx_defrag_failed: 0
dev.vtnet.1.tx_defragged: 0
dev.vtnet.1.tx_tso_not_tcp: 0
dev.vtnet.1.tx_tso_bad_ethtype: 0
dev.vtnet.1.tx_csum_bad_ethtype: 0
dev.vtnet.1.rx_task_rescheduled: 0
dev.vtnet.1.rx_csum_offloaded: 0
dev.vtnet.1.rx_csum_failed: 0
dev.vtnet.1.rx_csum_bad_proto: 0
dev.vtnet.1.rx_csum_bad_offset: 0
dev.vtnet.1.rx_csum_bad_ipproto: 0
dev.vtnet.1.rx_csum_bad_ethtype: 0
dev.vtnet.1.rx_mergeable_failed: 0
dev.vtnet.1.rx_enq_replacement_failed: 0
dev.vtnet.1.rx_frame_too_large: 0
dev.vtnet.1.mbuf_alloc_failed: 0
dev.vtnet.1.act_vq_pairs: 1
dev.vtnet.1.requested_vq_pairs: 0
dev.vtnet.1.max_vq_pairs: 1
dev.vtnet.1.%parent: virtio_pci3
dev.vtnet.1.%pnpinfo:
dev.vtnet.1.%location:
dev.vtnet.1.%driver: vtnet
dev.vtnet.1.%desc: VirtIO Networking Adapter


[2.4.3-RELEASE][admin@x.x.x]/root: ifconfig vtnet1
vtnet1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=6c00b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 1a:b3:6c:0f:3c:61
        hwaddr 1a:b3:6c:0f:3c:61
        inet6 fe80::18b3:6cff:fe0f:3c61%vtnet1 prefixlen 64 scopeid 0x2
        inet 192.168.0.1 netmask 0xffffff00 broadcast 192.168.0.255
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet 10Gbase-T <full-duplex>
        status: active

[mattund]

https://github.com/pfsense/pfsense/pull/3941

The PR was just merged into HEAD, it hopefully will make into the next release.

[A Former User]

@mattund said in Playing with fq_codel in 2.4:

https://github.com/pfsense/pfsense/pull/3941

The PR was just merged into HEAD, it hopefully will make into the next release.

This is great news! Thanks so much for your hard work here @mattund !

[dennypage]

@mattund said in Playing with fq_codel in 2.4:

https://github.com/pfsense/pfsense/pull/3941

The PR was just merged into HEAD, it hopefully will make into the next release.

Congratulations Matt. Great contribution!

[tman222]

@muppet said in Playing with fq_codel in 2.4:

Hi @tman222 thanks for the reply.

I am running pfSense virtualised on a Proxmox host.
I am using vtnet as my interfaces, they report themselves as 10Gb interfaces.

The only plugins I had active at the time were avahci and openvpn-exporter.

I'm unsure how to answer your question about queue sizes, I have included the output of a couple of commands in the hope they might capture the answer you're after?

hw.vtnet.rx_process_limit: 512
hw.vtnet.mq_max_pairs: 8
hw.vtnet.mq_disable: 0
hw.vtnet.lro_disable: 0
hw.vtnet.tso_disable: 0
hw.vtnet.csum_disable: 0
dev.vtnet.1.txq0.rescheduled: 0
dev.vtnet.1.txq0.tso: 0
dev.vtnet.1.txq0.csum: 0
dev.vtnet.1.txq0.omcasts: 2765830
dev.vtnet.1.txq0.obytes: 785586398380
dev.vtnet.1.txq0.opackets: 708668034
dev.vtnet.1.rxq0.rescheduled: 0
dev.vtnet.1.rxq0.csum_failed: 0
dev.vtnet.1.rxq0.csum: 198931285
dev.vtnet.1.rxq0.ierrors: 0
dev.vtnet.1.rxq0.iqdrops: 0
dev.vtnet.1.rxq0.ibytes: 116215930004
dev.vtnet.1.rxq0.ipackets: 482511639
dev.vtnet.1.tx_task_rescheduled: 0
dev.vtnet.1.tx_tso_offloaded: 0
dev.vtnet.1.tx_csum_offloaded: 0
dev.vtnet.1.tx_defrag_failed: 0
dev.vtnet.1.tx_defragged: 0
dev.vtnet.1.tx_tso_not_tcp: 0
dev.vtnet.1.tx_tso_bad_ethtype: 0
dev.vtnet.1.tx_csum_bad_ethtype: 0
dev.vtnet.1.rx_task_rescheduled: 0
dev.vtnet.1.rx_csum_offloaded: 0
dev.vtnet.1.rx_csum_failed: 0
dev.vtnet.1.rx_csum_bad_proto: 0
dev.vtnet.1.rx_csum_bad_offset: 0
dev.vtnet.1.rx_csum_bad_ipproto: 0
dev.vtnet.1.rx_csum_bad_ethtype: 0
dev.vtnet.1.rx_mergeable_failed: 0
dev.vtnet.1.rx_enq_replacement_failed: 0
dev.vtnet.1.rx_frame_too_large: 0
dev.vtnet.1.mbuf_alloc_failed: 0
dev.vtnet.1.act_vq_pairs: 1
dev.vtnet.1.requested_vq_pairs: 0
dev.vtnet.1.max_vq_pairs: 1
dev.vtnet.1.%parent: virtio_pci3
dev.vtnet.1.%pnpinfo:
dev.vtnet.1.%location:
dev.vtnet.1.%driver: vtnet
dev.vtnet.1.%desc: VirtIO Networking Adapter


[2.4.3-RELEASE][admin@x.x.x]/root: ifconfig vtnet1
vtnet1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=6c00b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 1a:b3:6c:0f:3c:61
        hwaddr 1a:b3:6c:0f:3c:61
        inet6 fe80::18b3:6cff:fe0f:3c61%vtnet1 prefixlen 64 scopeid 0x2
        inet 192.168.0.1 netmask 0xffffff00 broadcast 192.168.0.255
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet 10Gbase-T <full-duplex>
        status: active

Hi @muppet - do you know if there is an option to change the size of the RX and TX descriptors? You might also try increasing the interface queue length. More details on turning are available on the link below (although this seems to be written mainly for Intel based cards):

https://calomel.org/freebsd_network_tuning.html

Also, I'd probably increase the process_limit to 1024 or higher.

Unfortunately I'm not familiar with vtnet, but I hope this helps.

[pfsvrb]

Using newer builds of 2.4.4.a with the FQ_codel GUI options, I'm seeing a strange error when I go to enable a limiter. On an install without any existing limiters or traffic shaping enabled, if I click on Firewall/Traffic Shaper/Limiters, I see an error displayed at the top of the web browser.

I am able to create limiters and queues, and assign the queues to rules to enable QoS. FQ_Codel seems to work fine and buffer bloat scores are A+ on speedtest.net. However, when pfSense is rebooted I see errors in the log and the following is displayed when logging in to the router after it has booted up.

This seems to be related to the FQ_Codel GUI additions, I was not able to produce this behavior using a 6/30/18 build of pfSense 2.4.4.a. Is anyone else seeing these errors when enabling fq_codel Limiters via the GUI?

[10-Jul-2018 07:18:26 America/Chicago] PHP Warning: A non-numeric value encountered in /etc/inc/shaper.inc on line 121
[10-Jul-2018 07:18:26 America/Chicago] PHP Warning: A non-numeric value encountered in /etc/inc/shaper.inc on line 122
[10-Jul-2018 07:18:26 America/Chicago] PHP Warning: A non-numeric value encountered in /etc/inc/shaper.inc on line 131
[10-Jul-2018 07:18:26 America/Chicago] PHP Warning: A non-numeric value encountered in /etc/inc/shaper.inc on line 132
[10-Jul-2018 07:18:26 America/Chicago] PHP Warning: A non-numeric value encountered in /etc/inc/shaper.inc on line 82
[10-Jul-2018 07:18:26 America/Chicago] PHP Warning: A non-numeric value encountered in /etc/inc/shaper.inc on line 83
[10-Jul-2018 07:18:26 America/Chicago] PHP Warning: A non-numeric value encountered in /etc/inc/shaper.inc on line 95
[10-Jul-2018 07:18:26 America/Chicago] PHP Warning: A non-numeric value encountered in /etc/inc/shaper.inc on line 96
[10-Jul-2018 07:18:26 America/Chicago] PHP Warning: A non-numeric value encountered in /etc/inc/shaper.inc on line 121
[10-Jul-2018 07:18:26 America/Chicago] PHP Warning: A non-numeric value encountered in /etc/inc/shaper.inc on line 122
[10-Jul-2018 07:18:26 America/Chicago] PHP Warning: A non-numeric value encountered in /etc/inc/shaper.inc on line 131
[10-Jul-2018 07:18:26 America/Chicago] PHP Warning: A non-numeric value encountered in /etc/inc/shaper.inc on line 132
[10-Jul-2018 07:18:26 America/Chicago] PHP Warning: A non-numeric value encountered in /etc/inc/shaper.inc on line 121
[10-Jul-2018 07:18:26 America/Chicago] PHP Warning: A non-numeric value encountered in /etc/inc/shaper.inc on line 122
[10-Jul-2018 07:18:26 America/Chicago] PHP Warning: A non-numeric value encountered in /etc/inc/shaper.inc on line 131
[10-Jul-2018 07:18:26 America/Chicago] PHP Warning: A non-numeric value encountered in /etc/inc/shaper.inc on line 132
[10-Jul-2018 07:18:26 America/Chicago] PHP Warning: A non-numeric value encountered in /etc/inc/shaper.inc on line 121
[10-Jul-2018 07:18:26 America/Chicago] PHP Warning: A non-numeric value encountered in /etc/inc/shaper.inc on line 122
[10-Jul-2018 07:18:26 America/Chicago] PHP Warning: A non-numeric value encountered in /etc/inc/shaper.inc on line 131
[10-Jul-2018 07:18:26 America/Chicago] PHP Warning: A non-numeric value encountered in /etc/inc/shaper.inc on line 132
[10-Jul-2018 07:18:26 America/Chicago] PHP Warning: A non-numeric value encountered in /etc/inc/shaper.inc on line 82
[10-Jul-2018 07:18:26 America/Chicago] PHP Warning: A non-numeric value encountered in /etc/inc/shaper.inc on line 83
[10-Jul-2018 07:18:27 America/Chicago] PHP Warning: A non-numeric value encountered in /etc/inc/shaper.inc on line 95
[10-Jul-2018 07:18:27 America/Chicago] PHP Warning: A non-numeric value encountered in /etc/inc/shaper.inc on line 96
[10-Jul-2018 07:18:27 America/Chicago] PHP Warning: A non-numeric value encountered in /etc/inc/shaper.inc on line 121
[10-Jul-2018 07:18:27 America/Chicago] PHP Warning: A non-numeric value encountered in /etc/inc/shaper.inc on line 122
[10-Jul-2018 07:18:27 America/Chicago] PHP Warning: A non-numeric value encountered in /etc/inc/shaper.inc on line 131
[10-Jul-2018 07:18:27 America/Chicago] PHP Warning: A non-numeric value encountered in /etc/inc/shaper.inc on line 132
[10-Jul-2018 07:18:27 America/Chicago] PHP Warning: A non-numeric value encountered in /etc/inc/shaper.inc on line 121
[10-Jul-2018 07:18:27 America/Chicago] PHP Warning: A non-numeric value encountered in /etc/inc/shaper.inc on line 122
[10-Jul-2018 07:18:27 America/Chicago] PHP Warning: A non-numeric value encountered in /etc/inc/shaper.inc on line 131
[10-Jul-2018 07:18:27 America/Chicago] PHP Warning: A non-numeric value encountered in /etc/inc/shaper.inc on line 132
[10-Jul-2018 07:18:27 America/Chicago] PHP Warning: A non-numeric value encountered in /etc/inc/shaper.inc on line 121
[10-Jul-2018 07:18:27 America/Chicago] PHP Warning: A non-numeric value encountered in /etc/inc/shaper.inc on line 122
[10-Jul-2018 07:18:27 America/Chicago] PHP Warning: A non-numeric value encountered in /etc/inc/shaper.inc on line 131
[10-Jul-2018 07:18:27 America/Chicago] PHP Warning: A non-numeric value encountered in /etc/inc/shaper.inc on line 132
[10-Jul-2018 07:18:27 America/Chicago] PHP Warning: A non-numeric value encountered in /etc/inc/shaper.inc on line 82
[10-Jul-2018 07:18:27 America/Chicago] PHP Warning: A non-numeric value encountered in /etc/inc/shaper.inc on line 83
[10-Jul-2018 07:18:27 America/Chicago] PHP Warning: A non-numeric value encountered in /etc/inc/shaper.inc on line 95
[10-Jul-2018 07:18:27 America/Chicago] PHP Warning: A non-numeric value encountered in /etc/inc/shaper.inc on line 96
[10-Jul-2018 07:18:27 America/Chicago] PHP Warning: A non-numeric value encountered in /etc/inc/shaper.inc on line 121
[10-Jul-2018 07:18:27 America/Chicago] PHP Warning: A non-numeric value encountered in /etc/inc/shaper.inc on line 122
[10-Jul-2018 07:18:27 America/Chicago] PHP Warning: A non-numeric value encountered in /etc/inc/shaper.inc on line 131
[10-Jul-2018 07:18:27 America/Chicago] PHP Warning: A non-numeric value encountered in /etc/inc/shaper.inc on line 132
[10-Jul-2018 07:18:27 America/Chicago] PHP Warning: A non-numeric value encountered in /etc/inc/shaper.inc on line 121
[10-Jul-2018 07:18:27 America/Chicago] PHP Warning: A non-numeric value encountered in /etc/inc/shaper.inc on line 122
[10-Jul-2018 07:18:27 America/Chicago] PHP Warning: A non-numeric value encountered in /etc/inc/shaper.inc on line 131
[10-Jul-2018 07:18:27 America/Chicago] PHP Warning: A non-numeric value encountered in /etc/inc/shaper.inc on line 132
[10-Jul-2018 07:18:27 America/Chicago] PHP Warning: A non-numeric value encountered in /etc/inc/shaper.inc on line 82
[10-Jul-2018 07:18:27 America/Chicago] PHP Warning: A non-numeric value encountered in /etc/inc/shaper.inc on line 83
[10-Jul-2018 07:18:27 America/Chicago] PHP Warning: A non-numeric value encountered in /etc/inc/shaper.inc on line 95
[10-Jul-2018 07:18:27 America/Chicago] PHP Warning: A non-numeric value encountered in /etc/inc/shaper.inc on line 96
[10-Jul-2018 07:18:27 America/Chicago] PHP Warning: A non-numeric value encountered in /etc/inc/shaper.inc on line 121
[10-Jul-2018 07:18:27 America/Chicago] PHP Warning: A non-numeric value encountered in /etc/inc/shaper.inc on line 122
[10-Jul-2018 07:18:27 America/Chicago] PHP Warning: A non-numeric value encountered in /etc/inc/shaper.inc on line 131
[10-Jul-2018 07:18:27 America/Chicago] PHP Warning: A non-numeric value encountered in /etc/inc/shaper.inc on line 132

[TheNarc]

@pfsvrb Can you open that file (Diagnostics > Edit File) and paste a few of the lines being referenced?

[w0w]

@pfsvrb said in Playing with fq_codel in 2.4:

Is anyone else seeing these errors when enabling fq_codel Limiters via the GUI?

I don't see any errors.

[pfsvrb]

@thenarc Yes, here are the lines being referenced in the error.

Line 121: "target" => array("name" => "Target Delay (ms)", "type" => "number", "default" => get_single_sysctl("net.inet.ip.dummynet.codel.target") / 1000),

Line 122: "interval" => array("name" => "Interval (ms)", "type" => "number", "default" => get_single_sysctl("net.inet.ip.dummynet.codel.interval") / 1000),

Line 131: "target" => array("name" => "Target Delay (ms)", "type" => "number", "default" => get_single_sysctl("net.inet.ip.dummynet.fqpie.target") / 1000),

Line 132: "tupdate" => array("name" => "Interval (ms)", "type" => "number", "default" => get_single_sysctl("net.inet.ip.dummynet.fqpie.tupdate") / 1000),

[pfsvrb]

@w0w I've tried the following.

Delete all the queues and limiters. Check LAN outbound IP4/IP6 rules to verify that there are no references to the deleted QoS queues. Reboot pfSense.

Once it boots up, I click on Firewall/Traffic Shaper/Limiters and try to create a new limiter. I see the following error at the top of the browser, this mirrors what I then see in the error log/crash log when I visit the Dashboard in pfSense.

At this point, even if I do not create any new limiters, I still see the following error on the Dashboard:

This strange behavior doesn't seem to be impacting QoS but, it does seem to be related to the GUI FQ_Codel additions that were completed recently. Prior to that I cannot re-produce this odd error. If I can help post additional screenshots or logs let me know and I'm happy to provide the info.

[w0w]

@pfsvrb said in Playing with fq_codel in 2.4:

@thenarc Yes, here are the lines being referenced in the error.

Line 121: "target" => array("name" => "Target Delay (ms)", "type" => "number", "default" => get_single_sysctl("net.inet.ip.dummynet.codel.target") / 1000),

Line 122: "interval" => array("name" => "Interval (ms)", "type" => "number", "default" => get_single_sysctl("net.inet.ip.dummynet.codel.interval") / 1000),

Line 131: "target" => array("name" => "Target Delay (ms)", "type" => "number", "default" => get_single_sysctl("net.inet.ip.dummynet.fqpie.target") / 1000),

Line 132: "tupdate" => array("name" => "Interval (ms)", "type" => "number", "default" => get_single_sysctl("net.inet.ip.dummynet.fqpie.tupdate") / 1000),

I have compared the code and found no difference. But... what version of pfsense exactly do you have?
My version:
I have 2.4.4-DEVELOPMENT (amd64)
built on Sat Jul 07 17:23:30 EDT 2018
FreeBSD 11.2-RELEASE

BTW I've updated some older version in VM and found that it installed PHP7 version, my real hardware installation shows PHP 5.6.36 (cli) (built: Jul 4 2018 18:59:20) and VM version shows PHP 7.2.7 (cli) (built: Jul 4 2018 19:00:07) ( NTS ), you can also check this with 'php --version' command. This could be related to your error, but currently I can not reproduce it, even on this new PHP7 version. This could be some temporary error also or broken installation. Also there should be some switch to change the PHP version...

[pfsvrb]

Thank you for checking on this, it is very strange that I am seeing these errors.

Here is my current version:
2.4.4-DEVELOPMENT (amd64)
built on Tue Jul 10 06:09:20 EDT 2018
FreeBSD 11.2-RELEASE

Here is the output of a "php -i" command on this pfSense install:
phpinfo()
PHP Version => 7.2.7

System => FreeBSD pfSense.pfvm.vbox 11.2-RELEASE FreeBSD 11.2-RELEASE #36 79c8a561b61(RELENG_2_4_4): Tue Jul 10 06:14:32 EDT 2018 root@buildbot3:/builder/ce-master/tmp/obj/builder/ce-master/tmp/FreeBSD-src/sys/pfSense amd64
Build Date => Jul 4 2018 18:58:13

And, the output of "php --version":
PHP 7.2.7 (cli) (built: Jul 4 2018 19:00:07) ( NTS )
Copyright (c) 1997-2018 The PHP Group
Zend Engine v3.2.0, Copyright (c) 1998-2018 Zend Technologies
with Zend OPcache v7.2.7, Copyright (c) 1999-2018, by Zend Technologies

A few other items to note, I had installed this from 2.4.4.a built on June 2nd, 2018. I then have done two upgrades, one to a build on June 30th, 2018, and another upgrade yesterday bringing the build up to the current version displayed here.

It's also worth noting that I had previously enabled fq_codel using the previous threads listed above and using the ShellCMD function to enable fq_codel schedulers on reboot. I wonder if this somehow conflicted with the install of the newer version that had the GUI fq_codel shaping options? I will try a re-install tonight and do a backup restore and see if I can still duplicate the issue.

This is a non-prod box on a VM that I use for testing before I roll out to real hardware. If there's anything else I can do to provide more logs/testing, I'm open to it.

[w0w]

@pfsvrb said in Playing with fq_codel in 2.4:

wonder if this somehow conflicted with the install of the newer version that had the GUI fq_codel shaping options?

I think no, but yes it's the best solution to try clean install...

[pfsvrb]

@w0w I tried a clean install in a new VM. This is using a current build of 2.4.4.a downloaded from pfsense.org today.

Unfortunately, I am seeing identical results. When I go to create a new limiter, it shows the same shaper.inc error.

This is happening on a box stock install without a backup being restored. I've just installed pfSense, logged in, gone through the basic first time setup config wizard, and then clicked on Firewall/Traffic Shaper/Limiters "New Limiter"

[w0w]

@pfsvrb
I did the same and no error so far. It's definitely related to VM configuration, I think.

Yep. Reproduced. Sorry used wrong VM at testing
Shame on me!

[w0w]

@pfsvrb
I think it's time to report it on redmine!

[pfsvrb]

Just as a followup, I downloaded the 7/16 2.4.4.a build and tested in my VM today, and this error is gone.

The only oddity I noticed is that the speed limits specified seem to be ignored if in mbit/s. I had to use kbit/s values to get my limits enforced. However, this is a minor issue and it's great to be able to finally set FQ_Codel in the GUI. Great work guys!

[zwck]

I just updated to the latest beta and i am struggling to understand how to use the GUI. I want to have a very basic setup.

Is this correct?

and Lan rule

[pfsvrb]

@zwck Yes, those settings look fine for a "set it and forget it" fq_codel implementation. How are your bufferbloat scores on dslreports.com/speedtest using the settings in those screenshots??

[zwck]

@pfsvrb
Its A all around, however i noticed that when checking ipfw sched show periodically during both phases (up and down) there are quite some dropped packages, maybe my NICs are not that great, they are intel but not the recommended 350s

00001: 900.000 Mbit/s    0 ms burst 0
q65537  50 sl. 0 flows (1 buckets) sched 1 weight 0 lmax 0 pri 0 droptail
 sched 1 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 NoECN
   Children flowsets: 1
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
  0 ip           0.0.0.0/0             0.0.0.0/0     295104 439672423 275 410213 492
00002: 900.000 Mbit/s    0 ms burst 0
q65538  50 sl. 0 flows (1 buckets) sched 2 weight 0 lmax 0 pri 0 droptail
 sched 2 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 NoECN
   Children flowsets: 2
  0 ip           0.0.0.0/0             0.0.0.0/0     2405   101107  0    0   0

00001: 900.000 Mbit/s    0 ms burst 0
q65537  50 sl. 0 flows (1 buckets) sched 1 weight 0 lmax 0 pri 0 droptail
 sched 1 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 NoECN
   Children flowsets: 1
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
  0 ip           0.0.0.0/0             0.0.0.0/0      745    31427  0    0   0
00002: 900.000 Mbit/s    0 ms burst 0
q65538  50 sl. 0 flows (1 buckets) sched 2 weight 0 lmax 0 pri 0 droptail
 sched 2 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 NoECN
   Children flowsets: 2
  0 ip           0.0.0.0/0             0.0.0.0/0     614966 917451569 207 309100 616

i am quite sure there are some errors in my config, i exchanged to the 350 intel nics and bufferbloat is back to F