The traffic download limit only applies half of what I give it. why?

[nicknuke]

@enesas
Have you got this solved ?
I experience the same thing to one ISP.
Works fine in 2.6

[enesas]

@nicknuke

Unfortunately.
I set the location where I wanted to give 100mbit to 200mbit. I found such a temporary solution!

[nicknuke]

@enesas
lol no I can't accept that...
I will stay with 2.6 for now until i've found out why.
fyi : my setup is vm image.. so it's a virtio nic driver instead of real nic.

[mardacs27]

Has anyone got this fixed? Also experiencing the same issue with 23.09

[HLPPC]

@mardacs27 So, it seems that the upload speed and the download speed need to match eachother for fqcodel to function "correctly". If I have a 1Gbps connection both up and down from my ISP and limit the download to 800mbps to get an A+ on my download speeds, I may see 1000mbps on the actual upload and may see decrease performance on the upload as a result. of applications being confused. (I'
m talking to you smash ultimate :P)

It is important to note that some application and even linux itself already implement qdisc with fqcodel and these could be causing intermittent issues. Your bufferbloat score won't be the same if you are running the test in Windows while using programs like Edge or Chrome.

To make things a little bit more complicated, I have done some reasearch suggestig that the devd module (sockstat | grep devd) with its sequential packet applet alsos interfere with tcp streams, even though devd is supposedly deprecated. (sockstat | grep devd) and sockstat kill xxxxx the SID of seqpacket and devd.

pfsense 2.7.2 has the khelp module still, which I haven't played around with in pfsense CE, but it abruptly disappears in pfsense plus, I assume to be replaced by the devd module.

Flow control/Mac control frames are used by IGMP proxy and avahii on port 5353 for mdns. If your upload and download speeds aren't set to exactly the same you may notice some poorly performing applications especially if they cannot find where to send their mac control pause frames.

In the last few days I found that setting my upload and download limiters to be slightly over my ISP's bandwidth i.e. 1024 up and 1024 down and just allowing the applications to deal with the bufferbloat, was best. Videogames undoubtedly have mechanisms to handle bufferbloat.

Also, setting a tcp delay piggyback option of 20ms seems to interfere with fqcodel, suggesting you may need to alter the target and interval settings based on how long you are delaying ACKnowledgement.

[HLPPC]

@mardacs27 Are you using the default CPU mapping with machdep hyperthreading enabled?

I have read on a microsoft forum that setting your net.inet.rss.bits to acknowledge twice of your number of cpus (when not hyperthreading) and cpus+threads (logical cores) while hyperthreading) is the best way to go. So, if you have a four core, two thread system, you have 8 "cores" while hyperthreading so net.inet.rss.bits=4, so you have twice the number of bucket mappings (16 to handle hash function errors.)

[HLPPC]

This post is deleted!

[HLPPC]

here are some loader tunables I have been playing with
hw.pci.msix_rewrite_table="1"
hw.pci.enable_msi="0"
hw.pci.enable_msix="0"
machdep.hyperthreading_intr_allowed="1"
machdep.disable_msix_migration="1"
dev.igb.0.iflib.disable_msix="1"
dev.igb.0.iflib.core_offset="4"
dev.igb.0.iflib.override_nrxqs="2"
dev.igb.0.iflib.override_ntxqs="2"
dev.igb.1.iflib.disable_msix="1"
dev.igc.0.iflib.disable_msix="1"
dev.igc.1.iflib.disable_msix="1"
hw.igc.rx_abs_int_delay="64"
hw.igc.tx_abs_int_delay="64"
hw.igc.tx_int_delay="0"
hw.igc.rx_process_limit="-1"
net.isr.maxthreads="-1"
net.isr.defaultqlimit="2048"
net.inet.tcp.initcwnd_segments="16"

[HLPPC]

@HLPPC and here are some shell commands I have been using because loaders and system tunables refuse to play nice with the igb driver :)

            shellcmd sysctl dev.igb.0.tx_int_delay=0 /shellcmd
	shellcmd sysctl dev.igb.1.tx_int_delay=0 /shellcmd
	shellcmd sysctl dev.igc.0.tx_int_delay=0 /shellcmd
	shellcmd sysctl dev.igc.1.tx_int_delay=0 /shellcmd
	shellcmd sysctl dev.igb.0.tx_abs_int_delay=64 /shellcmd
	shellcmd sysctl dev.igb.1.tx_abs_int_delay=64 /shellcmd
	shellcmd sysctl dev.igc.0.tx_abs_int_delay=64 /shellcmd
	shellcmd sysctl dev.igc.1.tx_abs_int_delay=64 /shellcmd
	shellcmd sysctl dev.igb.0.rx_abs_int_delay=64 /shellcmd
	 shellcmd sysctl dev.igb.1.rx_abs_int_delay=64 /shellcmd
	 shellcmd sysctl dev.igc.0.rx_abs_int_delay=64 /shellcmd
	 shellcmd sysctl dev.igc.1.rx_abs_int_delay=64 /shellcmd

shellcmd ifconfig igc0 -wol -rxcsum -txcsum -rxcsum6 -txcsum6 -mextpg -tso4 -lro -tso6 -vlanhwfilter -vlanhwcsum -vlanhwtag -vlanhwtso -vlanmtu /shellcmd
shellcmd ifconfig igc1 -wol -rxcsum -rxcsum6 -txcsum -txcsum6 -mextpg -tso4 -lro -tso6 -vlanhwfilter -vlanhwcsum -vlanhwtag -vlanhwtso -vlanmtu /shellcmd
shellcmd ifconfig igb0 -wol -rxcsum -txcsum -rxcsum6 -txcsum6 -lro -tso4 -tso6 -vlanhwfilter -vlanhwcsum -vlanhwtag -vlanhwtso vlanmtu /shellcmd
shellcmd ifconfig igb1 -wol -rxcsum -txcsum -rxcsum6 -txcsum6 -mextpg -lro -tso4 -tso6 -vlanhwfilter -vlanhwcsum -vlanhwtag -vlanhwtso -vlanmtu /shellcmd
shellcmd ifconfig lo0 -rxcsum -txcsum -lro -tso4 -tso6 -vlanhwtso -vlanmtu -vlanhwfilter /shellcmd
shellcmd ifconfig re0 -wol -rxcsum -txcsum -rxcsum6 -txcsum6 -mextpg -lro -tso4 -tso6 vlanhwfilter -vlanhwcsum -vlanhwtag -vlanhwtso -vlanmtu /shellcmd
It is a lot of stuff to make it so that ONLY the fqcodel limiter is in charge of the bandwidth. Not the offloading, not tso/lro. Not the ISP's janky vlan on the wan. Limiters work great, but I suspect they send different checksums than the offloading checksums, especially if hw.inet.udp.checksum is enabled. It is sus because if you make fqcodel suuuuper aggressive, your NIC cannot keep up with the dropped packets some udp based games can desync and udp can fragment.

[david.woodward]

@enesas, same problem on pfsense+ 23.09.1 using intel NIC's. If it didn't work fine for the upload limiter then I'd assume there's some flaw in my logic/config. But the same config I used for upload should theoretically work for download, right?

For now I've just doubled my download speeds like you, but I agree with @nicknuke that this is unacceptable.

And, while I appreciate community members like @HLPPC who help us figure out ways to work around these issues, the primary reason I moved from Ubiquiti to pfSense was so that I could do more within the web UI and not have to dive into the shell as much. I'm fully capable of using SSH or a console cable if needed. I just prefer not to since whatever changes I make there are much harder to remember a year later when I might need to update those obscure settings.

Unfortunately, I may just go back to Ubiquiti. I've been on pfSense for about 6 months now and it feels like a fragile mess. It's stable as long as I don't touch it, but if I'm changing anything at all I have to say a prayer and coordinate an "internet outage" with the family first.

[sandersui]

Same issue, download is half of what I set, and the upload is normal. On pfsense 2.7.2.

[SteveITS]

Are you looking the traffic graph and running into this?
https://redmine.pfsense.org/issues/14078
“Traffic graph shows half actual throughput when switching back to the graph”

[HLPPC]

Hello again. I have been playing with fq_codel, iflib CPU mapping and fifo for a bit. I get half of what my bandwidth download is set to when I use fq_codel on the LAN download side, and always full speed on the upload. If I set up fq_codel on the down and upload wan I get full speeds as expected, but there are issues with local host when using unbound and whenever new connections are made to DNS randomly. I have cut appropriate crossover cables for my setup and think maybe Auto MDI-X messes with the limiters.

If Auto-MDIX is the same as my ISP's router/switch/ONT all in one combo, it uses a pseudo random number generator to determine which physical paths to use.

If you guys are getting bandwidth issues while using limiters, maybe try cutting appropriate crossover cables, and maybe try disabling auto-mdix entirely :P

Limiters also change the number of times packets pass through the firewall. DDWRT users usually set fqcodel on the LAN Download link and on the WAN upload link. I think fq_codel on uplinks maybe messes with applications that have weird UDP acknowledgements (transport layer) because of round-robin packet processing, so I have recently opted for an FQ-Codel on my LAN Download and only FIFO on my wan upload while using dnsmasq with DNS port forwarding everything to my LAN interfaces.

You have to disable all offloading and set localhost's mtu to 1500 to get dummynet to work perfectly, and (if you care about perfect fast UDP processing) this may include setting net.inet.udp.maxdgram="1500"

There are also issues when devices cannot use UPnP or Stun and have to try random ports that you may be using. For instance, the Nintendo Switch needs to punch holes during NAT but other devices want UPnP and I definitely don't let them have it.

I also resolved some of the graphics bugging issues by disabling unused APU cores in my mobo's bios, setting CStates on all cores = to the same frequency and improved fq_codel's timestamping resolution by setting:
kern.hz="1100"
kern.eventtimer.periodic="1"
kern.eventtimer.idletick="1"
hw.igc.rx_abs_int_delay="0"
hw.igc.tx_abs_int_delay="16"
hw.igc.rx_int_delay="0"
hw.igc.tx_int_delay="16"
(setting appropriate iflib interrupt settings to match on all NICS)
hw.igc.eee_setting="0"
vm.pmap.pti="1"
net.isr.bindthreads="1"

Limiters are supposedly heavily effected by interrupts according to RFC8290, otherwise I wouldn't bother with changing interrupts:
"When deploying a queue management algorithm such as FQ-CoDel, it is
important to ensure that the algorithm actually runs in the right
place to control the queue. In particular, lower layers of the
operating system networking stack can have queues of their own, as
can device drivers and hardware. Thus, it is desirable that the
queue management algorithm runs as close to the hardware as possible.
However, scheduling such complexity at interrupt time is difficult,
so a small standing queue between the algorithm and the wire is often
needed at higher transmit rates." - https://datatracker.ietf.org/doc/html/rfc8290

I have also seen improvements in performance by disabling fragmentation via sysctls and using suricata to assist with QoS by blocking bad checksums and invalid timestamps (whatever those are . ) Anyways, I hope this info is correct. It is based on intuition and loads of reading and hypothesizing.

[sandersui]

@SteveITS The traffic graph looks fine here

[sandersui]

@HLPPC Like others, I've just doubled the download speed in the limiters until someone finds the cause and can fix it.

Not sure why but this post get marked as spam so just typing some stuff in the hope that it get thru that way

[nhs7000]

I'm facing this issue too. Only worse! I have 2 wan connections grouped with load balance. For upload, I have to set half the desired speed (50 to get 100). For download double the desired speed (500 to get 250). This was reported from 9 months but has not been fixed until now.

[HLPPC]

@nhs7000 It happens when I run FQ_codel on LAN interfaces, either down on LAN and up on LAN or (like DDWRT) down on LAN and up on WAN. It has to do with how many times the packets are passed through the firewall and maybe how many times they are hashed. Sometimes I get unseen acknowledgements with TCP running fq_codel on any interface, and UDP checksum errors other times. I have even had videogames have their graphics pipes break because of what I think are these sorts of error, causing my TV to show two pictures at once. A few times I thought I overcame the issue but nope. I have even seen fqcodel start fragmenting UDP connections :P

[HLPPC]

@HLPPC FQ_Codel and DNS don't seem to get along either. Lots of new DNS connections can really ruin healthy priority, and localhost stepping out to do whatever it wants with NAT doesn't help either. I think unbound commonly interferes with fqcodel. I have tried keeping it away from fqcodel entirely but sometimes UDP packets seem to be in the same internal priority as UDP, especially if the DNS is over HTTPS and the UDP packets are riding that connection's sawtooth.

[nhs7000]

@HLPPC Thank you for your reply, I'm not even using FQ_codel, I used Tail Drop with the default scheduler worse case weighted fair queueing on the LAN interface. As some users had previously noted, version 2.6 didn't have this problem.

Maybe the new FreeBSD 14 caused some conflicts I'm not sure, but it should be an easy problem to reproduce and detect.

[nicknuke]

After trying it...
I end up using the limiter on Floating Rules.
I use Taildrop - Codel - Taildrop
Then put the queue on Floating rules with Out direction..
I finally get the expected upload / download results.

I think that has to be like that from 2.7.2