The traffic download limit only applies half of what I give it. why?

HLPPC

@HLPPC and here are some shell commands I have been using because loaders and system tunables refuse to play nice with the igb driver :)

            shellcmd sysctl dev.igb.0.tx_int_delay=0 /shellcmd
	shellcmd sysctl dev.igb.1.tx_int_delay=0 /shellcmd
	shellcmd sysctl dev.igc.0.tx_int_delay=0 /shellcmd
	shellcmd sysctl dev.igc.1.tx_int_delay=0 /shellcmd
	shellcmd sysctl dev.igb.0.tx_abs_int_delay=64 /shellcmd
	shellcmd sysctl dev.igb.1.tx_abs_int_delay=64 /shellcmd
	shellcmd sysctl dev.igc.0.tx_abs_int_delay=64 /shellcmd
	shellcmd sysctl dev.igc.1.tx_abs_int_delay=64 /shellcmd
	shellcmd sysctl dev.igb.0.rx_abs_int_delay=64 /shellcmd
	 shellcmd sysctl dev.igb.1.rx_abs_int_delay=64 /shellcmd
	 shellcmd sysctl dev.igc.0.rx_abs_int_delay=64 /shellcmd
	 shellcmd sysctl dev.igc.1.rx_abs_int_delay=64 /shellcmd

shellcmd ifconfig igc0 -wol -rxcsum -txcsum -rxcsum6 -txcsum6 -mextpg -tso4 -lro -tso6 -vlanhwfilter -vlanhwcsum -vlanhwtag -vlanhwtso -vlanmtu /shellcmd
shellcmd ifconfig igc1 -wol -rxcsum -rxcsum6 -txcsum -txcsum6 -mextpg -tso4 -lro -tso6 -vlanhwfilter -vlanhwcsum -vlanhwtag -vlanhwtso -vlanmtu /shellcmd
shellcmd ifconfig igb0 -wol -rxcsum -txcsum -rxcsum6 -txcsum6 -lro -tso4 -tso6 -vlanhwfilter -vlanhwcsum -vlanhwtag -vlanhwtso vlanmtu /shellcmd
shellcmd ifconfig igb1 -wol -rxcsum -txcsum -rxcsum6 -txcsum6 -mextpg -lro -tso4 -tso6 -vlanhwfilter -vlanhwcsum -vlanhwtag -vlanhwtso -vlanmtu /shellcmd
shellcmd ifconfig lo0 -rxcsum -txcsum -lro -tso4 -tso6 -vlanhwtso -vlanmtu -vlanhwfilter /shellcmd
shellcmd ifconfig re0 -wol -rxcsum -txcsum -rxcsum6 -txcsum6 -mextpg -lro -tso4 -tso6 vlanhwfilter -vlanhwcsum -vlanhwtag -vlanhwtso -vlanmtu /shellcmd
It is a lot of stuff to make it so that ONLY the fqcodel limiter is in charge of the bandwidth. Not the offloading, not tso/lro. Not the ISP's janky vlan on the wan. Limiters work great, but I suspect they send different checksums than the offloading checksums, especially if hw.inet.udp.checksum is enabled. It is sus because if you make fqcodel suuuuper aggressive, your NIC cannot keep up with the dropped packets some udp based games can desync and udp can fragment.

david.woodward

@enesas, same problem on pfsense+ 23.09.1 using intel NIC's. If it didn't work fine for the upload limiter then I'd assume there's some flaw in my logic/config. But the same config I used for upload should theoretically work for download, right?

For now I've just doubled my download speeds like you, but I agree with @nicknuke that this is unacceptable.

And, while I appreciate community members like @HLPPC who help us figure out ways to work around these issues, the primary reason I moved from Ubiquiti to pfSense was so that I could do more within the web UI and not have to dive into the shell as much. I'm fully capable of using SSH or a console cable if needed. I just prefer not to since whatever changes I make there are much harder to remember a year later when I might need to update those obscure settings.

Unfortunately, I may just go back to Ubiquiti. I've been on pfSense for about 6 months now and it feels like a fragile mess. It's stable as long as I don't touch it, but if I'm changing anything at all I have to say a prayer and coordinate an "internet outage" with the family first.

sandersui

Same issue, download is half of what I set, and the upload is normal. On pfsense 2.7.2.

SteveITS

Are you looking the traffic graph and running into this?
https://redmine.pfsense.org/issues/14078
“Traffic graph shows half actual throughput when switching back to the graph”

HLPPC

Hello again. I have been playing with fq_codel, iflib CPU mapping and fifo for a bit. I get half of what my bandwidth download is set to when I use fq_codel on the LAN download side, and always full speed on the upload. If I set up fq_codel on the down and upload wan I get full speeds as expected, but there are issues with local host when using unbound and whenever new connections are made to DNS randomly. I have cut appropriate crossover cables for my setup and think maybe Auto MDI-X messes with the limiters.

If Auto-MDIX is the same as my ISP's router/switch/ONT all in one combo, it uses a pseudo random number generator to determine which physical paths to use.

If you guys are getting bandwidth issues while using limiters, maybe try cutting appropriate crossover cables, and maybe try disabling auto-mdix entirely :P

Limiters also change the number of times packets pass through the firewall. DDWRT users usually set fqcodel on the LAN Download link and on the WAN upload link. I think fq_codel on uplinks maybe messes with applications that have weird UDP acknowledgements (transport layer) because of round-robin packet processing, so I have recently opted for an FQ-Codel on my LAN Download and only FIFO on my wan upload while using dnsmasq with DNS port forwarding everything to my LAN interfaces.

You have to disable all offloading and set localhost's mtu to 1500 to get dummynet to work perfectly, and (if you care about perfect fast UDP processing) this may include setting net.inet.udp.maxdgram="1500"

There are also issues when devices cannot use UPnP or Stun and have to try random ports that you may be using. For instance, the Nintendo Switch needs to punch holes during NAT but other devices want UPnP and I definitely don't let them have it.

I also resolved some of the graphics bugging issues by disabling unused APU cores in my mobo's bios, setting CStates on all cores = to the same frequency and improved fq_codel's timestamping resolution by setting:
kern.hz="1100"
kern.eventtimer.periodic="1"
kern.eventtimer.idletick="1"
hw.igc.rx_abs_int_delay="0"
hw.igc.tx_abs_int_delay="16"
hw.igc.rx_int_delay="0"
hw.igc.tx_int_delay="16"
(setting appropriate iflib interrupt settings to match on all NICS)
hw.igc.eee_setting="0"
vm.pmap.pti="1"
net.isr.bindthreads="1"

Limiters are supposedly heavily effected by interrupts according to RFC8290, otherwise I wouldn't bother with changing interrupts:
"When deploying a queue management algorithm such as FQ-CoDel, it is
important to ensure that the algorithm actually runs in the right
place to control the queue. In particular, lower layers of the
operating system networking stack can have queues of their own, as
can device drivers and hardware. Thus, it is desirable that the
queue management algorithm runs as close to the hardware as possible.
However, scheduling such complexity at interrupt time is difficult,
so a small standing queue between the algorithm and the wire is often
needed at higher transmit rates." - https://datatracker.ietf.org/doc/html/rfc8290

I have also seen improvements in performance by disabling fragmentation via sysctls and using suricata to assist with QoS by blocking bad checksums and invalid timestamps (whatever those are . ) Anyways, I hope this info is correct. It is based on intuition and loads of reading and hypothesizing.

sandersui

@SteveITS The traffic graph looks fine here

sandersui

@HLPPC Like others, I've just doubled the download speed in the limiters until someone finds the cause and can fix it.

Not sure why but this post get marked as spam so just typing some stuff in the hope that it get thru that way

nhs7000

I'm facing this issue too. Only worse! I have 2 wan connections grouped with load balance. For upload, I have to set half the desired speed (50 to get 100). For download double the desired speed (500 to get 250). This was reported from 9 months but has not been fixed until now.

HLPPC

@nhs7000 It happens when I run FQ_codel on LAN interfaces, either down on LAN and up on LAN or (like DDWRT) down on LAN and up on WAN. It has to do with how many times the packets are passed through the firewall and maybe how many times they are hashed. Sometimes I get unseen acknowledgements with TCP running fq_codel on any interface, and UDP checksum errors other times. I have even had videogames have their graphics pipes break because of what I think are these sorts of error, causing my TV to show two pictures at once. A few times I thought I overcame the issue but nope. I have even seen fqcodel start fragmenting UDP connections :P

HLPPC

@HLPPC FQ_Codel and DNS don't seem to get along either. Lots of new DNS connections can really ruin healthy priority, and localhost stepping out to do whatever it wants with NAT doesn't help either. I think unbound commonly interferes with fqcodel. I have tried keeping it away from fqcodel entirely but sometimes UDP packets seem to be in the same internal priority as UDP, especially if the DNS is over HTTPS and the UDP packets are riding that connection's sawtooth.

nhs7000

@HLPPC Thank you for your reply, I'm not even using FQ_codel, I used Tail Drop with the default scheduler worse case weighted fair queueing on the LAN interface. As some users had previously noted, version 2.6 didn't have this problem.

Maybe the new FreeBSD 14 caused some conflicts I'm not sure, but it should be an easy problem to reproduce and detect.

nicknuke

After trying it...
I end up using the limiter on Floating Rules.
I use Taildrop - Codel - Taildrop
Then put the queue on Floating rules with Out direction..
I finally get the expected upload / download results.

I think that has to be like that from 2.7.2

patrick.pesegodinski

Is it a bug or configuration change?

I have the same problem.

sandersui

@patrick-pesegodinski bug

bipton

Has to be a bug. After updating (which was a task in and of itself) from 2.7.0 my limiters aren't working as before. With a good amount of testing, I ended up basically doubling the limiters bandwidth to reach what it should be limiting to. Hopefully they find the reason for it, what a pain in the butt last few days.

mrmanuel

I can confirm the same issue on six different pfSense firewalls. This is for sure a bug somewhere in the conversion.

Here what I experienced:

10240 Mbit/s limit --> Traffic does not pass anymore, like a blocking rule
50 Mbit/s limit --> about 46 Mbit/s
51200 Kbit/s limit --> about 49 Mbit/s
52428800 Bit/s limit --> about 50 Mbit/s

Can others confirm the same?

Edit: Found another thread with the same issue, that the set limit does not match the applied limit.

Traffic Shaper-- Limiters -- Not working as expected

bipton

I have since situated my limiters and have got them working properly now. Before upgrading to 2.7.2 (which was a task due to a bug) I installed an additional fail-over wan, which didn't have rules for the limiters. My rules were set per lan/vlan to use my gateway group and it's respective limiter. I had only made limiters for upload/download. My problem seemed to situate once I put my rules into the floating ruleset, made my limiters simple tail-drop with worst-case waieghted fair queueing, then made a queue for each limiter utilizing PIE with ecn checked. The rules used default gateway rather than gateway group (pfSense gateway is set to use the gateway group rather now) and the limiters were the actual queues I set up under the limiters. Hope it helps someone out there.