Lag spikes despite fq_codel limiters

Inety

Hello everyone,

I am struggling with severe lag spikes on my network whenever my Storage Server (running a node-based system for distributed storage) fully saturates my upload bandwidth. The server runs 10 nodes in total — 8 over VPN tunnels, and 2 without VPN.

My Setup:

pfSense 2.8.1
4 interfaces total: 1x WAN, 3x additional ports bridged into a single LAN bridge → all clients share the same 192.168.0.0/24 subnet
Each physical interface and the bridge currently has an “allow any” rule
Internet connection: 300 Mbit/s down, 30 Mbit/s up (as measured by speedtest.net)

Details about the Storage Server traffic:

The server establishes 8 VPN connections (for the VPN nodes). These appear as states in the floating limiter rule as expected.
The 2 non-VPN nodes do not appear in the floating limiter rule. Instead, their states show up in the port-forwarding rule on the WAN interface.
Whenever the nodes saturate the uplink, lag spikes occur across the entire network (high ping, delayed responses).

What I have done so far:

Followed this official Netgate guide on fq_codel limiters: Configuring CoDel Limiters for Bufferbloat
I can see counters increasing under Diagnostics → Limiter Info, and VPN-related states appear there.
Also my PC and other clients on the network seem to get correctly limited.
However, the server still uploads above the set limits (e.g. up to 30-32 Mbit/s upload, measured directly on the server via btop. Bursts up to 50 Mbit/s upload).
Most importantly, lag spikes across the network remain.

My questions:

Am I misapplying the limiter due to my LAN bridge setup? Should limiter rules only be on the bridge, or also on physical interfaces?
Why are the 2 non-VPN nodes bypassing the floating limiter rule and only showing up under the WAN port-forward states? How can I ensure they are also shaped?
Should I add an ACK/ICMP priority limiter on top, or should fq_codel alone be enough to avoid latency spikes?
Why does the server still upload above the configured 25 Mbit/s upload limit?

Any hints on how to correctly apply fq_codel limiters would be highly appreciated.

Thanks!

SteveITS

@Inety First off I’ve not used FQ_CODEL much. It notably lowered the top speed on the 2100 on which I tested.

In general though it sounds a bit like you’re applying the limiters to the individual connections and not all of WAN? It should be the latter and then FQ_CODEL shapes all traffic.

You may just want to limit those connections though to save bandwidth. The catch there is the connection matches the limiter as it is created, so a limit on a web server download would be on the incoming connection to the web server. Because eventually if the pipe is full it’s full.

Inety

Thanks @SteveITS

I want to clarify that I am already applying fq_codel globally to my entire WAN interface by following the official Netgate tutorial and using floating rules. After setting this up I also reset the firewall state table so that all connections had to be rebuilt. Other clients in my network are shaped correctly this way. I tested this, by setting the limits to 50% of my maximum bandwidth (upload and download). My PC's bandwidth got cut in half, whereas the storage server still uploaded with 100% of my maximum upload bandwidth (with bursts above 100%, as measured with btop)

What confuses me is that the Storage Server still does not appear to be shaped. All other devices respect the fq_codel limits, yet the Storage Server continues to burst above the configured bandwidth and the lag spikes remain whenever it is active. This is very puzzling to me, since my expectation was that fq_codel at the WAN level should catch all traffic.

My goal is to get bufferbloat under control, because the added latency is very noticeable while gaming and also during normal web browsing.

Do you have an idea why the Storage Server in particular might be bypassing the limiter, even though everything else seems to be shaped correctly?