Traffic limiter has no effect on outbound reply traffic spikes
-
I want to limit the upload traffic on a particular interface that's used to serve video streams. I have the limiter set up as follows:
I know the limiter is at least partly working, because if I do a speedtest from a host on that interface, I see the speed limited (slightly slower actually, but I think it's a mibi/mega difference):
The streaming server encodes videos into chunks, and sends each chunk individually. I'm seeing that each chunk is sent out at maximum bandwidth:
I thought at first it might be that the chunks are just too small to fill the queue, but from the ipfw man page, it should be ~600Kbits by default, which is much smaller than my chunks.
Now I'm wondering if the issue is that this streaming traffic is not hitting the firewall rule because the connection isn't initiated by a host on my side of the interface; it's actually hitting a port forward rule. I've tried firewall rules on the interface itself and floating:
(I also tried switching source and destination)
None of these rules seem to limit traffic sent in response to an externally-initiated connection.
Interestingly, I also have bufferbloat traffic shapers set up as instructed by this recipe, and while they do fix bufferbloat when I run a test, they do nothing to limit traffic initiated externally either.
My suspicion, then, is that it doesn't have anything to do with the spiky nature of the traffic, but instead that none of my rules are set up to limit reply traffic.
How do I set up my firewall rules/limiters to limit reply traffic? TIA!
-
It seems that if I add the limiter to the firewall rule associated with the port forward, then things work as expected. But what if I want to add the limiter to all traffic coming from WAN destined for a particular interface (or subnet)? Would I add another rule? But how do I add a rule that assigns traffic to a limiter without passing the traffic?
-
Thanks to some helpful folks on reddit, turns out my suspicion was correct. The rule in the recipe in the docs just doesn't apply to incoming connections. I ended up solving this by adding a floating rule on the WAN interface that matches traffic bound for my DMZ interface (specified as a subnet) that tags a sepearate queue on the same limiter with low priority.
PS - if anyone sees this and is using it, make sure your floating firewall rules are set to MATCH and not PASS!
-
@thearamadon Use NordVPN and then pass all the traffic using your limiter using the VPN interface so you don't have to worry as much? Or using floating rules. Port Forwarding from the OpenVPN interface to the local net doesn't seem to work. Or, if you can give your video server DSCP markings, you can use them to force traffic into dummynet pipes with shapers.
-
@thearamadon I was able to successfully get fqcodel to work with CBQ shaping, PRIQ shaping and DSCP values set in windows. I have issues with inbound traffic too. Not to mention tons of out of order TCP from what I presume is io_fast switching between forwarding packets and shaping them with fqcodel. Some of the games I play use TCP Fast Open, and it seems to evade the fqcodel, probably why I get unseen ack errors. My limters have in/out queues too. Do your limiters only have in queues in both directions? Does that result in a half duplex connection?
-
@thearamadon "Caveats: By default, pfSense software only matches the first packet of a connection, which is the packet that creates an entry in the state table. If a connection starts with a different DSCP value, has no DSCP value in the starting packet, or otherwise changes DSCP values during the connection, the traffic will not be classified as expected.
Tip:
This can be worked around by using “no state” rules, but crafting these rules in a secure manner is difficult, so it is not a viable workaround for most environments."https://docs.netgate.com/pfsense/en/latest/trafficshaper/dscp.html
This makes me wonder if I can make the limiters operate statelessly
for now at least, some videogames in windows only use certain fqcodel limiters if tagged with EF. Really neat.
I'll give these a shot
-
@HLPPC Can you share your shaper settings? I have a single fq_codel limiter with the idea that everything should sort of just work without having to set up manual queues. But I'm seeing some gaming issues because of the spiky nature of the traffic.
-
@thearamadon Yeah. I have been trying to tune this issue out for a bit over a year now. You get out of order TCP flows sporadically with fq_codel, in part because of net.inet.ip.dummynet.io_fast switching between limiting and forwarding the traffic, and partially due to what I think is there still being a necessity for a shaper to begin with, which is used to regulate convex and concave bandwidth curves.
So, because of the out of order TCP, I found a possible solution in reordering TCP using PRIQ or CBQ. Set up the shapers through the wizard, delete all the rules, and use all of Netgate's advice for setting up shapers found here:
https://docs.netgate.com/pfsense/en/latest/trafficshaper/advanced.html#editing-shaper-queues
Particularly pay attention to this bit:
"Shaper Rule Matching Tips
Connections can be tricky to match properly due to several factors, including:
NAT applies before outbound firewall rules can match connections, so for connections that have outbound NAT applies as they leave a WAN-type interface, the private IP address source is hidden by NAT and cannot be matched by a rule. Some protocols such as Bittorrent will use random ports or the same ports as other services. Multiple protocols using the same port cannot be distinguished by the firewall. A protocol may use a range of ports so wide that it cannot be distinguished from other traffic."and
"While many of these cannot be solved by the firewall directly, there are ways to work around these limitations in a few cases.
To match by a private address source outbound in WAN floating rules, first tag the traffic as it passes in on a local interface. For example, match inbound on LAN and use the advanced Tag field to set a value, and then use the Tagged field on the WAN-side floating rule to match the same connection as it exits the firewall. Alternately, queue the traffic as it enters the LAN with a pass rule instead of when it exits a WAN."
Since I am using NordVPN over OpenVPN anyways, my LAN allow rules are already tagged with KS for policy filtering. A decent habit to have to mitigate leaky DNS issues over multiple wan-type interfaces. KS stands for Killswitch and blocks data trying to get out of the WAN with a matching firewall rule.
I delete all of the generated PRIQ or CBQ rules, and make another floating match rule that tags all traffic again with KS that passes through the WAN/VPN interface in any direction.
Next, instead of using ports to match my PRIQ rules, I just use qACK and qGAMES for tcp in match rules, and qGAMES for UDP. In these match rules make sure to include the KS tag in the tagged field. I also assign the shaper queues to the LAN rules (not the fqcodel limiters)
Additionally, you can match DSCP here and stick Expedited Forwarded games in higher priority queues, although, I think Steam prefers af21, and Discord af11. You could also try giving games af41. Here is a guide on how to assign DSCP in Windows for outbound.
https://www.reddit.com/r/GlobalOffensive/comments/24xq5f/qos_how_to_stop_other_programs_from_making_your/?rdt=36966
I tried giving a game's anti-cheat af41 today and it didn't seem to like that. But changing the priority of the ACKS seemed to help in-game a lot.
To try and solve the issue of TCP reordering, I put qACK as a lower priority than qGAMES on all interfaces, even though it is generated by the wizard with a higher priority. Also, I make sure my interfaces are all set to 1000MB full-duplex with no flow control and disable flow control in sysctls. It may also be a good idea to set net.inet.tcp.ecn.maxretries=0 in a system tunable or loader tunable. Other people have solved the issue by giving downloads and uploads 1ms of delay in fqcodel, and preventing io_fast from forwarding and then limiting traffic randomly, but I have found that the delay breaks Windows and other applications more than it helps.
-
@HLPPC Also, I change these like, every day. So the stuff I posted the other day isn't even there anymore :)
-
@HLPPC Wish I could tag my Nintendo Switch's traffic with DSCP values without having to first route it through Linux or something crazy. Maybe a Microtik or Cisco managed switch could do so. Also, I'm not even sure if the PRIQ/CBQ priority fixes the out of order TCP and duplications in Wireshark, but it makes Smash Ultimate lag less, and I saw good results in ARK Ascended over the last two days. I have tried like, hundreds of combinations of fq_codel target/interval/quantum settings.
The Nintendo Switch trips out Suricata and Snort so much that I was thinking it is Deprecated DNS Z flags or cURL attempts or freeBSD exploits lagging my games. I have actually seen freeBSD metasploit exploits initiated from my LAN by Cloudflare which is crazy. They might not like the Deprecated DNS or something. I don't know how to block deprecated DNS as there are no options to other than in Suricata. And blocking it breaks some games.
-
@HLPPC For the record, here is an easy way to see if DSCP is actually going through the router:
I switched to a dumbed down version of Windows 11 for gaming because of all of the bloatware and unsolicited traffic.
-
@thearamadon "that tags a sepearate queue on the same limiter with low priority." How do you manage to prioritize fqcodel flows?
"To try and solve the issue of TCP reordering (and dup acks and ack unseen segments), I put qACK as a lower priority than qGAMES on all interfaces (after assigning TCP to qACK/qGAMES)" but haven't seen a way to prioritize traffic with limiters other than with DSCP or VLAN.
A moderate vs strict NAT matters a lot here. So does using UPnP. 🥵
Also, suricata and snort break ALTQ on whatever interface the ids is assigned to because of its own parser.
