Traffic shaping with CoDel and OpenVPN
-
Hi,
I have been using pfSense for a couple of months now. The first issue I encountered was, of course, bufferbloat. I set up CoDel limiters using this guide:
https://forum.netgate.com/topic/112527/playing-with-fq_codel-in-2-4/815
and this did resolve most of my issues, except one.One of the clients in the LAN subnet is using OpenVPN connection to external OpenVPN server, to upload/download data. The throughput/performance of the OVPN server is much lower (say half) than my DSL link's performance (300/50Mbit up/down).
When transfer to/from OVPN server is nearing its capacity, high latency and frame drops appear, even for my other LAN clients. Basically it behaves the same way as if my DSL link was saturated (while it's not). When this specific client is uploading/downloading data to other external servers (without using OVPN tunnel), everything works fine, no latency, no frame drops.I'm just trying to figure out how to set it up so that traffic to OVPN server from this client isn't impacting traffic for other servers, causing high latency and frame drops for them.
As a first workaround I tried setting up OVPN client on pfSense device itself and route traffic from this LAN client through OVPN tunnel on pfSense (while rest of clients isn't routed through this tunnel). That works but the same issue appears.
So I tried playing with separate queues for VPN and floating rules that would assign OVPN traffic to dedicated queues but that didn't work for me.How should I do it ? Create VPN-dedicated queues (up/down) under same limiters (as I did) ? Or create another set of limiters for VPN with one queue in each ?
How do I set up floating rules to use these queues ? I put a pair of additional match rules on the WAN interface before the 'general' match rules for CoDel and tried to set destination port to udp/1194 for out rule and source port udp/1194 for in rule but no traffic seems to match these rules. I know that would redirect all traffic to external OVPN servers to these queues but that not a problem for me (I have no other OVPN clients in my LAN subnet).Thanks,
Zbigniew -
Fixing openvpn I've been meaning to get around to for a long time. It too has (several) overlarge and undermanaged internal buffers for low bandwidths, and when it gets behind on crypting things as it appears to be in your case, it bloats there.
But it shouldn't affect the nonvpn-ed traffic . It also sounds as though you might be running out of cpu on your device which is affecting the other traffic?
vpns that do bad queue management internally are legion. The ipsec implementation in linux interoperates with fq_codel pretty well, but openvpn can be really poor, and even when wireguard runs out of cpu, it's bad:
http://blog.cerowrt.org/post/wireguard/
I've been trying to encourage vpn designers to pay attention to the bufferbloat work, and use tests like the rrul test above to make sure interactive traffic can fly through it. I've longed to produce a userspace version of openvpn/wireguard/etc that implemented fq_codel and "crypto queue limits", modeled on the successful BQL and AQL implementations in linux - but lack the budget to try myself.
-
It's certainly not lack of CPU power on pfSense device. pfSense is running on ESXi VM which in turn runs on Intel NUC with 2C/4T i3-8109U CPU. Basically CPU is rather severely underutilized. Neither amount of RAM is an issue here.
It's quite strange that I had no such issues with bufferbloating at all when the router (that pfSense replaced recently) was Asus RT-AC68U. But Asus ran out of scalability in certain areas and I wanted better functionality here an there (think pfBlockerNG but not only this) so I switched to pfSense and made Asus just another WiFi AP. This put me into bufferbloat issues instead.
Another weird thing is that I have another client in LAN that connects to another VPN server (this time this is Cisco AnyConnect + Cisco ASA on the remote end, so IPSEC) and I cannot observe similar issues here. I mean it does not create issues for other LAN clients when data is copied over this tunnel (but that client with OVPN tunnel does impact this IPSEC client).
Since for whatever reasons OVPN tunnel bloats in certain conditions, I just would like to try dedicating separate queues for just OVPN traffic. I think Codel works as expected by dropping frames in bufferbloat conditions (generated by OVPN) but since there's only one up/down queue, it drops frames for other traffic that should be unaffected.
Can someone give me a hint how to configure it so that OVPN traffic from LAN to WAN (or traffic from specific LAN client) is assigned to separate Codel queues ? I'm hoping it will help. -
My big goto test is flent's rrul test (on and off the vpn) and a packet capture. No idea what you are actually dealing with here. Not every network weirdness in the world is actually due to bufferbloat.
-
Actually what happens is that I have packed drops/high latency when transfers over VPN are getting very slow, not fast. Then VPN server can easily reach half of the speed of my download DSL link (i.e 300Mbit/2=150Mbit) and then everything is OK. There are no issues when VPN is not used at all either. Problem is when the remote end behind VPN (=torrent sources) isn't that fast and download speed drops to say 10Mbit. Then torrent transfers are causing high latency/high packet drop on my link.
This is very similar case to this one (unresolved issue): https://forum.netgate.com/topic/125639/lots-of-packet-loss-and-high-ping-when-torrenting-through-pia-vpn
But it's not PIA VPN that I'm using (it's NordVPN).What is surprising to me that, as said before, I had no such issues when Asus RT-AC68U was my router.