Consitent intermittent latency/loss spikes with wireguard tunnel.
-
[SOLVED]
My current setup involves a Wireguard tunnel with a Mullvad wireguard server configured as peer. The interface "WG1" is assigned and enabled, and a gateway exists for it.I use this tunnel to route certain LAN clients through using firewall rules (policy-based routing). All of this works really well, except I noticed my gateway (monitoring the other end, i.e. the Mullvad server) has some intermittent latency:
I noticed the same issue using Smokeping, which is going through the wireguard tunnel:
These latency/loss spikes only happen through the tunnel (edit: seems to be every 10 minutes on the dot), monitoring of the Mullvad server IP address over clear net and many other targets also over the clear net don't show any loss or spikes. I also tried changing Mullvad servers and the issue continues.
Has anyone else experienced this? I am trying to isolate the issue (pfsense/wireguard/freebsd) or Mullvad, but didn't want to create tons of other VPN accounts with other providers to check. Any feedback is appreciated :)
EDIT: changing the Mullvad wireguard server to another city completely didn't seem to eliminate this issue, I will keep monitoring this new server location to see if the pattern changes.
EDIT (3/11/21): this issue seems to be unique to Mullvad wireguard tunnels, please see below
-
@xparanoik Funny I ran into this post, but I use the same VPN provider and get the same issue. As I'm playing Stadia latency will pop up at a regular interval. I am monitoring the dashboard and literally nothing changes.
This is something with Pfsense, changing the VPN location doesn't fix anything. Funny enough, this doesn't happen with OpenVPN.
Does ANYONE have any ideas on this?
-
@heatmiser I've had the same issue, along with others, and have reverted to OpenVPN for now. Sucks too because wireguard is so much easier to set up and the performance really is great. Here's to hoping a fix comes sooner than later.
-
@heatmiser i actually catch my gateway latency (monitoring the mullvad wireguard server peer interface) going up slightly on the pfsense dashboard, then reverting to normal. Some on Reddit said this does not occur when they use NordVPN, but it does with Mullvad. Don't know for a fact if that person was using pfSense.
@gabacho4 I haven't been able to pinpoint the issue, going to have to setup a tunnel with a VM somewhere on the cloud to see if I can repeat it usinf pfsense as the client. Then might also need to test it between two Linux VMs to see if it's related to pfSense or not. Hard to tell just yet.
-
@gabacho4 Yep, the Wireguard config is just SO much simpler to implement and so much less overhead in trying to get extra performance. I see dropoffs in the form of connectivity issues on a regular basis while playing Stadia. Ironically these issues don't seem to translate to streaming, perhaps because of buffering, I'm not sure.
@xparanoik Do keep us in the loop on this, I think we've now become a small community since we're the first to really report this (that I've seen).
I'm going to keep testing and going to other forums to see if anybody else has run into this. Perhaps a Dev can come by and give their opinion?
-
@heatmiser I'll definitely post here again once/if I find more info on this issue.
-
For those of you seeing the periodic spikes in latency, do you have anything on the firewall which is reloading the filter at those times? Could be from a number of sources, but scheduled rules are a common cause of periodic filter reloads. If it were those, however, I'd expect the spikes to be every 15 minutes, not 10, and they'd happen around the quarter hours marks, not in between.
The regularity suggests some kind of periodic process, though, since at least on one of the graphs above it is about every 10 minutes.
Check the system logs, cron entries, etc. See if anything correlates with the time at or just before the spikes.
-
@jimp Thanks for checking this thread. That was something I thought of doing, glad you suggested it. I went ahead and checked pretty much all of the logs available through the GUI, and didn't see anything recurring that happens at/near the same time. The firewall reload due to hostname aliases seems to be every 15 minutes. Checked my temporary smokeping again and it does seem ito happen at exactly every 10 minutes, lasts 1 to 2 minutes and comes back to normal immediately.
I am going to spin up a VM somewhere and create a WG tunnel to see if I can rule out Mullvad as the culprit.
-
Update: I created a VM on DigitalOcean, configured wireguard and then created a new tunnel on pfSense (same setup as before) and changed my PBR rules to force smokeping traffic through this new tunnel instead.
Looks like this issue is isolated to Mullvad and not tied to pfsense/freebsd/wireguard.
-
@xparanoik This is great information. Guess we’ll need to pass this on to Mullvad support. I wonder what they have to say?
-
@heatmiser I shared my findings with them, if they reply with anything useful I will post it here :)
-
@xparanoik Excellent work! The followup on this forum is incredible.
-
@xparanoik great find! I'll give mullvad a few days to respond and then consider moving to nord or rolling my own vps if needs be. Thanks for your work!
-
Spun up a Linode instance with wireguard and so far, absolutely no packet loss despite streaming and doing speedtests etc from multiple devices at the same time. Definitely appears to be a Mullvad issue. I generally like not having another box to administer and update and all but, in this case, I might just roll over to Linode permanently.
-
@gabacho4 you know, a VPS for $5/mo is about the same price as Mullvad, so really it's not that bad of an option. If you set up the firewall properly, and do regular security patching, it should be a good alternative. But yeah, one more box to manage lol
-
@xparanoik did Mullvad get back to you? I sent them a note too and they wanted me to do a trace route to the server I use while connected to the VPN and when on WAN. I'll play with it in a while but so far, on Linode, I've had no packet loss at all. So I don't believe it's an issue with my internet provider or my end at all. But I'll wait till it's later when everyone else in town/the country starts to connect to the internet and see how things look. I still think my Linode performance has been better.
-
@gabacho4 Not yet. They originally asked me for traceroutes too, and suggested internet peering issues, etc... They never offered to look into their own systems, but now that I showed them the issue seems to be unique to their tunnels, I am waiting to see what they say.
-
@xparanoik ok. I feel the same way. I have no issues when not connected to Mullvad, via OpenVPN and wireguard with other providers. I have crap internet as a result of packet loss when I am connected with Mullvad. Doesn't really matter what traceroute says or doesn't say.
-
@gabacho4 yep, I am 99% sure it's something on their end.
-
@xparanoik I put a question out to them as well. I wonder if they're going to ignore us?
