OpenVPN RTT Way Too High
-
So, I’ve been having this peculiar issue lately. About a couple of weeks ago, I started noticing that my real IP was leaking on Apple TV as I tried to watch shows across various streaming services and would get told that it was unavailable in my region. So, I’d run a quick SpeedTest app on Apple TV, and it’d show me my IP. It made no sense why it’d do that since pfSense is configured to route only VPN traffic to Apple TV (but with no killswitch).
I started investigating, went to the main page of pfSense, and enabled Gateway monitoring. And I noticed that my OpenVPN RTT was going way too high, in a few hundred to even thousands — which would cause the gateway to go offline, let my WAN traffic go through to Apple TV, and leak my IP.
I’m attaching a couple of screenshots so you can see how bad the ping goes, and it only starts to go up when I try to use any device with VPN routing on.
And just now, as of 30mins ago, I ran into this issue again with my ping going just FUBAR. It's super annoying.
I have absolutely zero issues using the internet through pfSense when it’s not routed through OpenVPN, as I have an alias list of IPs that includes my desktop and that I exclude from using any VPN traffic. I am on my computer practically all day doing various things, including gaming, and I never face any issues. It is only happening on the VPN gateway.
Now, I’ve done ping tests to this VPN IP overnight, over hours, and so on and on, and the avg ping never goes beyond 300 even, which is fitting because an avg ping from my location to that VPN’s location is supposed to be around 280 anyway. I’m also attaching a screenshot of WinMTR of that result, so you can see there isn’t any issue with ping when pinging that VPN server IP directly from my ISP internet in a desktop environment.
This started happening about a couple of weeks ago; it was totally fine before that. After running into these issues, I updated my pfSense+ to the latest firmware, but before that, it was running fine for hours on end, streaming for about a month.
One more thing I should note if it helps someone understand why I’m having this issue is that whenever I do a SpeedTest of the VPN via OpenVPN environment through pfSense, as in if I’m doing it from Apple TV or desktop, my ping shows about double on SpeedTest like 500-600. But if I connect to the same host via L2TP/IPSec via Windows PC, the ping stays around 240ms or so for SpeedTests.
So, as you can see, the server itself is fine; I can ping it, and the service is good and all. But when using OpenVPN via pfSense, it causes a severe spike to RTT, and it only does that sometimes, it doesn’t go too high hall the time, sometimes it’ll go and chill around 400, but sometimes it’ll go 600, 800, 1000 and more.
Any help for this would be greatly appreciated, I’ve been trying to troubleshoot this for about two weeks now and spent quite a bit of time with my ISP to figure out if they were throttling my server IP for any reason, but that’s not the case. I even thought maybe my ONT was the issue, but we investigated and ruled that out, too.
Thank you!
p.s. the following info might also help.
Client: pfSense+ 23.05.1-RELEASE running on HP t730 Thin Client - AMD R-Series RX-427BB
OpenVPN Server: UniFi setup at my office in another country. -
@UmerF
Your WAN gateway monitoring checks just the internal interface of the local router.
So first of all set a public monitoring address for the WAN gateway to get an idea about your real internet latency. -
@viragomann that's a good idea, mate. Thanks for the heads up. I don't know why I never bothered putting a monitor IP on WAN. Probably cause none of the tutorials from Lawrence Systems and others I saw putting one, I guess. So, it never registered.
I just put the secondary CloudFlare IP for latency check there and will continue to monitor it. next time my latency on VPN goes way too high. But even right now, at 400ms reported on VPN, my WAN is reporting the right ping as it should.
-
@viragomann oh welp, literally two minutes later as I'm trying to stream a show on Peacock using VPN, ping shot up high and here's some results of WAN and VPN gateway from monitoring.
It's so weird as well cause it's not ALWAYS this high; sometimes for hours it'll just stay around 300-400, and then all of a sudden, it'll spike. Makes no sense what I'm doing wrong here, especially considering it worked totally fine for over a month.
-
@UmerF
Maybe there is something wrong in the OpenVPN settings, e.g. an MTU mismatch. Do you have some special custom settings?Enhance the log level to 4 and check if there are some hints then.
Is the server under your control, so that you can verify that there is all right on the server side?
-
@viragomann so, the VPN settings for this server are straight out of the ovpn file, which was generated by UniFi itself, where the server actually is hosted. Screenshot below from UniFi OS.
After that, I simply used the export wizard facility for this particular VPN and didn't change anything, and let it roll with whatever settings are supposed to be applied automatically. I actually upgraded from CE to pfSense+ just for this, I was using CE for months prior to that with public VPNs but I got tired of their IPs getting blacklisted from streaming services that I had to make a switch.
The server is under my control pretty much I don't have direct access to it at all times, but if there are any changes to be made for this particular OpenVPN setting on the server's end, that's entirely possible. But it's pretty straight forward with UniFi OS and it isn't as much in-depth to set it up, I believe.
Can you elaborate on the log level to 4, please? Do you just want me to change Log Retention Count to 4?
-
@UmerF
That's just the number of log files the system is holding available. And this might be the Unify OS, which I'm not familiar with.
I was talking about the clients log. At the bottom of the settings page you can specify the verbosity level for the logging.
If there is any configuration mismatch it should be written to the log then.
-
@viragomann ahhhhh, my bad. Okay, currently it's set to 3, so I'll bump it up to 4.
Any idea what to look for in logs, any particular keywords and such? It's such a mess in there, hard to make sense of it all. And I'm assuming I'd ideally be checking OpenVPN logs or Gateway logs?
-
@UmerF
As mentioned, could be due to an MTU or MSS mismatch between the OpenVPN server and client. If so OpenVPN writes this into the log.Basically the issue could appear due to packet fragmentation.. So you can also check with ping, what's the max packet size, you can get over the VPN.
-
@viragomann alright, I looked up for MTU/MSS keywords under OpenVPN for system logs in pfSense, and keep seeing this pop up.
Does that tell us anything that we need to know? From my understanding, my PC has a 1500 default MTU, and we probably want to keep it that way. If not, I'm open to making changes. As for pfSense, which MTU/MSS would I want to change? Is it the OpenVPN gateway or my WAN? It's blank in there for it all, and I'm assuming that's also set at 1500?
The screenshot above mentions the payload being 1600 and 1768. Does that mean I have to increase my MTU or something? I'm sorry; first time dealing with this stuff, and I really appreciate your input.
