Troubleshooting OpenVPN packet loss
I believe my ISP has an issue, and I'm trying to convince them to fix it, but I'm looking for a way to give them better data to help convince them to fix it.
I'm located in Nepal. I have two ISPs, each with a fiber connection. From each of the WANs I have three VPN tunnels setup to the USA, one to NordVPN and two to private servers running pfSense. I'm running pfSense 2.4.4. I have load balancers set up so that my LAN shares the various VPN connections.
Everything on the VPNs had been working fine. However, a couple of weeks ago one ISP called and said they needed to replace my modem. The previous box was just a modem, but the new box is actually a router/wifi box as well. After some work over about a week to get it to bridge the public IP address to pfSense, everything seemed to be working well.
However, I started to notice that I was getting a lot of packet loss on my VPNs. After doing some monitoring, I discovered that all three VPN tunnels from this ISP are getting packet loss in the evenings. It starts about 6pm, increases gradually until about 9:30pm (when it is around 35-40% packet loss) and then gets better until being back to 0% loss by midnight. This is reflected in the quality graphs of each of the VPN gateways under "Monitoring" as well - a blip every night since they have installed the new modem. However, I don't get any packet loss on the main WAN gateway for that ISP - only the VPN tunnels. This tells me that perhaps the connection to their server is fine, but the issue is from their server going out on the network (possibly in their pipe going out of the country?). But it also seems to be tied to my new modem.
In the evenings my VPN connection is basically unusable. What I've done for now is to set a schedule, and every evening from 6pm-midnight it ignores the one ISP (uses only gateways / VPN tunnels through the other ISP). But of course I would rather be able to use the full bandwidth I am paying for.
I could see it potentially being an MTU issue but that doesn't really make sense with the time-specific nature of the problem.
Am I correct in thinking it must be an ISP issue? Is there anything else I can give them from pfSense to help them troubleshoot the issue? I have sent them the quality graphs and screenshots of the gateways showing the packet loss. Basically all they are doing is calling me at various times through the day and asking if it is working. I don't get the feeling they are doing any troubleshooting on their end.