OpenVPN stability issues - error 55
-
Hi,
I am routing my whole internal network through openvpn. This works perfect and I have no issues during normal operation.
However, I sometimes get really weird hickups where the OpenVPN client totally fucks up and it cannot reconnect. Actually I cannot even ping SOME addresses from the shell when this happends.
Errors in logs show:
OPT1_VPNV4 193.xxx.xxx.xxx: sendto error: 55
over and over…(replaced real IP with x's).
Weird thing is that I can ping some addresses in shell - other does not work. I am unable to ping VPN server for example.
I have tried increasing ALL kinds of buffers in the system, disabling hardware offloading in drivers, increased TX queuelen, etc, etc. But now I am out of ideas.
NIC's are built in 2xRealtek (Gigabyte motherboard with 4x Celeron cores).
The only solution so far to this problem has been to reboot.
Between the problems, that comes every week or so, I have no other problems. Everything works perfect with no weird things happening on the NIC's or in the network.
-
This is a question for the OpenVPN forum.
What version of pfSense are you using?
-
I am on 2.3.2.
Well, it might seem like an OpenVPN issue but since it triggers a behaviour that is system wide (even normal ping through the shell fails) I am not sure openvpn is the root cause of this issue.
I am interested in the error 55 and how you can get the root cause of it, most of the answers so far has been "out of buffers" but I cannot understand which buffers. I have increased mbufs and socket buffers…
-
Anything in the OpenVPN log? Increase the verbosity to get more detail. Anything in the System log around the time that the problem happens, such as gateway alarms?
-
System logs -> Gateways shows:
Oct 23 20:46:42 dpinger OPT1_VPNV4 193.183.116.193: sendto error: 55
Oct 23 20:46:43 dpinger OPT1_VPNV4 193.183.116.193: sendto error: 55
Oct 23 20:46:43 dpinger OPT1_VPNV4 193.183.116.193: sendto error: 55
Oct 23 20:46:44 dpinger OPT1_VPNV4 193.183.116.193: sendto error: 55
Oct 23 20:46:44 dpinger OPT1_VPNV4 193.183.116.193: sendto error: 55
Oct 23 20:46:45 dpinger OPT1_VPNV4 193.183.116.193: sendto error: 55
Oct 23 20:46:45 dpinger OPT1_VPNV4 193.183.116.193: sendto error: 55
Oct 23 20:46:46 dpinger OPT1_VPNV4 193.183.116.193: sendto error: 55
Oct 23 20:46:46 dpinger OPT1_VPNV4 193.183.116.193: sendto error: 55
Oct 23 20:46:47 dpinger OPT1_VPNV4 193.183.116.193: sendto error: 55and so on…
OpenVPN log is already erased but at verbosity 3 it only showed that it could not find host, so I guess it can't even reach the DNS server due to this error 55.
During this time the ordinary gateway is reachable at all times and shows as green in the gateways status tab.
If it was an OpenVPN issue, I don't think the ping from shell would fail. To me this is some kind of system wide issue that is triggered by openvpn, or what do you guys think?
-
Is this a physical or virtual install?
-
Normal physical install, x64. Nothing weird really, standard setup.
-
There are definitely some quirks with OpenVPN on pfSense. I had to stop using it completely as I found that if the VPN wasn't 100% stable then pfSense would constantly be toggling the firewall up and down, causing constant connectivity issues with all traffic. (does it really need to keep restarting the WHOLE firewall and not just the VPN rules?)
This was extremely annoying, as I only used the VPN for certain IP ranges so if the VPN is having problems it should fail gracefully rather than bringing the whole Internet crashing down.
-
Strange. I've been using it for years with some remote employees and it's worked perfectly for us. We have a 100/100 link that is mostly idle so I don't know how it performs under heavy load, but it's been rock-solid for us.
-
@Alex:
There are definitely some quirks with OpenVPN on pfSense.
I think the quirks might be with your particular setup, not with OpenVPN on pfSense in general.
I've been using it for years at over a dozen sites, and the worst I've encountered is the process crashing and not restarting until you go into the shell and kill a phantom process. I have never had OpenVPN effect anything except the OpenVPN users.
Back on-topic-
OP, Any way you could test on a box with decent nics? Could be hardware, or just general realtek suckiness. -
I found my problem. When OpenVPN disconnects, which happends like once every week, I seem to loose my default gateway! OpenVPN seems to delete its own gateway, but also the default internet gateway. Then nothing works, of course, and if I re-add my default gateway the system is fully functional again.
I guess I need to dig a bit more to find out what is happening, or does someone have any idea about this?
-
No, sorry. Perhaps disable gateway monitoring and see if that makes any difference.
-
Ok thanks for answering.
One more question though: If gateway monitor notices that the vpn gateway is down, will it try to remove it from routing table?
If not, does the gateway monitor ever touch the routing table?
-
Just to report back. I have (hopefully) found my issue.
First issue was that I had chosen openvpn gateway as the default gateway, this is unneccessary as OpenVPN overrides default gateway anyway.
Second issue is more interesting. PfSense forces the "persist-tun" option to the client config file. This will instruct openvpn service to not run up or down scripts, so when you get a ping-timeout and a SIGUSR1 restart, openvpn will disconnect and try to reconnect without touching the routes.
This means that the, now default, gateway pushed by openvpn server earlier will remain. And trying to reconnect OpenVPN using the gateway on the OpenVPN network will of course not work. This is problem I am facing.
Now, since PfSense forces the persist-tun (I have not found any way to remove it), my only solution was to add the option "remap-usr1 SIGHUP". This will not even try to do a nice connection reset and keeping all the settings. It will more or less do a complete restart of openvpn when you get SIGUSR1 signal.
Why is the persist-tun forced into the client file? Options for every single line that file would be better!
-
See here too:
https://forum.pfsense.org/index.php?topic=117557.msg651859#msg651859
