[RESOLVED] OpenVPN client failure - No buffer space available (code=55)
-
Running 2.2-RC (i386)
built on Thu Dec 11 03:41:12 CST 2014
FreeBSD 10.1-RELEASE-p1Had this happen on Monday, and again today (Friday). Using OpenVPN client as a site to site tunnel to a remote office 24/7.
On Monday, I was running one of the betas from the previous Wednesday (12/3).Connection seems to randomly stop working. RRD graphs show packet loss, gateway monitor shows down, OpenVPN log becomes flooded (few 100 msgs / second) with:
write UDPv4: No buffer space available (code=55)
Forcing a SIGTERM[hard] & restart brings the link back up again.
This system was upgraded from the 2.1.x series. Is there a system level buffer that is now too small after the upgrade?
Should I report this as a bug in redmine?
The GUI doesn't show what happened before the "No buffer space" messages. Is that contained in a raw log somewhere? Would it be useful? -
CMB commented about this error here: https://forum.pfsense.org/index.php?topic=40405.msg208614#msg208614, and another discussion is here: https://code.google.com/p/tunnelblick/issues/detail?id=44#
I have no personal experience with it, but based on above it appears to be triggered by a routing issue. Not sure why a working 2.1.x config would suddenly break 2.2 though.
@PJ2:
The GUI doesn't show what happened before the "No buffer space" messages. Is that contained in a raw log somewhere? Would it be useful?
You can increase the number of log lines presented in the gui on the 'status', 'system log', 'settings' tab. Or you can use the console and 'clog /var/log/openvpn.log |less' to see the entire set of the latest logs. Also, note that verbosity of the openvpn logging is set in the 'advanced' section of each openvpn connection definition.
-
That's almost certainly the situation noted in the thread charliem linked.
https://forum.pfsense.org/index.php?topic=40405.msg208614#msg208614Nothing in routing would change just by upgrading. That also shouldn't be inconsistent, though it could be if something else outside OpenVPN is modifying the routing table.
What routes are you pushing across? Only private networks, or some public networks there as well? Any dynamic routing protocols (OSPF, BGP, etc.) running?
On the client side, check Diag>Routes for any entries for the OpenVPN server's public IP when it's an issue. The only possibility to get that log should be if the public IP ends up having a route inside the VPN.
-
Charliem and CMB - thanks for the feedback.
I'll check my routing for a feedback loop. I've reworked things a number of times, so it's possible I lost track of something. Document, document, document…
As a side note, the raw log file isn't very helpful at the default logging level. Hundreds / thousands of no buffer space lines recorded over a 2-3 second period, and then the restart, but that's all that's in there. I'm assuming the log has a rollover limit somewhere. For future reference, is it possible for me to change that? I've got plenty of disk space to sacrifice.
-
Status->System Logs, Settings has a Log File Size setting from 2.2 on.
Worth a try, but I suspect in this particular case there is going to be a huge number of the repeated messages that swamp even a large circular log. -
Thanks Phil - Yes - I typically set that for 500 lines, but in this case, that only covered a second or two of messages. I was hoping that was just a setting that controlled how many lines the GUI displayed, and that the actual log files had a different rollover setting. I really don't want to display 50k lines in my GUI if I can get it from the command line.
Appreciate the feedback.
-
The log settings now have the number of linse to display in the GUI (was there prior to 2.2) and a new box "Log File Size (Bytes)".
Put a bigger number in there, and it will keep more on disk. Then you can look at it with "clog" from the command line. -
Awesome! I can tell I need to spend some time going through the details of each page so I can get a feel for new options / changes.
I did find some unnecessary (and potentially harmful) routes being setup in the OpenVPN config, which I have removed. I'm not sure how to attempt to trigger the buffer issue, so I'll keep an eye on this and update the subject line to resolved if I don't see it again in the next 10 day or so.
Thanks for the guidance and the features!