OpenVPN client using 100% of the processor [SOLVED]



  • Hi there, I've been having a really annoying problem since I updated to 2.1.5 from 2.1.1. (I was on x32 before, now I'm using the x64 version).

    I'm running pfSense on a D2500CC Atom board (dual-core 1.8GHz). I'm on a 25Mbit internet connection. I have two OpenVPN client connections that are always connected.
    I route hosts on my LAN to whichever OpenVPN connection I want using LAN rules.

    My issue is that whenever the router is under heavy load (speed tests, usenet downloading, anything that takes me close to my 25Mbit ceiling), my OpenVPN clients (sometimes only one of them) takes up 100% use of the core that it's using, and then stays like that until I restart the OpenVPN service that's causing the issue.

    I've provided screen shots of the "system activity" page which shows processes and CPU usage (look at the top two items).
    I've also provided the errors showing in the OpenVPN log and the graph showing processor usage. You can see in the graph that it started happening yesterday and I didn't catch it, so the process that was causing the issue stayed at 100% until I restarted the services this morning.

    This is a real annoyance because when this happens the temperature steadily increases. I'm using a fanless setup and therefore need to keep this as cool as possible.

    Has anyone run into this before or have any ideas on what I can do?

    Oh, another thing.. in the OpenVPN log the entries are saying 'Client lost connection' (or something to that affect), but they actually haven't… I'm still able to connect through those tunnels when it's saying that.

    Is there some type of scripting that I can add that could detect when a process hits 100% and automatically restarts the service? I don't know...

    Thanks in advance for the help!







  • Is that the entirety of the OpenVPN log during the affected time? Doesn't seem like it's doing anything unusual there. Anything in the gateways log?



  • That's the entirety of the VPN log for that time, yes.
    Nothing unusual in the gateway logs.

    Problem started as soon as I upgraded to 2.1.5, worked for over a year when I installed 2.1.1.

    I even did a fresh install of 2.1.5 after I upgraded through the pfsense GUI and I realized this was an issue.



  • I downgraded and did a complete new install to 2.1.1 x32, and so far my issue has not repeated itself.


  • Netgate Administrator

    Did you try 2.1.5 32bit?

    Steve



  • I did, but not through a complete fresh install.
    I tried it when I updated 2.1.1 via the pfSense GUI.


  • Netgate Administrator

    Just wondering, if you upgraded across architectures (and have a full install) whether you somehow ended up with the wrong file type somewhere. If you'd gone the other way and wound up with a 64bit file in a 32bit install you just get errors but this way I'm unsure.

    Steve



  • @killerb81:

    That's the entirety of the VPN log for that time, yes.
    Nothing unusual in the gateway logs.

    Problem started as soon as I upgraded to 2.1.5, worked for over a year when I installed 2.1.1.

    I even did a fresh install of 2.1.5 after I upgraded through the pfsense GUI and I realized this was an issue.

    This use to happen to me!
    I did a fresh install and moved from i386 to amd64 and the issue has not come back… I do get spikes of cpu utilization but what I am seen when it happens is that the system suffers a disconnect from the vpn provider and its trying to reconnect causing a spike in cpu usage.

    Also if I start running large download over the vpn it degregate's my WAN connection causing packet loss on the WAN and making the CPU spike and the Gateway monitor to go crazy...



  • The problem serialdie described was fixed in some 2.1.x release and is definitely different from OP's described scenario.

    You need to go forward, not backward. Try 2.2 and see if it still happens. Running an OpenVPN client vulnerable to Heartbleed connecting out as a client to some untrusted source is nuts. No one under any circumstance should be running 2.1 nor 2.1.1 on account of Heartbleed.



  • So I've upgraded again to 2.1.5 using the GUI auto-updater.
    Right away my processor usage hit 50% (one core at 100%).

    Here's a few things I noticed:

    1. Gateway log is stating: apinger: ALARM: PRIVATEINTERNETACCESSCANADA_VPNV4(10.121.1.5) *** down ***
    and apinger: ALARM: PRIVATEINTERNETACCESSUS_VPNV4(10.141.1.5) *** down ***

    These are my two VPN connections. Although, they are not down. On the dashboard, under the "Client instance statistics" it shows both my VPNs as being connected and shows the IP addresses they've been assigned. However, under the Gateways sections of the dashboard it shows them as being offline (this has always been the case, even in 2.1.1). See screen shots.

    2. When I restart the two OpenVPN services and then go back to the dashboard it shows the Gateways as being connected, but only for a few seconds. Then they go back to an offline state. I've never noticed this before. See screen shot.

    Does pfSense think my VPNs aren't connected and then try forever to keep connecting? Is that why the service stays at 100%?
    Why would the Gateways show disconnected, if in fact the VPNs are connected?
    Is this a new check since version 2.1.1? In 2.1.1 it acted the same way but I have never seen the 100% processor usage issue.

    Anyone have any ideas?

    Thanks!












  • So I went in to the gateway settings for those two VPN connections and turned off gateway monitoring.
    So far, so good. :)

    I restarted the VPN services and so far they're acting as they should.

    I'll post later to confirm fix. Thanks everyone!



  • @cmb:

    The problem serialdie described was fixed in some 2.1.x release and is definitely different from OP's described scenario.

    You need to go forward, not backward. Try 2.2 and see if it still happens. Running an OpenVPN client vulnerable to Heartbleed connecting out as a client to some untrusted source is nuts. No one under any circumstance should be running 2.1 nor 2.1.1 on account of Heartbleed.

    Thanks for the clarification cmb.

    Regards.


  • Netgate Administrator

    I notice the gateways are in the 10.X.X.X subnet. Have they always been? Perhaps there was some routing issue there. The remote IPs are not in that subnet.
    I recently had to change the monitoring IP on my home connection after some upgrade my ISP made rendered their gateway unpingable.  >:( Hard to see how that relates to updating pfSense though. Perhaps your VPN provider sees the differing SSL versions and puts you in a subnet with all the other heartbleed vulnerables!  ;)

    Steve



  • Haha. Well I hope that's not the case.

    Their gateway addresses have always been in the 10.x.x.x range from what I can remember.



  • So I may have spoke too soon.

    It was running all day with no issues after I did the auto-update to 2.1.5 (x32) under relatively low internet use (I was at work all day).
    When I got home I decided to do a fresh install using the x64 version of 2.1.5. I made the same change to it (turned off gateway monitoring for my VPNs) then went and watched some TV.

    Came back just now and the processor was at 57% (one core @ 100%) and had been that way for almost two hours according to the RRD graph.

    Nothing in the gateway logs, and messages like these in the OpenVPN logs:

    Nov 10 20:10:34 openvpn[88844]: /sbin/route add -net 10.125.1.1 10.125.1.5 255.255.255.255
    Nov 10 20:10:34 openvpn[88844]: Initialization Sequence Completed
    Nov 10 20:15:25 openvpn[88844]: MANAGEMENT: Client connected from /var/etc/openvpn/client1.sock
    Nov 10 20:15:25 openvpn[88844]: MANAGEMENT: CMD 'state 1'
    Nov 10 20:15:25 openvpn[88844]: MANAGEMENT: CMD 'status 2'
    Nov 10 20:15:25 openvpn[88844]: MANAGEMENT: Client disconnected
    Nov 10 20:15:25 openvpn[55911]: MANAGEMENT: Client connected from /var/etc/openvpn/client2.sock
    Nov 10 20:15:25 openvpn[55911]: MANAGEMENT: CMD 'state 1'
    Nov 10 20:15:25 openvpn[55911]: MANAGEMENT: CMD 'status 2'
    Nov 10 20:15:25 openvpn[55911]: MANAGEMENT: Client disconnected

    I'm not sure what it means by client disconnected as both tunnels are still up and running fine.

    Any ideas?  Starting to get frustrated with this one.



  • @killerb81:

    So I may have spoke too soon.

    It was running all day with no issues after I did the auto-update to 2.1.5 (x32) under relatively low internet use (I was at work all day).
    When I got home I decided to do a fresh install using the x64 version of 2.1.5. I made the same change to it (turned off gateway monitoring for my VPNs) then went and watched some TV.

    Came back just now and the processor was at 57% (one core @ 100%) and had been that way for almost two hours according to the RRD graph.

    Nothing in the gateway logs, and messages like these in the OpenVPN logs:

    Nov 10 20:10:34 openvpn[88844]: /sbin/route add -net 10.125.1.1 10.125.1.5 255.255.255.255
    Nov 10 20:10:34 openvpn[88844]: Initialization Sequence Completed
    Nov 10 20:15:25 openvpn[88844]: MANAGEMENT: Client connected from /var/etc/openvpn/client1.sock
    Nov 10 20:15:25 openvpn[88844]: MANAGEMENT: CMD 'state 1'
    Nov 10 20:15:25 openvpn[88844]: MANAGEMENT: CMD 'status 2'
    Nov 10 20:15:25 openvpn[88844]: MANAGEMENT: Client disconnected
    Nov 10 20:15:25 openvpn[55911]: MANAGEMENT: Client connected from /var/etc/openvpn/client2.sock
    Nov 10 20:15:25 openvpn[55911]: MANAGEMENT: CMD 'state 1'
    Nov 10 20:15:25 openvpn[55911]: MANAGEMENT: CMD 'status 2'
    Nov 10 20:15:25 openvpn[55911]: MANAGEMENT: Client disconnected

    I'm not sure what it means by client disconnected as both tunnels are still up and running fine.

    Any ideas?  Starting to get frustrated with this one.

    This is what happens to me with one of my VPN end points and what causes my CPU spikes as well.
    Basically you are loosing your connection and the system is trying to reconnect and than it establish the connection which by than your CPU should go down on utilization.



  • Thing is, I didn't loose my connection.
    As a test, I started streaming Netflix on my desktop that was routed through one of those tunnels.

    On my other screen I was watching the "System Activity" screen in pfSense.

    Netflix connected and started streaming. As it started to stream the "OpenVPN –config" process started climbing the list of % processor usage.
    It got to 100% of the core it was using then stayed there, even after I shut down Netflix and anything else that was using the connection.

    That's where it sat... at 100%.



  • I am going to BumP this thread.

    I have this same issue and is causing major cpu spikes.



  • I went back to 2.1.1.. it's the only release I've tried that doesn't have this issue.
    I'll try 2.2 when it's stable… until then it looks like it'll have to do.



  • Hi,

    We had problems with high cpu. enabling device polling reduced the cpu from 80% to 40%. you may want to read this post. https://forum.pfsense.org/index.php?topic=77963.msg462019#msg462019

    Nadir Latif



  • @nadir.latif:

    Hi,

    We had problems with high cpu. enabling device polling reduced the cpu from 80% to 40%. you may want to read this post. https://forum.pfsense.org/index.php?topic=77963.msg462019#msg462019

    Nadir Latif

    That's not recommended.
    My issue has stabilized after I worked with my traffic shapers.



  • bump.

    :(

    I upgraded to 2.3.1 -p5 and the issue is back.
    I even reinstalled completely after I saw the issue come back… it was working fine for a long time and then... bam - back.

    Does anyone have any deffinitive solutions for this?  Would be greatly appreciated.

    Thing is, I don't lose my connections - they're still connected and working fine.  The CPU spikes and the temperature starts rising.
    I have to restart the OpenVPN Client services to get it to calm down.



  • @serialdie

    What exactly did you do to your traffic shaping to resolve the issue?
    I too have traffic shaping configured.



  • So, it's interesting.. but I shut off my traffic shaping and the issue seems to have stopped.

    Does anyone know why this might be?



  • *** SOLVED ***  (I think)

    So after having my traffic shaper turned off for a day, the problem has not been seen.
    I'm not sure WHY this is, after all, it was the OpenVPN process that was using all the CPU.

    I did a little digging into my traffic shaper before turning it off and noticed that it wasn't setup how I originally configured it.
    I had previously set it up using HFSC ques, but just before turning it off I noticed it was using CBQ based ques and some of the other settings had been changed.

    I could be wrong, but I think during an update of pfSense these settings somehow got changed… because that's when I noticed the VPN processes ramping up, right after an update.
    In the next few days I'm going to setup my traffic shaper again from scratch and see what happens... I'll post back here after I do some testing.

    Thanks to all!



  • I was just monitoring my firewall after a power outage and found this issue.
    I removed the simple traffic-shaper I recently put in place for VoIP and the CPU usage fell to sensible numbers.
    I tried putting the shaper back (CBQ) with the wizard but the openvpn usage went back to 100%, so it is not fixed

    2.3.1-RELEASE-p5 (amd64)