OpenVPN client using 100% of the processor [SOLVED]

killerb81 · Jul 9, 2016, 1:37 PM

Hi there, I've been having a really annoying problem since I updated to 2.1.5 from 2.1.1. (I was on x32 before, now I'm using the x64 version).

I'm running pfSense on a D2500CC Atom board (dual-core 1.8GHz). I'm on a 25Mbit internet connection. I have two OpenVPN client connections that are always connected.
I route hosts on my LAN to whichever OpenVPN connection I want using LAN rules.

My issue is that whenever the router is under heavy load (speed tests, usenet downloading, anything that takes me close to my 25Mbit ceiling), my OpenVPN clients (sometimes only one of them) takes up 100% use of the core that it's using, and then stays like that until I restart the OpenVPN service that's causing the issue.

I've provided screen shots of the "system activity" page which shows processes and CPU usage (look at the top two items).
I've also provided the errors showing in the OpenVPN log and the graph showing processor usage. You can see in the graph that it started happening yesterday and I didn't catch it, so the process that was causing the issue stayed at 100% until I restarted the services this morning.

This is a real annoyance because when this happens the temperature steadily increases. I'm using a fanless setup and therefore need to keep this as cool as possible.

Has anyone run into this before or have any ideas on what I can do?

Oh, another thing.. in the OpenVPN log the entries are saying 'Client lost connection' (or something to that affect), but they actually haven't… I'm still able to connect through those tunnels when it's saying that.

Is there some type of scripting that I can add that could detect when a process hits 100% and automatically restarts the service? I don't know...

Thanks in advance for the help!

cmb · Nov 8, 2014, 5:31 PM

Is that the entirety of the OpenVPN log during the affected time? Doesn't seem like it's doing anything unusual there. Anything in the gateways log?

killerb81 · Nov 8, 2014, 5:40 PM

That's the entirety of the VPN log for that time, yes.
Nothing unusual in the gateway logs.

Problem started as soon as I upgraded to 2.1.5, worked for over a year when I installed 2.1.1.

I even did a fresh install of 2.1.5 after I upgraded through the pfsense GUI and I realized this was an issue.

killerb81 · Nov 8, 2014, 10:26 PM

I downgraded and did a complete new install to 2.1.1 x32, and so far my issue has not repeated itself.

stephenw10 · Nov 9, 2014, 12:26 AM

Did you try 2.1.5 32bit?

Steve

killerb81 · Nov 9, 2014, 12:28 AM

I did, but not through a complete fresh install.
I tried it when I updated 2.1.1 via the pfSense GUI.

stephenw10 · Nov 9, 2014, 1:27 AM

Just wondering, if you upgraded across architectures (and have a full install) whether you somehow ended up with the wrong file type somewhere. If you'd gone the other way and wound up with a 64bit file in a 32bit install you just get errors but this way I'm unsure.

Steve

serialdie · Nov 9, 2014, 6:03 AM

@killerb81:

That's the entirety of the VPN log for that time, yes.
Nothing unusual in the gateway logs.

Problem started as soon as I upgraded to 2.1.5, worked for over a year when I installed 2.1.1.

I even did a fresh install of 2.1.5 after I upgraded through the pfsense GUI and I realized this was an issue.

This use to happen to me!
I did a fresh install and moved from i386 to amd64 and the issue has not come back… I do get spikes of cpu utilization but what I am seen when it happens is that the system suffers a disconnect from the vpn provider and its trying to reconnect causing a spike in cpu usage.

Also if I start running large download over the vpn it degregate's my WAN connection causing packet loss on the WAN and making the CPU spike and the Gateway monitor to go crazy...

cmb · Nov 10, 2014, 5:46 AM

The problem serialdie described was fixed in some 2.1.x release and is definitely different from OP's described scenario.

You need to go forward, not backward. Try 2.2 and see if it still happens. Running an OpenVPN client vulnerable to Heartbleed connecting out as a client to some untrusted source is nuts. No one under any circumstance should be running 2.1 nor 2.1.1 on account of Heartbleed.

killerb81 · Nov 10, 2014, 3:44 PM

So I've upgraded again to 2.1.5 using the GUI auto-updater.
Right away my processor usage hit 50% (one core at 100%).

Here's a few things I noticed:

1. Gateway log is stating: apinger: ALARM: PRIVATEINTERNETACCESSCANADA_VPNV4(10.121.1.5) *** down ***
and apinger: ALARM: PRIVATEINTERNETACCESSUS_VPNV4(10.141.1.5) *** down ***

These are my two VPN connections. Although, they are not down. On the dashboard, under the "Client instance statistics" it shows both my VPNs as being connected and shows the IP addresses they've been assigned. However, under the Gateways sections of the dashboard it shows them as being offline (this has always been the case, even in 2.1.1). See screen shots.

2. When I restart the two OpenVPN services and then go back to the dashboard it shows the Gateways as being connected, but only for a few seconds. Then they go back to an offline state. I've never noticed this before. See screen shot.

Does pfSense think my VPNs aren't connected and then try forever to keep connecting? Is that why the service stays at 100%?
Why would the Gateways show disconnected, if in fact the VPNs are connected?
Is this a new check since version 2.1.1? In 2.1.1 it acted the same way but I have never seen the 100% processor usage issue.

Anyone have any ideas?

Thanks!

killerb81 · Nov 10, 2014, 4:06 PM

So I went in to the gateway settings for those two VPN connections and turned off gateway monitoring.
So far, so good. :)

I restarted the VPN services and so far they're acting as they should.

I'll post later to confirm fix. Thanks everyone!

serialdie · Nov 10, 2014, 4:24 PM

@cmb:

The problem serialdie described was fixed in some 2.1.x release and is definitely different from OP's described scenario.

You need to go forward, not backward. Try 2.2 and see if it still happens. Running an OpenVPN client vulnerable to Heartbleed connecting out as a client to some untrusted source is nuts. No one under any circumstance should be running 2.1 nor 2.1.1 on account of Heartbleed.

Thanks for the clarification cmb.

Regards.

stephenw10 · Nov 10, 2014, 7:57 PM

I notice the gateways are in the 10.X.X.X subnet. Have they always been? Perhaps there was some routing issue there. The remote IPs are not in that subnet.
I recently had to change the monitoring IP on my home connection after some upgrade my ISP made rendered their gateway unpingable. >:( Hard to see how that relates to updating pfSense though. Perhaps your VPN provider sees the differing SSL versions and puts you in a subnet with all the other heartbleed vulnerables! ;)

Steve

killerb81 · Nov 10, 2014, 8:25 PM

Haha. Well I hope that's not the case.

Their gateway addresses have always been in the 10.x.x.x range from what I can remember.

killerb81 · Nov 11, 2014, 1:17 AM

So I may have spoke too soon.

It was running all day with no issues after I did the auto-update to 2.1.5 (x32) under relatively low internet use (I was at work all day).
When I got home I decided to do a fresh install using the x64 version of 2.1.5. I made the same change to it (turned off gateway monitoring for my VPNs) then went and watched some TV.

Came back just now and the processor was at 57% (one core @ 100%) and had been that way for almost two hours according to the RRD graph.

Nothing in the gateway logs, and messages like these in the OpenVPN logs:

Nov 10 20:10:34 openvpn[88844]: /sbin/route add -net 10.125.1.1 10.125.1.5 255.255.255.255
Nov 10 20:10:34 openvpn[88844]: Initialization Sequence Completed
Nov 10 20:15:25 openvpn[88844]: MANAGEMENT: Client connected from /var/etc/openvpn/client1.sock
Nov 10 20:15:25 openvpn[88844]: MANAGEMENT: CMD 'state 1'
Nov 10 20:15:25 openvpn[88844]: MANAGEMENT: CMD 'status 2'
Nov 10 20:15:25 openvpn[88844]: MANAGEMENT: Client disconnected
Nov 10 20:15:25 openvpn[55911]: MANAGEMENT: Client connected from /var/etc/openvpn/client2.sock
Nov 10 20:15:25 openvpn[55911]: MANAGEMENT: CMD 'state 1'
Nov 10 20:15:25 openvpn[55911]: MANAGEMENT: CMD 'status 2'
Nov 10 20:15:25 openvpn[55911]: MANAGEMENT: Client disconnected

I'm not sure what it means by client disconnected as both tunnels are still up and running fine.

Any ideas? Starting to get frustrated with this one.

serialdie · Nov 11, 2014, 1:32 AM

@killerb81:

So I may have spoke too soon.

It was running all day with no issues after I did the auto-update to 2.1.5 (x32) under relatively low internet use (I was at work all day).
When I got home I decided to do a fresh install using the x64 version of 2.1.5. I made the same change to it (turned off gateway monitoring for my VPNs) then went and watched some TV.

Came back just now and the processor was at 57% (one core @ 100%) and had been that way for almost two hours according to the RRD graph.

Nothing in the gateway logs, and messages like these in the OpenVPN logs:

Nov 10 20:10:34 openvpn[88844]: /sbin/route add -net 10.125.1.1 10.125.1.5 255.255.255.255
Nov 10 20:10:34 openvpn[88844]: Initialization Sequence Completed
Nov 10 20:15:25 openvpn[88844]: MANAGEMENT: Client connected from /var/etc/openvpn/client1.sock
Nov 10 20:15:25 openvpn[88844]: MANAGEMENT: CMD 'state 1'
Nov 10 20:15:25 openvpn[88844]: MANAGEMENT: CMD 'status 2'
Nov 10 20:15:25 openvpn[88844]: MANAGEMENT: Client disconnected
Nov 10 20:15:25 openvpn[55911]: MANAGEMENT: Client connected from /var/etc/openvpn/client2.sock
Nov 10 20:15:25 openvpn[55911]: MANAGEMENT: CMD 'state 1'
Nov 10 20:15:25 openvpn[55911]: MANAGEMENT: CMD 'status 2'
Nov 10 20:15:25 openvpn[55911]: MANAGEMENT: Client disconnected

I'm not sure what it means by client disconnected as both tunnels are still up and running fine.

Any ideas? Starting to get frustrated with this one.

This is what happens to me with one of my VPN end points and what causes my CPU spikes as well.
Basically you are loosing your connection and the system is trying to reconnect and than it establish the connection which by than your CPU should go down on utilization.

killerb81 · Nov 11, 2014, 1:38 AM

Thing is, I didn't loose my connection.
As a test, I started streaming Netflix on my desktop that was routed through one of those tunnels.

On my other screen I was watching the "System Activity" screen in pfSense.

Netflix connected and started streaming. As it started to stream the "OpenVPN –config" process started climbing the list of % processor usage.
It got to 100% of the core it was using then stayed there, even after I shut down Netflix and anything else that was using the connection.

That's where it sat... at 100%.

serialdie · Nov 13, 2014, 4:04 AM

I am going to BumP this thread.

I have this same issue and is causing major cpu spikes.

killerb81 · Nov 13, 2014, 6:50 PM

I went back to 2.1.1.. it's the only release I've tried that doesn't have this issue.
I'll try 2.2 when it's stable… until then it looks like it'll have to do.

nadir.latif · Nov 18, 2014, 5:07 PM

Hi,

We had problems with high cpu. enabling device polling reduced the cpu from 80% to 40%. you may want to read this post. https://forum.pfsense.org/index.php?topic=77963.msg462019#msg462019

Nadir Latif