OpenVPN client using 100% of the processor [SOLVED]

stephenw10 · Nov 9, 2014, 1:27 AM

Just wondering, if you upgraded across architectures (and have a full install) whether you somehow ended up with the wrong file type somewhere. If you'd gone the other way and wound up with a 64bit file in a 32bit install you just get errors but this way I'm unsure.

Steve

serialdie · Nov 9, 2014, 6:03 AM

@killerb81:

That's the entirety of the VPN log for that time, yes.
Nothing unusual in the gateway logs.

Problem started as soon as I upgraded to 2.1.5, worked for over a year when I installed 2.1.1.

I even did a fresh install of 2.1.5 after I upgraded through the pfsense GUI and I realized this was an issue.

This use to happen to me!
I did a fresh install and moved from i386 to amd64 and the issue has not come back… I do get spikes of cpu utilization but what I am seen when it happens is that the system suffers a disconnect from the vpn provider and its trying to reconnect causing a spike in cpu usage.

Also if I start running large download over the vpn it degregate's my WAN connection causing packet loss on the WAN and making the CPU spike and the Gateway monitor to go crazy...

cmb · Nov 10, 2014, 5:46 AM

The problem serialdie described was fixed in some 2.1.x release and is definitely different from OP's described scenario.

You need to go forward, not backward. Try 2.2 and see if it still happens. Running an OpenVPN client vulnerable to Heartbleed connecting out as a client to some untrusted source is nuts. No one under any circumstance should be running 2.1 nor 2.1.1 on account of Heartbleed.

killerb81 · Nov 10, 2014, 3:44 PM

So I've upgraded again to 2.1.5 using the GUI auto-updater.
Right away my processor usage hit 50% (one core at 100%).

Here's a few things I noticed:

1. Gateway log is stating: apinger: ALARM: PRIVATEINTERNETACCESSCANADA_VPNV4(10.121.1.5) *** down ***
and apinger: ALARM: PRIVATEINTERNETACCESSUS_VPNV4(10.141.1.5) *** down ***

These are my two VPN connections. Although, they are not down. On the dashboard, under the "Client instance statistics" it shows both my VPNs as being connected and shows the IP addresses they've been assigned. However, under the Gateways sections of the dashboard it shows them as being offline (this has always been the case, even in 2.1.1). See screen shots.

2. When I restart the two OpenVPN services and then go back to the dashboard it shows the Gateways as being connected, but only for a few seconds. Then they go back to an offline state. I've never noticed this before. See screen shot.

Does pfSense think my VPNs aren't connected and then try forever to keep connecting? Is that why the service stays at 100%?
Why would the Gateways show disconnected, if in fact the VPNs are connected?
Is this a new check since version 2.1.1? In 2.1.1 it acted the same way but I have never seen the 100% processor usage issue.

Anyone have any ideas?

Thanks!

killerb81 · Nov 10, 2014, 4:06 PM

So I went in to the gateway settings for those two VPN connections and turned off gateway monitoring.
So far, so good. :)

I restarted the VPN services and so far they're acting as they should.

I'll post later to confirm fix. Thanks everyone!

serialdie · Nov 10, 2014, 4:24 PM

@cmb:

The problem serialdie described was fixed in some 2.1.x release and is definitely different from OP's described scenario.

You need to go forward, not backward. Try 2.2 and see if it still happens. Running an OpenVPN client vulnerable to Heartbleed connecting out as a client to some untrusted source is nuts. No one under any circumstance should be running 2.1 nor 2.1.1 on account of Heartbleed.

Thanks for the clarification cmb.

Regards.

stephenw10 · Nov 10, 2014, 7:57 PM

I notice the gateways are in the 10.X.X.X subnet. Have they always been? Perhaps there was some routing issue there. The remote IPs are not in that subnet.
I recently had to change the monitoring IP on my home connection after some upgrade my ISP made rendered their gateway unpingable. >:( Hard to see how that relates to updating pfSense though. Perhaps your VPN provider sees the differing SSL versions and puts you in a subnet with all the other heartbleed vulnerables! ;)

Steve

killerb81 · Nov 10, 2014, 8:25 PM

Haha. Well I hope that's not the case.

Their gateway addresses have always been in the 10.x.x.x range from what I can remember.

killerb81 · Nov 11, 2014, 1:17 AM

So I may have spoke too soon.

It was running all day with no issues after I did the auto-update to 2.1.5 (x32) under relatively low internet use (I was at work all day).
When I got home I decided to do a fresh install using the x64 version of 2.1.5. I made the same change to it (turned off gateway monitoring for my VPNs) then went and watched some TV.

Came back just now and the processor was at 57% (one core @ 100%) and had been that way for almost two hours according to the RRD graph.

Nothing in the gateway logs, and messages like these in the OpenVPN logs:

Nov 10 20:10:34 openvpn[88844]: /sbin/route add -net 10.125.1.1 10.125.1.5 255.255.255.255
Nov 10 20:10:34 openvpn[88844]: Initialization Sequence Completed
Nov 10 20:15:25 openvpn[88844]: MANAGEMENT: Client connected from /var/etc/openvpn/client1.sock
Nov 10 20:15:25 openvpn[88844]: MANAGEMENT: CMD 'state 1'
Nov 10 20:15:25 openvpn[88844]: MANAGEMENT: CMD 'status 2'
Nov 10 20:15:25 openvpn[88844]: MANAGEMENT: Client disconnected
Nov 10 20:15:25 openvpn[55911]: MANAGEMENT: Client connected from /var/etc/openvpn/client2.sock
Nov 10 20:15:25 openvpn[55911]: MANAGEMENT: CMD 'state 1'
Nov 10 20:15:25 openvpn[55911]: MANAGEMENT: CMD 'status 2'
Nov 10 20:15:25 openvpn[55911]: MANAGEMENT: Client disconnected

I'm not sure what it means by client disconnected as both tunnels are still up and running fine.

Any ideas? Starting to get frustrated with this one.

serialdie · Nov 11, 2014, 1:32 AM

@killerb81:

So I may have spoke too soon.

It was running all day with no issues after I did the auto-update to 2.1.5 (x32) under relatively low internet use (I was at work all day).
When I got home I decided to do a fresh install using the x64 version of 2.1.5. I made the same change to it (turned off gateway monitoring for my VPNs) then went and watched some TV.

Came back just now and the processor was at 57% (one core @ 100%) and had been that way for almost two hours according to the RRD graph.

Nothing in the gateway logs, and messages like these in the OpenVPN logs:

Nov 10 20:10:34 openvpn[88844]: /sbin/route add -net 10.125.1.1 10.125.1.5 255.255.255.255
Nov 10 20:10:34 openvpn[88844]: Initialization Sequence Completed
Nov 10 20:15:25 openvpn[88844]: MANAGEMENT: Client connected from /var/etc/openvpn/client1.sock
Nov 10 20:15:25 openvpn[88844]: MANAGEMENT: CMD 'state 1'
Nov 10 20:15:25 openvpn[88844]: MANAGEMENT: CMD 'status 2'
Nov 10 20:15:25 openvpn[88844]: MANAGEMENT: Client disconnected
Nov 10 20:15:25 openvpn[55911]: MANAGEMENT: Client connected from /var/etc/openvpn/client2.sock
Nov 10 20:15:25 openvpn[55911]: MANAGEMENT: CMD 'state 1'
Nov 10 20:15:25 openvpn[55911]: MANAGEMENT: CMD 'status 2'
Nov 10 20:15:25 openvpn[55911]: MANAGEMENT: Client disconnected

I'm not sure what it means by client disconnected as both tunnels are still up and running fine.

Any ideas? Starting to get frustrated with this one.

This is what happens to me with one of my VPN end points and what causes my CPU spikes as well.
Basically you are loosing your connection and the system is trying to reconnect and than it establish the connection which by than your CPU should go down on utilization.

killerb81 · Nov 11, 2014, 1:38 AM

Thing is, I didn't loose my connection.
As a test, I started streaming Netflix on my desktop that was routed through one of those tunnels.

On my other screen I was watching the "System Activity" screen in pfSense.

Netflix connected and started streaming. As it started to stream the "OpenVPN –config" process started climbing the list of % processor usage.
It got to 100% of the core it was using then stayed there, even after I shut down Netflix and anything else that was using the connection.

That's where it sat... at 100%.

serialdie · Nov 13, 2014, 4:04 AM

I am going to BumP this thread.

I have this same issue and is causing major cpu spikes.

killerb81 · Nov 13, 2014, 6:50 PM

I went back to 2.1.1.. it's the only release I've tried that doesn't have this issue.
I'll try 2.2 when it's stable… until then it looks like it'll have to do.

nadir.latif · Nov 18, 2014, 5:07 PM

Hi,

We had problems with high cpu. enabling device polling reduced the cpu from 80% to 40%. you may want to read this post. https://forum.pfsense.org/index.php?topic=77963.msg462019#msg462019

Nadir Latif

serialdie · Nov 19, 2014, 4:41 AM

@nadir.latif:

Hi,

We had problems with high cpu. enabling device polling reduced the cpu from 80% to 40%. you may want to read this post. https://forum.pfsense.org/index.php?topic=77963.msg462019#msg462019

Nadir Latif

That's not recommended.
My issue has stabilized after I worked with my traffic shapers.

killerb81 · Jun 17, 2016, 9:25 PM

bump.

:(

I upgraded to 2.3.1 -p5 and the issue is back.
I even reinstalled completely after I saw the issue come back… it was working fine for a long time and then... bam - back.

Does anyone have any deffinitive solutions for this? Would be greatly appreciated.

Thing is, I don't lose my connections - they're still connected and working fine. The CPU spikes and the temperature starts rising.
I have to restart the OpenVPN Client services to get it to calm down.

killerb81 · Jul 8, 2016, 6:20 PM

@serialdie

What exactly did you do to your traffic shaping to resolve the issue?
I too have traffic shaping configured.

killerb81 · Jul 8, 2016, 7:11 PM

So, it's interesting.. but I shut off my traffic shaping and the issue seems to have stopped.

Does anyone know why this might be?

killerb81 · Jul 9, 2016, 1:37 PM

*** SOLVED *** (I think)

So after having my traffic shaper turned off for a day, the problem has not been seen.
I'm not sure WHY this is, after all, it was the OpenVPN process that was using all the CPU.

I did a little digging into my traffic shaper before turning it off and noticed that it wasn't setup how I originally configured it.
I had previously set it up using HFSC ques, but just before turning it off I noticed it was using CBQ based ques and some of the other settings had been changed.

I could be wrong, but I think during an update of pfSense these settings somehow got changed… because that's when I noticed the VPN processes ramping up, right after an update.
In the next few days I'm going to setup my traffic shaper again from scratch and see what happens... I'll post back here after I do some testing.

Thanks to all!

alecp · Jul 15, 2016, 3:46 PM

I was just monitoring my firewall after a power outage and found this issue.
I removed the simple traffic-shaper I recently put in place for VoIP and the CPU usage fell to sensible numbers.
I tried putting the shaper back (CBQ) with the wizard but the openvpn usage went back to 100%, so it is not fixed

2.3.1-RELEASE-p5 (amd64)