OpenVPN client using 100% of the processor [SOLVED]

cmb

Is that the entirety of the OpenVPN log during the affected time? Doesn't seem like it's doing anything unusual there. Anything in the gateways log?

killerb81

That's the entirety of the VPN log for that time, yes.
Nothing unusual in the gateway logs.

Problem started as soon as I upgraded to 2.1.5, worked for over a year when I installed 2.1.1.

I even did a fresh install of 2.1.5 after I upgraded through the pfsense GUI and I realized this was an issue.

killerb81

I downgraded and did a complete new install to 2.1.1 x32, and so far my issue has not repeated itself.

stephenw10

Did you try 2.1.5 32bit?

Steve

killerb81

I did, but not through a complete fresh install.
I tried it when I updated 2.1.1 via the pfSense GUI.

stephenw10

Just wondering, if you upgraded across architectures (and have a full install) whether you somehow ended up with the wrong file type somewhere. If you'd gone the other way and wound up with a 64bit file in a 32bit install you just get errors but this way I'm unsure.

Steve

serialdie

@killerb81:

That's the entirety of the VPN log for that time, yes.
Nothing unusual in the gateway logs.

Problem started as soon as I upgraded to 2.1.5, worked for over a year when I installed 2.1.1.

I even did a fresh install of 2.1.5 after I upgraded through the pfsense GUI and I realized this was an issue.

This use to happen to me!
I did a fresh install and moved from i386 to amd64 and the issue has not come back… I do get spikes of cpu utilization but what I am seen when it happens is that the system suffers a disconnect from the vpn provider and its trying to reconnect causing a spike in cpu usage.

Also if I start running large download over the vpn it degregate's my WAN connection causing packet loss on the WAN and making the CPU spike and the Gateway monitor to go crazy...

cmb

The problem serialdie described was fixed in some 2.1.x release and is definitely different from OP's described scenario.

You need to go forward, not backward. Try 2.2 and see if it still happens. Running an OpenVPN client vulnerable to Heartbleed connecting out as a client to some untrusted source is nuts. No one under any circumstance should be running 2.1 nor 2.1.1 on account of Heartbleed.

killerb81

So I've upgraded again to 2.1.5 using the GUI auto-updater.
Right away my processor usage hit 50% (one core at 100%).

Here's a few things I noticed:

1. Gateway log is stating: apinger: ALARM: PRIVATEINTERNETACCESSCANADA_VPNV4(10.121.1.5) *** down ***
and apinger: ALARM: PRIVATEINTERNETACCESSUS_VPNV4(10.141.1.5) *** down ***

These are my two VPN connections. Although, they are not down. On the dashboard, under the "Client instance statistics" it shows both my VPNs as being connected and shows the IP addresses they've been assigned. However, under the Gateways sections of the dashboard it shows them as being offline (this has always been the case, even in 2.1.1). See screen shots.

2. When I restart the two OpenVPN services and then go back to the dashboard it shows the Gateways as being connected, but only for a few seconds. Then they go back to an offline state. I've never noticed this before. See screen shot.

Does pfSense think my VPNs aren't connected and then try forever to keep connecting? Is that why the service stays at 100%?
Why would the Gateways show disconnected, if in fact the VPNs are connected?
Is this a new check since version 2.1.1? In 2.1.1 it acted the same way but I have never seen the 100% processor usage issue.

Anyone have any ideas?

Thanks!

Gateway_Alarms.JPG_thumb

Gateways.JPG_thumb

VPN_Connections.JPG_thumb

System_Activity.JPG_thumb

Gateway_after_restarting_service.JPG_thumb

killerb81

So I went in to the gateway settings for those two VPN connections and turned off gateway monitoring.
So far, so good. :)

I restarted the VPN services and so far they're acting as they should.

I'll post later to confirm fix. Thanks everyone!

serialdie

@cmb:

The problem serialdie described was fixed in some 2.1.x release and is definitely different from OP's described scenario.

You need to go forward, not backward. Try 2.2 and see if it still happens. Running an OpenVPN client vulnerable to Heartbleed connecting out as a client to some untrusted source is nuts. No one under any circumstance should be running 2.1 nor 2.1.1 on account of Heartbleed.

Thanks for the clarification cmb.

Regards.

stephenw10

I notice the gateways are in the 10.X.X.X subnet. Have they always been? Perhaps there was some routing issue there. The remote IPs are not in that subnet.
I recently had to change the monitoring IP on my home connection after some upgrade my ISP made rendered their gateway unpingable. >:( Hard to see how that relates to updating pfSense though. Perhaps your VPN provider sees the differing SSL versions and puts you in a subnet with all the other heartbleed vulnerables! ;)

Steve

killerb81

Haha. Well I hope that's not the case.

Their gateway addresses have always been in the 10.x.x.x range from what I can remember.

killerb81

So I may have spoke too soon.

It was running all day with no issues after I did the auto-update to 2.1.5 (x32) under relatively low internet use (I was at work all day).
When I got home I decided to do a fresh install using the x64 version of 2.1.5. I made the same change to it (turned off gateway monitoring for my VPNs) then went and watched some TV.

Came back just now and the processor was at 57% (one core @ 100%) and had been that way for almost two hours according to the RRD graph.

Nothing in the gateway logs, and messages like these in the OpenVPN logs:

Nov 10 20:10:34 openvpn[88844]: /sbin/route add -net 10.125.1.1 10.125.1.5 255.255.255.255
Nov 10 20:10:34 openvpn[88844]: Initialization Sequence Completed
Nov 10 20:15:25 openvpn[88844]: MANAGEMENT: Client connected from /var/etc/openvpn/client1.sock
Nov 10 20:15:25 openvpn[88844]: MANAGEMENT: CMD 'state 1'
Nov 10 20:15:25 openvpn[88844]: MANAGEMENT: CMD 'status 2'
Nov 10 20:15:25 openvpn[88844]: MANAGEMENT: Client disconnected
Nov 10 20:15:25 openvpn[55911]: MANAGEMENT: Client connected from /var/etc/openvpn/client2.sock
Nov 10 20:15:25 openvpn[55911]: MANAGEMENT: CMD 'state 1'
Nov 10 20:15:25 openvpn[55911]: MANAGEMENT: CMD 'status 2'
Nov 10 20:15:25 openvpn[55911]: MANAGEMENT: Client disconnected

I'm not sure what it means by client disconnected as both tunnels are still up and running fine.

Any ideas? Starting to get frustrated with this one.

serialdie

@killerb81:

So I may have spoke too soon.

It was running all day with no issues after I did the auto-update to 2.1.5 (x32) under relatively low internet use (I was at work all day).
When I got home I decided to do a fresh install using the x64 version of 2.1.5. I made the same change to it (turned off gateway monitoring for my VPNs) then went and watched some TV.

Came back just now and the processor was at 57% (one core @ 100%) and had been that way for almost two hours according to the RRD graph.

Nothing in the gateway logs, and messages like these in the OpenVPN logs:

Nov 10 20:10:34 openvpn[88844]: /sbin/route add -net 10.125.1.1 10.125.1.5 255.255.255.255
Nov 10 20:10:34 openvpn[88844]: Initialization Sequence Completed
Nov 10 20:15:25 openvpn[88844]: MANAGEMENT: Client connected from /var/etc/openvpn/client1.sock
Nov 10 20:15:25 openvpn[88844]: MANAGEMENT: CMD 'state 1'
Nov 10 20:15:25 openvpn[88844]: MANAGEMENT: CMD 'status 2'
Nov 10 20:15:25 openvpn[88844]: MANAGEMENT: Client disconnected
Nov 10 20:15:25 openvpn[55911]: MANAGEMENT: Client connected from /var/etc/openvpn/client2.sock
Nov 10 20:15:25 openvpn[55911]: MANAGEMENT: CMD 'state 1'
Nov 10 20:15:25 openvpn[55911]: MANAGEMENT: CMD 'status 2'
Nov 10 20:15:25 openvpn[55911]: MANAGEMENT: Client disconnected

I'm not sure what it means by client disconnected as both tunnels are still up and running fine.

Any ideas? Starting to get frustrated with this one.

This is what happens to me with one of my VPN end points and what causes my CPU spikes as well.
Basically you are loosing your connection and the system is trying to reconnect and than it establish the connection which by than your CPU should go down on utilization.

killerb81

Thing is, I didn't loose my connection.
As a test, I started streaming Netflix on my desktop that was routed through one of those tunnels.

On my other screen I was watching the "System Activity" screen in pfSense.

Netflix connected and started streaming. As it started to stream the "OpenVPN –config" process started climbing the list of % processor usage.
It got to 100% of the core it was using then stayed there, even after I shut down Netflix and anything else that was using the connection.

That's where it sat... at 100%.

serialdie

I am going to BumP this thread.

I have this same issue and is causing major cpu spikes.

killerb81

I went back to 2.1.1.. it's the only release I've tried that doesn't have this issue.
I'll try 2.2 when it's stable… until then it looks like it'll have to do.

nadir.latif

Hi,

We had problems with high cpu. enabling device polling reduced the cpu from 80% to 40%. you may want to read this post. https://forum.pfsense.org/index.php?topic=77963.msg462019#msg462019

Nadir Latif

serialdie

@nadir.latif:

Hi,

We had problems with high cpu. enabling device polling reduced the cpu from 80% to 40%. you may want to read this post. https://forum.pfsense.org/index.php?topic=77963.msg462019#msg462019

Nadir Latif

That's not recommended.
My issue has stabilized after I worked with my traffic shapers.