OpenVPN not working after Upgrade from 2.1.2 > 2.1.3

[cthomas]

I've got two pfSense boxes (HA pair) that aren't establishing OpenVPN connections (peer2peer) after the upgrade to 2.1.3 - Service appears to be running but it doesn't look like the client is attempting to initiate the connection.

These are Dell R210's, dual onboard Broadcom NIC with Intel Quad-Port add-in NIC.

Name pfsense.local
Version 2.1.3-RELEASE (amd64)
built on Thu May 01 15:52:13 EDT 2014
FreeBSD 8.3-RELEASE-p16

Platform 			pfSense
CPU Type 			Intel(R) Xeon(R) CPU E3-1270 V2 @ 3.50GHz 8 CPUs: 1 package(s) x 4 core(s) x 2 SMT threads
Uptime 				09 Hours 55 Minutes 42 Seconds
Current date/time 		Sat May 10 19:23:49 EDT 2014

DNS server(s) 			127.0.0.1, 208.67.222.222, 208.67.220.220

Last config change 		Sat May 10 11:52:55 EDT 2014
State table size 		0% (18/3271000)
MBUF Usage 			8% (10246/131072)
Temperature 	   		29.8°C
Load average 			0.00, 0.00, 0.00
CPU usage 	  		0%
Memory usage 			1% of 32716 MB
SWAP usage 	  		0% of 65536 MB
Disk usage 	  		0% of 1.7T 

I stopped and restarted the service for the OpenVPN Client - these are the log file entries immediately following

May 10 19:32:07 	php: rc.start_packages: Restarting/Starting all packages.
May 10 19:32:05 	check_reload_status: Starting packages
May 10 19:32:05 	php: rc.newwanip: pfSense package system has detected an ip change -> 10.51.0.2 ... Restarting packages.
May 10 19:32:05 	check_reload_status: Reloading filter
May 10 19:32:05 	php: rc.newwanip: rc.newwanip: on (IP address: 10.51.0.2) (interface: []) (real interface: ovpnc2).
May 10 19:32:05 	php: rc.newwanip: rc.newwanip: Informational is starting ovpnc2.
May 10 19:32:03 	check_reload_status: rc.newwanip starting ovpnc2
May 10 19:32:03 	kernel: ovpnc2: link state changed to UP
May 10 19:32:03 	check_reload_status: Reloading filter
May 10 19:32:03 	php: /status_services.php: The command '/sbin/route -q delete 10.51.0.2' returned exit code '1', the output was 'route: writing to routing socket: No such process'
May 10 19:31:48 	syslogd: kernel boot file is /boot/kernel/kernel
May 10 19:31:48 	syslogd: exiting on signal 15
May 10 19:31:28 	check_reload_status: Reloading filter
May 10 19:31:28 	kernel: ovpnc2: link state changed to DOWN

May 10 19:32:03 	openvpn[82904]: UDPv4 link remote: [AF_INET]x.x.x.x:1207
May 10 19:32:03 	openvpn[82904]: UDPv4 link local (bound): [AF_INET]x.x.x.x
May 10 19:32:03 	openvpn[81062]: /usr/local/sbin/ovpn-linkup ovpnc2 1500 1560 10.51.0.2 10.51.0.1 init
May 10 19:32:03 	openvpn[81062]: /sbin/ifconfig ovpnc2 10.51.0.2 10.51.0.1 mtu 1500 netmask 255.255.255.255 up
May 10 19:32:03 	openvpn[81062]: do_ifconfig, tt->ipv6=1, tt->did_ifconfig_ipv6_setup=0
May 10 19:32:03 	openvpn[81062]: TUN/TAP device /dev/tun2 opened
May 10 19:32:03 	openvpn[81062]: TUN/TAP device ovpnc2 exists previously, keep at program end
May 10 19:32:03 	openvpn[81062]: NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
May 10 19:32:03 	openvpn[81062]: OpenVPN 2.3.2 amd64-portbld-freebsd8.3 [SSL (OpenSSL)] [LZO] [eurephia] [MH] [IPv6] built on Mar 27 2014

Apparently I forgot to check the 'create backup' checkbox, so I cannot revert to 2.1.2.

Any help would be greatly appreciated!

-ct

[divsys]

If the setup was operational before the upgrade, it may be helpful to add "verb 5" (or higher) to the custom settings in the OpenVPN server and client to get more log information.

It may be completely unrelated, but I recently had a scenario where I was replacing certificates on the fly with an OpenVPN pair and manged to get one instance running with an old certificate and I couldn't kill it through the GUI.  I had to shell in and find the process manually, once it was stopped I could restart and all was well.

Just a thought…..

[cthomas]

I'm able to start and stop the service(s) without a problem, and I've deleted and re-created the client profiles.  (I don't recall if I rebooted between the delete and re-create, might try that.)

[divsys]

A reboot is worth a try at least….

What had bothered me with my scenario, is that I could stop in start in the GUI as well.
But when I did a "ps aux | grep openvpn" in the shell, there was still a "phantom" instance of OpenVPN running.
It's not something I've seen before and was very likely due to me playing around in the GUI doing starts and stops without letting things catch up to me.....

The verb 5 additions are sometimes useful in these cases although you get huge log files, you can watch the OpenVPN instances proceed and sometimes see where they go astray.

Like many others here I end up learning something new every time I dig into the details of pfsense.

[kanzi]

Maybe i got the same problem after migration from 2.0.1 to 2.1.3. In 2.0.1 the openvpn connection works well, but with unchanged clients there is no answer in 2.1.3.

Curiously there seems to be no such process, altough the UI says 'openvpn - OpenVPN server: VPN - status  Running'

$ ps aux | grep openvpn
root    7029  0.0  0.2   548   400  ??  R     4:54PM   0:00.00 grep openvpn

[divsys]

Have you tried to turn on logging of the Firewall->Rules->OpenVPN  rule on the server to verify that traffic is indeed attempting to connect from the client?

If so I would try a full reboot and/or adding the "verb 5" option to the custom option section of the OpenVPN server to see what's happening with the client traffic.

[cthomas]

@divsys:

Have you tried to turn on logging of the Firewall->Rules->OpenVPN  rule on the server to verify that traffic is indeed attempting to connect from the client?

Issue resolved - I had 'also' updated the OpenVPN client to use the CARP VIP instead of WAN address just before the Upgrade, and I wasn't permitting the correct IP on the Server side.

This is why you make one change at a time.  Divsys - thanks.

[divsys]

I'm happy to take the credit - when you do all the work…... ;)