OpenVPN up but unable to contact daemon

phil.davis

Just want to document that this happens, and can happen even on an APU system with 2GB memory and running 2.2. I was connected from the office to my test system at home over the OpenVPN site-to-site link, so the link was definitely up. But the dashboard said "unable to contact daemon".
There was no PID file:

[2.2-BETA][root@apu22.localdomain]/var/etc/openvpn: ls -l /var/run
total 96
-rw-r--r--  1 root  wheel    6 Oct 26 21:28 apinger.pid
-rw-r--r--  1 root  wheel   72 Oct 29 09:21 apinger.status
srw-rw-rw-  1 root  wheel    0 Oct 26 21:28 check_reload_status
-rw-------  1 root  wheel    5 Oct 28 03:01 cron.pid
-rw-------  1 root  wheel    3 Oct 26 21:28 devd.pid
srw-rw-rw-  1 root  wheel    0 Oct 26 21:28 devd.pipe
srw-rw-rw-  1 root  wheel    0 Oct 26 21:28 devd.seqpacket.pipe
-rw-------  1 root  wheel    5 Oct 26 21:28 dhclient.re1.pid
-rw-r--r--  1 root  wheel    6 Oct 26 21:29 dnsmasq.pid
-rw-r--r--  1 root  wheel    6 Oct 26 21:29 expire_accounts.pid
-rw-r--r--  1 root  wheel    4 Oct 29 09:21 filter_reload_status
-rw-r--r--  1 root  wheel    6 Oct 26 21:28 filterlog.pid
-rw-------  1 root  wheel    5 Oct 26 21:29 inetd.pid
-r--r--r--  1 root  wheel  173 Oct 26 21:28 ld-elf.so.hints
-r--r--r--  1 root  wheel  188 Oct 28 03:01 ld-elf.so.hints.sudo-amd64
-r--r--r--  1 root  wheel  139 Oct 26 21:28 ld-elf32.so.hints
-r--r--r--  1 root  wheel  174 Oct 28 03:01 ld-elf32.so.hints.sudo-amd64
-rw-r--r--  1 root  wheel    6 Oct 26 21:29 lighty-webConfigurator.pid
srw-rw-rw-  1 root  wheel    0 Oct 26 21:28 log
srw-------  1 root  wheel    0 Oct 26 21:28 logpriv
-rw-r--r--  1 root  wheel    5 Oct 28 03:02 ntpd.pid
-rw-r--r--  1 root  wheel    3 Oct 26 21:28 php-fpm.pid
srw-------  1 root  wheel    0 Oct 26 21:28 php-fpm.socket
-rw-r--r--  1 root  wheel   58 Oct 26 21:28 php_modules_load_errors.txt
-rw-r--r--  1 root  wheel    6 Oct 26 21:29 ping_hosts.pid
-rw-r--r--  1 root  wheel    5 Oct 26 21:28 sshd.pid
-rw-------  1 root  wheel    4 Oct 26 21:28 syslog.pid
-rw-r--r--  1 root  wheel    6 Oct 26 21:29 update_alias_url_data.pid
-rw-r--r--  1 root  wheel    5 Oct 26 21:29 updaterrd.sh.pid
-rw-r--r--  1 root  wheel    0 Oct 26 21:28 utmp
-rw-r--r--  1 root  wheel  394 Oct 29 09:11 utx.active

Then I made one to match the running process:

[2.2-BETA][root@apu22.localdomain]/var/etc/openvpn: ps aux | grep openvpn
root   66268   0.0  0.3  21592  5452  -  Ss   Tue03AM    0:06.99 /usr/local/sbin/openvpn --config /var/etc/openvpn/client1.conf
root   50638   0.0  0.1  18816  2304  0  S+    9:22AM    0:00.00 grep openvpn
[2.2-BETA][root@apu22.localdomain]/var/etc/openvpn: echo 66268 > /var/run/openvpn_client1.pid

and then the dashboard widget (and Status:Services…) could find the process and report it up.
I know this has been reported in the past, somewhere in the forums.
So there is some way that the PID file can go missing, probably in amongst WAN going down/up. One day someone (me?) will work out the timing or whatever issue that causes this.

kejianshi

Yep - I have seen this on a vm running under esxi
Quite annoying.

jimp

Is this client using a hostname to find the server, by chance?

kejianshi

DNS name.

jimp

I don't see it right now but I think there is an open redmine ticket for that. Something happens to OpenVPN when it stops and restarts when using a hostname and infinite resolve. Somehow it ends up with a different process ID disconnected from the management port.

kejianshi

Yes - I get that on some installs but not others.
Seems it never really stops anything from working for me.

phil.davis

My clients all use a public DynDNS name that is kept up-to-date by the server-end pfSense (that has multi-WAN and offers the server end on the highest tier WAN of a gateway group…).
I haven't tried hard to find a reproducible test case, it just happens from time to time. When I am at home I will try some combinations of failing the client-end link, switching the server end link and name... to see if I can induce it.
From an end-user perspective it is not a show-stopper - user traffic is still passing through the tunnel.

cmb

That's probably this.
https://redmine.pfsense.org/issues/3894

I haven't had time to dig far enough to find out where and why it's getting started multiple times in that circumstance.

kejianshi

Do you believe that running a pfsense instance in esxi might make this condition more likely to occur?
Its the only time I ever see it myself and the setting on physical machines are not really any different.

cmb

If there's something about your ESX environment or setup in general that makes those VMs network connectivity take longer to come online, possibly. Generally speaking that wouldn't be the case though. I was testing and replicating it on a VK-T40E4 appliance where I blocked or degraded its network access upstream on its WAN network. Same end result with a VM.

kejianshi

Possibly. Slower HDD access?
I think the one in question is spread pretty thin. The drive is running on a NAS connected with a gigabit switch and maybe 10 VMs sharing access to the NAS on that one switch.
I could see it running abit slow at times. That is the one that is consistently having this issue no matter if it is wiped and completely reinstalled.

charliem

I just saw this today, first time I updated snapshots after installing OpenVPN. Bare hardware, not stretched at all (Atom D2550). Client connection to Private Internet Access was shown as down, yet it was up & running. Kill by hand, restart by GUI, and things are fine.