Any recent changes? Getting hard reset.

JKnott

When I tried to connect with OpenVPN recently, the connection failed. A packet capture shows a few attempts failing with hard reset. This VPN worked well until now and fails in both Linux & Windows.

Suggestions?

tnx jk

JKnott

@JKnott

Forgot to mention, the status shows this:
[error] Unable to contact daemon Service not running?

I also clicked on the Start Service button, but no different.

Gertjan

@JKnott

Something like this :

Connect to the console, or ssh, option 8 :

[25.03-BETA][root@pfSense.bhf.tld]/root: ps aux | grep openvpn
root    93661   0.0  0.2  21260   9528  -  Ss   Wed09      0:01.56 /usr/local/sbin/openvpn --config /var/etc/openvpn/server1/config.ovpn

openvpn, this is a server instance, is running with process ID 93661 - yours will be different.

Keep the console / ssh session open.

On the dashboard, stop the openvpn server process.

On the console /ssh, execute the same command again.
Is it process 93661 now gone ?
If not, fire up a browser and ask how to kill / remove zombie processes.

It's gone, execute :

tail -f /var/log/openvpn.log

and in the dashboard, start the openvpn.
Now look at the logs produced : any errors ? If there was a (startup) error, correct the issue.

Rico

What do you mean by 'hard reset'? Is your pfSense rebooting?

-Rico

Gertjan

@JKnott said in Any recent changes? Getting hard reset.:

[error] Unable to contact daemon Service not running?

This message is generated when the dashboard widget that shows the connected users, if any.

The dashboard script connects to an openVPN service port (socket) :

25.03-BETA][root@pfSense.bhf.tld]/root: sockstat  | grep 'openvpn'
root     openvpn    93661 3   stream /var/etc/openvpn/server1/sock
....

and asks through this socket the server status, and if users are connected.
If that socket doesn't exists, or the process openvpn doesn't exist, you'll see the message " Unable to contact daemon Service not running".

Extra test : when you've stopped the process, go look for the 'file' /var/etc/openvpn/server1/sock
It shouldn't exist anymore. If it is still there, 'kill' (delete) it.
When openvpn is started, it should exist.

edit : the owner of the socket file is the same process ID - 93661 in my case.

JKnott

@Rico said in Any recent changes? Getting hard reset.:

What do you mean by 'hard reset'? Is your pfSense rebooting?

No. That's in the packets from the client to the server. The full text, as displayed in Wireshark, is P_CONTROL_HARD_RESET_CLIENT_V2.

JKnott

@Gertjan

/root: ps aux | grep openvpn
root 25176 0.0 0.1 12752 2356 0 S+ 10:31 0:00.00 grep openvpn

This indicates there is no openvpn process running, only the ps aux command looking for it.

There are 2 buttons on the dashboard, a plain X and X on red circle. Neither stop the process. The plain X is to kill the connection and the red circle one is to halt it. I also tried using the kill command on the process number 12752 and get "No such process".

JKnott

@Gertjan said in Any recent changes? Getting hard reset.:

Extra test : when you've stopped the process, go look for the 'file' /var/etc/openvpn/server1/sock
It shouldn't exist anymore. If it is still there, 'kill' (delete) it.

: cat /var/etc/openvpn/server1/sock
cat: /var/etc/openvpn/server1/sock: Connection refused

Gertjan

@JKnott said in Any recent changes? Getting hard reset.:

only the ps aux command looking for it.

@JKnott said in Any recent changes? Getting hard reset.:

tried using the kill command on the process number 12752

12752 was the grep command you started at the command line.
It was already terminated right after showing you what it found (it found itself ^^)

The issue s much clearer now I go for : : openvpn can't start because of a config error.

@JKnott said in Any recent changes? Getting hard reset.:

cat /var/etc/openvpn/server1/sock

It a command pipe. You can't 'cat' it like that.
You have to put command in it, and then read from it to get the reply back.
Example : let's do what the GUI does - the openvpn widget :

Here : https://github.com/pfsense/pfsense/blob/7573eafea9025f1a895886cf24d20fc5762c7b7d/src/etc/inc/openvpn.inc#L1998

So : let's put stuff (your keyboard) inn the pipe and see what comes back : time to use the good old telnet :

[25.03-BETA][root@pfSense.bhf.tld]/root: telnet /var/etc/openvpn/server1/sock
Trying /var/etc/openvpn/server1/sock...
Connected to /var/etc/openvpn/server1/sock.
Escape character is '^]'.
>INFO:OpenVPN Management Interface Version 5 -- type 'help' for more info
status 2
TITLE,OpenVPN 2.6.12 amd64-portbld-freebsd15.0 [SSL (OpenSSL)] [LZO] [LZ4] [PKCS11] [MH/RECVDA] [AEAD] [DCO]
TIME,2025-02-27 17:25:22,1740673522
HEADER,CLIENT_LIST,Common Name,Real Address,Virtual Address,Virtual IPv6 Address,Bytes Received,Bytes Sent,Connected Since,Connected Since (time_t),Username,Client ID,Peer ID,Data Channel Cipher
HEADER,ROUTING_TABLE,Virtual Address,Common Name,Real Address,Last Ref,Last Ref (time_t)
GLOBAL_STATS,Max bcast/mcast queue length,0
GLOBAL_STATS,dco_enabled,1
END

No one connected right now.

Anyway, the real issue is : openvpn isn't running.
As said, I go for a config issue.
So, as shown above, tail the openvlog file, and in the GUI, visit the server settings page and save that page. The config gets rebuild, and then the openvpn is stopped (it was already stopped) and then started. If there are any issue, they will show up in the log. If the error is a fatal error, the process will stop.

JKnott

@Gertjan said in Any recent changes? Getting hard reset.:

It a command pipe. You can't 'cat' it like that.

It works in Linux.

: time to use the good old telnet :

/root: telnet /var/etc/openvpn/server1/sock
Trying /var/etc/openvpn/server1/sock...
/var/etc/openvpn/server1/sock: Connection refused

root: tail -f /var/log/openvpn
tail: /var/log/openvpn: No such file or directory

I get the same before and after trying to start it. I had already saved the config.

johnpoz

@JKnott said in Any recent changes? Getting hard reset.:

tail -f /var/log/openvpn

that not going to work because the file is openvpn.log

JKnott

@johnpoz

That's what @Gertjan said to do and that command would work in Linux.

Gertjan

@JKnott said in Any recent changes? Getting hard reset.:

root: tail -f /var/log/openvpn
tail: /var/log/openvpn: No such file or directory

Sorry :

tail -f /var/log/openvpn.log

@JKnott said in Any recent changes? Getting hard reset.:

That's what @Gertjan said to do and that command would work in Linux.

My pfSense uses FreeBSD ^^
A socket/pipe is actually some sort of file, I agree.
For content to be available so it can be listed on the screen, this is what cat does, you need to inject a command first so to the outgoing data (== the file content) becomes available. By default is an empty file == no info ...

Anyway : you found info about why your openvpn doesn't start ?

JKnott

@Gertjan said in Any recent changes? Getting hard reset.:

Sorry :

tail -f /var/log/openvpn.log

/root: tail -f /var/log/openvpn.log
Feb 27 12:35:27 firewall openvpn[37666]: library versions: OpenSSL 3.0.12 24 Oct 2023, LZO 2.10
Feb 27 12:35:27 firewall openvpn[37666]: DCO version: FreeBSD 14.0-CURRENT amd64 1400094 #1 RELENG_2_7_2-n255948-8d2b56da39c: Wed Dec 6 20:45:47 UTC 2023 root@freebsd:/var/jenkins/workspace/pfSense-CE-snapshots-2_7_2-main/obj/amd64/StdASW5b/var/jenkins/workspace/pfSense-CE-snapshots-2_7_2-main/sources/F
Feb 27 12:35:27 firewall openvpn[37699]: WARNING: using --duplicate-cn and --client-config-dir together is probably not what you want
Feb 27 12:35:27 firewall openvpn[37699]: NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
Feb 27 12:35:27 firewall openvpn[37699]: WARNING: experimental option --capath /var/etc/openvpn/server1/ca
Feb 27 12:35:27 firewall openvpn[37699]: TUN/TAP device ovpns1 exists previously, keep at program end
Feb 27 12:35:27 firewall openvpn[37699]: TUN/TAP device /dev/tun1 opened
Feb 27 12:35:27 firewall openvpn[37699]: /sbin/ifconfig ovpns1 172.16.255.1/24 mtu 1500 up
Feb 27 12:35:27 firewall openvpn[37699]: FreeBSD ifconfig failed: external program exited with error status: 1
Feb 27 12:35:27 firewall openvpn[37699]: Exiting due to fatal error

My pfSense uses FreeBSD ^^

I have been running Linux for over 25 years. While most commands work the same or very similar to BSD, there are some differences that get in my way. For example, ifconfig has been deprecated on Linux for years, replaced by ip. However, it can still be installed if you want it. Depending on what I'm doing, I'll use either.

Gertjan

There you have your answer, it was in the logs all the time (Status > System Logs > OpenVPN shows the same thing btw) :
@JKnott said in Any recent changes? Getting hard reset.:

Exiting due to fatal error

That universal OS language for :
No OpenVPN for you.

My OpenVPN server startup sequence :

I don't have any TAP/TUN lines.
I do have a "GDG: problem writing to routing socket: No such process (errno=3) " but that's probably a new thing present in "pfSense 25.03-BETA (amd64)".

This is strange :

@JKnott said in Any recent changes? Getting hard reset.:

FreeBSD ifconfig failed: external program exited with error status: 1

Your "/sbin/ifconfig ovpns1 172.16.255.1/24 mtu 1500 up" seems to fail.

can't tell you why .....
No issues with my :

/sbin/ifconfig ovpns1 192.168.3.1/24 mtu 1500 up

JKnott

@Gertjan said in Any recent changes? Getting hard reset.:

Your "/sbin/ifconfig ovpns1 172.16.255.1/24 mtu 1500 up" seems to fail.

can't tell you why .....
No issues with my :

I couldn't seen any reason either, so I rebooted pfSense and that seemed to clear it.