OpenVPN reconnect storm: UNDEF clients and UDP socket drops

PJHaan

Hi all,
I’m troubleshooting a pfSense/OpenVPN reconnect-storm issue and I’m trying to determine whether this is a
FreeBSD/pfSense/OpenVPN stack regression, a tuning issue, or some kind of processing bottleneck.
Scenario:
- pfSense Plus test server: 26.03.1-RELEASE
- FreeBSD: 16.0-CURRENT
- OpenVPN: 2.6.20
- OpenSSL: 3.5.5
- OpenVPN UDP server, port 1194
- Around 300+ site routers/clients can reconnect at the same time after an OpenVPN/pfSense restart
- Clients are TP-Link/Omada routers, so I cannot add client-side reconnect jitter/backoff
- The same OpenVPN config is used on another production pfSense system that currently holds 800+ clients fine
Symptoms on the test server during reconnect storm:
- Many OpenVPN status entries stay in UNDEF / no virtual IP
- with_vip remains much lower than total
- UDP “dropped due to full socket buffers” increases heavily
- Example during storm:
total=157 with_vip=21 undef=94 no_vip=136
or:
total=301 with_vip=122 undef=163 no_vip=179
Comparing production vs test:
Production VPN1:
- pfSense Plus: 24.03-RELEASE
- FreeBSD: 15.0-CURRENT
- OpenVPN: 2.6.8
- OpenSSL: 3.0.13
- 4 vCPU
- Current OpenVPN status:
total=837 with_vip=837 undef=0 no_vip=0
Test VPN:
- pfSense Plus: 26.03.1-RELEASE
- FreeBSD: 16.0-CURRENT
- OpenVPN: 2.6.20
- OpenSSL: 3.5.5
- 2 vCPU
- During reconnect storm:
lots of UNDEF/no_vip
UDP full socket buffer drops increase rapidly
Important: the generated OpenVPN runtime configs are effectively identical, apart from hostname/public IP.
Relevant sysctls are the same on both:
kern.ipc.maxsockbuf: 4262144
net.inet.udp.recvspace: 42080
net.inet.udp.maxdgram: 57344
kern.ipc.nmbclusters: 1000000
net.inet.ip.intr_queue_maxlen: 1000
net.route.netisr_maxqlen: 1024
pf state limits do not appear to be hit:
Production:
states hard limit: 812000
current states: ~1246
Test:
states hard limit: 608000
current states: ~122
state-limit: 0
src-limit: 0
UDP drop ratio is very different:
Production:
~589M datagrams received
~2.5M dropped due to full socket buffers
roughly 0.4%
Test:
~1.6M datagrams received
~800k dropped due to full socket buffers
roughly 50%
I also tested increasing:
net.inet.udp.recvspace = 262144 and 524288
Even after restarting OpenVPN so it opened a new UDP socket, this did not clearly fix the issue. UDP full socket buffer drops
still increased, and UNDEF/no_vip remained high.
I also tested OpenVPN throttling:
connect-freq 20 10
connect-freq-initial 20 10
connect-freq 25 10
connect-freq-initial 25 10
connect-freq 30 10
connect-freq-initial 30 10
connect-freq 40 10
connect-freq-initial 40 10
30/10 looked best so far, but still not a full fix.
The OpenVPN config includes:
client-connect /usr/local/sbin/openvpn.attributes.sh
client-disconnect /usr/local/sbin/openvpn.attributes.sh
tls-verify "/usr/local/sbin/ovpn_auth_verify tls '<hostname>' 1"
So pfSense’s tls-verify/client-connect helper path may be part of the bottleneck under a reconnect storm.

  Questions:                                                                                                                       
  
  1. Has anyone seen OpenVPN 2.6.20 / pfSense Plus 26.03.1 / FreeBSD 16 behave worse under large UDP reconnect storms than         
  pfSense 24.03 / OpenVPN 2.6.8?                                                                                                   
  
  2. Are there known FreeBSD 16 UDP socket buffer or network stack changes that could explain the high “dropped due to full        
  socket buffers” count?                                                                                                           
  
  3. Is `connect-freq` / `connect-freq-initial` the recommended way to protect OpenVPN from this kind of reconnect storm, or is    
  there a better pfSense-specific approach?                                                                                        
  
  4. Could the pfSense `tls-verify` / `client-connect` PHP/fcgicli helper path become the bottleneck here?                         
  
  5. Would increasing vCPU from 2 to 4 likely help, given that the production system has 4 vCPU and behaves much better?           
  
  6. Are there other sysctls or OpenVPN options worth testing besides `net.inet.udp.recvspace`, `sndbuf/rcvbuf`, and               
  `connect-freq`?                                                                                                                  
  I’m not currently seeing pf state limits, socket limits, file limits, or mbuf exhaustion being hit. The main observable          
  failure is OpenVPN accumulating UNDEF/no_vip entries while UDP full socket buffer drops rise quickly.

MalagaFirewall8

I would test this first as a server-side bottleneck, not a pf state problem. The big difference I see is 2 vCPU vs 4 vCPU plus the helper scripts in the auth/connect path. During the storm, watch CPU per thread/process from the shell and see if openvpn, php-fpm/fcgicli, or interrupts are pegged. If possible, do one lab run with the tls-verify/client-connect helpers temporarily simplified or removed; if UNDEF drops sharply, you have found the choke point.

I would also try matching the production vCPU count before chasing too many sysctls. connect-freq-initial 30 10 is a reasonable guard, but if the PHP/helper path serializes enough work, throttling only masks it. For UDP buffers, make sure any OpenVPN rcvbuf/sndbuf values and the system udp recvspace are increased before the daemon opens the socket, then confirm with sockstat/netstat during the test rather than assuming the new value was used.

PJHaan

Zeker PJ — concept voor forumreactie:

Hi all,

We did some additional testing around this issue and it strongly looks related to the pfSense/OpenVPN helper path after upgrading, not simply CPU capacity.

Environment comparison

Working system before upgrade:

pfSense Plus: 24.03
FreeBSD: 15.0-CURRENT
OpenVPN: 2.6.8
CPU: 4 vCPU

Problematic test system:

pfSense Plus: 26.03.1
FreeBSD: 16.0-CURRENT
OpenVPN: 2.6.20
CPU: 2 vCPU

We also observed the same type of issue on a 4 vCPU system after upgrade, so this does not appear to be purely a CPU sizing problem.

Symptoms

During a mass reconnect storm, many clients remain stuck as UNDEF / without virtual IP. With the normal helper path enabled, reconnect recovery is very poor. With the OpenVPN helper scripts temporarily bypassed, clients recover much faster and receive virtual IPs normally.

Example with helpers enabled on the 26.03.1 test system:

total=216 with_vip=61 no_vip=155 undef=143

Example with both helper scripts temporarily replaced by exit 0:

total=267 with_vip=260 no_vip=7 undef=3

So the OpenVPN daemon itself seems capable of accepting the reconnect storm, but the pfSense helper path appears to become the bottleneck.

Helpers tested

The relevant OpenVPN config calls:

script-security 3
client-connect /usr/local/sbin/openvpn.attributes.sh
client-disconnect /usr/local/sbin/openvpn.attributes.sh
tls-verify "/usr/local/sbin/ovpn_auth_verify tls '<hostname>' 1"

We tested:

both helpers enabled
both helpers disabled/no-op
only tls-verify disabled
only client-connect/client-disconnect disabled
replacing openvpn.connect_async.sh on the 26.03.1 system with the older version from another working system

Results:

both helpers enabled: bad reconnect behavior, high UNDEF
both helpers disabled: reconnect behavior much better
only tls-verify disabled: still bad
only client-connect/client-disconnect disabled: still bad
old openvpn.connect_async.sh: did not solve it

This suggests the problem is not just one shell script, but the pfSense helper/runtime path behind it.

Process observations

During the bad reconnect state we see multiple helper/runtime processes, including:

/usr/local/sbin/fcgicli -f /etc/inc/openvpn.tls-verify.php ...
/usr/local/bin/php-cgi -f /usr/local/sbin/openvpn_connect_async.php
/bin/sh /usr/local/sbin/openvpn.connect_async.sh
/bin/sh /usr/local/sbin/ovpn_auth_verify ...

CPU was not fully saturated during the test. On the 26.03.1 test system we still saw idle CPU at times, while UNDEF continued building. This looks more like helper-runtime latency/blocking/locking/deferred-connect behavior than simply insufficient CPU.

File/version differences

The direct helper scripts are very similar between the older working system and the newer system:

/usr/local/sbin/ovpn_auth_verify
/usr/local/sbin/openvpn.attributes.sh

openvpn.connect_async.sh changed more substantially in the newer version, but replacing it with the older version did not fix the issue.

The deeper PHP/runtime files differ as expected between versions, for example:

/usr/local/sbin/openvpn_connect_async.php
/etc/inc/openvpn.tls-verify.php

Current conclusion

This appears to be a regression or performance issue in the pfSense/OpenVPN helper runtime path after upgrade, likely around tls-verify, client-connect, fcgicli, openvpn_connect_async.php, PHP runtime, or OpenVPN deferred client-connect handling.

Temporarily disabling helpers is not a safe production fix because it bypasses pfSense/OpenVPN checks and connect/disconnect handling, but it is a useful diagnostic: it shows the reconnect storm only becomes manageable when the helper path is removed from the hot path.
Has anyone else seen this behavior on newer pfSense Plus / OpenVPN versions, especially with large numbers of clients reconnecting simultaneously after an OpenVPN restart or pfSense upgrade?

MalagaFirewall8

That result points much more at the pfSense-side helper/runtime path than at UDP buffers. If only disabling both helper paths makes UNDEF drop, I’d try to get one more run with timestamps around each helper call and PHP process count/CPU during the storm, so it is clear whether jobs are queueing or timing out. Also worth testing the same upgraded version with the production 4 vCPU shape, just to separate version regression from capacity.

PJHaan

@MalagaFirewall8

I tested the same scenario again after increasing the test system to 4 vCPU.

The issue still occurs, so this does not look like a simple “not enough CPU” problem.

Test system during this run

pfSense Plus: 26.03.1
FreeBSD: 16.0-CURRENT
OpenVPN: 2.6.20
CPU: 4 vCPU

For this run I restored the original helper stack and added temporary timestamp wrappers around:

/usr/local/sbin/ovpn_auth_verify
/usr/local/sbin/openvpn.attributes.sh

The wrappers logged helper start/end timestamps and return codes, then called the original scripts. I also sampled OpenVPN counters, CPU, load, and helper/PHP process counts during the reconnect storm.

Result

The problem still appears with 4 vCPU:

total=469 with_vip=417 no_vip=52 undef=51
total=536 with_vip=420 no_vip=116 undef=98

CPU did spike during the storm, for example:

cpu us=62 sy=29 id=8

But it was not permanently pinned at 0% idle. The behavior is wave-like: connections build up, UNDEF / no_vip increases, then the status sometimes drops back down before the next reconnect wave.

During the storm we saw the expected helper/runtime processes:

fcgicli
php-cgi -f /usr/local/sbin/openvpn_connect_async.php
/bin/sh /usr/local/sbin/openvpn.connect_async.sh
/bin/sh /usr/local/sbin/ovpn_auth_verify
php-fpm
openvpn

The timestamp wrappers showed many helper calls completing with rc=0, so it is not simply that every helper process hangs forever. However, under reconnect storm conditions the helper/runtime path still falls behind badly enough that clients remain in UNDEF / no virtual IP and later hit TLS/handshake timeouts.

Earlier tests also showed:

both helpers enabled → bad reconnect behavior / high UNDEF
both helpers disabled/no-op → reconnect behavior much better
only tls-verify disabled → still bad
only client-connect/client-disconnect disabled → still bad
replacing openvpn.connect_async.sh with an older version → did not solve it

So my current conclusion is:

This is probably not just UDP buffers and not just CPU capacity. It looks more like a regression or bottleneck in the pfSense/OpenVPN helper runtime path on the newer stack — likely around fcgicli, PHP helper execution, openvpn_connect_async.php, openvpn.tls-verify.php, or OpenVPN deferred client-connect handling under mass reconnect load.

Disabling the helpers is not a safe production workaround, but it remains the clearest diagnostic: removing the helper path from the hot path makes reconnect recovery much better.

PJHaan

@MalagaFirewall8 What else can we try, or is this a bug?

Gertjan

@PJHaan

Focus more on : don't restart the system, and make sure the WAN interface 'never' goes down.
This means : get an UPS, and protect your uplink ISP device (if present), and protect the power of pfSense also.
pfSense updates can't be a real issue, because Netgate doesn't upgrade/update xx times a year.
After all : a process that never goes down doesn't need to restart.
If there us an update, you could plan it when there are less connected users.

Btw : because (PHP) scripts are used to identify.check the connection (the certificates ?), and scripts == interpreted code takes thousand time more time to execute a native binary code, this will be a bottle neck for you, all of us.
Not really a bug, but a 'total' rewrite will be necessary to really solve this.

Last but not least : if the Openvpn becomes really important for you, forget about the pfSense OpenVPN solution.
Get a dedicated 'native' (open source) OpenVPN device on the pfSense LAN. I'm pretty sure this care of your issue.

Btw : these are just suggestions.

PJHaan

@Gertjan

Thanks, but this is in a datacenter, so UPS/power/uplink protection is not the issue.

The real issue is mass OpenVPN reconnect behavior after pfSense/OpenVPN restart/update. Avoiding restarts helps operationally, but does not solve the technical problem: many clients reconnecting at once leads to UNDEF/no-VIP/TLS/auth problems and poor recovery.

I’m looking for insight into pfSense/OpenVPN reconnect-storm handling, auth/client-connect helper bottlenecks, PHP/fcgicli overhead, tunables, or known version regressions. A dedicated Linux OpenVPN server may be a long-term workaround, but I’d still like to understand the root cause.

tinfoilmatt

How much memory are these systems allocated? Surprised to not see that mentioned anywhere yet...

PJHaan

@tinfoilmatt These systems have 8GB of memory.

PJHaan

Does anyone have any other ideas what it could be?