OpenVPN first user to connect gets in and nobody after

frgeeks

Hi pfExperts! I've dug around and found this has come up for others but none of the solutions have worked here. This is what's going on with our Netgate 2100 with 22.05, even after a full re-image and manual rebuild of the entire config.

The first OpenVPN user to connect after a reboot or some unknown long timeout has no problem getting an IP address and reaching any internal device they please. Any subesquent users are able to connect OpenVPN successfully, but no traffic passes the firewall's LAN IP.

It used to all work perfectly but we switched from Comcast to Lumen (CenturyLink) fiber, and now this weird bonked behavior. We had 5 statics on Comcast and now have just one via PPPoE, because that's how Lumen rolls.

Users authenticate against our Windows AD via LDAP and that never fails, not counting badly entered passwords. Everyone uses the same certificate. Again, it worked before switching away from Comcast so I don't believe that's a cause.

10.0.2.0/24 is the private network and 10.0.4.0/24 is OpenVPN. There are currently two users connected. The user with 10.0.4.2 can reach everything, the user with 10.0.4.3 doesn't even appear in the pfSense routing table and can't reach anything:

IPv4 Routes
default	207.225.112.2	UGS	492480	1492	pppoe0	
10.0.2.0/24	link#2	U	35991694	1500	mvneta1	
10.0.2.1	link#2	UHS	3	16384	lo0	
10.0.4.0/24	10.0.4.2	UGS	8	1500	ovpns1	
10.0.4.1	link#14	UHS	0	16384	lo0	
10.0.4.2	link#14	UH	0	1500	ovpns1	
10.0.8.0/24	link#13	U	2178390	1500	mvneta1.8

Shouldn't each connected user's 10.0.4... IP appear in the pfSense routing table? It feels like the OpenVPN server is spacing out and forgetting to add other client IPs to it.

While pinging from the second (broken) user, an ICMP packet capture on the OpenVPN interface shows this

15:14:58.103971 IP 10.0.4.3 > 10.0.2.8: ICMP echo request, id 1, seq 52, length 40
15:14:58.104894 IP 10.0.2.8 > 10.0.4.3: ICMP echo reply, id 1, seq 52, length 40
15:15:03.098962 IP 10.0.4.3 > 10.0.2.8: ICMP echo request, id 1, seq 53, length 40
15:15:03.099828 IP 10.0.2.8 > 10.0.4.3: ICMP echo reply, id 1, seq 53, length 40
15:15:08.082446 IP 10.0.4.3 > 10.0.2.8: ICMP echo request, id 1, seq 54, length 40
15:15:08.083204 IP 10.0.2.8 > 10.0.4.3: ICMP echo reply, id 1, seq 54, length 40
15:15:13.080830 IP 10.0.4.3 > 10.0.2.8: ICMP echo request, id 1, seq 55, length 40
15:15:13.081533 IP 10.0.2.8 > 10.0.4.3: ICMP echo reply, id 1, seq 55, length 40

and from the LAN

15:16:34.355617 IP 10.0.4.3 > 10.0.2.8: ICMP echo request, id 1, seq 60, length 40
15:16:34.356493 IP 10.0.2.8 > 10.0.4.3: ICMP echo reply, id 1, seq 60, length 40
15:16:39.089825 IP 10.0.4.3 > 10.0.2.8: ICMP echo request, id 1, seq 61, length 40
15:16:39.090770 IP 10.0.2.8 > 10.0.4.3: ICMP echo reply, id 1, seq 61, length 40
15:16:44.085051 IP 10.0.4.3 > 10.0.2.8: ICMP echo request, id 1, seq 62, length 40
15:16:44.086052 IP 10.0.2.8 > 10.0.4.3: ICMP echo reply, id 1, seq 62, length 40
15:16:49.092612 IP 10.0.4.3 > 10.0.2.8: ICMP echo request, id 1, seq 63, length 40
15:16:49.093681 IP 10.0.2.8 > 10.0.4.3: ICMP echo reply, id 1, seq 63, length 40

but the client PC never sees the response. The first user has no problems getting ping replies in a decent 15ms. Gah!

Has anyone seen a fix that doesn't involve toggling every single option in the OpenVPN server config to no avail? And yes, I tried just about all of them. We've rolled out dozens of Netgates over the years and this is the only box having this sort of problem. What other pertinent details can I share?

Thanks for any brilliant ideas, O Great Collective Community Brain!

David

jake

@frgeeks What does the OpenVPN log show when the second user connects?

frgeeks

@jake Hi! I don't have access at the moment, but I was able to work with TAC late Friday and we tracked the problem down to a known bug (https://redmine.pfsense.org/issues/13358). That didn't come up in all my searching beforehand, of course. We have a simple workaround of disabling DCO. I could have sworn I tried that, but I tried so many things over a couple of weeks it was easy to lose track. Thanks for the note!
David