OpenVPN could not be established after upgrade to 23.01 on SG-3100

jkibbey

The above fix and/or patch worked for me and VPN functionality is working fine with the only perceived exception: when trying to access the pfSense web gui through VPN, it loads some of the page then hangs. From then forward, it seems to corrupt the gui service (PHP?) for any client until the firewall is rebooted (restarting webConfigurator and restarting PHP-FPM did not resolve the issue, reboot was required.) I also tried different browsers, different client machines, and forcing a cache refresh.)

I've tested this through site-to-site as well as remote access through OpenVPN. Everything else through VPN is working fine. Testing included SSH, Remote Desktop, UniFi web gui.

I'm using a Netgate 3100.

tennbaum

@peekay said in OpenVPN could not be established after upgrade to 23.01 on SG-3100:

works for me as well (3100)
kldxref /boot/kernel
all working after (did not check the pfsense gui though)

stephenw10

@jkibbey said in OpenVPN could not be established after upgrade to 23.01 on SG-3100:

From then forward, it seems to corrupt the gui service (PHP?) for any client until the firewall is rebooted

Hmm, that's odd. Do you mean it happens one time after updating the linker hints? Or Whenever you try to access the gui over the VPN even after the reboot?

And does that then also apply to local clients accessing it?

Steve

jkibbey

@stephenw10 It happens anytime I try to access the pfSense gui via the OpenVPN tunnel. All other communication through VPN is fine and the pfSense gui functions like normal if I do not access it via VPN and if I have not attempted to use it via the tunnel after a reboot, even if it was from another client.

stephenw10

Hmm, is it only the webgui, can you ssh to the firewall across the VPN?

JayB

Dealing with similar result (different cause?) this morning on our 2440. Upgraded HO on Saturday and We've no connectivity this morning (first day office is back since upgrade)

Tunnel is stuck on "Adding Routes to System" and the RO (which did not get the upgrade) is showing "reconnecting: ping-restart". The logs show GDG: problem writing to routing socket. Tried the kldxref /boot/kernel fix but to no effect.

Unable to roll back to previous stable (not offered) so we're putting a spare that's on 22.05 online to replace the borked one.

stephenw10

What is it connecting to? The only other report of this was at the other end of a link to a 3100 that had also been updated and required the above fix. But if that was the case here it would still fail after you replaced the local end.

You did not also see logged:

ERROR: FreeBSD route add command failed: external program exited with error status: 1

Which would indicate some sort of route conflict.

Steve

JayB

@stephenw10 Both ends are SG-2440's That said, the RO is running 2.6.0 (we recently restored it and haven't re-upgraded to Plus yet) Maybe it's a 23 <> 2.6 issue? We're also still running peer to peer on a shared key but seems it's still allowed?

We're back on our "spare" 2440 which is on 22.01 and the tunnel is up.

I didn't see any error messages, just the GDG warning.

stephenw10

Shared key will still work in 23.01. It likely won't work in future versions when OpenVPN stops supporting it upstream.
Was there anything logged at the other end?
I would try turning up the logging level in OpenVPN and retesting of you can.

Steve

JayB

@stephenw10 re: shared key...that's what I understaood but worth mentioning.

The remote log shows no errors or warnings.

Looks normal except there's no Peer Connection Initiated with [AF_INET]XXX.XXX.XXX.XXX:1194 at the top of the log.

stephenw10

Can you show more of the OpenVPN client log showing the failures? Preferable with the logging level raised.

jkibbey

@stephenw10 Yes, when this happens I can still SSH to it using putty. This is how I've been restarting it and how I was able to try restarting the web configurator, etc. Other VPN traffic works fine as well ie: web gui for unifi.

stephenw10

Ok, well we need more info here to try to reproduce the issue really. Any additional error logs you might be able to get. Anything unusual in your config.

techscribe

@stephenw10 said in OpenVPN could not be established after upgrade to 23.01 on SG-3100:

kldxref /boot/kernel

Finally! This fixed both OpenVPN and Tailscale. Thank you so much for posting this. SG-3100 here.

jkibbey

@stephenw10 I just reproduced it on a completely different 3100 at a different location. I have nothing special going on regarding configuration. All I did to reproduce it is connect to openvpn from a windows 10 machine running the openvpn connect client. I visited the pfsense web gui and logged in. I held shift while clicking refresh. 3/4 of the page loaded then it froze completely. Dropdowns stopped working and broken images. I disconnected VPN and reconnected hard wire and still could not get the web gui to properly respond. I rebooted the 3100 and everything is back in order. I can get to the web gui like normal. I can get you whatever logs or configs you'd like.

stephenw10

Hmm, do you see anything logged when that happens? Nothing logged as blocked in the firewall log?
The only vaguely related thing I'm aware of here is this: https://redmine.pfsense.org/issues/13938
Obviously there's no kernel panic here but you are still accessing nginx via a virtual interface. You might try disabling unmapped mbufs as shown there to see if it makes any difference.

Steve

jkibbey

@stephenw10 I didn't see anything logged in system/gui or firewall pointing to an issue. Disabling sendfile on one of them seems to have solved the problem. Doing that and kern.ipc.mb_use_ext_pgs=0 on the other seems to have solved the crashing and need for a reboot, though it doesn't resolve the gui loading through vpn issue.

stephenw10

Hmm, that's interesting. I would only have expected to need those workarounds at the remote side of the tunnel.

So with those in place you are still unable to access the webgui on the remote pfSense over the VPN? But doing so no longer prevents clients local to it accessing it after that?

jkibbey

@stephenw10 In both test cases they were remote side. I tested each with multiple clients and connections. One 3100 issue was resolved. The other didn't completely resolve with either of the two fixes I mentioned. It continues to fail to load the web gui when accessed through the tunnel but it no longer prevents local access to it prior to a reboot.

mmrk

@techscribe said in OpenVPN could not be established after upgrade to 23.01 on SG-3100:

Tail

Hey @techscribe Did you have to do anything extra to get Tailscale to work? My Tailscale didn't return after entering kldxref /boot/kernel into the command prompt. Any insight would be appreciated, thank you!