OpenVPN could not be established after upgrade to 23.01 on SG-3100
-
Can you show more of the OpenVPN client log showing the failures? Preferable with the logging level raised.
-
Referenced by
KpuCko
-
@stephenw10 Yes, when this happens I can still SSH to it using putty. This is how I've been restarting it and how I was able to try restarting the web configurator, etc. Other VPN traffic works fine as well ie: web gui for unifi.
-
Ok, well we need more info here to try to reproduce the issue really. Any additional error logs you might be able to get. Anything unusual in your config.
-
@stephenw10 said in OpenVPN could not be established after upgrade to 23.01 on SG-3100:
kldxref /boot/kernel
Finally! This fixed both OpenVPN and Tailscale. Thank you so much for posting this. SG-3100 here.
-
@stephenw10 I just reproduced it on a completely different 3100 at a different location. I have nothing special going on regarding configuration. All I did to reproduce it is connect to openvpn from a windows 10 machine running the openvpn connect client. I visited the pfsense web gui and logged in. I held shift while clicking refresh. 3/4 of the page loaded then it froze completely. Dropdowns stopped working and broken images. I disconnected VPN and reconnected hard wire and still could not get the web gui to properly respond. I rebooted the 3100 and everything is back in order. I can get to the web gui like normal. I can get you whatever logs or configs you'd like.
-
Hmm, do you see anything logged when that happens? Nothing logged as blocked in the firewall log?
The only vaguely related thing I'm aware of here is this: https://redmine.pfsense.org/issues/13938
Obviously there's no kernel panic here but you are still accessing nginx via a virtual interface. You might try disabling unmapped mbufs as shown there to see if it makes any difference.Steve
-
@stephenw10 I didn't see anything logged in system/gui or firewall pointing to an issue. Disabling sendfile on one of them seems to have solved the problem. Doing that and kern.ipc.mb_use_ext_pgs=0 on the other seems to have solved the crashing and need for a reboot, though it doesn't resolve the gui loading through vpn issue.
-
Hmm, that's interesting. I would only have expected to need those workarounds at the remote side of the tunnel.
So with those in place you are still unable to access the webgui on the remote pfSense over the VPN? But doing so no longer prevents clients local to it accessing it after that?
-
@stephenw10 In both test cases they were remote side. I tested each with multiple clients and connections. One 3100 issue was resolved. The other didn't completely resolve with either of the two fixes I mentioned. It continues to fail to load the web gui when accessed through the tunnel but it no longer prevents local access to it prior to a reboot.
-
@techscribe said in OpenVPN could not be established after upgrade to 23.01 on SG-3100:
Tail
Hey @techscribe Did you have to do anything extra to get Tailscale to work? My Tailscale didn't return after entering
kldxref /boot/kernelinto the command prompt. Any insight would be appreciated, thank you! -
Referenced by W wavesound
-
having the same issue after 23.01 on a custom build tried the fix did not work, removed my defined network from "IPv4 Tunnel Network" openvpn connects again.
-
@mmrk All I did was run the command, then restarted both my OpenVPN and Tailscale services.
-
@sphillips said in OpenVPN could not be established after upgrade to 23.01 on SG-3100:
removed my defined network from "IPv4 Tunnel Network" openvpn connects again
That sounds more like a subnet conflict. Perhaps something else changed coincidentally with the upgrade? Or possibly simply making any change and resaving the OpenVPN settings re-created the conf file. I would have expected to see errors logged in either of those cases though.
Steve
-
@stephenw10 said in OpenVPN could not be established after upgrade to 23.01 on SG-3100:
That sounds more like a subnet conflict. Perhaps something else changed coincidentally with the upgrade? Or possibly simply making any change and resaving the OpenVPN settings re-created the conf file. I would have expected to see errors logged in either of those cases though.
SteveI did not see any other errors besides the "Exiting due to fatal error", i dont recall making any changes between the upgrade and noticing the issue, i confess it took e a few days to notice the issue since i dont use vpn everyday.
i just got frusted with the problem and went back to pfsense CE will revisit pfsense plus when theres an update or time allows, thank you. -
Ok, probably some other issue then. Unfortunately "Exiting due to fatal error" isn't particularly helpful in the default OpenVPN output. We'll probably need to turn up the logging level if nothing else is shown.
-
@stephenw10 Thank you, this simple fix also did it for me!
-
@stephenw10 This worked for me, as well! Thanks!
-
@stephenw10 Just adding for the web record that I also encountered this issue when upgrading my 3100 this morning. The manual update to the linker file followed by a reboot of the Firewall fixed it.
I obviously have no data on how pervasive this issue is, but like with the 1100 and 2100 maybe it's worth pulling the 23.01 upgrade for the time being on the 3100? Yes I fixed it by going to Netgate forums and see this is one of the top OpenVPN threads but there's lots of folks with these appliances out there that are going to be pulling their hair out over this especially when the interfaces are still in place. Just seems like a big headache for them waiting to happen. Doubly so if they're at remote sites. Either way, the fix was simple! Seems like this should be petted manually and reintroduced as a new release for the 3100 to cover this bug.
-
I'm not sure it would be possible to make that change without rebuilding and all that implies.
I'll certainly confirm that though. -
Thanks for much for posting the solution - really appreciate the great advice. Confirming OpenVPN still works after reboot.
I'm quite disappointed that Netgate would release "stable" software with a major flaw! Next time I get a survey will voice that.
Rocker
-
kldxref /boot/kernel and a restart of the OpenVPN service worked for me on my Sg3100. I issued the command through the web gui.
-
@mikej47 Yes, that worked for me, too. Problem comes when you're upgrading a bunch of firewalls and you connect to them via OpenVPN to administer and you can't restart the service after a reboot because you've been kicked off the VPN due to this bug.
-
@mmrk said in OpenVPN could not be established after upgrade to 23.01 on SG-3100:
kldxref
It's fixed in 23.05. https://redmine.pfsense.org/issues/13963
I would always advise having some out of band access for an upgrade though if at all possible. Especially a large step like this. A lot of other changes happened in OpenVPN.
Steve
-
@jvcomputers
What I do on the remote firewall is to allow access to ports 22 and 443, but only specifically from:- one public IP address within my primary location's static IP address range, and
- an unused public IP address within a nearby satellite location's public IP address range
I've run into situations before where the VPN goes down, so 1) comes in handy to get the VPN back up and running without needing to travel to the remote location. I've also run into a situation before where my ISP actually reassigned the static IP address at my primary location to another client, causing me to lose that static IP address (and subsequently my VPN and the remote access to all of my satellite locations!) So 2) protects against that from happening again....at least I only need to travel to a nearby location to re-establish all the VPNs.
It's not exactly out-of-band, but it serves my needs.
-
@cneep That's an idea, but one I'd have to carefully consider. I don't like the idea of opening administrative ports directly to the internet, even if I lock them down to my IP.
-
If it's an SSH port you can also use key based auth only. The risk of that combined with a limited source IP is close to zero.
Steve
-
@stephenw10 True, providing no future vulnerabilities are found in the SSH services which is, unfortunately, not unheard of.
-
Right. But don't leave the port open after the upgrade is complete and the VPN is back up.
If you really wanted to you could put the rule on a schedule so it closed again automatically if you were still unable to access it for some other reason. -
@stephenw10 I feel dumb, I should've thought of only opening it during an upgrade procedure. Good idea.
