OpenVPN could not be established after upgrade to 23.01 on SG-3100
-
@maxk-0 said in OpenVPN could not be established after upgrade to 23.01 on SG-3100:
contingency plan to ensure you can fallback to a known operational state
This is the way ;)
Be it your in an enterprise or just a home user - this is the way! ;)
-
@johnpoz
sure there's a way, I just, dammit, don't want to go onsite! -
Opened a bug for this: https://redmine.pfsense.org/issues/13963
Updating the linker file manaully fixed it for me. Run:
kldxref /boot/kernel
Steve
-
@stephenw10 This works for me!
-
working fine and is reboot safe. Excellent. Thanks.
I won't reflash and wait for a possible bugfix release. -
@shpokas I hear ya - but we always have someone on hand in the DC when doing any sort of upgrade.. Just in case.. The best laid plans can always fail..
I left some boxes at sites on old pfsense for longest time because of covid, and just pita to either go there or have someone available..
And also hear ya that damn it this box should just work.. And I should just be able to click update.. But no matter how easy it should be or how many times it has worked in the past, etc. And this is not some minor update, this is a major change to freebsd, and an upgrade to php as well, etc..
I haven't pulled the trigger yet on my 4860 here at home.. But I did get the new 23.01 image from tac already, and will burn it to usb and be ready to clean install if something goes horrible wrong.. Nothing will I hope - but just in case, I don't want to be down for any period of time. I also still have 22.05 image if need be to clean install too, etc.
Also notice that I might have an issue with my 4860 "may have issues with the ichsmb0 and/or ehci0" so have that info handy to put in the /boot/loader.conf.local
If something crashes and burns - I wouldn't be happy no.. But also know that things like this can happen.. I need to update the dsm version on my nas for example.. And had many an update go smooth in the past, but since the last update I have created a non supported configuration with nvme as storage.. People say shouldn't be a problem - but just in case I am currently backing up my plex and dockers that reside on the nvme storage - just in case that doesn't survive the update.. And I don't want to go through the hassle recreating them, nor using a backup from a few days ago.. So running a backup just before I do the update later this morning ;)
-
I would add the line to disable ichsmb on the 4860 before upgrading @johnpoz
You lose nothing by having it there. We will likely add that by default in 23.05.
If you're not on the console you won't notice anything anyway. -
@stephenw10 thanks I was thinking of that.. Any reason why a clean install might be good - change to the zfs layout or anything. I was thinking I should be able to test the zfs rollback feature with this update as well.
So should I be able to rollback to 22.05 after an update to 23.01?
-
Yes you can roll back to 22.05 from 23.01. But not to 22.01.
There's no particular reason a clean install should be any better there. IMO at least!
-
@stephenw10 might have to give that a test run ;) I really don't ever see any need for the feature myself - but might be nice to have at least tested it in case of anyone having questions on it at some point.
-
-
-
Same here (SG-3100), OpenVPN service cannot start after 23.01 update.
Is the command kldxref /boot/kernel (described in this thead as a fix) the offcial recommended solution, or should we wait for another workaround? If done, would it be a permanent fix?
Thank you
-
I have use that and seen no issues since. It does survive a reboot.
-
Ok great, thank you!
I will also follow your bug report status.. -
Same here, kldxref /boot/kernel fixed for me as well. 3100
-
I also had this issue with a 3100 and the kldxref /boot/kernel command worked for me too.
-
-
-
-
-
@stephenw10 I ended up having this issue on a 1537. Applied the patch from the redmine post and issue is now resolved.
-
@stephenw10 Is this ran through SSH console? I ran via SSH and ran into the following error: kldxref: can't create /boot/lhint.h6bsDi: Permission denied
-
You probably need to be logged in as root or admin to run that.
It should work via Diag > Command Prompt in the webgui though.
-
@stephenw10 worked for me. Thanks for this!
Rankldxref /boot/kernel
in Diagnostics > Command Prompt on 3100 and restarted OpenVPN services. fired up immediately. -
The above fix and/or patch worked for me and VPN functionality is working fine with the only perceived exception: when trying to access the pfSense web gui through VPN, it loads some of the page then hangs. From then forward, it seems to corrupt the gui service (PHP?) for any client until the firewall is rebooted (restarting webConfigurator and restarting PHP-FPM did not resolve the issue, reboot was required.) I also tried different browsers, different client machines, and forcing a cache refresh.)
I've tested this through site-to-site as well as remote access through OpenVPN. Everything else through VPN is working fine. Testing included SSH, Remote Desktop, UniFi web gui.
I'm using a Netgate 3100.
-
@peekay said in OpenVPN could not be established after upgrade to 23.01 on SG-3100:
works for me as well (3100)
kldxref /boot/kernel
all working after (did not check the pfsense gui though) -
@jkibbey said in OpenVPN could not be established after upgrade to 23.01 on SG-3100:
From then forward, it seems to corrupt the gui service (PHP?) for any client until the firewall is rebooted
Hmm, that's odd. Do you mean it happens one time after updating the linker hints? Or Whenever you try to access the gui over the VPN even after the reboot?
And does that then also apply to local clients accessing it?
Steve
-
@stephenw10 It happens anytime I try to access the pfSense gui via the OpenVPN tunnel. All other communication through VPN is fine and the pfSense gui functions like normal if I do not access it via VPN and if I have not attempted to use it via the tunnel after a reboot, even if it was from another client.
-
Hmm, is it only the webgui, can you ssh to the firewall across the VPN?
-
Dealing with similar result (different cause?) this morning on our 2440. Upgraded HO on Saturday and We've no connectivity this morning (first day office is back since upgrade)
Tunnel is stuck on "Adding Routes to System" and the RO (which did not get the upgrade) is showing "reconnecting: ping-restart". The logs show GDG: problem writing to routing socket. Tried the kldxref /boot/kernel fix but to no effect.
Unable to roll back to previous stable (not offered) so we're putting a spare that's on 22.05 online to replace the borked one.
-
What is it connecting to? The only other report of this was at the other end of a link to a 3100 that had also been updated and required the above fix. But if that was the case here it would still fail after you replaced the local end.
You did not also see logged:
ERROR: FreeBSD route add command failed: external program exited with error status: 1
Which would indicate some sort of route conflict.
Steve
-
@stephenw10 Both ends are SG-2440's That said, the RO is running 2.6.0 (we recently restored it and haven't re-upgraded to Plus yet) Maybe it's a 23 <> 2.6 issue? We're also still running peer to peer on a shared key but seems it's still allowed?
We're back on our "spare" 2440 which is on 22.01 and the tunnel is up.
I didn't see any error messages, just the GDG warning.
-
Shared key will still work in 23.01. It likely won't work in future versions when OpenVPN stops supporting it upstream.
Was there anything logged at the other end?
I would try turning up the logging level in OpenVPN and retesting of you can.Steve
-
@stephenw10 re: shared key...that's what I understaood but worth mentioning.
The remote log shows no errors or warnings.
Looks normal except there's no Peer Connection Initiated with [AF_INET]XXX.XXX.XXX.XXX:1194 at the top of the log.
-
Can you show more of the OpenVPN client log showing the failures? Preferable with the logging level raised.
-
-
@stephenw10 Yes, when this happens I can still SSH to it using putty. This is how I've been restarting it and how I was able to try restarting the web configurator, etc. Other VPN traffic works fine as well ie: web gui for unifi.
-
Ok, well we need more info here to try to reproduce the issue really. Any additional error logs you might be able to get. Anything unusual in your config.
-
@stephenw10 said in OpenVPN could not be established after upgrade to 23.01 on SG-3100:
kldxref /boot/kernel
Finally! This fixed both OpenVPN and Tailscale. Thank you so much for posting this. SG-3100 here.
-
@stephenw10 I just reproduced it on a completely different 3100 at a different location. I have nothing special going on regarding configuration. All I did to reproduce it is connect to openvpn from a windows 10 machine running the openvpn connect client. I visited the pfsense web gui and logged in. I held shift while clicking refresh. 3/4 of the page loaded then it froze completely. Dropdowns stopped working and broken images. I disconnected VPN and reconnected hard wire and still could not get the web gui to properly respond. I rebooted the 3100 and everything is back in order. I can get to the web gui like normal. I can get you whatever logs or configs you'd like.
-
Hmm, do you see anything logged when that happens? Nothing logged as blocked in the firewall log?
The only vaguely related thing I'm aware of here is this: https://redmine.pfsense.org/issues/13938
Obviously there's no kernel panic here but you are still accessing nginx via a virtual interface. You might try disabling unmapped mbufs as shown there to see if it makes any difference.Steve
-
@stephenw10 I didn't see anything logged in system/gui or firewall pointing to an issue. Disabling sendfile on one of them seems to have solved the problem. Doing that and kern.ipc.mb_use_ext_pgs=0 on the other seems to have solved the crashing and need for a reboot, though it doesn't resolve the gui loading through vpn issue.
-
Hmm, that's interesting. I would only have expected to need those workarounds at the remote side of the tunnel.
So with those in place you are still unable to access the webgui on the remote pfSense over the VPN? But doing so no longer prevents clients local to it accessing it after that?
-
@stephenw10 In both test cases they were remote side. I tested each with multiple clients and connections. One 3100 issue was resolved. The other didn't completely resolve with either of the two fixes I mentioned. It continues to fail to load the web gui when accessed through the tunnel but it no longer prevents local access to it prior to a reboot.
-
@techscribe said in OpenVPN could not be established after upgrade to 23.01 on SG-3100:
Tail
Hey @techscribe Did you have to do anything extra to get Tailscale to work? My Tailscale didn't return after entering
kldxref /boot/kernel
into the command prompt. Any insight would be appreciated, thank you! -
-
having the same issue after 23.01 on a custom build tried the fix did not work, removed my defined network from "IPv4 Tunnel Network" openvpn connects again.