OpenVPN could not be established after upgrade to 23.01 on SG-3100

stephenw10 · Feb 28, 2023, 9:15 PM

Ok, probably some other issue then. Unfortunately "Exiting due to fatal error" isn't particularly helpful in the default OpenVPN output. We'll probably need to turn up the logging level if nothing else is shown.

systemofapwne · Mar 5, 2023, 9:34 PM

@stephenw10 Thank you, this simple fix also did it for me!

JVComputers · Mar 11, 2023, 2:50 AM

@stephenw10 This worked for me, as well! Thanks!

rune-san · Mar 12, 2023, 4:28 PM

@stephenw10 Just adding for the web record that I also encountered this issue when upgrading my 3100 this morning. The manual update to the linker file followed by a reboot of the Firewall fixed it.

I obviously have no data on how pervasive this issue is, but like with the 1100 and 2100 maybe it's worth pulling the 23.01 upgrade for the time being on the 3100? Yes I fixed it by going to Netgate forums and see this is one of the top OpenVPN threads but there's lots of folks with these appliances out there that are going to be pulling their hair out over this especially when the interfaces are still in place. Just seems like a big headache for them waiting to happen. Doubly so if they're at remote sites. Either way, the fix was simple! Seems like this should be petted manually and reintroduced as a new release for the 3100 to cover this bug.

stephenw10 · Mar 13, 2023, 1:11 AM

I'm not sure it would be possible to make that change without rebuilding and all that implies.
I'll certainly confirm that though.

Lavarocker · Mar 16, 2023, 5:53 PM

@peekay

Thanks for much for posting the solution - really appreciate the great advice. Confirming OpenVPN still works after reboot.

I'm quite disappointed that Netgate would release "stable" software with a major flaw! Next time I get a survey will voice that.

Rocker

mikej47 · Mar 21, 2023, 8:14 PM

kldxref /boot/kernel and a restart of the OpenVPN service worked for me on my Sg3100. I issued the command through the web gui.

JVComputers · Mar 21, 2023, 10:01 PM

@mikej47 Yes, that worked for me, too. Problem comes when you're upgrading a bunch of firewalls and you connect to them via OpenVPN to administer and you can't restart the service after a reboot because you've been kicked off the VPN due to this bug.

stephenw10 · Mar 21, 2023, 10:49 PM

@mmrk said in OpenVPN could not be established after upgrade to 23.01 on SG-3100:

kldxref

It's fixed in 23.05. https://redmine.pfsense.org/issues/13963

I would always advise having some out of band access for an upgrade though if at all possible. Especially a large step like this. A lot of other changes happened in OpenVPN.

Steve

cneep · Mar 22, 2023, 4:14 AM

@jvcomputers
What I do on the remote firewall is to allow access to ports 22 and 443, but only specifically from:

one public IP address within my primary location's static IP address range, and
an unused public IP address within a nearby satellite location's public IP address range

I've run into situations before where the VPN goes down, so 1) comes in handy to get the VPN back up and running without needing to travel to the remote location. I've also run into a situation before where my ISP actually reassigned the static IP address at my primary location to another client, causing me to lose that static IP address (and subsequently my VPN and the remote access to all of my satellite locations!) So 2) protects against that from happening again....at least I only need to travel to a nearby location to re-establish all the VPNs.

It's not exactly out-of-band, but it serves my needs.

JVComputers · Mar 22, 2023, 4:14 AM

@cneep That's an idea, but one I'd have to carefully consider. I don't like the idea of opening administrative ports directly to the internet, even if I lock them down to my IP.

stephenw10 · Mar 22, 2023, 4:28 PM

If it's an SSH port you can also use key based auth only. The risk of that combined with a limited source IP is close to zero.

Steve

JVComputers · Mar 22, 2023, 4:37 PM

@stephenw10 True, providing no future vulnerabilities are found in the SSH services which is, unfortunately, not unheard of.

stephenw10 · Mar 22, 2023, 4:49 PM

Right. But don't leave the port open after the upgrade is complete and the VPN is back up.
If you really wanted to you could put the rule on a schedule so it closed again automatically if you were still unable to access it for some other reason.

JVComputers · Mar 22, 2023, 4:52 PM

@stephenw10 I feel dumb, I should've thought of only opening it during an upgrade procedure. Good idea.

alvinestevez · Apr 14, 2023, 8:50 PM

After I upgraded my Netgate SG-3100 to pfsense 23.01-RELEASE (arm) from 22.05.1, the OpenVPN demon was unable to start.

Logs show: Cannot open TUN/TAP dev /dev/tun1: No such file or directory (errno=2)
I was unable to start the OpenVPN Deamon

Solution:

goto: Diagnostics, then click Command Prompt
execute: kldxref /boot/kernel
Reboot shortly after executing step 2

After I rebooted the my Openvpn server and and Client tunnel both were running!!!!!

This instructions on these threads are accurate!

jkalber · Apr 27, 2023, 1:09 PM

@mikej47 said in OpenVPN could not be established after upgrade to 23.01 on SG-3100:

kldxref /boot/kernel and a restart of the OpenVPN service worked for me on my Sg3100. I issued the command through the web gui.

Yup same here as well! Had a slight heart attack when the VPN tunnels didn't come back online though. Thankfully we have site to site IPSec tunnels in place or else I would have been making a 2hr round trip commute into the office to finish up everything onsite.

SteveITS · Apr 27, 2023, 3:12 PM

@jkalber There was also a kernel panic bug accessing the web GUI over IPSec and an nginx crash with OpenVPN, though I'm guessing from your message you weren't using pfSense for your remote access?

You might consider having an extra rule you can enable only during upgrades to allow access directly from your IP to WAN.

jkalber · Apr 27, 2023, 3:48 PM

@steveits Hey Steve! I manage a couple offices here in Texas and then a few others out of state. We use OpenVPN for our VPN service and I use the standard Remote Desktop app to remote into the FW's remotely to perform upgrades etc. I haven't ran into any issues aside from last night's little hiccup. Usually I will perform the upgrade and leave OpenVPN opened and reconnecting. Eventually once the FW comes back online the VPN reconnects and I re-gain access. Obviously with last nights upgrade it nuked the OpenVPN services so after about 1hr of it not connecting I disconnected from that sites VPN to my main corporate site. Once I was connected I remoted into our server, and then browsed to the FW that I was working on previously. I'm still new to networking so I hope this makes sense. Your recommendation on creating a rule to allow my static IP access to WAN for upgrades sounds like a great idea and I will look into this. I'm just happy the kernal panic bug you are referring to didn't affect me, guess I lucked out.

m7infratec · Apr 30, 2023, 6:38 PM

Version 23.01 was the worst of all. I have 2 Netgate 2100 in HA, I updated it first and when it restarted it didn't come back, I had to open a ticket, download the image, reinstall the system and redo everything from 0. Now comes the problem of OpenVPN that doesn't work, it even connects but I can't access it the other network, I tested everything and nothing. I'm seriously thinking about removing pfsense and netgate from all customers, because the level of stress and headache it's causing me is not worth it and the worst thing is that we don't see a light at the end of the tunnel, just a palliative solution that It doesn't work, just a waste of time.