OpenVPN could not be established after upgrade to 23.01 on SG-3100

depam

Hi,

I am using SG-3100 after upgrade from 22.05 to 23.01, all my OpenVPN connection could not be initialized.

From system logs,

Feb 16 10:07:22	openvpn	63187	Exiting due to fatal error
Feb 16 10:07:22	openvpn	63187	Cannot open TUN/TAP dev /dev/tun1: No such file or directory (errno=2)
Feb 16 10:07:22	openvpn	63187	WARNING: experimental option --capath /var/etc/openvpn/server1/ca
Feb 16 10:07:22	openvpn	63187	NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
Feb 16 10:07:22	openvpn	63187	library versions: OpenSSL 1.1.1t-freebsd 7 Feb 2023, LZO 2.10
Feb 16 10:07:22	openvpn	63187	OpenVPN 2.6_beta1 armv7-portbld-freebsd14.0 [SSL (OpenSSL)] [LZO] [LZ4] [MH/RECVDA] [AEAD] [DCO]
Feb 16 10:05:51	openvpn	1169	Exiting due to fatal error
Feb 16 10:05:51	openvpn	1169	Cannot open TUN/TAP dev /dev/tun5: No such file or directory (errno=2)
Feb 16 10:05:51	openvpn	1169	Options error: Unrecognized option or missing or extra parameter(s) in [PUSH-OPTIONS]:1: block-outside-dns (2.6_beta1)
Feb 16 10:05:50	openvpn	81436	Exiting due to fatal error
Feb 16 10:05:50	openvpn	81436	Cannot open TUN/TAP dev /dev/tun4: No such file or directory (errno=2)

Anyone have any similar issues? Thanks

catloafsoft

I'm having the same issue. It seems the /dev/tun* devices are not being created, even though the kernel module for those seems to be loaded.

depam

@catloafsoft Yes. What worked for me is complete flash of the 23.01 provided by Support. I am facing other issues now such as pfblocker

catloafsoft

@depam Ugh, I'd rather wait for them to fix the openvpn package (or the kernel, whichever is at fault). Did you have to recreate your VPN configurations?

shpokas

/dev/tap1 missing here, but otherwise exactly the same issue on the same hardware SG-3100

2023-02-16 09:03:59 us=269241 OpenVPN 2.6_beta1 armv7-portbld-freebsd14.0 [SSL (OpenSSL)] [LZO] [LZ4] [MH/RECVDA] [AEAD] [DCO]
2023-02-16 09:03:59 us=269522 library versions: OpenSSL 1.1.1t-freebsd  7 Feb 2023, LZO 2.10
2023-02-16 09:03:59 us=270896 MANAGEMENT: unix domain socket listening on /var/etc/openvpn/server1/sock
2023-02-16 09:03:59 us=271390 NOTE: when bridging your LAN adapter with the TAP adapter, note that the new bridge adapter will often take on its own IP address that is different from what the LAN adapter was previously set to
2023-02-16 09:03:59 us=272082 NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
2023-02-16 09:03:59 us=272302 Initializing OpenSSL support for engine 'devcrypto'
2023-02-16 09:03:59 us=277015 Diffie-Hellman initialized with 4096 bit key
2023-02-16 09:03:59 us=281213 WARNING: experimental option --capath /var/etc/openvpn/server1/ca
2023-02-16 09:03:59 us=282306 Outgoing Control Channel Encryption: Cipher 'AES-256-CTR' initialized with 256 bit key
2023-02-16 09:03:59 us=282830 Outgoing Control Channel Encryption: Using 256 bit message hash 'SHA256' for HMAC authentication
2023-02-16 09:03:59 us=283167 Incoming Control Channel Encryption: Cipher 'AES-256-CTR' initialized with 256 bit key
2023-02-16 09:03:59 us=283459 Incoming Control Channel Encryption: Using 256 bit message hash 'SHA256' for HMAC authentication
2023-02-16 09:03:59 us=283841 TLS-Auth MTU parms [ mss_fix:0 max_frag:0 tun_mtu:1250 tun_max_mtu:0 headroom:126 payload:1600 tailroom:126 ET:0 ]
**2023-02-16 09:03:59 us=284554 Cannot open TUN/TAP dev /dev/tap1: No such file or directory (errno=2)**
2023-02-16 09:03:59 us=284994 Exiting due to fatal error

depam

@catloafsoft No it was taken from the backup. But still "downtime" :(. And my pfblocker and Snort is not functioning properly now. This happened to me few times but on 22.0x it seems the upgrade process is stable. I never learned my lesson to hop in on new release though I know it will always have issue.

innostus

Same issue here. Netgate should not release software that's not been tested for such basic functions!

routeless

Also got this issue on Netgate 3100

Dayve

Same issue here. I upgraded last night and lost all openvpn connections. Anyone know of a fix coming or know how to resolve?

fertig

Same issue here.

Cannot create tun-Interfaces manually either

[23.01-RELEASE][root@fw]/dev: openvpn --mktun --dev tun1
2023-02-16 13:27:50 Note: --mktun does not support DCO. Creating TUN interface.
2023-02-16 13:27:50 Options error: options --mktun and --rmtun are not available on your operating system.  Please check 'man tun' (or 'tap'), whether your system supports using 'ifconfig tun0 create' / 'destroy' to create/remove persistent tunnel interfaces.
2023-02-16 13:27:50 Exiting due to fatal error

[23.01-RELEASE][root@fw]/dev: ifconfig tun1 create
ifconfig: SIOCIFCREATE2 (tun1): Invalid argument

[23.01-RELEASE][root@fw]/dev: mknod /dev/tun1 c 10 200
mknod: /dev/tun1: No such file or directory

There are not many devices at all in /dev, perhaps there's more broken

shpokas

@fertig
why can't they release a new update, instead forcing me to go onsite to reflash the device?

fertig

@shpokas said in OpenVPN could not be established after upgrade to 23.01 on SG-3100:

@fertig
why can't they release a new update, instead forcing me to go onsite to reflash the device?

I think the reflash is a workaround, they'll probably fix it - the problem is about a half a day old

MaxK 0

@innostus Mission critical software development, testing, deployment, and support has been my career since 1985. First rule for applying an update: develop and test a contingency plan to ensure you can fallback to a known operational state if anything goes wrong during or after the update.

You can blame Netgate for missing a defect but we are all responsible for ensuring that we follow industry established best practices to ensure a rapid recovery from an unplanned event or disaster. If I upgraded my environment without testing or having a fallback plan and I am suffering from unexpected or degraded performance, I am to blame - not Netgate.

If I don’t have the time or resources to test or recover then I wait to upgrade and monitor forums like this to see if there are any issues that may impact my environment.

Finally, thank you very much to everyone who did discover issues and provided the valuable information for Netgate to address them.

innostus

@maxk-0 Thanks for replying. You seem to assume that I do not have a fallback. However, one of the reasons someone buys an appliance, is to have decent reliability and support. Also, the 3100 is aimed at small business and private persons, who do not always have a spare device to test with. My own mistakes: I missed the news items on the 23.01 beta, RC and release. They followed rapidly after each other. That should have tipped me off into waiting a little longer.

Have a nice day!

Arnold

johnpoz

@maxk-0 said in OpenVPN could not be established after upgrade to 23.01 on SG-3100:

contingency plan to ensure you can fallback to a known operational state

This is the way ;)

Youtube Video

Be it your in an enterprise or just a home user - this is the way! ;)

shpokas

@johnpoz
sure there's a way, I just, dammit, don't want to go onsite!

stephenw10

Opened a bug for this: https://redmine.pfsense.org/issues/13963

Updating the linker file manaully fixed it for me. Run: kldxref /boot/kernel

Steve

innostus

@stephenw10 This works for me!

fertig

@stephenw10

working fine and is reboot safe. Excellent. Thanks.
I won't reflash and wait for a possible bugfix release.

johnpoz

@shpokas I hear ya - but we always have someone on hand in the DC when doing any sort of upgrade.. Just in case.. The best laid plans can always fail..

I left some boxes at sites on old pfsense for longest time because of covid, and just pita to either go there or have someone available..

And also hear ya that damn it this box should just work.. And I should just be able to click update.. But no matter how easy it should be or how many times it has worked in the past, etc. And this is not some minor update, this is a major change to freebsd, and an upgrade to php as well, etc..

I haven't pulled the trigger yet on my 4860 here at home.. But I did get the new 23.01 image from tac already, and will burn it to usb and be ready to clean install if something goes horrible wrong.. Nothing will I hope - but just in case, I don't want to be down for any period of time. I also still have 22.05 image if need be to clean install too, etc.

Also notice that I might have an issue with my 4860 "may have issues with the ichsmb0 and/or ehci0" so have that info handy to put in the /boot/loader.conf.local

If something crashes and burns - I wouldn't be happy no.. But also know that things like this can happen.. I need to update the dsm version on my nas for example.. And had many an update go smooth in the past, but since the last update I have created a non supported configuration with nvme as storage.. People say shouldn't be a problem - but just in case I am currently backing up my plex and dockers that reside on the nvme storage - just in case that doesn't survive the update.. And I don't want to go through the hassle recreating them, nor using a backup from a few days ago.. So running a backup just before I do the update later this morning ;)

stephenw10

I would add the line to disable ichsmb on the 4860 before upgrading @johnpoz
You lose nothing by having it there. We will likely add that by default in 23.05.
If you're not on the console you won't notice anything anyway.

johnpoz

@stephenw10 thanks I was thinking of that.. Any reason why a clean install might be good - change to the zfs layout or anything. I was thinking I should be able to test the zfs rollback feature with this update as well.

So should I be able to rollback to 22.05 after an update to 23.01?

stephenw10

Yes you can roll back to 22.05 from 23.01. But not to 22.01.

There's no particular reason a clean install should be any better there. IMO at least!

johnpoz

@stephenw10 might have to give that a test run ;) I really don't ever see any need for the feature myself - but might be nice to have at least tested it in case of anyone having questions on it at some point.

fred7911

Same here (SG-3100), OpenVPN service cannot start after 23.01 update.

Is the command kldxref /boot/kernel (described in this thead as a fix) the offcial recommended solution, or should we wait for another workaround? If done, would it be a permanent fix?

Thank you

stephenw10

I have use that and seen no issues since. It does survive a reboot.

fred7911

Ok great, thank you!
I will also follow your bug report status..

bdzero

Same here, kldxref /boot/kernel fixed for me as well. 3100

jsnaza

I also had this issue with a 3100 and the kldxref /boot/kernel command worked for me too.

crogerson

@stephenw10 I ended up having this issue on a 1537. Applied the patch from the redmine post and issue is now resolved.

acardenas

@stephenw10 Is this ran through SSH console? I ran via SSH and ran into the following error: kldxref: can't create /boot/lhint.h6bsDi: Permission denied

stephenw10

You probably need to be logged in as root or admin to run that.

It should work via Diag > Command Prompt in the webgui though.

PeeKay

@stephenw10 worked for me. Thanks for this!
Ran kldxref /boot/kernel in Diagnostics > Command Prompt on 3100 and restarted OpenVPN services. fired up immediately.

jkibbey

The above fix and/or patch worked for me and VPN functionality is working fine with the only perceived exception: when trying to access the pfSense web gui through VPN, it loads some of the page then hangs. From then forward, it seems to corrupt the gui service (PHP?) for any client until the firewall is rebooted (restarting webConfigurator and restarting PHP-FPM did not resolve the issue, reboot was required.) I also tried different browsers, different client machines, and forcing a cache refresh.)

I've tested this through site-to-site as well as remote access through OpenVPN. Everything else through VPN is working fine. Testing included SSH, Remote Desktop, UniFi web gui.

I'm using a Netgate 3100.

tennbaum

@peekay said in OpenVPN could not be established after upgrade to 23.01 on SG-3100:

works for me as well (3100)
kldxref /boot/kernel
all working after (did not check the pfsense gui though)

stephenw10

@jkibbey said in OpenVPN could not be established after upgrade to 23.01 on SG-3100:

From then forward, it seems to corrupt the gui service (PHP?) for any client until the firewall is rebooted

Hmm, that's odd. Do you mean it happens one time after updating the linker hints? Or Whenever you try to access the gui over the VPN even after the reboot?

And does that then also apply to local clients accessing it?

Steve

jkibbey

@stephenw10 It happens anytime I try to access the pfSense gui via the OpenVPN tunnel. All other communication through VPN is fine and the pfSense gui functions like normal if I do not access it via VPN and if I have not attempted to use it via the tunnel after a reboot, even if it was from another client.

stephenw10

Hmm, is it only the webgui, can you ssh to the firewall across the VPN?

JayB

Dealing with similar result (different cause?) this morning on our 2440. Upgraded HO on Saturday and We've no connectivity this morning (first day office is back since upgrade)

Tunnel is stuck on "Adding Routes to System" and the RO (which did not get the upgrade) is showing "reconnecting: ping-restart". The logs show GDG: problem writing to routing socket. Tried the kldxref /boot/kernel fix but to no effect.

Unable to roll back to previous stable (not offered) so we're putting a spare that's on 22.05 online to replace the borked one.

stephenw10

What is it connecting to? The only other report of this was at the other end of a link to a 3100 that had also been updated and required the above fix. But if that was the case here it would still fail after you replaced the local end.

You did not also see logged:

ERROR: FreeBSD route add command failed: external program exited with error status: 1

Which would indicate some sort of route conflict.

Steve