OpenVPN fails with 2.50

JKnott · Feb 28, 2021, 4:30 AM

As I have said several times, this worked well for several years until I upgraded to 2.5.0. In fact there have been times I mentioned to you that I was doing that. You may recall a topic I started recently about the Windows client not working. While that problem had nothing to do with this issue, at that time I was on 2.4.5 and the Linux client worked and continued to work until 2.5.0 You can see from the packet capture that the connection is started, with responses from pfsense, but suddenly stops. This tells me that the ISP is not blocking anything.

While the capture I provided was from pfsense, I see the same thing with Wireshark on my notebook. That is I am seeing the limited response from pfsense. Again, the ISP/modem is not blocking anything.

BTW, the 2.5.0 Windows client doesn't have the problem I mentioned.

JKnott · Feb 28, 2021, 4:34 AM

@johnpoz

BTW, WRT hairpinning, since it's different subnets, it wouldn't be happening in the modem. It would have to go back to the head end. How would this be any different than a neighbour on the same ISP doing it?

bleeuw · Mar 18, 2021, 10:40 AM

I happen to experience the same issue as JKnott.
I also have en OpenVPN server instance that i use for connecting older Yealink SIP phone's.
This has been working just perfectly since i ever created the VPN back in 2016 until i upgraded pfSense last weekend from 2.4.5 to 2.5.0.
Since 2016 the config has never changed, neither has the topology !

So, there must be some change of behaviour since 2.5.0 as JKnott detailed described already.

However, the issue seems to depend on which combination of the following settings are used...

We use multiple OpenVPN server instances, for different purposes.

for Windows OpenVPN clients to access office-network : remained working perfectly
TCP4 (TUN) 1194
Mode Remote Access ( SSL/TLS)
Ciphers AES-256-GCM, AES-128-GCM, BF-CBC
SHA1 / DH 2048
for Windows/iPad OpenVPN client to access service-network: failed after update to 2.5.0
UDP4 (TUN) 1196
Mode Remote Access ( SSL/TLS + User Auth )
Ciphers AES-128-CBC, AES-128-GCM, AES-256-GCM, BF-CBC
SHA1 / DH 1024
*** After changing mode from SSL/TLS + User Auth to User Auth-only, clients were able to connect again (!) ***
for site-to-site central management of customers with pfSense: remained working perfectly
TCP4 (TUN) 1199
Mode Peer to Peer ( SSL/TLS )
Ciphers AES-256-GCM, AES-128-GCM, AES-128-CBC
SHA1 / DH 1024
for connecting Yealink SIP phones through a T28 client-export: failed after update 2.5.0
UDP4 (TUN) 1201
Mode Remote Access ( SSL/TLS )
Ciphers BF-CBC, AES-128-CBC, AES-128-GCM
SHA1 / DH 1024
Client only supports BF-CBC and is configured in that way in de config-file i created back in 2016.

So, the issue lies in a combination of the fact that either TCP or UDP is used combined with the use of SSL/TLS and/or the Cipher.

This must be an issue that more users are experiencing, using one of these combinations.

Any suggestions (other than already mentioned to JKnott) are welcome.

JKnott · Mar 19, 2021, 7:53 AM

@bleeuw said in OpenVPN fails with 2.50:

So, there must be some change of behaviour since 2.5.0 as JKnott detailed described already.

As described above, my problem was not caused by OpenVPN. For some reason, I couldn't connect when using my 2nd IPv4 address, though I could if I tethered through my cell phone. This also affected ssh.

nicole4pt · Mar 18, 2021, 7:27 PM

I am also pulling my hair out over the same problem. My PFSense box will now no longer connect as a client to an OpenVPNAS server.

I tried changing settings and making sure to match ciphers and algorithms but Nothing has worked.
I just keep getting in my VPN Server..
Authenticate/Decrypt packet error: packet HMAC authentication failed'
TLS Error: incoming packet authentication failed from..

This all worked before upgrading.
/var/etc/openvpn/client1: openssl ciphers -v | grep TLSv1.2
Shows what OpenSSL has available. But still no combination I have tried works.
What got broken. Is any developer responding about this?

I am using OpenVPNAS on my server side. I shot them a plea for help but this really seems like some sort of PFSense/OpenSSL weirdness.

Anyway just another person saying.. What broke in the update :(

johnpoz · Mar 18, 2021, 7:34 PM

What version of openvpn-as are you running. I have a connection as client to an openvpn-as server I run, and never missed a bit..

I am running 2.8.7 of AS..

nicole4pt · Mar 18, 2021, 7:54 PM

@johnpoz I have 2 servers
AcessServerVersion: 2.6.1 TLS Min = 1.2
ASV: 2.7.3 = TLS Min = 1.1

What connection protocols are you using? It was working as 2.4X and right after the upgrade -- not. Same settings.

johnpoz · Mar 18, 2021, 8:01 PM

Both of those are quite OLD.. Why would you not be running 2.8.7?

bennyc · Mar 18, 2021, 8:13 PM

Well... I found one of my openvpn's down this morning. Didn't had time then to troubleshoot, cycling the client (pfSense 2.5.0) didn't instantly help, but changed small setting in the client config (from Gateway "Both" to "IPv4") and it re-connected to the server (pfSense 2.5.0) again.
Looking a bit to the client log files now, and I have these new strange entries in the clients openvpn log:

Mar 14 21:46:10 openvpn 23056 TLS Error: cannot locate HMAC in incoming packet from [AF_INET]185.200.118.41:48846
Mar 14 10:30:44 openvpn 23056 TLS Error: cannot locate HMAC in incoming packet from [AF_INET]185.200.118.79:49851
Mar 14 04:23:53 openvpn 23056 TLS Error: cannot locate HMAC in incoming packet from [AF_INET]146.88.240.4:56098

They seem to be random Public IP's, but coming from where?
I don't see those IP's in the server log on corresponding time.
Given it would be on the server side, I would maybe consider them as rogue ip's trying to connect, but on the client side? (side info; this tunnel is shared key only, no ssl/tls)
Also strange, I see them randomly in clients log, during tunnel up, during tunnel down, mid initialisation sequence. Server log doesn't show anything relevant (or haven't found it yet)
Weird, can't recall having seen that before (tunnel exists since many years)

Can't point it yet to anything, just adding the info here in hope it can help somehow....

nicole4pt · Mar 18, 2021, 9:25 PM

@johnpoz Because management does not like downtime and quite often when you upgrade you also have to force users to download new clients.

So besides asking why something is old, I guess you too are out of ideas?
It sounds like you are saying, once you upgrade to PFS 2.5.X you had better be using a brand new server and version of OpenVPN or it won't work. :(

So PfSense needs specific data. Besides what is in the config file, How do you query the openVPN server find out out what entries are needed?
How can you find out if there is a cipher mismatch and what it may be?

Also if you say you are working, what are your settings to perhaps compare?

nicole4pt · Mar 18, 2021, 10:22 PM

@johnpoz
I spun up a test VPN on a cloud site and indeed, it seems to be true that Pfsense 2.5 will work with a brand new 2.8.5 openVPNas (on a new version of OS) server.
(Although needing RSA-Sha1 even though my config says SHA256)

But so far Not with an older 2.6 or 2.7 version OpenVPNas

So some backward compatibility seems missing.

johnpoz · Mar 18, 2021, 10:26 PM

@nicole4pt said in OpenVPN fails with 2.50:

So some backward compatibility seems missing

Quite possible... I get it people don't like downtime and change.. But your version of openvpnas is couple years old - you know how many security fixes have been included.. Why would you not spin up 2.8.7 which is current vs 2.8.5?

nicole4pt · Mar 18, 2021, 11:11 PM

@johnpoz

This is why I hate forums for asking questions. (or is it just because I'm a woman?)
So far no real help ... no providing configs... and just why don't you have the latest and greatest. How dare you. How about you ask Debian and Digital Ocean why they provide it. Maybe because as with PFS 2.5 all thats new is not always better and as usual with upgrades it seems to always break something and demand other upgrades on down the line. :(

(It's also one of the reasons I have moved away from FreeBSD for a number of things. They force an upgrades on production servers or you risk never being able to find an older package or even upgrade it if you're not fast enough. Their long term support is exceptionally minimal these days)

2.5 Breaking something like backward compatibility with things like OpenVPN is going to ruin a lot of peoples day the hard way. Especially since there seems to be no warning about... yet. I wonder if anything in 2.5.1-RC addresses this?

johnpoz · Mar 18, 2021, 11:30 PM

Why don't you post your configs?

As you saw updating your AS its working. Your welcome...

My stuff is current - like I said never had any issues moving up in versions of AS as they came out, and moving up to new versions of openvpn in pfsense as it was updated. Because I stayed current..

But now your all ticked that you updated one side and it broke because the other side is antiquated.. ?? That is somehow pfsense problem?

nicole4pt · Mar 19, 2021, 12:17 AM

@johnpoz
Exactly. When you asked why was I using older versions of OpenVPNas, you just made the case for why people do not upgrade.

BSD has developed a bad reputation for breaking backward compatibility in the name of forced security.

Sadly for things like PfSense it creates unannounced breakage. Which doesn't help it.

I'm glad you are current on everything. However in a production environment and known issues of upgrading, many people will not be able to spend the time and possible downtime will likely wind up here for when they finally do, hoping for a, well it won't work, or help. But seriously, not, works for me, sucks for you.

Gertjan · Mar 19, 2021, 7:12 AM

@nicole4pt said in OpenVPN fails with 2.50:

So some backward compatibility seems missing.

pfSense didn't invent the VPN part, the OpenVPN part (they somewhat did so with WireGaurd)
About the compatibility, I guess OpenVPN as a whole is as complex as is pfSense.
Both have one point in common : the backwards compatibility comes after... security.
So, yeah, check out the FAQ and manuals about the 2.5.0 (identical version number - it's 2.5.1 already) : they did, for example, remove old crypto stuff that's known to be weak now. Another aspect is : a tunnel was always over IPv4. That fades out now, as it could also be IPv6. So 'config option' get renamed, added, removed.

Also, VPN access has become very important for a lot of people since March 2020.
If companies wanted easy-of-use first, they would have stayed also with XP - or, as some are still doing, use Win 7 - and RDP - on both sides.

@nicole4pt said in OpenVPN fails with 2.50:

BSD has developed a bad reputation for breaking backward compatibility in the name of forced security.

Yep, and glad I does. BSD is also known as the "OS" with one of the best network stacks. That's why its used for pfSense (also, true ;) for legacy reasons - but changes the OS is like creating a new product).

@nicole4pt said in OpenVPN fails with 2.50:

I'm glad you are current on everything

Because he (@johnpoz ) is probably both the OpenVPN admin and OpenVPN (road warrior) user, so he is using the OpenVPN desktop traybar tool tool that shows the client- log- connecting-to-the-server initial phase. This small log windows is not some gadget, but part of the security process.
As soon as there are red "depreciated" lines, he translates that to "not- appreciated", no need for a science background that make that translation, and acts upon it, so the client follows the VPN server version number.

An OpenVPN basic end user should have a "what to do" list which stated that if these "depreciated" show up, the log should be Ctrl-C Ctrl-V and mailed to the vpn administrator, so a teamviewer session can be planned so the admin can update the client when he see fits.

bennyc · Mar 19, 2021, 7:21 AM

Hmm, it might very well be the backwards compatibility is an issue, but then again as in my situation it is implemented from pfSense to pfSense and both are on the same latest level, then imho compatibility shouldn't be such an issue.
And, to get back on topic (please), there are strange things seen and reported here by others and myself in this topic, where the jury is still out if it's configuration related...

Gertjan · Mar 19, 2021, 7:53 AM

@bennyc said in OpenVPN fails with 2.50:

Hmm, it might very well be the backwards compatibility is an issue, but then again as in my situation it is implemented from pfSense to pfSense and both are on the same latest level, then imho compatibility shouldn't be such an issue.

I agree, pfSense to pfSense, 2.5.0 to 2.5.0, identical settings, from client to server, should work.

topic

@jknott said in OpenVPN fails with 2.50:

@bleeuw said in OpenVPN fails with 2.50:

So, there must be some change of behaviour since 2.5.0 as JKnott detailed described already.

As described above, my problem was not caused by OpenVPN. For some reason, I couldn't connect when using my 2nd IPv4 address, though I could if I tethered through my cell phone. This also affected ssh.

The topic title "OpenVPN fails with 2.50 " was wrong.
The initial poster had issues that didn't start with OpenVPN.

bennyc · Mar 19, 2021, 8:17 AM

True, though he didn't know that initially because it affected openvpn functionality.

Can we agree on the fact there is an issue? (even though it isn't pinned yet to a specific item? )

I have the feeling it doesn't affect a lot of people, otherwise there would be more noise here in the forum. (or it is yet to come)
I also don't see yet how to start troubleshooting this, as I only see limited side-effects in my situation... (All hints welcome)

bleeuw · Apr 29, 2021, 10:08 PM

For anyone who might be interested (after running into the OpenVPN issues i experienced with multiple customer after upgrading both our customers and our own pfSense boxes to 2.5.0:
All the OpenVPN related issues dissappeared "like snow in the sun" after upgrading to 2.5.1 !!
(So, @bennyc i think we can agree that there was an issue... ;) )

However.... a big advice to those who have multiple WAN-connections (wether or not using load-balancing or fail-over mechanism's...) before upgrading to 2.5.1: do not upgrade!!!
Those customers we have with multiple WAN's had serious connectivity-issues (not only over VPN but also from/to the internet) after upgrading from 2.5.0 to 2.5.1!

Rightnow i don't have the time (or desire) to look into this new introduced issue in 2.5.1... i'll just wait for the next patch or major update and have another go with upgrading a multi-WAN customer....

Cheers.