OpenVPN fails with 2.50

JKnott

I just tried with ssh and the same thing happens. If I use my 2nd IPv4 address, Packet Capture shows the attempt, but no connection. I can connect with ssh if I tether to my phone. This shows the problem is with FreeBSD and not OpenVPN.

Here's what ifconfig shows:
inet 99.246.125.121 netmask 0xfffffe00 broadcast 255.255.255.255

This shows the appropriate /23 subnet mask. My 2nd address (99.245.223.190) is also a /23, but in a different subnet.

On the other hand, ping works in both directions. What is it that's causing this issue with OpenVPN & ssh?

JKnott

@jknott

Here's a packet capture, showing ping, OpenVPN & ssh (along with some other junk) from my notebook computer to pfsense, using my 2nd IPv4 address. Ping works, but OpenVPN & ssh fail.packetcapture (7).cap Both OpenVPN and ssh work if I tether through my phone.

jimp

If it is in that second subnet only and affects things other than OpenVPN, then it isn't specific to OpenVPN, so the thread title and category here need to be adjusted.

In that packet capture the packet that fails is the largest, so perhaps there is an interface MTU or MSS issue at play. Not sure why it would affect just that one subnet, but it's the first place I'd look.

JKnott

@jimp

Well, it works fine if I tether to my phone, instead of using the 2nd address and it worked for years with the 2nd address, until I updated to 2.5.0.

I agree it's something beyond OpenVPN that also affects ssh. Also, I don't know how it could be an MTU or MSS issue, when both interfaces are connected to the same modem, with the same ISP. I just checked the MTU of the ssh session and it shows 1280 on the notebook end (I don't know why that is), which shouldn't be a problem for pfsense, which has 1500 MTU. Also, UDP doesn't use MTU. Why is pfsense not properly responding to OpenVPN or ssh? It seems to start, then nothing. I also tried with the Windows client. With it, it appeared to connect, but there was no tunnel shown in ipconfig and it couldn't pass any data. That capture was taken with Packet Capture, which means pfsense is receiving, but for some reason failing the connection attempts, but was responding to pings.

As for the 1280 MTU, that's the minimum allowed for IPv6. For the first six years I had IPv6 I was using a tunnel, which forced 1280 MTU. That was never an issue for anything I'm aware of.

jimp

UDP doesn't use MSS but it does respect MTU.

It does respond but once the packets are sufficiently large it fails, which is a common problem associated with MTU issues.

As to why it's different for another subnet, I don't know, that's likely due to differences in your modem/ISP but it's tough to say without more info.

It may work from your phone because your phone connection probably has an MTU lower than whatever is causing the problem.

JKnott

@jimp

UDP respects MTU, but doesn't communicate it with the other end the way TCP does. In looking through the packet capture, I don't see anything longer than 1188 bytes, so even that is below the 1280 MTU.

As for tethering to my phone, the MTU that way is 1500.

I doubt my ISP changed anything when I updated pfsense. I used the VPN, through the 2nd address the day before I updated, as I had for years. I've had this particular modem for a few months, since I updated to IPTV, but have had at least two other modems in the time I've been using the 2nd address.

I never noticed that 1280 MTU before, as I never had any reason to check it. However, it's my notebook computer that gets it, not pfsense, which has 1500, so any UDP coming from it should have already been limited by it. TCP would negotiate the MTU used accordingly.

I wonder if there's anyone else here who can try this, with the 2nd address. I'm on Rogers.

johnpoz

@jknott said in OpenVPN fails with 2.50:

which are not in the same /23 subnet and I normally use the 2nd address for testing

So your having to hairpin up to your ISP to get to your pfsense, from your laptop..

You can see ssh answering there for example - sniffing on your laptop I take you never get those.. So it really has nothing to do with pfsense or your laptop. But your ISP.

JKnott

@johnpoz

As I have said several times, this worked well for several years until I upgraded to 2.5.0. In fact there have been times I mentioned to you that I was doing that. You may recall a topic I started recently about the Windows client not working. While that problem had nothing to do with this issue, at that time I was on 2.4.5 and the Linux client worked and continued to work until 2.5.0 You can see from the packet capture that the connection is started, with responses from pfsense, but suddenly stops. This tells me that the ISP is not blocking anything.

While the capture I provided was from pfsense, I see the same thing with Wireshark on my notebook. That is I am seeing the limited response from pfsense. Again, the ISP/modem is not blocking anything.

BTW, the 2.5.0 Windows client doesn't have the problem I mentioned.

JKnott

@johnpoz

BTW, WRT hairpinning, since it's different subnets, it wouldn't be happening in the modem. It would have to go back to the head end. How would this be any different than a neighbour on the same ISP doing it?

bleeuw

I happen to experience the same issue as JKnott.
I also have en OpenVPN server instance that i use for connecting older Yealink SIP phone's.
This has been working just perfectly since i ever created the VPN back in 2016 until i upgraded pfSense last weekend from 2.4.5 to 2.5.0.
Since 2016 the config has never changed, neither has the topology !

So, there must be some change of behaviour since 2.5.0 as JKnott detailed described already.

However, the issue seems to depend on which combination of the following settings are used...

We use multiple OpenVPN server instances, for different purposes.

for Windows OpenVPN clients to access office-network : remained working perfectly
TCP4 (TUN) 1194
Mode Remote Access ( SSL/TLS)
Ciphers AES-256-GCM, AES-128-GCM, BF-CBC
SHA1 / DH 2048
for Windows/iPad OpenVPN client to access service-network: failed after update to 2.5.0
UDP4 (TUN) 1196
Mode Remote Access ( SSL/TLS + User Auth )
Ciphers AES-128-CBC, AES-128-GCM, AES-256-GCM, BF-CBC
SHA1 / DH 1024
*** After changing mode from SSL/TLS + User Auth to User Auth-only, clients were able to connect again (!) ***
for site-to-site central management of customers with pfSense: remained working perfectly
TCP4 (TUN) 1199
Mode Peer to Peer ( SSL/TLS )
Ciphers AES-256-GCM, AES-128-GCM, AES-128-CBC
SHA1 / DH 1024
for connecting Yealink SIP phones through a T28 client-export: failed after update 2.5.0
UDP4 (TUN) 1201
Mode Remote Access ( SSL/TLS )
Ciphers BF-CBC, AES-128-CBC, AES-128-GCM
SHA1 / DH 1024
Client only supports BF-CBC and is configured in that way in de config-file i created back in 2016.

So, the issue lies in a combination of the fact that either TCP or UDP is used combined with the use of SSL/TLS and/or the Cipher.

This must be an issue that more users are experiencing, using one of these combinations.

Any suggestions (other than already mentioned to JKnott) are welcome.

JKnott

@bleeuw said in OpenVPN fails with 2.50:

So, there must be some change of behaviour since 2.5.0 as JKnott detailed described already.

As described above, my problem was not caused by OpenVPN. For some reason, I couldn't connect when using my 2nd IPv4 address, though I could if I tethered through my cell phone. This also affected ssh.

nicole4pt

I am also pulling my hair out over the same problem. My PFSense box will now no longer connect as a client to an OpenVPNAS server.

I tried changing settings and making sure to match ciphers and algorithms but Nothing has worked.
I just keep getting in my VPN Server..
Authenticate/Decrypt packet error: packet HMAC authentication failed'
TLS Error: incoming packet authentication failed from..

This all worked before upgrading.
/var/etc/openvpn/client1: openssl ciphers -v | grep TLSv1.2
Shows what OpenSSL has available. But still no combination I have tried works.
What got broken. Is any developer responding about this?

I am using OpenVPNAS on my server side. I shot them a plea for help but this really seems like some sort of PFSense/OpenSSL weirdness.

Anyway just another person saying.. What broke in the update :(

johnpoz

What version of openvpn-as are you running. I have a connection as client to an openvpn-as server I run, and never missed a bit..

I am running 2.8.7 of AS..

nicole4pt

@johnpoz I have 2 servers
AcessServerVersion: 2.6.1 TLS Min = 1.2
ASV: 2.7.3 = TLS Min = 1.1

What connection protocols are you using? It was working as 2.4X and right after the upgrade -- not. Same settings.

johnpoz

Both of those are quite OLD.. Why would you not be running 2.8.7?

bennyc

Well... I found one of my openvpn's down this morning. Didn't had time then to troubleshoot, cycling the client (pfSense 2.5.0) didn't instantly help, but changed small setting in the client config (from Gateway "Both" to "IPv4") and it re-connected to the server (pfSense 2.5.0) again.
Looking a bit to the client log files now, and I have these new strange entries in the clients openvpn log:

Mar 14 21:46:10 openvpn 23056 TLS Error: cannot locate HMAC in incoming packet from [AF_INET]185.200.118.41:48846
Mar 14 10:30:44 openvpn 23056 TLS Error: cannot locate HMAC in incoming packet from [AF_INET]185.200.118.79:49851
Mar 14 04:23:53 openvpn 23056 TLS Error: cannot locate HMAC in incoming packet from [AF_INET]146.88.240.4:56098

They seem to be random Public IP's, but coming from where?
I don't see those IP's in the server log on corresponding time.
Given it would be on the server side, I would maybe consider them as rogue ip's trying to connect, but on the client side? (side info; this tunnel is shared key only, no ssl/tls)
Also strange, I see them randomly in clients log, during tunnel up, during tunnel down, mid initialisation sequence. Server log doesn't show anything relevant (or haven't found it yet)
Weird, can't recall having seen that before (tunnel exists since many years)

Can't point it yet to anything, just adding the info here in hope it can help somehow....

nicole4pt

@johnpoz Because management does not like downtime and quite often when you upgrade you also have to force users to download new clients.

So besides asking why something is old, I guess you too are out of ideas?
It sounds like you are saying, once you upgrade to PFS 2.5.X you had better be using a brand new server and version of OpenVPN or it won't work. :(

So PfSense needs specific data. Besides what is in the config file, How do you query the openVPN server find out out what entries are needed?
How can you find out if there is a cipher mismatch and what it may be?

Also if you say you are working, what are your settings to perhaps compare?

nicole4pt

@johnpoz
I spun up a test VPN on a cloud site and indeed, it seems to be true that Pfsense 2.5 will work with a brand new 2.8.5 openVPNas (on a new version of OS) server.
(Although needing RSA-Sha1 even though my config says SHA256)

But so far Not with an older 2.6 or 2.7 version OpenVPNas

So some backward compatibility seems missing.

johnpoz

@nicole4pt said in OpenVPN fails with 2.50:

So some backward compatibility seems missing

Quite possible... I get it people don't like downtime and change.. But your version of openvpnas is couple years old - you know how many security fixes have been included.. Why would you not spin up 2.8.7 which is current vs 2.8.5?

nicole4pt

@johnpoz

This is why I hate forums for asking questions. (or is it just because I'm a woman?)
So far no real help ... no providing configs... and just why don't you have the latest and greatest. How dare you. How about you ask Debian and Digital Ocean why they provide it. Maybe because as with PFS 2.5 all thats new is not always better and as usual with upgrades it seems to always break something and demand other upgrades on down the line. :(

(It's also one of the reasons I have moved away from FreeBSD for a number of things. They force an upgrades on production servers or you risk never being able to find an older package or even upgrade it if you're not fast enough. Their long term support is exceptionally minimal these days)

2.5 Breaking something like backward compatibility with things like OpenVPN is going to ruin a lot of peoples day the hard way. Especially since there seems to be no warning about... yet. I wonder if anything in 2.5.1-RC addresses this?