OpenVPN fails with 2.50

Gertjan

Wow, detail are not missing here ....

This is what I have :

The settings :

.....

Ok, cut in half :

When exporting :

Note : I do accept the new "2.5.0 OpenVPN stuff".

When I export, I only use one of these button :

Result : using the iPhone OpenVPN Connect App (the official one ?!) it connects using IPv4 - IP ok - also received an IPv6 (correct address) but IPv6 is out.
I can live with that.

@home, I use the OpenVPN connect version 2.4.9 or 2.4.10 (have to check - it from last year) NOT the latest 2.5.1 from yesterday or so (and yes, shame or Netgate for not pushing that one out the minute it came out of the oven, I'm with you here )

It connects with any (or just one) warnings just fine !
IPv4 and IPv6 are up !

My OpenVPN servers bind to my WAN = 192.168.10.3 - I have an upstream ISP router with its NAT rule stuff for VPN on port 1194-UDP.
"Works for yers" .

Now this seems strange. I presumed that I needed to upgrade at least my Windows 10 home OpenVPN connect App, but no ... it was working and kept on working.

Btw : my routes :

I never use my VPN client facilities (I do have a xxxxVPN to play with) - they are just there so I can fool around with it so I can maybe once understand some one wants to tunnel my already TLS encrypted traffic.
I'm using he.net for my IPv6 because, well, you know why.

edit : Maybe I should raise my cyphering. Dono.
The VPN is just for me so I can 'do things at work' while not being at work.

JKnott

@gertjan

The VPN is working, just not when I use my 2nd IPv4 address. As I mentioned above, it does work when my notebook is tethered to my phone. My ISP provides 2 IPv4 addresses, which are not in the same /23 subnet and I normally use the 2nd address for testing. Testing this way has worked for years, until I updated to pfsense 2.5.0. I'd rather not have to tether to my phone for testing, but it appears I may have to.

What changed in pfsense or OpenVPN to cause this?

jimp

@jimp said in OpenVPN fails with 2.50:

"UDP IPv4 and IPv6 on all interfaces (multihome)"

Set that and try it again -- doesn't matter why you have the settings you have now or if it worked before.

JKnott

@jimp

It still fails when I use my second IPv4 address.

JKnott

@jknott

I just tried with ssh and the same thing happens. If I use my 2nd IPv4 address, Packet Capture shows the attempt, but no connection. I can connect with ssh if I tether to my phone. This shows the problem is with FreeBSD and not OpenVPN.

Here's what ifconfig shows:
inet 99.246.125.121 netmask 0xfffffe00 broadcast 255.255.255.255

This shows the appropriate /23 subnet mask. My 2nd address (99.245.223.190) is also a /23, but in a different subnet.

On the other hand, ping works in both directions. What is it that's causing this issue with OpenVPN & ssh?

JKnott

@jknott

Here's a packet capture, showing ping, OpenVPN & ssh (along with some other junk) from my notebook computer to pfsense, using my 2nd IPv4 address. Ping works, but OpenVPN & ssh fail.packetcapture (7).cap Both OpenVPN and ssh work if I tether through my phone.

jimp

If it is in that second subnet only and affects things other than OpenVPN, then it isn't specific to OpenVPN, so the thread title and category here need to be adjusted.

In that packet capture the packet that fails is the largest, so perhaps there is an interface MTU or MSS issue at play. Not sure why it would affect just that one subnet, but it's the first place I'd look.

JKnott

@jimp

Well, it works fine if I tether to my phone, instead of using the 2nd address and it worked for years with the 2nd address, until I updated to 2.5.0.

I agree it's something beyond OpenVPN that also affects ssh. Also, I don't know how it could be an MTU or MSS issue, when both interfaces are connected to the same modem, with the same ISP. I just checked the MTU of the ssh session and it shows 1280 on the notebook end (I don't know why that is), which shouldn't be a problem for pfsense, which has 1500 MTU. Also, UDP doesn't use MTU. Why is pfsense not properly responding to OpenVPN or ssh? It seems to start, then nothing. I also tried with the Windows client. With it, it appeared to connect, but there was no tunnel shown in ipconfig and it couldn't pass any data. That capture was taken with Packet Capture, which means pfsense is receiving, but for some reason failing the connection attempts, but was responding to pings.

As for the 1280 MTU, that's the minimum allowed for IPv6. For the first six years I had IPv6 I was using a tunnel, which forced 1280 MTU. That was never an issue for anything I'm aware of.

jimp

UDP doesn't use MSS but it does respect MTU.

It does respond but once the packets are sufficiently large it fails, which is a common problem associated with MTU issues.

As to why it's different for another subnet, I don't know, that's likely due to differences in your modem/ISP but it's tough to say without more info.

It may work from your phone because your phone connection probably has an MTU lower than whatever is causing the problem.

JKnott

@jimp

UDP respects MTU, but doesn't communicate it with the other end the way TCP does. In looking through the packet capture, I don't see anything longer than 1188 bytes, so even that is below the 1280 MTU.

As for tethering to my phone, the MTU that way is 1500.

I doubt my ISP changed anything when I updated pfsense. I used the VPN, through the 2nd address the day before I updated, as I had for years. I've had this particular modem for a few months, since I updated to IPTV, but have had at least two other modems in the time I've been using the 2nd address.

I never noticed that 1280 MTU before, as I never had any reason to check it. However, it's my notebook computer that gets it, not pfsense, which has 1500, so any UDP coming from it should have already been limited by it. TCP would negotiate the MTU used accordingly.

I wonder if there's anyone else here who can try this, with the 2nd address. I'm on Rogers.

johnpoz

@jknott said in OpenVPN fails with 2.50:

which are not in the same /23 subnet and I normally use the 2nd address for testing

So your having to hairpin up to your ISP to get to your pfsense, from your laptop..

You can see ssh answering there for example - sniffing on your laptop I take you never get those.. So it really has nothing to do with pfsense or your laptop. But your ISP.

JKnott

@johnpoz

As I have said several times, this worked well for several years until I upgraded to 2.5.0. In fact there have been times I mentioned to you that I was doing that. You may recall a topic I started recently about the Windows client not working. While that problem had nothing to do with this issue, at that time I was on 2.4.5 and the Linux client worked and continued to work until 2.5.0 You can see from the packet capture that the connection is started, with responses from pfsense, but suddenly stops. This tells me that the ISP is not blocking anything.

While the capture I provided was from pfsense, I see the same thing with Wireshark on my notebook. That is I am seeing the limited response from pfsense. Again, the ISP/modem is not blocking anything.

BTW, the 2.5.0 Windows client doesn't have the problem I mentioned.

JKnott

@johnpoz

BTW, WRT hairpinning, since it's different subnets, it wouldn't be happening in the modem. It would have to go back to the head end. How would this be any different than a neighbour on the same ISP doing it?

bleeuw

I happen to experience the same issue as JKnott.
I also have en OpenVPN server instance that i use for connecting older Yealink SIP phone's.
This has been working just perfectly since i ever created the VPN back in 2016 until i upgraded pfSense last weekend from 2.4.5 to 2.5.0.
Since 2016 the config has never changed, neither has the topology !

So, there must be some change of behaviour since 2.5.0 as JKnott detailed described already.

However, the issue seems to depend on which combination of the following settings are used...

We use multiple OpenVPN server instances, for different purposes.

for Windows OpenVPN clients to access office-network : remained working perfectly
TCP4 (TUN) 1194
Mode Remote Access ( SSL/TLS)
Ciphers AES-256-GCM, AES-128-GCM, BF-CBC
SHA1 / DH 2048
for Windows/iPad OpenVPN client to access service-network: failed after update to 2.5.0
UDP4 (TUN) 1196
Mode Remote Access ( SSL/TLS + User Auth )
Ciphers AES-128-CBC, AES-128-GCM, AES-256-GCM, BF-CBC
SHA1 / DH 1024
*** After changing mode from SSL/TLS + User Auth to User Auth-only, clients were able to connect again (!) ***
for site-to-site central management of customers with pfSense: remained working perfectly
TCP4 (TUN) 1199
Mode Peer to Peer ( SSL/TLS )
Ciphers AES-256-GCM, AES-128-GCM, AES-128-CBC
SHA1 / DH 1024
for connecting Yealink SIP phones through a T28 client-export: failed after update 2.5.0
UDP4 (TUN) 1201
Mode Remote Access ( SSL/TLS )
Ciphers BF-CBC, AES-128-CBC, AES-128-GCM
SHA1 / DH 1024
Client only supports BF-CBC and is configured in that way in de config-file i created back in 2016.

So, the issue lies in a combination of the fact that either TCP or UDP is used combined with the use of SSL/TLS and/or the Cipher.

This must be an issue that more users are experiencing, using one of these combinations.

Any suggestions (other than already mentioned to JKnott) are welcome.

JKnott

@bleeuw said in OpenVPN fails with 2.50:

So, there must be some change of behaviour since 2.5.0 as JKnott detailed described already.

As described above, my problem was not caused by OpenVPN. For some reason, I couldn't connect when using my 2nd IPv4 address, though I could if I tethered through my cell phone. This also affected ssh.

nicole4pt

I am also pulling my hair out over the same problem. My PFSense box will now no longer connect as a client to an OpenVPNAS server.

I tried changing settings and making sure to match ciphers and algorithms but Nothing has worked.
I just keep getting in my VPN Server..
Authenticate/Decrypt packet error: packet HMAC authentication failed'
TLS Error: incoming packet authentication failed from..

This all worked before upgrading.
/var/etc/openvpn/client1: openssl ciphers -v | grep TLSv1.2
Shows what OpenSSL has available. But still no combination I have tried works.
What got broken. Is any developer responding about this?

I am using OpenVPNAS on my server side. I shot them a plea for help but this really seems like some sort of PFSense/OpenSSL weirdness.

Anyway just another person saying.. What broke in the update :(

johnpoz

What version of openvpn-as are you running. I have a connection as client to an openvpn-as server I run, and never missed a bit..

I am running 2.8.7 of AS..

nicole4pt

@johnpoz I have 2 servers
AcessServerVersion: 2.6.1 TLS Min = 1.2
ASV: 2.7.3 = TLS Min = 1.1

What connection protocols are you using? It was working as 2.4X and right after the upgrade -- not. Same settings.

johnpoz

Both of those are quite OLD.. Why would you not be running 2.8.7?

bennyc

Well... I found one of my openvpn's down this morning. Didn't had time then to troubleshoot, cycling the client (pfSense 2.5.0) didn't instantly help, but changed small setting in the client config (from Gateway "Both" to "IPv4") and it re-connected to the server (pfSense 2.5.0) again.
Looking a bit to the client log files now, and I have these new strange entries in the clients openvpn log:

Mar 14 21:46:10 openvpn 23056 TLS Error: cannot locate HMAC in incoming packet from [AF_INET]185.200.118.41:48846
Mar 14 10:30:44 openvpn 23056 TLS Error: cannot locate HMAC in incoming packet from [AF_INET]185.200.118.79:49851
Mar 14 04:23:53 openvpn 23056 TLS Error: cannot locate HMAC in incoming packet from [AF_INET]146.88.240.4:56098

They seem to be random Public IP's, but coming from where?
I don't see those IP's in the server log on corresponding time.
Given it would be on the server side, I would maybe consider them as rogue ip's trying to connect, but on the client side? (side info; this tunnel is shared key only, no ssl/tls)
Also strange, I see them randomly in clients log, during tunnel up, during tunnel down, mid initialisation sequence. Server log doesn't show anything relevant (or haven't found it yet)
Weird, can't recall having seen that before (tunnel exists since many years)

Can't point it yet to anything, just adding the info here in hope it can help somehow....