PFSense Release 2.5 + OpenVPN 2.5 broken? Any fixes?

A Former User

@n8lbv, I have different sites, some with 2.4, 2.5 and 2.5.2, the version doesn't matter, it always works the first time...!

just try it without configuring anything else extra.

Gertjan

@silence said in PFSense Release 2.5 + OpenVPN 2.5 broken? Any fixes?:

some with 2.4, 2.5 and 2.5.2

And these 2.4.5 and 2.5.2 are "OpenVPN" "site-to-site" interconnected ?
I guess it is possible, but "VPN" settings on both sides have to be fine tuned.

pfSense 2.4.5px uses the OpenVPN version 2.4.8 (or 9).
pfSense 2.5.2 uses :

OpenVPN 2.5.2 amd64-portbld-freebsd12.2 [SSL (OpenSSL)] [LZO] [LZ4] [MH/RECVDA] [AEAD] built on Jun 24 2021
....

That's 2.5.2; pure coincidence that pfSense and OpenVPN have the same version number.
Btw : the latest OpenVPN, today, is 2.5.5

Between OpenVPN 2.4.x and OpenVPN 2.5.2, there were enough changes to make things break: RTFCL here : See "Overview of changes since OpenVPN 2.4".

As soon as you see a "soon to be depreciated'" in the **OpenVPN log today, you should redo the setup asap so that "depreciated option" message are not shown any-more.
The thing is : this is nearly never done. "As it works right now - I don't touch it" and "I'm not looking at the logs now and won't start doing so tomorrow ;)".

When the the big upgrade is presented, like pfSense from 2.4.5 to 2.5.x, this includes OpenVPN 2.4.x to 2.5.0, a major upgrade version shift - the connection breaks.

Btw : pfSense OpenVPN site to site - client to VPN like "Exp*ssVPN" or OpenVPN as a remote pfSense admin access works for for me
I rarely use the site to site, less often the client OpenVPN, but most often the "have to work from home" option.

mrpushner

@gertjan Hi, I had 3 total sites all on 2.4.4. I updated (via GUI) my Server site to 2.5.2. SITE TO SITE open VPN's continued to work fine. So two client sites on 2.4.4 to Server site on 2.5.2.

As soon I updated one of my client sites to 2.5.2, VPN crashed.

I had to go to the site and restore back to 2.4.4 to get it to work again.

Since then, I did a fresh install of 2.5.2 on that client site (on my spare PFS hardware), and applied my 2.4.4 backup config, and it came back online normally.

I did not determine the exact cause of this problem.

MP

N8LBV

@mrpushner This seem to be the state I am in and what has happened.

I no longer have backup configs from 2.4.5 or 2.4.4 so I do not have the option to
restore from those on my sites that I have updated to 2.5 about 1 year ago.
But it may be an option for me on all of the sites that are still on 2.4.5 if that proves to be a way of fixing them.

Right now I have a couple of sites that have been upgraded around a year ago that are now on 2.5.2 and would like to add VPN but cannot because of this problem.

And doing a fresh install and manually restoring all of the settings would be a lot of work :)
The reality is that I may have to do that for this issue.

I can also try restoring from a current backup after a fresh install.
To see if the problem follows the config or not.
I am NOT backing up ANY VPN settings configs or certificates.
In all cases the VPN setup is brand new.
And is still broken on any system I try to bring up that has not been a fresh install.

Gertjan

@mrpushner said in PFSense Release 2.5 + OpenVPN 2.5 broken? Any fixes?:

Hi, I had 3 total sites all on 2.4.4. I updated (via GUI) my Server site to 2.5.2. SITE TO SITE open VPN's continued to work fine. So two client sites on 2.4.4 to Server site on 2.5.2.
As soon I updated one of my client sites to 2.5.2, VPN crashed.

Oh ... great.
My "common sense" would say :
if 2.4.4 (really ?? 2.4.4 ??? 2.4.4 is way to old) to 2.4.4 works.
and
2.4.4 to 2.5.2 works
then
2.5.2 to 2.5.2 should also work.

A bit like
"Windows 7 networking" to "Windows 7 networking" worked.
"Windows 7 networking" to "Windows 10 networking" worked.
"Windows 10 networking 10" to "Windows 10 networking" fails ....

Ok, sorry for me ranting.

@mrpushner said in PFSense Release 2.5 + OpenVPN 2.5 broken? Any fixes?:

I did not determine the exact cause of this problem.

Two things are needed :
Clean restart of VPN server and the logs from that moment.
Clean start of the client and the logs from that moment.

Important : there are no 2.4.4 users on this forum. As 2.4.4 dates from .... dono, 2019 - 2018 ? I can't recall what the details were from that old OpenVPN version - and the 2.4.4 pfSense GUI - and the old pfSense quirks.

So, tests should be done with a "2.5.2" to "2.5.2".

Please, don't paste log text here in the forum. Use https://pastebin.com/ and paste the link here.

mrpushner

@gertjan Hi, yes, and my 2.5.2 to 2.5.2 would not work even though all the VPN settings looked normal and identical from the upgrade via the GUI.

A clean install of 2.5.2 with a restored 2.4.4 config worked.

So we indeed have here a case of:

"Windows 7 networking" to "Windows 10 networking" worked.
"Windows 10 networking" to "Windows 10 networking" failed.....until I started with a fresh install of Windows 10.

MP

N8LBV

@gertjan Please don't get stuck in a circle about the too old thing. We all know that 2.4.4 is no longer "supported", no need go on about that too much.
Thanks.

All of my systems that I am presenting here were updated to 2.5.0 and 2.5.2 a long time ago.
Fact is -some people are still on 2.4.5 including some of my sites due to the upgrading issues not being fully figured out yet.

Also I'm not really inclined to signup for another account on an external system just to post
log snippets here. This forum provides for easily posting inline log snippets or examples.
I did not really understand you "please don't post logs here" comment.
pastebin might be great for some people and if you like it, by all means suggest and promote it.

But I don't undertand your asking not to post logs here unless you are talking about
large pieces, pages & pages.
In that case what you are asking makes complete sense.
I try to keep any log postings here short and relevant unless otherwise a full longer log is needed. Thanks! :)

stephenw10

@n8lbv said in PFSense Release 2.5 + OpenVPN 2.5 broken? Any fixes?:

It is not working on any of my production systems that have both been upgraded from 2.4.5 to 2.5.2 and have two static public IP addresses configured.

This still sounds like what I initially suggested in this thread, this bug:
https://redmine.pfsense.org/issues/11545

How do you have the IPs configured if not as VIPs on WAN?

If it is like that then check the WAN interface status and make sure it's using the expected IP address. The symptoms of this are exactly what you're describing.

Steve

N8LBV

@stephenw10 Thanks - I will check into this. Does it only affect or apply to IPSEC?
I am currently only testing/troubleshooting with OPENVPN until I get that working first and then planned to circle back to IPSEC.
Mainly just to keep this less confusing and have more of a single point of focus to work on and troubleshoot, even though I know BOTH are not working on my systems ever since upgrading past 2.4.5 and now currently at 2.5.2 :)

Steve

N8LBV

@stephenw10 Holy Crap!
That might have effing worked!
I'll get back on this and let you know.
I remember trying this a long long time ago without any luck.
But that would have been combined with old configs carried over versus me trying with new
server certs and tunnels built from scratch after all of the old stuff was deleted.
:)

stephenw10

Yeah, when you hit that it affects both IPSec and OpenVPN if they are set to listen on 'WAN address'. Some users seem to hit this regularly but I have never managed to replicate it locally and, as far as I know, neither have any of the devs which makes it impossible to pin down.

A possible workaround is to use the VIP address for the VPN because that does not change. That's not suitable for everyone though.

Steve

N8LBV

@stephenw10 This is cool that I got it working now.
I have a number on non-critical "friend and family" sites where VPN has been IPOP since
upgrading past 2.4.5 around Feb. 2020
And I kept customer production systems that needed VPN functionality on 2.4.5

Not exactly sure what you mean by using the VIP because it does not change as the WAN address in any of my cases do not change either :) and have never changed.

But yes!

stephenw10

The bug here is that when you have VIPs defined on an interface the primary IP address should always be that defined in the interface config and VIP listed below that in the ifconfig output.
But under some unknown conditions ifconfig can start returning one of the VIP addresses at the too if the list. You can check it by running ifconfig manually or on the Status > Interfaces page where the IP address shown is what ifconfig returns.
That means that services using 'WAN address' can end up using a VIP address instead and clients are still using the correct address > failure!
If you set the VPN to use a VIP address instead of the main WAN it will always be that IP even if you hit this bug. But that means either changing the clients to use the VIP address or swapping the main interface and VIP IPs. That may not be practical in some setups.

Steve

N8LBV

@stephenw10 Cool!
I'm still not quite over the shock and thrill that it's working again :)
Did without it for quite a long time and poked at it here & there over that time.

jimp

@stephenw10 said in PFSense Release 2.5 + OpenVPN 2.5 broken? Any fixes?:

If you set the VPN to use a VIP address instead of the main WAN it will always be that IP even if you hit this bug. But that means either changing the clients to use the VIP address or swapping the main interface and VIP IPs. That may not be practical in some setups.

Or bind to localhost and use port forward in on WAN port to the instance on localhost.

Or bind to all interfaces/multihome and control access with rules.

Only real reason to bind to a specific interface would be on a client if it had to source from a specific address, but even that is of questionable use these days.

stephenw10

Yup that^. Works well for OpenVPN.

N8LBV

This will be a topic for another thread, I had the problem "come back"
After I had tried to create an additional site to site shared key openvpn server on a different port.
I was able to get the tunnel to come up and work between the two PFSense systems,
I could ping and reach hosts on both LANS local and remote. (from either PFSense box)
I setup the remote site as a client.
But I could not get any traffic from one LAN to the other (not being routed fully to the other LAN).
I checked and double checked all of my firewall settings and have the usual "allow all to pass" type rules in place on the openvpn interfaces on each pfsense system-
And the routing tables look good on both systems local and remote.
And I can reach both openvpn interface addresses from both LANs so some routing is occurring
part way though at least to the far end openvpn interface.
Anyhow that's not working as planned or expected and it's weird.
I also tried a packet capture on the openvpn interface at the far end while trying to ping a host on the far end LAN, and did not see any traffic here.
The packets originate of the local LAN and are destined to the far end LAN,
I have all of the usual and expected stuff in place, LAN to any rule on both ends etc.
Routing table firewall rules all look proper on both ends and looks like it should work.
reboots made no difference.
I deleted that attempt and was left with my main VPN server on 1194 not responding again until I re-did the save WAN interface settings trick.

stephenw10

That sounds like it must be a routing issue. Do you have policy based routing that might be overriding the routes added for OpenVPN?

N8LBV

@stephenw10 1- I don't think so. 2- I don't think so because I'm not familiar with policy based routing. I will attempt to get familiar.

N8LBV

@stephenw10 I did also do a packet capture on the WAN interface /gateway while trying to ping a host on the far end LAN just to confirm it was not taking the default route to the Internet and it was not seen there either.