PFSense Release 2.5 + OpenVPN 2.5 broken? Any fixes?

N8LBV

@mrpushner Not blocking the packets, and I am logging the rule and seeing the pass logs.
If you scroll up just a little bit I showed one example.
:)

N8LBV

@silence It's not too much, I intend to figure it out, learn something and then share back to this thread what I learned and what fixed it as this may also help anyone else that runs into this.

A Former User

@n8lbv ok so if it's not too much for you i can help you here, what i would recommend is to do a backup and then a fresh install of pfsense the latest stable version and try to upload your clean config please...this should work for you simple and fast in 15 minutes you have your vpn working without problem.

N8LBV

@silence I will try this.
I think I already did try this back when I was troubleshooting 2.5.0
But it's been so long now I am not sure so yes I need to retry that.
Thanks. :)

N8LBV

@n8lbv I think you mean to try to restore my backup after a fresh install.. I will try that soon and report back with results.

A Former User

@n8lbv, I have different sites, some with 2.4, 2.5 and 2.5.2, the version doesn't matter, it always works the first time...!

just try it without configuring anything else extra.

Gertjan

@silence said in PFSense Release 2.5 + OpenVPN 2.5 broken? Any fixes?:

some with 2.4, 2.5 and 2.5.2

And these 2.4.5 and 2.5.2 are "OpenVPN" "site-to-site" interconnected ?
I guess it is possible, but "VPN" settings on both sides have to be fine tuned.

pfSense 2.4.5px uses the OpenVPN version 2.4.8 (or 9).
pfSense 2.5.2 uses :

OpenVPN 2.5.2 amd64-portbld-freebsd12.2 [SSL (OpenSSL)] [LZO] [LZ4] [MH/RECVDA] [AEAD] built on Jun 24 2021
....

That's 2.5.2; pure coincidence that pfSense and OpenVPN have the same version number.
Btw : the latest OpenVPN, today, is 2.5.5

Between OpenVPN 2.4.x and OpenVPN 2.5.2, there were enough changes to make things break: RTFCL here : See "Overview of changes since OpenVPN 2.4".

As soon as you see a "soon to be depreciated'" in the **OpenVPN log today, you should redo the setup asap so that "depreciated option" message are not shown any-more.
The thing is : this is nearly never done. "As it works right now - I don't touch it" and "I'm not looking at the logs now and won't start doing so tomorrow ;)".

When the the big upgrade is presented, like pfSense from 2.4.5 to 2.5.x, this includes OpenVPN 2.4.x to 2.5.0, a major upgrade version shift - the connection breaks.

Btw : pfSense OpenVPN site to site - client to VPN like "Exp*ssVPN" or OpenVPN as a remote pfSense admin access works for for me
I rarely use the site to site, less often the client OpenVPN, but most often the "have to work from home" option.

mrpushner

@gertjan Hi, I had 3 total sites all on 2.4.4. I updated (via GUI) my Server site to 2.5.2. SITE TO SITE open VPN's continued to work fine. So two client sites on 2.4.4 to Server site on 2.5.2.

As soon I updated one of my client sites to 2.5.2, VPN crashed.

I had to go to the site and restore back to 2.4.4 to get it to work again.

Since then, I did a fresh install of 2.5.2 on that client site (on my spare PFS hardware), and applied my 2.4.4 backup config, and it came back online normally.

I did not determine the exact cause of this problem.

MP

N8LBV

@mrpushner This seem to be the state I am in and what has happened.

I no longer have backup configs from 2.4.5 or 2.4.4 so I do not have the option to
restore from those on my sites that I have updated to 2.5 about 1 year ago.
But it may be an option for me on all of the sites that are still on 2.4.5 if that proves to be a way of fixing them.

Right now I have a couple of sites that have been upgraded around a year ago that are now on 2.5.2 and would like to add VPN but cannot because of this problem.

And doing a fresh install and manually restoring all of the settings would be a lot of work :)
The reality is that I may have to do that for this issue.

I can also try restoring from a current backup after a fresh install.
To see if the problem follows the config or not.
I am NOT backing up ANY VPN settings configs or certificates.
In all cases the VPN setup is brand new.
And is still broken on any system I try to bring up that has not been a fresh install.

Gertjan

@mrpushner said in PFSense Release 2.5 + OpenVPN 2.5 broken? Any fixes?:

Hi, I had 3 total sites all on 2.4.4. I updated (via GUI) my Server site to 2.5.2. SITE TO SITE open VPN's continued to work fine. So two client sites on 2.4.4 to Server site on 2.5.2.
As soon I updated one of my client sites to 2.5.2, VPN crashed.

Oh ... great.
My "common sense" would say :
if 2.4.4 (really ?? 2.4.4 ??? 2.4.4 is way to old) to 2.4.4 works.
and
2.4.4 to 2.5.2 works
then
2.5.2 to 2.5.2 should also work.

A bit like
"Windows 7 networking" to "Windows 7 networking" worked.
"Windows 7 networking" to "Windows 10 networking" worked.
"Windows 10 networking 10" to "Windows 10 networking" fails ....

Ok, sorry for me ranting.

@mrpushner said in PFSense Release 2.5 + OpenVPN 2.5 broken? Any fixes?:

I did not determine the exact cause of this problem.

Two things are needed :
Clean restart of VPN server and the logs from that moment.
Clean start of the client and the logs from that moment.

Important : there are no 2.4.4 users on this forum. As 2.4.4 dates from .... dono, 2019 - 2018 ? I can't recall what the details were from that old OpenVPN version - and the 2.4.4 pfSense GUI - and the old pfSense quirks.

So, tests should be done with a "2.5.2" to "2.5.2".

Please, don't paste log text here in the forum. Use https://pastebin.com/ and paste the link here.

mrpushner

@gertjan Hi, yes, and my 2.5.2 to 2.5.2 would not work even though all the VPN settings looked normal and identical from the upgrade via the GUI.

A clean install of 2.5.2 with a restored 2.4.4 config worked.

So we indeed have here a case of:

"Windows 7 networking" to "Windows 10 networking" worked.
"Windows 10 networking" to "Windows 10 networking" failed.....until I started with a fresh install of Windows 10.

MP

N8LBV

@gertjan Please don't get stuck in a circle about the too old thing. We all know that 2.4.4 is no longer "supported", no need go on about that too much.
Thanks.

All of my systems that I am presenting here were updated to 2.5.0 and 2.5.2 a long time ago.
Fact is -some people are still on 2.4.5 including some of my sites due to the upgrading issues not being fully figured out yet.

Also I'm not really inclined to signup for another account on an external system just to post
log snippets here. This forum provides for easily posting inline log snippets or examples.
I did not really understand you "please don't post logs here" comment.
pastebin might be great for some people and if you like it, by all means suggest and promote it.

But I don't undertand your asking not to post logs here unless you are talking about
large pieces, pages & pages.
In that case what you are asking makes complete sense.
I try to keep any log postings here short and relevant unless otherwise a full longer log is needed. Thanks! :)

stephenw10

@n8lbv said in PFSense Release 2.5 + OpenVPN 2.5 broken? Any fixes?:

It is not working on any of my production systems that have both been upgraded from 2.4.5 to 2.5.2 and have two static public IP addresses configured.

This still sounds like what I initially suggested in this thread, this bug:
https://redmine.pfsense.org/issues/11545

How do you have the IPs configured if not as VIPs on WAN?

If it is like that then check the WAN interface status and make sure it's using the expected IP address. The symptoms of this are exactly what you're describing.

Steve

N8LBV

@stephenw10 Thanks - I will check into this. Does it only affect or apply to IPSEC?
I am currently only testing/troubleshooting with OPENVPN until I get that working first and then planned to circle back to IPSEC.
Mainly just to keep this less confusing and have more of a single point of focus to work on and troubleshoot, even though I know BOTH are not working on my systems ever since upgrading past 2.4.5 and now currently at 2.5.2 :)

Steve

N8LBV

@stephenw10 Holy Crap!
That might have effing worked!
I'll get back on this and let you know.
I remember trying this a long long time ago without any luck.
But that would have been combined with old configs carried over versus me trying with new
server certs and tunnels built from scratch after all of the old stuff was deleted.
:)

stephenw10

Yeah, when you hit that it affects both IPSec and OpenVPN if they are set to listen on 'WAN address'. Some users seem to hit this regularly but I have never managed to replicate it locally and, as far as I know, neither have any of the devs which makes it impossible to pin down.

A possible workaround is to use the VIP address for the VPN because that does not change. That's not suitable for everyone though.

Steve

N8LBV

@stephenw10 This is cool that I got it working now.
I have a number on non-critical "friend and family" sites where VPN has been IPOP since
upgrading past 2.4.5 around Feb. 2020
And I kept customer production systems that needed VPN functionality on 2.4.5

Not exactly sure what you mean by using the VIP because it does not change as the WAN address in any of my cases do not change either :) and have never changed.

But yes!

stephenw10

The bug here is that when you have VIPs defined on an interface the primary IP address should always be that defined in the interface config and VIP listed below that in the ifconfig output.
But under some unknown conditions ifconfig can start returning one of the VIP addresses at the too if the list. You can check it by running ifconfig manually or on the Status > Interfaces page where the IP address shown is what ifconfig returns.
That means that services using 'WAN address' can end up using a VIP address instead and clients are still using the correct address > failure!
If you set the VPN to use a VIP address instead of the main WAN it will always be that IP even if you hit this bug. But that means either changing the clients to use the VIP address or swapping the main interface and VIP IPs. That may not be practical in some setups.

Steve

N8LBV

@stephenw10 Cool!
I'm still not quite over the shock and thrill that it's working again :)
Did without it for quite a long time and poked at it here & there over that time.

jimp

@stephenw10 said in PFSense Release 2.5 + OpenVPN 2.5 broken? Any fixes?:

If you set the VPN to use a VIP address instead of the main WAN it will always be that IP even if you hit this bug. But that means either changing the clients to use the VIP address or swapping the main interface and VIP IPs. That may not be practical in some setups.

Or bind to localhost and use port forward in on WAN port to the instance on localhost.

Or bind to all interfaces/multihome and control access with rules.

Only real reason to bind to a specific interface would be on a client if it had to source from a specific address, but even that is of questionable use these days.