PFSense Release 2.5 + OpenVPN 2.5 broken? Any fixes?
-
Looking for some help here.
I am asking a pretty simple question.
Is there a well known issue or a bug causing this?Or do we need to look at some logs dig in more here? (I asked this earlier)
What I am seeing on my end is that the windows client is timing out trying to reach UDP 1194
on the server.
And server is showing no signs of activity from the remote client trying to connect in the logs on port 1194.Firewall rules for port 1194 all look good like they should be allowing this as they always have.
In the web interface I tried setting Certificate Depth drop down to "Do Not Check".
This did not make any difference.A packet capture on the WAN does see the inbound packets from the remote client trying connect but there is no response from the server where the packet capture is being done.
Over & over again.I can post my client and server logs here and what I am stating above will be quite obvious.
Packet capture of UDP:1194
18:12:36.170797 IP C.C.C.C.58290 > S.S.S.S.1194: UDP, length 42No response from S.S.S.S (where the packet capture is running).
-
@n8lbv Doesn't look like any of the typical OpenVPN issues associated with 2.5.0 at the moment.
"server is showing no signs of activity" is not at all typical of the 2.5.0 OpenVPN issues that have been reported
You say the "Firewall rules for port 1194 all look good" does the byte counter on the Server side show any increase in allowed data if you force a try from the Client side?
If not, the Client is not trying the correct address for the Server or something else is getting in the way first.If you do see the byte count increasing, then attempts are being made and you might want to increase the OpenVPN logging level on the Server, Client or both to see what's happening with the connection.
-
@divsys Thanks for the feedback on this not being typical.
I really didnl;t want to spin my wheels on chasing this if there were already some well known issue out of the gate with 2.5
That I am somehow missing in my forum searches.
I did see enough people having problems that I thought maybe possibly there was something..
Like the certificate depth setting.. that may have broken it
for everyone globally. But I was not sure.I will do more testing soon as I have time.
So far what I have is that the inbound rule for UDP 1194 looks good.
Local packet capture on the PFSense box WAN sees the incoming packets for the connection,
But no response packets going back.
at the same time while testing that-
with default log levels and using the log view interface
on PFSense I do not see anything getting logged about the inbound requests or any connections. or any kind of failures etc.
Only thing I see i nthe log is the startup stuff when OpenVPN is
started up.This box that I have been doing most of my testing on was an upgrade to 2.5 from the latest 2.4.x release before it.
I have not yet done a full test round on a newly installed from scratch box.
Nor have I yet tried deleting the server instance on the upgraded box and recreating it from scratch in PFS 2.5
-
@n8lbv The other thing you might try is to leave the existing server instance as is and create a new one on a different port, but using the same WAN interface. Then you can update one or more of your clients to use the new port for testing.
-
Or run from the console :
sockstat -Lv | grep '1194'It should show the server process "openvpn" :
root openvpn 64342 7 udp4 192.168.10.3:1194 *:*running on the/a WAN interface, using the listening port '1194'.
( 192.168.10.3 is my WAN IP - I also NATted my upstream ISP router )If NAT works, then 'no answers' should probably mean : no one is listening.
Btw : The day I upgraded to pfSense 2.5.0, I checked if my Phone using the OpenVPN Connect App, could still connect to pfSense/OpenVPN (I knew a major VPN upgrade).
One of the main Highlights was :
OpenSSL upgraded.
OpenVPN upgraded to 2.5.0 :- OpenVPN 2.5.0 now mandates data cipher negotiation, but also tries to be friendly to older clients
- ChaCha20-Poly1305 is now supported, which is the same cipher used by WireGuard and may offer speed improvements on some platforms
- OpenVPN now disables compression by default because it is insecure, but it can still decompress traffic received from clients while not transmitting compressed packets
maybe I'm just lucky, as my iPhone always auto installs the latest version of the VPN Connect client : my Phone could connect right away.
@home, that evening, my OpenVPN 2.4.x Connect client could also connect just fine.
TodayI did not install a single patch into 2.5.0 CE.
I upgraded the client anyway. -
@gertjan I had similar success with some of my 2.5.0 OPenVPN converts.
For me, the issue seems to be some funkyness with the way 2.5.0 deals with certificates, sometimes. As seen in other threads the length or content of CN's in a certificate can cause a fail in 2.5.0 where it worked in 2.4.5.I've been able to work around the issue successfully (and not too painfully) so all's good for me.
I'm waiting on 2.5.1 before I convert the rest of my sites just to eliminate variables I don't need. -
@divsys Thanks for the reply.
Do you have any reference as to how you worked around it?
I already tried: In the web interface I setting Certificate Depth drop down to "Do Not Check"
as mentioned up above a few lines.Thanks!
-
@n8lbv My issues were all around the handshake that occurs between Server and Client during the connection. Using the logs I was able to narrow it down.
So far your issues are not clear as to whether the server is even running or the NAT rules are passing requests. gertjan makes a good point about checking if an instance of the OpenVPN server is even running to receive the packets forwarded by the firewall rule for 1194.
Until you can verify the individual pieces on the server side, it's hard to pint to what may be wrong.
Can you get to a command line to try some diagnostics? -
Most certainly I can!
I am a bit overwhelmed with fixing hacked exchange servers at the moment so I will dig in a little here as I can.
I'll take a deeper look. here soon and verify the openvpn server process is running etc.
It says it's there in the GUI but we all know how that is.All I had verified so far was that the request packets were coming into the WAN
and there I didn't see any kind of response packets going back out the WAN
and no log activity from the requests.I will take a look and see if the process is running/listening to UDP1194.
Don't really know my way around BSD networking but not biggie if you've seen one system you've seen them all far as I see things.
-
I have still not had time to dig into this any further..
However just a quick mention for now that all of my site to site IPSEC vpns are also broken since the upgrade to 5.0I had site to site IPSEC vpns set up to a number of friends and family locations that are still all running the latest PFsense 2.4.5
And those are also broke snice the update to 2.5
-
Quick followup-
Not had time to mess with it.
2.5 did not work out for me on any of the three systems that I ran the update on.
Both IPSEC and OpenVPN broken after update both no longer connect/work.
All 3 sites had a caching DNS and no longer work.
I can easily tell from all the posts that it is not just me or my setups.
For now- had to reinstall older version and restore from backup.Had to get the latest 2.4 from another users because somebody thinks it's some kind of good
idea to INSTANTLY make the previous WORKING version no longer available.
Complete and total arse headed move...
And who ever makes this decision can seriously stick it.
I'll come back to this and troubleshoot when I have time.Too many exchange servers to fix right now keeping me busy and great pay.
Maybe all of these broken issues will be fixed in 2.5.1
2.5 is obviously not release worthy and nothing was checked on it for upgradeability.
And I have the simplest setups to upgrade about as basic and featureless as it gets.2.4 is rock solid and has always worked well.
-
Hi,
I feel the same. First I was fighting openvpn after the upgrade. I had to reinstall and restore from backup to get openvpn working. And now I am experiencing an issue with unbound ignoring pfblockerng blocklists. The only solution to solve this seems reinstalling and restoring from backup. But I am asking myself seriously if I want te reinstall version 2.5 or just go back to 2.4.5 where everything was working fine. And indeed bad move from netgate in immediately removing 2.4.5 from the downloads site. I found the old version on some transit.NL mirror :( -
Very bad move and I just made a separate post about it.
How can that possibly be good or helpful in any way? -
Hi, any update to this? I just ran an update to ver. 2.5 and am seeing the same thing. OpenVPN stops working.
In my setup, I was in version 2.4.4 for multiple sites. I upgraded the site that has the VPN SERVERS setup, and the remote VPN's (still on 2.4.4) continued to work normally, even after re-boots.
As soon as I upgraded one of the REMOTE sites to 2.5.2, they stop working! Configs exactly the same. I did FRESH installs of 2.5.2, then simply had the new install grab my backup config file off of USB.
So this must have something to do with the CLIENT side of OPENVPN if the SERVER side updates to 2.5 without issue.
I will also note that the new install in the remote location would not allow any access to the internet either, so its like DNS or something is broken as well. ???
Any update or ideas on any fix would be appreciated.
Thanks,
MP
-
As asked about above make sure the incoming traffic is actually opening states. Check the packet counters on the WAN firewall rule(s). Check the state table.
One thing that can present as seen here is this: https://redmine.pfsense.org/issues/11545
If you have VIPs on WAN go to Status > Interfaces and make sure the WAN is actually using the correct IP. If it isn't go to Interfaces > WAN and resave it.If your OpenVPN server is set to listen on WAN address that bug will break it if you hit it.
Steve
-
@mrpushner Hi MP:
No updates on this yet.
This is on my list for beginning of the year projects.
I have a number of small business PFS 2.4.5 VPN servers deployed.
Some are IPSEC site-to-site.
And others are OpenVPN servers with mobile client users.
All are very simple setups, what I would call "nearly textbook".
Nothing special and nearly default wizard type setups without anything special or unique
other than the specific public IPs involved and pre-shared keys.
I have determined that none of them can be upgraded in place as per my previous posts.
I've also done some troubleshooting and really not made any progress yet.
I've checked and double checked the firewall rules performed packet captures, looked at logs-
I see packets coming in but do not see the vpn servers responding to them per the logs.
I've tried completely deleting all VPN configs (from the GUI) rebooting and re-creating them and recreating others for test, none of which worked.
I've tried manually creating the IPSEC firewall inbound allow rules even though the documentation states these rules are created automatically and are hidden when you
Create an IPSEC config from the GUI because I have read in some posts that people stated they needed to do this.The only things (my next steps) I have not yet tried are
- Posting detailed packet captures and logs here one step at a time and trying to get help.
- Installing TWO clean 2.5.2 systems and trying to bring up a VPN connection between them.
I have a number of non-critical "friend and family" PFSense systems that I have upgraded to
2.5.2 and all of them have lost VPN functionality.
I would really like to both get those back working again and to get all of my business customers up to date
that are on 2.4.5
I also have a new customer that is waiting on me to deploy a VPN server for them.
They are currently on 2.4.5
And I don't want to do this new VPN on 2.4.5 knowing they need to upgrade soon
and that it will break when they do.
Fresh installing 2.5.2 and having to manually build out all of the stuff that has been done on their existing setup would be a lot of work.
So I will indeed be digging in and figuring this out soon.Let's keep in touch on this one MP!
Steve
-
None of them with multiple IPs on WAN? VIPs?
The only other thing that occasionally causes a problem is that with pfSense 2.5 came (confusingly!) an update to OpenVPN 2.5. That has options that are not compatible with earlier clients but it is backwards compatible when upgraded. If you have 2.4 clients connecting though and are exporting new configs be sure to check:
Legacy Client: Do not include OpenVPN 2.5 settings in the client configuration.Steve
-
@n8lbv N8, I never knew this was quite the issue that it really is as I'm finding more thread on this problem.
And I will let you know here if I find out anything new.
Just so you know how mine are currently setup.
I have both PEER and Remote VPN'S setup. I have 3 different physical locations with the Servers all setup in one location, then 2 peer clients, and two remote clients.
I upgraded the server location to 2.5.2. ALL VPN's still worked.
If I update a Peer client, they fail. So VPN SERVER at 2.5.2 and 2.4.4 Peer Clients works, 2.5.2 Server to 2.5.2 clients peer VPN's fail.
I have quickly perused the logs but get impatient as I don't even know what I'm looking for, lol.
I did not try all you did yet. My next step is to remove a peer client that is not working, and recreate it fresh, apply the key, and see what happens.
If that does not work I will fresh Server and Client on each and see if that does anything. Based on what you just said you did, I will not hold my breath that its going to work however.
SIDE NOTE: How do you guys do dev testing of multiple PF boxes from the same Lan/Wan so as to be able to TEST the VPN's without having to have a different actual WAN setup? Is that even possible? (I currently take them home to test)
MP
-
@stephenw10 Hi, yes I did see this in another thread, but then my VPN's setup as REMOTES are still working ok. Its only my PEER VPN's that have failed with the upgrades.
No, I'm not using multiple WAN or Virtual IP's in my setups.
I will note that my PF boxes that are failing ALSO are losing all internet access as well. Can't even ping 8.8.8.8 from the PF box.
That may be another issue and I'm going to double check my DNS and Gateway routing to see if something is a miss there.
Will let you know what I may find.
Thanks,
MP
-
@stephenw10 Thanks, I will check into this first.
