PFSense Release 2.5 + OpenVPN 2.5 broken? Any fixes?

[stephenw10]

Hmm, odd. Does it look like it's trying to connect before resaving?

[N8LBV]

@stephenw10 Yes, just from memory something like "ping timeout retrying"..
But I'm not exactly sure.
Will have to see if I can get it to fail again.

[N8LBV]

@n8lbv I'm able to re-create to problem by restarting the server.
If I restart only the client it comes back and connects.
If I restart the server the connection never comes back.
On the client side it says: " reconnecting; ping-restart" under status.

Local address and remote host both have "pending".

Saving and applying WAN on the client this time is not fixing it.

Also just tried reapplying wan settings on the server.
And it's not working.
I now have a broken connection that is not working and not sure how to fix since rebooting the server end.

[N8LBV]

@n8lbv Some kind of unknown combination of restarting the client and server services
and reapplying the WAN settings finally got it to come back after about ten minutes of trying all four things over & over again.

I'll have to work at it and try to narrow it down to something more definitive.
For now - restarting the Pfsense box with the openvpn server on it initiates the problem,
The tunnel does not come back and remains broken until I start fiddling with it as stated above.

This box does have TWO static public IPs and two VPN servers, one on port 1194 and another on 1195.

And it's the client connection using 1195 that is failing.

Probably a tad more complicated of a setup than we'd prefer to troubleshoot such a problem.

[N8LBV]

@n8lbv Once the connection is up it works great and is solid and seems to run fine for days.
It gets unhappy when I reboot the server.
And stays unhappy.

I'll dig in more over the next few days and try to narrow this down a bit more than what I have presented now which is a bit of a mess.

[stephenw10]

Ok, interesting.
So is the server on port 1195 running on the WAN IP dirrectly or a VIP on the WAN?
Or do you mean it has two WANs?

The first thing to check here is that the WAN is not coming up at boot on the server end with the wrong IP as the primary address since we know that can happen and it will cause this.

Steve

[N8LBV]

@stephenw10 Directly on the WAN.
And sorry what I meant was two static IPs that are accessible via the WAN.
I am not using the VIP for VPN services.

Also this is getting messy as my main VPN server on port 1194 is back to not working at all since that reboot yesterday.
And reapplying the WAN settings is not fixing it this time.

I will need some time to get a handle on this.
It's broken in multiple places right now.

[N8LBV]

--- Restarted the service for the 1194 server (main) and it works now.
Plenty of weirdness going on here.
Too many variables right now to be useful I realize.
Will try to narrow it down to something simpler than the problem storm it is now.

[stephenw10]

Cool, thanks for gathering data.
It still feels like it at least could still be the incorrect primary IP issue.
Though I note some others who were hitting that frequently are no longer in 2.6.

Steve

[N8LBV]

@stephenw10 Minor tiny update.
Worked for several weeks then broke.
Connection is not maintaining itself.
At the far end client (PFsense 2.6) same one we have been testing. in this thread-
Status was stopped and local address pending...
Clicked the start button (in the client ipsec status page) and it came back up and is working.
This is in no way configured to stop or timeout like this.
It should always reconnect on its own if it were ever to lose connection for any reason.
Or if the far end client has a power loss or restart.

Serverside is battery backed has continuous uptime, not been restarted since my last email
concerning this project.

[N8LBV]

@n8lbv ```
ar 2 04:08:39 openvpn 33449 Peer Connection Initiated with [AF_INET]S.S.S.S:1195
Mar 2 04:08:40 openvpn 33449 Initialization Sequence Completed
Mar 17 08:40:05 openvpn 33449 write UDPv4: No route to host (code=65)
Mar 17 08:40:15 openvpn 33449 write UDPv4: No route to host (code=65)
Mar 17 08:40:25 openvpn 33449 write UDPv4: No route to host (code=65)
Mar 17 08:40:35 openvpn 33449 write UDPv4: No route to host (code=65)
Mar 17 08:40:46 openvpn 33449 write UDPv4: No route to host (code=65)
Mar 17 08:40:49 openvpn 33449 Inactivity timeout (--ping-restart), restarting
Mar 17 08:40:49 openvpn 33449 SIGUSR1[soft,ping-restart] received, process restarting
Mar 17 08:40:54 openvpn 33449 NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
Mar 17 08:40:54 openvpn 33449 Re-using pre-shared static key
Mar 17 08:40:54 openvpn 33449 Preserving previous TUN/TAP instance: ovpnc1
Mar 17 08:40:54 openvpn 33449 TCP/UDP: Preserving recently used remote address: [AF_INET]P.P.P.P:1195
Mar 17 08:40:54 openvpn 33449 TCP/UDP: Socket bind failed on local address [AF_INET]C.C.C.C:0: Can't assign requested address (errno=49)
Mar 17 08:40:54 openvpn 33449 Exiting due to fatal error
Mar 17 08:40:54 openvpn 33449 /usr/local/sbin/ovpn-linkdown ovpnc1 1500 1572 44.11.11.2 44.11.11.1 init

[Gertjan]

@n8lbv
Log from the client side, right ?
"Socket not found" and "Socket bind failed on local address" : The interface used is down or no IP ?

[stephenw10]

@n8lbv said in PFSense Release 2.5 + OpenVPN 2.5 broken? Any fixes?:

Clicked the start button (in the client ipsec status page) and it came back up and is working.

Did you mean IPSec here?

[N8LBV]

@stephenw10 No sorry only testing openvpn at this point.
Must have been really tired when I posted that.
Also it's been up & working 18 days since that post.
We're both on battery backup.
It just that when it gets shutdown or rebooted that thing go wonky again
and I have to mess with it to get it to work.

[N8LBV]

This continues to be an issue.
Had to restart the OPenVPN physical server and lose the vpn client connection.
For this reason I have been unable to use any openVPN site to site persistant tunnels.

I'm guessing anyone doing this has to use IPSEC for this purpose.
The tunnel stayed up since my last post here back in March.
But the moment that either end of the tunnel gets rebooted or loses power the tunnel is gone and never comes back on its own ever.
Took me a half-hour of messing with it restarting services, re-saving the WAN connection settings to get it to work again.

Same as before.

I will need to look around and research if anyone else has reported or fixed this or related issues since March of 2022.

-Steve

[stephenw10]

I have several OpenVPN site to site tunnels and have done in all pfSense versions without any significant issues.
Currently I'm running 22.05 there and will shortly be moving to 23.01 snapshots for testing.

PSK will be going away in OpenVPN at some point so if you're using that for S2S tunnels it's worth switching to ssl/tls now. Though that shouldn't be causing any problems currently.

Steve

[N8LBV]

@stephenw10 Thanks for the reply..
When you say "all versions" not exactly sure what that means or if it matters too much but
the more data points the better for sure.

The problem might have something to do with the fact I am running multiple static IP (two in use at the moment) on the
router public Internet facing versus using only one IP address.
And I guess I may be in a minority here being the combination of this and running an openvpn
server.

This is also my first attempt to ever use OpenVPN for site to site..
IPSEC works find always has and I think most people are using IPSEC for S2S even with PFSense.

This time around I wanted to be "different" and see if I could use OPENVPN instead for what I have
always done in the past with IPSEC on PFSense.

I've not tried using this type of configuration in production with any clients yet.
For now it's just a single test connection going to a friend's house.

I will need to do some work if I want to troubleshoot it further and work on a fix.
narrow it down to if a single IP address fixes it and such.

And simplify the arrangement as much as possible.
Right now I'm running two VPN serves on two different ports.
I don't think that's part of the problem but I'll need to re-test that.
I seem to remember trying that to rule it out but didn't document it unless I documented it here
on this thread which I'll have to have a full lookback before I go back to work on it.

The connection had been up for months and I had to reboot the router at the friends house
and had to visit this all over again as it takes some significant fussing about it restarting services
and re-saving the settings on the WAN page to get the connection to work again.

I need to better track that process and determine what exactly does get the connection to work again when it finally does...
As of now it's just random restarting things and re-saving the WAN page on BOTH ends until it works.

Which is no position to actually start troubleshooting this from.
I really need to read my previous nodes (above) to remind myself of what I had determined already on this. :)

Thanks for the heads-up on PSK going away. (I've not been keeping up lately).
I will start to get familiar with that now that you mentioned it.

I am super happy that AES-NI did not end up being a requirement as it was going to be.
SO many of my implementations do not need it and so many of my customers are on
Little J1900 no moving parts firewalls that do a great job for their needs.

[stephenw10]

Yeah, I'd have to re-read it all too.

But I would expect that to work without issues. I have run s2s OpenVPN tunnels in each version as we've released them and I've not seen an issue with tunnels failing that couldn't be fixed with a config change.
So I guess I'd say let's try to see exactly what's happening now and what's logged when it happens.

Steve

[N8LBV]

Probably not the most elegant solution but I just switched the family and friends VPN server to 2.7
This appears to have fixed it.
I actually just moved the edge router (PFSense) from ESXI6 over to a Hyper-V server.
I ran into all sorts of issues with speed to-from PFSense in Hyper-V and Hyper-V
and other VMs on the server.
Was getting 2mbps upload and around 3mbps download between servers.
and to the gigabit lan and a 2.5gb lan.
Read the huge PFS/FreeBSD/RSC Reddit thread for an hour or so.

Turning off RSC globally fixed the issue to the wired networks, but still had slow
transfers from PFsense to the other VMs.
Threw in the towel and switched to 2.7 and everything appears to be great.

This also appears to have fixed the ongoing issue I've had with OPENVPN tunnel server connections not being able to re-establish themselves (after a reboot/restart) without requiring manually going in and re-saving the WAN settings page as well as restarting the two OpenVPN servers I have on my PFsense system.

[stephenw10]

Yeah, that Hyper-V RSC issue in 2.6 was painful. But as you said is fixed in 2.7, RSC support in the hn driver is disabled by default.
Good news on the OpenVPN server there too. Would have been nice to know exactly what the cause was. There have been a number of fixes gone in for boot issues though. Many of them are only ever hit on some systems because of timing issues. I could imagine this being one of them.

Steve