pfSense 2.4.5 with OpenVPN: connection issue on turning hotspot on phone on/off

brma

I'm using OpenVPN-server on a pfSense 2.4.5 and trying to connect from a Windows 10 client running OpenVPN client 2.4.8 over WLAN to a hotspot provided by an Android-phone running version 9 (Android Pie).

The server is configured for remote access (SSL/TSL + user authentication) listening on UDP 1194. A firewall-rule on the WAN-interface to allow UDP/1194 is created and working.

The issue is as follows:
When I connect via OpenVPN-client the first time, it works. and also keeps working if I do only re-connect the OpenVPN-Client without turning the hotspot on the Android phone off and on again.
As soon as I turn off the hotspot on the Android-phone and turn it on again, re-connecting the WLAN of the laptop to the hotspot and trying to re-establish the connection to the OpenVPN-server I'm receiving the following log-entries on the server (the IP-adress is replaced by xxx.xxx.xxx.xxx):

Apr 7 13:01:25 	openvpn 	7851 	xxx.xxx.xxx.xxx:36524 TLS Error: TLS handshake failed
Apr 7 13:01:25 	openvpn 	7851 	xxx.xxx.xxx.xxx:36524 TLS Error: TLS key negotiation failed to occur within 60 seconds (check your network connectivity)

The client log shows:

Tue Apr 07 13:35:09 2020 OpenVPN 2.4.8 x86_64-w64-mingw32 [SSL (OpenSSL)] [LZO] [LZ4] [PKCS11] [AEAD] built on Oct 31 2019
Tue Apr 07 13:35:09 2020 Windows version 6.2 (Windows 8 or greater) 64bit
Tue Apr 07 13:35:09 2020 library versions: OpenSSL 1.1.0l  10 Sep 2019, LZO 2.10
Enter Management Password:
Tue Apr 07 13:35:28 2020 TCP/UDP: Preserving recently used remote address: [AF_INET]xxx.xxx.xxx.xxx:1194
Tue Apr 07 13:35:28 2020 UDP link local (bound): [AF_INET][undef]:1194
Tue Apr 07 13:35:28 2020 UDP link remote: [AF_INET]xxx.xxx.xxx.xxx::1194

and then retries connecting to the server having the same error.
Some remarks:

Looking in the pfSense at Status/OpenVPN shows the connection with the proper client IP-address but having "UNDEF" as CN
if I try exactly the same one day later, the same happens again: connects are successful as long as I do not turn off the hotspot. If I turn off the hotspot and on again, I need to wait about 24h to be able to re-connect again
if I'm using the same client with the same hotspot connecting to another OpenVPN server with a compareable server-configuration (just different servername/certificate/username/password) - and of course adapted client-configuration for the other server - it works fine. Even if the hotspot is turned off and on again.
Neither rebooting the windows-laptop nore the Android cell-phone shows any effect
Firewall logs do not show any blocks for the client-adress and also allowing all traffic does not show any different behaviour
OpenVPN is allowed in the client-firewall (F-Secure and Windows-Firewall)
All other IP-traffic over the hotspot works without any issues, also after turning the hotspot off and on and re-connecting the WLAN-connection

Any ideas?

Any help would be highly appreciated!

Pippin

@brma said in pfSense 2.4.5 with OpenVPN: connection issue on turning hotspot on phone on/off:

As soon as I turn off the hotspot on the Android-phone and turn it on again, re-connecting the WLAN of the laptop to the hotspot

Switching the hotspot on/off will "mess up" the routing table on your Windows.
Try remove/comment:

persist-tun

from/in the client config.

brma

@Pippin Thank you for the hint but this does not do the trick. A reboot did neither, what would have solved the issue with the routing table as well....
One additional remark: if I'm establishing the connection directly with the OpenVPN on the Android cell phone it works like a charme... so there must be something wrong with the hotspot and this special connection as the other VPN connects also after the hotspot has been turned off/on
It's completely weird...

Pippin

Compare the windows client routing table

normal connection to hotspot
after starting the vpn
after disabling hotspot
after reenabling hotspot

Gertjan

@brma said in pfSense 2.4.5 with OpenVPN: connection issue on turning hotspot on phone on/off:

WAN-interface to allow UDP/1194 is created and working.

In that case, it won't take long to run a packet capture on port 1194 - WAN (UDP or TCP ??) Interface.
Did it show you packets are coming in == solid proof that the connection actually made it into to pfSense ?

brma

@Pippin First - thanks for supporting me on this. I'll post the results below - to me everything looks right, but I'm not an expert on this. Could you please take a look into it, if there is a wrong routing after the restart of the hotspot?

Normal connection to the hotspot - the laptop receives IP-adress 192.168.43.107 from the hotspot - please have a look at the gateway because this is going to change with a re-start of the hotspot later on - the IP-adress will stay the same:
ipconfig:

   Drahtlos-LAN-Adapter WLAN:

   Verbindungsspezifisches DNS-Suffix:
   Verbindungslokale IPv6-Adresse  . : fe80::b161:382d:e905:6665%18
   IPv4-Adresse  . . . . . . . . . . : 192.168.43.107
   Subnetzmaske  . . . . . . . . . . : 255.255.255.0
   Standardgateway . . . . . . . . . : 192.168.43.238

route print:

IPv4-Routentabelle
===========================================================================
Aktive Routen:
     Netzwerkziel    Netzwerkmaske          Gateway    Schnittstelle Metrik
          0.0.0.0          0.0.0.0   192.168.43.238   192.168.43.107     35
        127.0.0.0        255.0.0.0   Auf Verbindung         127.0.0.1    331
        127.0.0.1  255.255.255.255   Auf Verbindung         127.0.0.1    331
  127.255.255.255  255.255.255.255   Auf Verbindung         127.0.0.1    331
     192.168.43.0    255.255.255.0   Auf Verbindung    192.168.43.107    291
   192.168.43.107  255.255.255.255   Auf Verbindung    192.168.43.107    291
   192.168.43.255  255.255.255.255   Auf Verbindung    192.168.43.107    291
        224.0.0.0        240.0.0.0   Auf Verbindung         127.0.0.1    331
        224.0.0.0        240.0.0.0   Auf Verbindung    192.168.43.107    291
  255.255.255.255  255.255.255.255   Auf Verbindung         127.0.0.1    331
  255.255.255.255  255.255.255.255   Auf Verbindung    192.168.43.107    291
===========================================================================
Ständige Routen:
  Keine

After the successful connect to the OpenVPN-Server, it looks like this:
ipconfig:

Unbekannter Adapter LAN-Verbindung:

   Verbindungsspezifisches DNS-Suffix: brma.loc
   Verbindungslokale IPv6-Adresse  . : fe80::3c49:7c0d:8a9a:e94c%9
   IPv4-Adresse  . . . . . . . . . . : 10.75.0.2
   Subnetzmaske  . . . . . . . . . . : 255.255.255.0
   Standardgateway . . . . . . . . . :

Drahtlos-LAN-Adapter WLAN:

   Verbindungsspezifisches DNS-Suffix:
   Verbindungslokale IPv6-Adresse  . : fe80::b161:382d:e905:6665%18
   IPv4-Adresse  . . . . . . . . . . : 192.168.43.107
   Subnetzmaske  . . . . . . . . . . : 255.255.255.0
   Standardgateway . . . . . . . . . : 192.168.43.238

route print:

IPv4-Routentabelle
===========================================================================
Aktive Routen:
     Netzwerkziel    Netzwerkmaske          Gateway    Schnittstelle Metrik
          0.0.0.0          0.0.0.0   192.168.43.238   192.168.43.107     35
        10.75.0.0    255.255.255.0   Auf Verbindung         10.75.0.2    281
        10.75.0.2  255.255.255.255   Auf Verbindung         10.75.0.2    281
      10.75.0.255  255.255.255.255   Auf Verbindung         10.75.0.2    281
        127.0.0.0        255.0.0.0   Auf Verbindung         127.0.0.1    331
        127.0.0.1  255.255.255.255   Auf Verbindung         127.0.0.1    331
  127.255.255.255  255.255.255.255   Auf Verbindung         127.0.0.1    331
     192.168.43.0    255.255.255.0   Auf Verbindung    192.168.43.107    291
   192.168.43.107  255.255.255.255   Auf Verbindung    192.168.43.107    291
   192.168.43.255  255.255.255.255   Auf Verbindung    192.168.43.107    291
     192.168.75.0    255.255.255.0        10.75.0.1        10.75.0.2    281
        224.0.0.0        240.0.0.0   Auf Verbindung         127.0.0.1    331
        224.0.0.0        240.0.0.0   Auf Verbindung         10.75.0.2    281
        224.0.0.0        240.0.0.0   Auf Verbindung    192.168.43.107    291
  255.255.255.255  255.255.255.255   Auf Verbindung         127.0.0.1    331
  255.255.255.255  255.255.255.255   Auf Verbindung         10.75.0.2    281
  255.255.255.255  255.255.255.255   Auf Verbindung    192.168.43.107    291
===========================================================================
Ständige Routen:
  Keine

After disconnecting windows from the hotspot and turning off the hotspot, it looks like this:
route print:

IPv4-Routentabelle
===========================================================================
Aktive Routen:
     Netzwerkziel    Netzwerkmaske          Gateway    Schnittstelle Metrik
        127.0.0.0        255.0.0.0   Auf Verbindung         127.0.0.1    331
        127.0.0.1  255.255.255.255   Auf Verbindung         127.0.0.1    331
  127.255.255.255  255.255.255.255   Auf Verbindung         127.0.0.1    331
        224.0.0.0        240.0.0.0   Auf Verbindung         127.0.0.1    331
  255.255.255.255  255.255.255.255   Auf Verbindung         127.0.0.1    331
===========================================================================
Ständige Routen:
  Keine

Connecting to the hotspot again, shows the following picture (please note the changed standard-gateway while the IP-address stays the same!):
ipconfig:

Drahtlos-LAN-Adapter WLAN:

   Verbindungsspezifisches DNS-Suffix:
   Verbindungslokale IPv6-Adresse  . : fe80::b161:382d:e905:6665%18
   IPv4-Adresse  . . . . . . . . . . : 192.168.43.107
   Subnetzmaske  . . . . . . . . . . : 255.255.255.0
   Standardgateway . . . . . . . . . : 192.168.43.11

However, the routes seem to reflect the change as well, but the connect to the OpenVPN-server does not work anymore:
route print:

IPv4-Routentabelle
===========================================================================
Aktive Routen:
     Netzwerkziel    Netzwerkmaske          Gateway    Schnittstelle Metrik
          0.0.0.0          0.0.0.0    192.168.43.11   192.168.43.107     40
        127.0.0.0        255.0.0.0   Auf Verbindung         127.0.0.1    331
        127.0.0.1  255.255.255.255   Auf Verbindung         127.0.0.1    331
  127.255.255.255  255.255.255.255   Auf Verbindung         127.0.0.1    331
     192.168.43.0    255.255.255.0   Auf Verbindung    192.168.43.107    296
   192.168.43.107  255.255.255.255   Auf Verbindung    192.168.43.107    296
   192.168.43.255  255.255.255.255   Auf Verbindung    192.168.43.107    296
        224.0.0.0        240.0.0.0   Auf Verbindung         127.0.0.1    331
        224.0.0.0        240.0.0.0   Auf Verbindung    192.168.43.107    296
  255.255.255.255  255.255.255.255   Auf Verbindung         127.0.0.1    331
  255.255.255.255  255.255.255.255   Auf Verbindung    192.168.43.107    296
===========================================================================
Ständige Routen:
  Keine

Do you see anything being wrong here?

brma

@Gertjan - thank you for the answer. It's the UDP-interface on port 1194 and yes, the packet-capture works without any issues. Don't worry that the IP-address of the firewall is 192.168.0.2 - that's because my cable-modem's provides this IP-address to the firewall but it's configured in a way that the firewall is the DMZ-host and all trafic is forwarded from the cable-modem to the firewall.
The IP-address of the phone stays the same with both requests... the working and the failing request...

Some questions to your suggestion:

If there would be an issue with the firewall-rule in pfSense, why would it work the first time? And again after about 12 to 24 hours? Would this make sense?
Would the log in the pfSense talk about "TLS Error: TLS handshake failed" if the packets would not make it to the OpenVPN-Server? In this case they would have been dropped by the firewall before - right?

Result of the packet capture on UDP port 1194:

08:45:07.441054 IP 46.125.249.93.11326 > 192.168.0.2.1194: UDP, length 54
08:45:07.441562 IP 192.168.0.2.1194 > 46.125.249.93.11326: UDP, length 66
08:45:09.534732 IP 192.168.0.2.1194 > 46.125.249.93.11326: UDP, length 54
08:45:09.953605 IP 46.125.249.93.11326 > 192.168.0.2.1194: UDP, length 54
08:45:09.953784 IP 192.168.0.2.1194 > 46.125.249.93.11326: UDP, length 62
08:45:13.142574 IP 192.168.0.2.1194 > 46.125.249.93.11326: UDP, length 54
08:45:13.694062 IP 46.125.249.93.11326 > 192.168.0.2.1194: UDP, length 54
08:45:13.694165 IP 192.168.0.2.1194 > 46.125.249.93.11326: UDP, length 62
08:45:21.234197 IP 46.125.249.93.11326 > 192.168.0.2.1194: UDP, length 54
08:45:21.234282 IP 192.168.0.2.1194 > 46.125.249.93.11326: UDP, length 66

Gertjan

@brma said in pfSense 2.4.5 with OpenVPN: connection issue on turning hotspot on phone on/off:

And again after about 12 to 24 hours?

WAN IP changed ?

This is my snif :

10:25:09.726091 IP 92.184.108.238.51760 > 192.168.10.2.1194: UDP, length 54
10:25:09.726726 IP 192.168.10.2.1194 > 92.184.108.238.51760: UDP, length 66
10:25:09.805921 IP 92.184.108.238.51760 > 192.168.10.2.1194: UDP, length 262
10:25:09.812106 IP 192.168.10.2.1194 > 92.184.108.238.51760: UDP, length 62
10:25:09.815627 IP 192.168.10.2.1194 > 92.184.108.238.51760: UDP, length 1148
10:25:09.819623 IP 192.168.10.2.1194 > 92.184.108.238.51760: UDP, length 1148
10:25:09.946964 IP 92.184.108.238.51760 > 192.168.10.2.1194: UDP, length 1316

192.168.10.2 is my WAN IP, as you, I'm behind an ISP upstream router.
As you can see, after the initial handshake, packet size ramps up quickly. Normal, the certs have to be send over, and these are not some 64 or 62 bytes size.

Your alternating 64 and 66 bytes packet size says to me : your VPN client and server do not speak the same "language" : or : settings are not equal on both sides.
Like : example : compression is set on one side - none on the other.

edit : try the newest VPN video from Netgate, from last week.

brma

@Gertjan said in pfSense 2.4.5 with OpenVPN: connection issue on turning hotspot on phone on/off:

WAN IP changed ?

Definitely no - the wan address is always the same (and verified just this moment - my ISP always assigns the same IP-address on reboots).
At the "first" connect, my snif looks the same as yours. Just when disabling the hotspot on the phone and enabling it again, I do have the "small" packages only.

I double checked the settings on both sides - they are equal and I guess it wouldn't work for the "first" times if they wouldn't be...? To prevent another possible cause of failure I even dropped the negotiable cryptographic parameters to force AES-256-CBC and - of course - also adapted the client-configuration by using ncp-disable instead ncp-ciphers...

@Gertjan said in pfSense 2.4.5 with OpenVPN: connection issue on turning hotspot on phone on/off:

edit : try the newest VPN video from Netgate, from last week.

I just searched the netgate-website for videos dealing with OpenVPN from last week but couldn't find any - could you please post me the link?

Thanks for your help!

Gertjan

@brma said in pfSense 2.4.5 with OpenVPN: connection issue on turning hotspot on phone on/off:

just searched the netgate-website

They use Youtube ... https://www.youtube.com/channel/UC3Cq2kjCWM8odzoIzftS04A

Pippin

@brma said in pfSense 2.4.5 with OpenVPN: connection issue on turning hotspot on phone on/off:

After disconnecting windows from the hotspot and turning off the hotspot

I have missed that in your openings post.
But this is normal behaviour, when you disconnect from WiFi OpenVPN will disconnect too.
So manually connecting is neccesary.
Maybe rooting your Android and using something like Tasker can automate reconnecting.... No experience with that.

brma

@Gertjan Thank you for the link. Well, I watched the video and that's the procedure I went through several times already. However - the "first" connects works! Today again! But as already mentioned: after turning off/on the hotspot on the phone and restarting the WLAN-connection to the hotspot and then opening the OpenVPN connection fails.

To me it looks as parts of the "old" connection is buffered and the re-enabling of the hotspot changes something in there. After the timeout occured there seems to be a reset and after that the "first" connect works... any idea what that could be?

The only thing I've found and that comes to my mind is the changed IP-gateway on the phone with the hotspot on every time the hotspot is disabled/enabled? But if it is - how can I manually "reset" the buffer that keeps that information?

brma

@Pippin I'm afraid there is a misundersting: my issue is not to manually re-connect the hotspot. My issue is, that if I turn off the hotspot and later on again (let's say 30min later because I moved to another working place) I cannot re-connect to the OpenVPN-server. I can access the "normal" IP-connection without any issues - surfing, etc. but NOT re-connecting to the OpenVPN-server.

After some time (it looks to me about 12 hours), it works again. As long as I do not turn off the Android hotspot.

Gertjan

I tried to replicate this.
Using pfSense 2.4.5 + package OpenVPNClient.

I created a User, "WIN7", member of the OpenVPN OpenVPN user group. I added a certifcate, because I'm using remote access type SSL/TLS.

I exported this executable

and installed it on a Wifi capable device - a Windows 7 pro PC - that had never OpenVPN (client) installed before.

I activated the Hotspot on my iPhone, so it was using LTE/4G for Internet access, and offerring a Wifi local AP type network.
I connected my Windows 7 PC to my hostspot-iPhone network.

Started the OpenVPNGUI and activated the VPN.
I was connected. Using an IPv4 and IPv6 ....

I put my iPhone in Flight mode (all connections lost) and waited for 30 seconds.
The Wifi connection on my Windows 7 PC was lost ....

Re activated my iPhone.
Re established the Wifi connection.

I did not had to do anything with the OpenVPN Client (no disconnect and - connect or re-connect) : the VPN connection came back by itself.

I repeated these steps several time.

Note : I have no control over what IP and/or Gateway is assigned by my iPhone to the PC. Because I wasn't moving, I guess these stay the same.

@brma said in pfSense 2.4.5 with OpenVPN: connection issue on turning hotspot on phone on/off:

But if it is - how can I manually "reset" the buffer that keeps that information?

Easy : disconnect using the OpenVPN GUI menu command. And re connect.
It shouldn't matter that the IP or anything else changes. There is no persistent information that last between VPN sessions.

Check : Be sure that the network and IP assigned to your PC by your phone isn't in conflict conflict with the tunnel network used by your VPN.

brma

@Gertjan First of all - thanks for the effort spent!

As I mentioned before I do have a second OpenVPN-server with the same configuration (of course besides IP-address/dns-name/certificate/user/password) and there I do not have any issues with the same client and the same hotspot - also after restarting the hotspot!

That's why I'm having such a hard time determining the cause of this issue!

The process you went through and described is exactly the same I did and also did several times for different server/clients. The main-difference seems to be that you're using an iPhone whereas I am using an Android device.

If the connect is successful, it looks comparable to your installation:

@Gertjan said in pfSense 2.4.5 with OpenVPN: connection issue on turning hotspot on phone on/off:

But if it is - how can I manually "reset" the buffer that keeps that information?
Easy : disconnect using the OpenVPN GUI menu command. And re connect.

That's exactly what I'm doing... but without success...

@Gertjan said in pfSense 2.4.5 with OpenVPN: connection issue on turning hotspot on phone on/off:

Check : Be sure that the network and IP assigned to your PC by your phone isn't in conflict conflict with the tunnel network used by your VPN.

I checked this several times - it isn't (at least - I do not see one):
Local network: 192.168.75.0/24
Tunnel: 10.75.0.0/24
Hotspot IP: 46.125.249.xxx (xxx varies - the ISP seems to issue addresses out of the 46.125.249.0/24 subnet)
Laptop (IP received from hotspot on the android phone): 192.168.43.107

And also always the same question in the back of my head:
if the config would be wrong, why would it always work the first time, also with re-connects by the OpenVPN-client as long as I do not disable the hotspot?

If you have any other idea - everything is welcome!

Another thought: could it be that the router of the ISP at the server side keeps a state for the connection and there is a difference because of the restart of the hotspot? And therefore packets come to the server but are not routed back properly? Unfortunatelly I cannot check this as the routing-table is not accessible for me...

Gertjan

@brma said in pfSense 2.4.5 with OpenVPN: connection issue on turning hotspot on phone on/off:

Laptop (IP received from hotspot on the android phone): 192.168.43.107

A /24 right ?

@brma said in pfSense 2.4.5 with OpenVPN: connection issue on turning hotspot on phone on/off:

why would it always work the first time

Like : what is the DHCP lease time on your Laptop ? 12 hours delay looks like a DHCP kind of time out.
What happens if you ask a new lease ?
What happens if you assign a static IP (+DNS + Gateway) on the VPNClient side ? In the range of the tunnel pool of course.

brma

@Gertjan said in pfSense 2.4.5 with OpenVPN: connection issue on turning hotspot on phone on/off:

A /24 right ?

Well, I'm always receiving the same IP-address - it seems somehow fixed. Only the gateway within the 192.168.43.0/24 changes every time I turn the hotspot off/on.

What happens if you ask a new lease ?

I'll get exactly the same again and cannot connect (same behaviour). Only turning the hotspot off/on gives me a new gateway with the same IP-address. I do also believe it cannot really be an DHCP issue because how would I be able to connect to the server and produce an TLS-handshake error? Without valid IP - no connect. I do have more the impression the packets don't find their way back to the phone...

What happens if you assign a static IP (+DNS + Gateway) on the VPNClient side ? In the range of the tunnel pool of course.

In my understanding, the tunnel can only be established as soon as the TLS-handshake has been done and this causes the error at the very moment. However - I tried it - no change.

Gertjan

@brma said in pfSense 2.4.5 with OpenVPN: connection issue on turning hotspot on phone on/off:

In my understanding, the tunnel can only be established as soon as the TLS-handshake has been don

Euh, noop.
An IP connection will exist first. The entire SSL/TLS/whatever exists in the packet's payload - the "data".
Traffic info like IP-source and IP-destination, ports and so are always 'visible'.

brma

@Gertjan said in pfSense 2.4.5 with OpenVPN: connection issue on turning hotspot on phone on/off:

Euh, noop.
An IP connection will exist first. The entire SSL/TLS/whatever exists in the packet's payload - the "data".
Traffic info like IP-source and IP-destination, ports and so are always 'visible'.

Well - even if I have a wrong understanding here (my thought was the connect is between the "physical" addresses and the tunneled IP-adresses are already inside the encryption), the result is as follows (as I already tried it several times):
I even cannot connect to the server anymore - so no incoming packets on pfSense. I guess this is because the changing gateway on any new hotspot off/on has sense and the manuel configured one is simply false. So no proper routing takes place.

brma

Just in case anybody has this issue as well - I finally was able to resolve it: the issue seems to be that the routing table on the Android phone (version 9 - Pie) seems to have corrupted in some way when the OpenVPN-connection routed is closed and re-opened.
The solution is either rebooting the phone or (much faster) turning on and off flight-mode - this seems to reset the routing table and OpenVPN-connections can be initiated again.