NAT issue on OpenVPN Remote Access

[ozus82]

Hi all,

Recently one of my pfsense box (2.6.0) crashed. This box hosts couple site to site vpn connections and couple remote access vpns.

Upon installing 2.7.2 and restore old config, the site to site vpn connections resumed to work as normal, but the remote access vpn got some weird problems.

A remote access client connecting to this new 2.7.2 box will have these problems :

1. It cannot access the internet
2. It cannot access other networks on the new 2.7.2 box
3. It can access other networks from other sites. Those sites is still on 2.6.0 (not really a problem but I think related to previous and should get fixed too)

To troubleshoot #1 I have tried the following:

1. Add or move the 'allow everything' rule to the OpenVPN tab (previously it is on the remote vpn tab). Still no go
2. Enabled logging on the 'allow everything' rules. I can see that the request from the remote access client is handled by the rule but still doesn't seem to work.
3. My Outbound NAT tab is on automatic. I can see the remote client private ip (/24) when it is connected.

For #2 I guess it is the similar issue with #1 and for #3, probably it works because the networks is behind a 2.6.0

I read on couple posts that there were changes in openvpn in 2.7.2 that make 'wrong config' that used to work not work anymore but I'm at lost at what is wrong.

Appreciate if anyone can point me to the right directions.

Thanks in advance

[viragomann]

@ozus82 said in NAT issue on OpenVPN Remote Access:

It can access other networks from other sites. Those sites is still on 2.6.0 (not really a problem but I think related to previous and should get fixed too)

So there are two boxes now, a 2.6 and a 2.7.2 aside?

Which one is the default gateway in the local subnets?

[ozus82]

@viragomann

So box A (2.6.0) is the server for both site to site vpn (box B, box C all using 2.6.0) and remote access vpn (android D using openvpn connect)

A is typical router box with 2 interfaces 1 for WAN and 1 for LAN. The LAN interface got couple vlans under it. Same with B and C

For rules, outbound nat is automatic in all ABC, port forwarding, and the usual common rules in each ABC to restrict access between vlans, etc

The site to site vpn is a SSL/TLS with a /24 tunnel network.
The remote access vpn is also SSL/TLS + user auth also with a /24 tunnel network.
I have Client Overide for all the connections so that I can assign each corresponding IPs when connected to A.

With this, D, when connected to A via remote access, can access all networks (behind A,B,C) and have internet access via A (nat)

After the crash A got reinstalled with 2.7.2 and restored config from backup

Now when D is connected to A via remote access:

1. D can no longer access networks behind A
2. D have no internet
3. But D still can access network behind B,C (note B,C is still in 2.6.0)

Additionally, I found out that sometimes D is able to have internet but only for a brief period of time.
I tried with other device, E (an IOS device also with openvpn connect) with the same profile and can verify the same behavior
During these time (when have internet access) access to network behind A failed but access to network behind B,C is good all time.

When A was in 2.6.0 everything works ok. Now with 2.7.2 everything is 'ok' (hosts inside A can reach to B, C and vice versa) except the weird D's problem. I don't know if I change B, C to 2.7.2 will it break the connections between hosts in ABC similar to what happened to D.

Based on my knowledge, I can't find where is the problem, especially with the recent finding that it's able to work, granted only for a brief of time.

Seems like my understanding of network is wrong and the 'wrong 2.6.0' helped me to get away with it.

Thanks in advance

[viragomann]

@ozus82
Pretty strange. When you can access the internet from an access server client, I'd expect, that you also can access the A subnets, presumed the firewall rules allow this.
Also ensure, that the destination devices allows access. However, if it permits access from an s2s remote network, road warrior access should work as well.

What are your cryptographic and tunnel settings of the access server?

Try disable the compression if enabled and reexport the client config and install it on the client.

Also the server and client logs may content some hints on what's going wrong.

[ozus82]

@viragomann

Yes very strange indeed. I think access rules should be ok as it is a restore config and 'will work for a brief time'.

Here is the config with some part redacted
For the remote access is

dev ovpns4
verb 1
dev-type tun
dev-node /dev/tun4
writepid /var/run/openvpn_server4.pid
#user nobody
#group nobody
script-security 3
daemon
keepalive 10 60
ping-timer-rem
persist-tun
persist-key
proto udp4
auth SHA256
up /usr/local/sbin/ovpn-linkup
down /usr/local/sbin/ovpn-linkdown
client-connect /usr/local/sbin/openvpn.attributes.sh
client-disconnect /usr/local/sbin/openvpn.attributes.sh
local <the_wan_ip>
tls-server
server 192.168.15.0 255.255.255.0
client-config-dir /var/etc/openvpn/server4/csc
plugin /usr/local/lib/openvpn/plugins/openvpn-plugin-auth-script.so /usr/local/sbin/ovpn_auth_verify_async user <redacted> false server4 1197
tls-verify "/usr/local/sbin/ovpn_auth_verify tls '<redacted>' 1"
lport 1197
management /var/etc/openvpn/server4/sock unix
max-clients 3
push "redirect-gateway def1"
remote-cert-tls client
capath /var/etc/openvpn/server4/ca
cert /var/etc/openvpn/server4/cert
key /var/etc/openvpn/server4/key
dh /etc/dh-parameters.4096
tls-auth /var/etc/openvpn/server4/tls-auth 0
data-ciphers AES-256-GCM:AES-128-GCM:CHACHA20-POLY1305:AES-256-CBC
data-ciphers-fallback AES-256-CBC
allow-compression no
topology subnet
inactive 300

For the site to site

dev ovpns5
verb 1
dev-type tun
dev-node /dev/tun5
writepid /var/run/openvpn_server5.pid
#user nobody
#group nobody
script-security 3
daemon
keepalive 10 60
ping-timer-rem
persist-tun
persist-key
proto udp4
auth SHA256
up /usr/local/sbin/ovpn-linkup
down /usr/local/sbin/ovpn-linkdown
local <the_wan_ip>
tls-server
server 192.168.16.0 255.255.255.0
client-config-dir /var/etc/openvpn/server5/csc
ifconfig 192.168.16.1 192.168.16.2
tls-verify "/usr/local/sbin/ovpn_auth_verify tls '<redacted>' 1"
lport 1198
management /var/etc/openvpn/server5/sock unix
max-clients 3
push "route all networks in A (lan and vlan) all /24 and non overlapping 
push ...
push "route 192.168.15.0 255.255.255.0"    ==> the network of remote access 
remote-cert-tls client
route 'network from connecting sites, all /24 and non overlapping' 
...
capath /var/etc/openvpn/server5/ca
cert /var/etc/openvpn/server5/cert
key /var/etc/openvpn/server5/key
dh /etc/dh-parameters.4096
tls-auth /var/etc/openvpn/server5/tls-auth 0
data-ciphers AES-256-GCM:AES-128-GCM:CHACHA20-POLY1305:AES-256-CBC
data-ciphers-fallback AES-256-CBC
allow-compression no
topology subnet
explicit-exit-notify 1

Another weird observation is as I'm typing this, the connection works just fine. D connected to A, have internet through A and can access hosts behind ABC just like expected. I do noticed that earlier when D 'failed', the mobile connection might not be so good, but still it doesn't explain the fact that in that 'bad connection' to A, D can still access hosts behind BC just no internet and no access to host behind A. This is what baffled me.

Another observation is when D 'failed', looking at firewall logs I found a lot of

"@9 block drop out log inet all label "Default deny rule IPv4" ridentifier 1000000104"

default deny invoked by the reply traffic of what D is trying to reach earlier

for example
VPN_REMOTE Default deny rule IPv4 (1000000104) 152.228.215.159:443 192.168.15.26:48662 TCP:FPA
VPN_REMOTE Default deny rule IPv4 (1000000104) 152.228.215.159:443 192.168.15.26:48662 TCP:FA

I searched and one topic that I think could explain this seems to be 'asymmetric routing', but then I can't wrap my logic around it.
I can't see how the road warrior become asymmetric (but it can still access BC) and time to time can work just fine for brief period of time (this indeed a symptom of asymmetric routing from what I read)

Seems like something simple but I can't pin point it. Like now it been working fine for 30+ mins, also tried disconnect and reconnect again, all no issues. Really baffled me.

Any ideas what to check ?

Thanks in advance

[Gertjan]

@ozus82 said in NAT issue on OpenVPN Remote Access:

So box A (2.6.0) is the server for ....

You come from far, far away. "2.6.0" was pretty good, and "2.7.2" is even better.

But something happened
Read the top most forum post from here - this is the post I tals about HEADS UP: OpenVPN deprecating shared key mode, requires TLS, deprecating cipher selection.
'OpenVPN', server and client, has news things added, other stuff deprecated or simple removed.

[ozus82]

@Gertjan

Yeah it been rock solid on 2.6.0, I think it has a year plus uptime before it finally gave away (the disk).

I did see the post you linked, but I can't see how it apply to my config. I have not used shared keys, compression for my setup in 2.6.0
The same config 'migrated' to 2.7.2 as you can see the config in above post.

Is the config version 22.2 (from 2.6.0) incompatible with version 23.3? When installing 2.7.2, it gladly accept it via config.xml.

I did read a post that 2.6.0 was forgiving for 'bad config', but I can't pin point what was wrong in 2.6.0 that 2.7.2 is now enforcing and create the problem.

I guess I'm now experiencing it, sometime works sometime it doesn't. This baffled me

Anyone can help point out what could be the reasons for this inconsistent behavior ?

Thanks in advance