Site2Site does not work/route in both directions

[demux]

Hi!

I do have a strange problem with OpenVPN routing and I am pretty sure it was working some time ago.

It is a site to site link conecting 172.27.72.0/24 (remote) with 10.252.200.0/22. 10.252.200.254 is the OpenVPN server on pfSense, 172.27.72.254 is the client side pfSense. OpenVPN tunnel network is 172.30.14.0/24 (172.30.14.1=server, 172.30.14.2=client).

From remote (172.27.72.x) I can ping and reach the full 10.252.20x.y network.
But I cannot reach the remote devices from the server side's network. I can only ping the remote pfSense but no other device.

What I get is this from server's network - ping:
PING 172.27.72.1 (172.27.72.1) 56(84) bytes of data.
From 172.30.14.1 icmp_seq=1 Time to live exceeded
From 172.30.14.1 icmp_seq=2 Time to live exceeded
From 172.30.14.1 icmp_seq=3 Time to live exceeded
From 172.30.14.1 icmp_seq=4 Time to live exceeded
...

What I get is this from server's network - traceroute:
traceroute to 172.27.72.1 (172.27.72.1), 30 hops max, 60 byte packets
1 10.252.200.254 0.338 ms 0.362 ms 0.356 ms
2 172.30.14.1 27.970 ms 35.843 ms 42.439 ms
3 172.30.14.1 70.223 ms 70.251 ms 74.688 ms
...
29 172.30.14.1 881.594 ms 871.410 ms 885.545 ms
30 172.30.14.1 907.684 ms 895.962 ms 896.176 ms

This traceroute is really strange.

ssh from pfSense server to a device on the client network says:
ssh -l user 172.27.72.1
ssh: connect to host 172.27.72.1 port 22: No route to host

But I see the route, below (***).

The routing table on the server looks like this (two WANs and two additional OpenVPN servers):
Destination Gateway Flags Netif Expire
default 24.134.xx.yyy UGS igb1
1.0.0.1 82.82.xx.yyy UGHS pppoe0
1.1.1.1 24.134.xx.yyy UGHS igb1
4.2.2.1 24.134.xx.yyy UGHS igb1
4.2.2.2 82.82.xx.yyy UGHS pppoe0
4.2.2.3 24.134.xx.yyy UGHS igb1
4.2.2.4 pppoe0 UHS pppoe0
8.8.4.4 82.82.xx.yyy UGHS pppoe0
8.8.8.8 24.134.xx.yyy UGHS igb1
9.9.9.10 24.134.xx.yyy UGHS igb1
10.252.200.0/22 link#3 U igb2
10.252.200.254 link#3 UHS lo0
10.254.154.54 link#3 UHS lo0
10.254.154.54/32 link#3 U igb2
24.134.xx.yyy/30 link#2 U igb1
24.134.xx.yyy link#2 UHS lo0
82.82.xx.yyy link#12 UH pppoe0
127.0.0.1 link#6 UH lo0
172.25.254.0/24 link#11 U igb2.222
172.25.254.254 link#11 UHS lo0
172.27.72.0/24 172.30.14.2 UGS ovpns6 (***)
172.30.1.0/24 172.30.1.2 UGS ovpns1
172.30.1.1 link#17 UHS lo0
172.30.1.2 link#17 UH ovpns1
172.30.2.0/24 172.30.2.2 UGS ovpns3
172.30.2.1 link#16 UHS lo0
172.30.2.2 link#16 UH ovpns3
172.30.14.1 link#14 UHS lo0
172.30.14.2 link#14 UH ovpns6
172.30.15.1 link#15 UHS lo0
172.30.15.2 link#15 UH ovpns7
176.95.xxx.yyy link#12 UHS lo0
192.168.55.0/24 link#10 U igb2.55
192.168.55.254 link#10 UHS lo0

On the client I have the following routing table (two additional OpenVPN servers):
Destination Gateway Flags Netif Expire
default 90.187.xxx.yyy UGS igb0
10.252.200.0/22 172.30.14.1 UGS ovpnc1
90.187.xxx.yyy/30 link#1 U igb0
90.187.xxx.yyy link#1 UHS lo0
127.0.0.1 link#5 UH lo0
172.27.72.0/24 link#3 U igb2
172.27.72.254 link#3 UHS lo0
172.30.14.1 link#11 UH ovpnc1
172.30.14.2 link#11 UHS lo0
172.30.15.1 link#12 UH ovpnc2
172.30.15.2 link#12 UHS lo0
172.30.50.0/24 172.30.50.2 UGS ovpns3
172.30.50.1 link#9 UHS lo0
172.30.50.2 link#9 UH ovpns3
172.30.51.0/24 172.30.51.2 UGS ovpns4
172.30.51.1 link#10 UHS lo0
172.30.51.2 link#10 UH ovpns4
192.168.55.0/24 link#8 U igb2.55
192.168.55.254 link#8 UHS lo0

The server config looks like this:
dev ovpns6
verb 1
dev-type tun
dev-node /dev/tun6
writepid /var/run/openvpn_server6.pid
#user nobody
#group nobody
script-security 3
daemon
keepalive 10 60
ping-timer-rem
persist-tun
persist-key
proto udp4
cipher AES-128-CBC
auth SHA1
up /usr/local/sbin/ovpn-linkup
down /usr/local/sbin/ovpn-linkdown
local 24.134.xxx.yyy
engine cryptodev
ifconfig 172.30.14.1 172.30.14.2
lport 1196
management /var/etc/openvpn/server6.sock unix
max-clients 16
route 172.27.72.0 255.255.255.0
secret /var/etc/openvpn/server6.secret
fast-io
sndbuf 1048576
rcvbuf 1048576

The client config looks like this:
dev ovpnc1
verb 1
dev-type tun
dev-node /dev/tun1
writepid /var/run/openvpn_client1.pid
#user nobody
#group nobody
script-security 3
daemon
keepalive 10 60
ping-timer-rem
persist-tun
persist-key
proto udp4
cipher AES-128-CBC
auth SHA1
up /usr/local/sbin/ovpn-linkup
down /usr/local/sbin/ovpn-linkdown
local 90.187.xxx.yyy
engine cryptodev
lport 0
management /var/etc/openvpn/client1.sock unix
remote server 1196
ifconfig 172.30.14.2 172.30.14.1
route 10.252.200.0 255.255.252.0
secret /var/etc/openvpn/client1.secret
resolv-retry infinite
fast-io
sndbuf 1048576
rcvbuf 1048576

On the client I have some sort of fail-over using a gateway group. See https://forum.netgate.com/topic/147314/fail-over-client-config/9

Can someone please have a look and help me find what's wrong? What am I missing?
I need to reach all devices from both sides of the VPN.

Thanks a lot!

[kiokoman]

both side are on the same subnet ?

it seems the packet are bounced between the hops

3 172.30.14.1 70.223 ms 70.251 ms 74.688 ms
...
29 172.30.14.1 881.594 ms 871.410 ms 885.545 ms
30 172.30.14.1 907.684 ms 895.962 ms 896.176 ms

it's a routing problem

[demux]

No, they are not on the same subnet.
Server is on 10.252.200.0/22, client is on 172.27.72.0/24.
Server pfSense has 10.252.200.254, client pfSense has 172.27.72.254.
Both share the same tunnel network (172.30.14.0/24) with server pfSense at 172.30.14.1 and client pfSense on 172.30.14.2.
Yes, it seems to be a routing problem. But I cannot see where. The routing table looks good to me.
From client pfSense I can ping the whole 172.27.72.0/24 network.
From client network I can ping the whole 10.252.200.0/22 network via VPN.
From server pfSense network I can ping nothing on the client network, except client pfSense on client network. Not even the server pfSense can ping anything else than the client pfSense.
And I don't see why!

I just checked, the firewalls on both ends do not block anything.

[kiokoman]

@demux said in Site2Site does not work/route in both directions:
client:
172.27.72.0/24 link#3 U igb2
172.27.72.254 link#3 UHS lo0
172.30.14.1 link#11 UH ovpnc1
172.30.14.2 link#11 UHS lo0
10.252.200.0/22 172.30.14.1 UGS ovpnc1

server:
10.252.200.0/22 link#3 U igb2
10.252.200.254 link#3 UHS lo0
172.30.14.1 link#14 UHS lo0
172.30.14.2 link#14 UH ovpns6
172.27.72.0/24 172.30.14.2 UGS ovpns6

from the client side it work but not from the server side

[demux]

Correct.

[kiokoman]

when did it stop working? after an update maybe?
a reboot didn't help i suppose?
i can't see what's wrong ...
can you make a screenshot of the firewall rules ?
do you have any floating rules?

[demux]

I did reboot. Did not update.
I am not sure if it ever worked. I believe, but I'm not sure.
Maybe it stopped working after that gateway fail-over thing mentioned in the link. I did reverse that temporarily but did not help.
To me everything looks fine. And why the problems into this direction, not the other or both.
I have real problems testing that, because the distance between them is about 800km. If something breaks I have to travel about 9 hours. Like last weekend...

[demux]

I have no clue what happened.
I deleted all servers and clients and started over again.
I strictly followed this documentation:
https://docs.netgate.com/pfsense/en/latest/book/openvpn/openvpn-and-multi-wan.html#
Server uses GW group and client uses remote ...
And now it works.
Whatever happened to my previous config.

[kiokoman]

the routing table now is the same ?
maybe it was something else on the configuration