Routing between 2 pfsense does not work with OpenVPN Tunnel

[Genpc]

Hello everybody,

I have 2 pfsense installed. The one pfSense is directly connected to the internet. The other pfsense is placed behind the first pfsense, the wan interface of the second pfsense is connected to a port of the first pfsense (All in all, its a classic network with a DMZ).

I installed OpenVPN on the first pfSense and I can reach the subnets behind the first firewall. I disabled outgoing NAT on the second firewall and created a new gateway on the first pfsense with the WAN IP of the second pfsense in it. Afterwards, I added a static route for the subnet to the firewall.

The firewall itself is able to ping the clients behind the second firewall (I did set the firewallrules accordingly), but I cannot connect to this client over VPN. I captured the traffic and the ping reply also arrives on the interface which is connected to the second pfSense, but it never arrives back on the OpenVPN Interface.

Thank you for your help  :)

[Derelict]

Specifics, man, diagrams, actual subnets and routes, etc. Impossible to help without details.

[Genpc]

I'll try it:

1. pfsense is connected to the internet and gets a static wan ip (xxx.xxx.xxx.xxx/27)
2. the pfsense is connected to a switch. The link from the switch is configured as vlan trunk.
3. The other pfsense is connected to the first pfsense over the switch (a vlan is configured on the pfsense and on the switch, the second firewall has the wan-ip 172.16.58.250/24)
4. An Interface has been created for the communication between both pfsense(s) (172.16.58.0/24)
5. NAT has been disabled on the second pfsense firewall (since there are all private IPs, NAT is not required.)
6. A new gateway has been added (172.16.58.250)
7. A new static route has been created (Destination network 172.16.20.0/24, gateway 172.16.58.250)
8. A ping from the first pfsense to a server located behind the second pfsense (Server has 172.16.20.1) works
9. A ping from the OpenVPN (tun, force all traffic through gateway, vpn subnet 192.168.68.0/24) does not work.

Packet capture shows that the package gets lost somewhere between the interface which connects both pfsenses (I can see the ping reply there) and the OpenVPN (I only see the ping request but NOT the ping reply).

I hope this small painting helps.

–-------------
--INTERNET--

-
        -
        - xxx.xxx.xxx.xxx/27
        -
        -
        -

---pfSense--- (with the OpenVPN Server)

- 172.16.58.254
        -
        -
        -
        -
        - 172.16.58.250

---pfSense---

- 172.16.20.254
        -
        -
        - 172.16.20.1

---Server---

[Derelict]

What are the source addresses of the connections coming in from across the OpenVPN to the 172.16.20.0/24 destinations? Are the source networks "known" to the node running the OpenVPN server (does that server have a route in its routing table for the source address as it is known to it) or are they arbitrary addresses (like various internet sources?)

[Genpc]

The clients connect to the pfSense OpenVPN Server. The subnet is known for the pfsense (since it does provide the VPN itself)

The 192.168.68.0/24 subnet is routed to the OpenVPN Server Address (192.168.68.1) which gets routed to #link8 (netif ovpns1)

The sourceaddress is 192.168.68.2.

I attached the routing table - the back entries are the public ip and the public gateway.

[Derelict]

Probably should post a quick packet capture filtered on icmp and the target host address taken on the 172.16.58.254 interface.

You can just ping from the 172.16.58.254 firewall selecting the OpenVPN interface as the source interface.

Then capture the same traffic running the pcap on the openvpn instance.

[Genpc]

I just installed another host which is connected to the first pfsense and this host cannot ping 172.16.20.1 either, so this is not an OpenVPN issue but a routing issue.

On the OpenVPN Interface (does also apply to the other server interface):

18:17:04.562619 IP 192.168.68.2 > 172.16.20.1: ICMP echo request, id 53527, seq 0, length 64
18:17:05.551177 IP 192.168.68.2 > 172.16.20.1: ICMP echo request, id 53527, seq 1, length 64
18:17:06.595303 IP 192.168.68.2 > 172.16.20.1: ICMP echo request, id 53527, seq 2, length 64
18:17:07.598748 IP 192.168.68.2 > 172.16.20.1: ICMP echo request, id 53527, seq 3, length 64

On the interface which connects both firewalls:

18:18:15.316407 IP 172.16.58.250 > xxx.xxx.xxx.193: ICMP echo request, id 21153, seq 5564, length 8
18:18:15.316952 IP 1xxx.xxx.xxx.193 > 172.16.58.250: ICMP echo reply, id 21153, seq 5564, length 8
18:18:15.321373 IP 172.16.58.250 > 172.16.58.254: ICMP echo request, id 21835, seq 5592, length 8
18:18:15.321385 IP 172.16.58.254 > 172.16.58.250: ICMP echo reply, id 21835, seq 5592, length 8

xxx.xxx.xxx.193 is the gateway IP of the public subnet. This also happens if I use an internal server which is not connected via OpenVPN. It looks like the backcoming packages are routed on the public gateway ip and not back to the subnet.

I attached 2 pictures which show the gateway configuration and the static route. The selected interface is the interface where both pfSense(s) are connected.