Packets disappearing between tun and wan

4632215

Hi,

I would like to ask the community for help understanding where some packets are disappearing as I've been pulling my hair out for a couple of days now!

Background

I have been successfully maintaining a site-to-site openvpn connection between two pfsense firewalls (let's call them site A and site B) for years now, so I'm confident that my approach is correct
I now have the need to add a third site (site C), which will connect to both site A and B using exactly the same setup. The only difference is that site C will be fronted by a teltonika 5G gateway
site C is able to connect to both sites A and B (as a client), can correctly forward DNS queries to the remote sites, can reach both remote LANs, everything works great

What doesn't work

site A and B's LANs (server-side LANs) cannot reach site C's LAN

Troubleshooting

I am focusing here on site A to site C but the same applies to site B.

site A - pfsense.sitea.lan
address in the vpn tunnel: 10.0.55.1/24
address on the lan side: 192.168.34.254/24
tun device: ovpns1
wan device: igb0

site B - trb500.sitec.lan
address in the vpn tunnel: 10.0.55.2/24
address on the lan side: 192.168.134.1/24

Site A can ping site C's openvpn client within the virtual tunnel network, so the vpn tunnel itself is fine

[2.6.0-RELEASE][root@pfSense.sitea.lan]/root: ping 10.0.55.2
PING 10.0.55.2 (10.0.55.2): 56 data bytes
64 bytes from 10.0.55.2: icmp_seq=0 ttl=64 time=46.099 ms
64 bytes from 10.0.55.2: icmp_seq=1 ttl=64 time=37.703 ms

I see traffic on the tun interface on the site A side, so I am putting packets in the tunnel

[2.6.0-RELEASE][root@pfSense.sitea.lan]/root: tcpdump -ni ovpns1 icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ovpns1, link-type NULL (BSD loopback), capture size 262144 bytes
11:37:03.224080 IP 10.0.55.1 > 192.168.134.1: ICMP echo request, id 28162, seq 0, length 64
11:37:04.236731 IP 10.0.55.1 > 192.168.134.1: ICMP echo request, id 28162, seq 1, length 64
11:37:05.242268 IP 10.0.55.1 > 192.168.134.1: ICMP echo request, id 28162, seq 2, length 64
11:37:06.255882 IP 10.0.55.1 > 192.168.134.1: ICMP echo request, id 28162, seq 3, length 64

So... site C is firewalling off ingress openvpn traffic, right? Nope, because I do not see the packets when I tcpdump on the site C side.

What I believe is happening is that those packets disappear between the ovpns1 interface and the wan device.

If I capture on the wan side I can see the encrypted packets, but only when I ping within the virtual tunnel.

shell1:

[2.6.0-RELEASE][root@pfSense.sitea.lan]/root: ping -s1400 -c4 10.0.55.2
PING 10.0.55.2 (10.0.55.2): 1400 data bytes
1408 bytes from 10.0.55.2: icmp_seq=0 ttl=64 time=648.205 ms
1408 bytes from 10.0.55.2: icmp_seq=1 ttl=64 time=99.991 ms
1408 bytes from 10.0.55.2: icmp_seq=2 ttl=64 time=106.075 ms
1408 bytes from 10.0.55.2: icmp_seq=3 ttl=64 time=104.884 ms

--- 10.0.55.2 ping statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 99.991/239.789/648.205/235.810 ms

shell2:

[2.6.0-RELEASE][root@pfSense.sitea.lan]/root: tcpdump -ni igb0 host 109.118.84.52 # this is site C public IP
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on igb0, link-type EN10MB (Ethernet), capture size 262144 bytes
11:40:28.730600 IP 192.168.1.2.1199 > 109.118.84.52.13981: UDP, length 1452
11:40:28.835884 IP 109.118.84.52.13981 > 192.168.1.2.1199: UDP, length 1452
11:40:29.744901 IP 192.168.1.2.1199 > 109.118.84.52.13981: UDP, length 1452
11:40:29.809449 IP 109.118.84.52.13981 > 192.168.1.2.1199: UDP, length 1452
11:40:30.759282 IP 192.168.1.2.1199 > 109.118.84.52.13981: UDP, length 1452
11:40:30.868418 IP 109.118.84.52.13981 > 192.168.1.2.1199: UDP, length 1452
11:40:31.774335 IP 192.168.1.2.1199 > 109.118.84.52.13981: UDP, length 1452
11:40:31.848450 IP 109.118.84.52.13981 > 192.168.1.2.1199: UDP, length 1452

If I try to ping site C LAN address no output is produced in my tcpdump capture, therefore pfsense must be the one dropping the packets, and I have no idea why.

To recap:

src	dst	packets on tun	packets on wan
10.0.55.1	10.0.55.2	yes	yes
10.0.55.2	10.0.55.1	yes	yes
192.168.34.254	192.168.134.1	yes	no
192.168.134.1	192.168.34.254	yes	yes

Further data

site A does know the route, so packets are not dropped due to that, or sent to a different device

[2.6.0-RELEASE][root@pfSense.sitea.lan]/root: netstat -r | grep 134
192.168.134.0/24   10.0.55.2          UGS      ovpns1

the route 'Uses' counter in the web ui goes up as expected (e.g. 5 pings yield +5 uses), so the route is both known and used
those missing icmps cannot be seen on any other interface via tcpdump
pfsense is not firewalling egress because I have a single allow all rule for openvpn
I have confirmed the allow all rule is triggered by the disappearing pings by turning on logging
the missing traffic does not trigger any block rule
I have tried to no avail disabling hardware checksum offloading as described here

At this point I'm really out of ideas, there must be something I'm overlooking!

viragomann

@4632215
Edit the server settings and state a /30 tunnel mask.

4632215

@viragomann ...and it worked[*]! But why? My two pfsense devices (unnecessarily) use a /24 tunnel subnet and need no additional configuration.

[*] the teltonika client was not setting its own IP so I had to add a push "ifconfig 10.0.55.2 10.0.55.1" and a push "route-gateway 10.0.55.1" directives

viragomann

@4632215
If you have a /30 tunnel there can only be two IPs inside, one is the server, the other one is the client. So all is clear.

If the tunnel network is bigger, there can be one server and multiple clients inside. So you have to tell the server, behind which client IP he can find the desired network, you want to send packets to. This can be done by the iroute directive in OpenVPN.
In pfSense you have to create a client specific override to set this, where you have to state the client sides remote network.
But if you only have one client anyway, you can spare this and easily set the tunnel mask to /30.