Packets disappearing between tun and wan
-
Hi,
I would like to ask the community for help understanding where some packets are disappearing as I've been pulling my hair out for a couple of days now!
Background
- I have been successfully maintaining a site-to-site openvpn connection between two pfsense firewalls (let's call them site A and site B) for years now, so I'm confident that my approach is correct
- I now have the need to add a third site (site C), which will connect to both site A and B using exactly the same setup. The only difference is that site C will be fronted by a teltonika 5G gateway
- site C is able to connect to both sites A and B (as a client), can correctly forward DNS queries to the remote sites, can reach both remote LANs, everything works great
What doesn't work
- site A and B's LANs (server-side LANs) cannot reach site C's LAN
Troubleshooting
I am focusing here on site A to site C but the same applies to site B.
site A -
pfsense.sitea.lan
address in the vpn tunnel:10.0.55.1/24
address on the lan side:192.168.34.254/24
tun device:ovpns1
wan device:igb0site B -
trb500.sitec.lan
address in the vpn tunnel:10.0.55.2/24
address on the lan side:192.168.134.1/24Site A can ping site C's openvpn client within the virtual tunnel network, so the vpn tunnel itself is fine
[2.6.0-RELEASE][root@pfSense.sitea.lan]/root: ping 10.0.55.2 PING 10.0.55.2 (10.0.55.2): 56 data bytes 64 bytes from 10.0.55.2: icmp_seq=0 ttl=64 time=46.099 ms 64 bytes from 10.0.55.2: icmp_seq=1 ttl=64 time=37.703 msI see traffic on the tun interface on the site A side, so I am putting packets in the tunnel
[2.6.0-RELEASE][root@pfSense.sitea.lan]/root: tcpdump -ni ovpns1 icmp tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on ovpns1, link-type NULL (BSD loopback), capture size 262144 bytes 11:37:03.224080 IP 10.0.55.1 > 192.168.134.1: ICMP echo request, id 28162, seq 0, length 64 11:37:04.236731 IP 10.0.55.1 > 192.168.134.1: ICMP echo request, id 28162, seq 1, length 64 11:37:05.242268 IP 10.0.55.1 > 192.168.134.1: ICMP echo request, id 28162, seq 2, length 64 11:37:06.255882 IP 10.0.55.1 > 192.168.134.1: ICMP echo request, id 28162, seq 3, length 64So... site C is firewalling off ingress openvpn traffic, right? Nope, because I do not see the packets when I tcpdump on the site C side.
What I believe is happening is that those packets disappear between the ovpns1 interface and the wan device.
If I capture on the wan side I can see the encrypted packets, but only when I ping within the virtual tunnel.
shell1:
[2.6.0-RELEASE][root@pfSense.sitea.lan]/root: ping -s1400 -c4 10.0.55.2 PING 10.0.55.2 (10.0.55.2): 1400 data bytes 1408 bytes from 10.0.55.2: icmp_seq=0 ttl=64 time=648.205 ms 1408 bytes from 10.0.55.2: icmp_seq=1 ttl=64 time=99.991 ms 1408 bytes from 10.0.55.2: icmp_seq=2 ttl=64 time=106.075 ms 1408 bytes from 10.0.55.2: icmp_seq=3 ttl=64 time=104.884 ms --- 10.0.55.2 ping statistics --- 4 packets transmitted, 4 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 99.991/239.789/648.205/235.810 msshell2:
[2.6.0-RELEASE][root@pfSense.sitea.lan]/root: tcpdump -ni igb0 host 109.118.84.52 # this is site C public IP tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on igb0, link-type EN10MB (Ethernet), capture size 262144 bytes 11:40:28.730600 IP 192.168.1.2.1199 > 109.118.84.52.13981: UDP, length 1452 11:40:28.835884 IP 109.118.84.52.13981 > 192.168.1.2.1199: UDP, length 1452 11:40:29.744901 IP 192.168.1.2.1199 > 109.118.84.52.13981: UDP, length 1452 11:40:29.809449 IP 109.118.84.52.13981 > 192.168.1.2.1199: UDP, length 1452 11:40:30.759282 IP 192.168.1.2.1199 > 109.118.84.52.13981: UDP, length 1452 11:40:30.868418 IP 109.118.84.52.13981 > 192.168.1.2.1199: UDP, length 1452 11:40:31.774335 IP 192.168.1.2.1199 > 109.118.84.52.13981: UDP, length 1452 11:40:31.848450 IP 109.118.84.52.13981 > 192.168.1.2.1199: UDP, length 1452If I try to ping site C LAN address no output is produced in my tcpdump capture, therefore pfsense must be the one dropping the packets, and I have no idea why.
To recap:
src dst packets on tun packets on wan 10.0.55.1 10.0.55.2 yes yes 10.0.55.2 10.0.55.1 yes yes 192.168.34.254 192.168.134.1 yes no 192.168.134.1 192.168.34.254 yes yes Further data
- site A does know the route, so packets are not dropped due to that, or sent to a different device
[2.6.0-RELEASE][root@pfSense.sitea.lan]/root: netstat -r | grep 134 192.168.134.0/24 10.0.55.2 UGS ovpns1- the route 'Uses' counter in the web ui goes up as expected (e.g. 5 pings yield +5 uses), so the route is both known and used
- those missing icmps cannot be seen on any other interface via tcpdump
- pfsense is not firewalling egress because I have a single allow all rule for openvpn
- I have confirmed the allow all rule is triggered by the disappearing pings by turning on logging
- the missing traffic does not trigger any block rule
- I have tried to no avail disabling hardware checksum offloading as described here
At this point I'm really out of ideas, there must be something I'm overlooking!
-
@4632215
Edit the server settings and state a /30 tunnel mask. -
@viragomann ...and it worked[*]! But why? My two pfsense devices (unnecessarily) use a /24 tunnel subnet and need no additional configuration.
[*] the teltonika client was not setting its own IP so I had to add a
push "ifconfig 10.0.55.2 10.0.55.1"and apush "route-gateway 10.0.55.1"directives -
@4632215
If you have a /30 tunnel there can only be two IPs inside, one is the server, the other one is the client. So all is clear.If the tunnel network is bigger, there can be one server and multiple clients inside. So you have to tell the server, behind which client IP he can find the desired network, you want to send packets to. This can be done by the iroute directive in OpenVPN.
In pfSense you have to create a client specific override to set this, where you have to state the client sides remote network.
But if you only have one client anyway, you can spare this and easily set the tunnel mask to /30.