Traffic blocked by default deny rule (1000000103) after Starlink reboots.

hpeyerl

Setup is fairly simple. I have a LAN port on re1 and re0.300 is connected to a Cisco switch with VLAN trunking to another Cisco switch that my Starlink router is connected to on VLAN300. This setup has been working for years and years.

Recently, it seems that when my Starlink dish reboots, pfsense blocks return traffic with the default deny rule, even though there are applicable rules to pass the traffic.

I confirmed this using tcpdump on both the LAN interface and WAN interface.

From a LAN host when this problem is happening, I try to ping some host on the internet.

tcpdump on the LAN interface:

[2.8.0-RELEASE][root@dr-pfSense.beer.org]/root: tcpdump -ni re1 icmp
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on re1, link-type EN10MB (Ethernet), snapshot length 262144 bytes
08:24:11.730355 IP 192.168.34.89 > 8.8.8.8: ICMP echo request, id 32480, seq 102, length 64
08:24:12.735497 IP 192.168.34.89 > 8.8.8.8: ICMP echo request, id 32480, seq 103, length 64
08:24:13.736581 IP 192.168.34.89 > 8.8.8.8: ICMP echo request, id 32480, seq 104, length 64
08:24:14.739492 IP 192.168.34.89 > 8.8.8.8: ICMP echo request, id 32480, seq 105, length 64
08:24:15.744011 IP 192.168.34.89 > 8.8.8.8: ICMP echo request, id 32480, seq 106, length 64
08:24:16.749043 IP 192.168.34.89 > 8.8.8.8: ICMP echo request, id 32480, seq 107, length 64
08:24:17.751250 IP 192.168.34.89 > 8.8.8.8: ICMP echo request, id 32480, seq 108, length 64
08:24:18.753267 IP 192.168.34.89 > 8.8.8.8: ICMP echo request, id 32480, seq 109, length 64
08:24:19.754864 IP 192.168.34.89 > 8.8.8.8: ICMP echo request, id 32480, seq 110, length 64

tcpdump on the WAN interface:

[2.8.0-RELEASE][root@dr-pfSense.beer.org]/root: tcpdump -ni re0.300 icmp
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on re0.300, link-type EN10MB (Ethernet), snapshot length 262144 bytes
08:24:09.720159 IP 100.99.170.112 > 8.8.8.8: ICMP echo request, id 32480, seq 100, length 64
08:24:09.769300 IP 8.8.8.8 > 100.99.170.112: ICMP echo reply, id 32480, seq 100, length 64
08:24:10.725245 IP 100.99.170.112 > 8.8.8.8: ICMP echo request, id 32480, seq 101, length 64
08:24:10.769296 IP 8.8.8.8 > 100.99.170.112: ICMP echo reply, id 32480, seq 101, length 64
08:24:11.730377 IP 100.99.170.112 > 8.8.8.8: ICMP echo request, id 32480, seq 102, length 64
08:24:11.769242 IP 8.8.8.8 > 100.99.170.112: ICMP echo reply, id 32480, seq 102, length 64
08:24:12.735517 IP 100.99.170.112 > 8.8.8.8: ICMP echo request, id 32480, seq 103, length 64
08:24:12.774872 IP 8.8.8.8 > 100.99.170.112: ICMP echo reply, id 32480, seq 103, length 64
08:24:13.736602 IP 100.99.170.112 > 8.8.8.8: ICMP echo request, id 32480, seq 104, length 64
08:24:13.770863 IP 8.8.8.8 > 100.99.170.112: ICMP echo reply, id 32480, seq 104, length 64
08:24:14.739522 IP 100.99.170.112 > 8.8.8.8: ICMP echo request, id 32480, seq 105, length 64
08:24:14.778897 IP 8.8.8.8 > 100.99.170.112: ICMP echo reply, id 32480, seq 105, length 64
08:24:15.744036 IP 100.99.170.112 > 8.8.8.8: ICMP echo request, id 32480, seq 106, length 64
08:24:15.782821 IP 8.8.8.8 > 100.99.170.112: ICMP echo reply, id 32480, seq 106, length 64
08:24:16.749063 IP 100.99.170.112 > 8.8.8.8: ICMP echo request, id 32480, seq 107, length 64
08:24:16.786837 IP 8.8.8.8 > 100.99.170.112: ICMP echo reply, id 32480, seq 107, length 64

Now, conspicuously, at least one LAN host is able to get packets through. Oddly, it's an LXC container on my Proxmox server. The Proxmox server can't get ping replies but the LXC container running on it CAN get ping replies. Most other hosts on the network are unable to get ping replies.

Note, that I'm just using 'ping' to debug this. The problem happens regardless of transport (ie: TCP4). Also, I'm just pinging 8.8.8.8 out of convenience but it happens to other known IP's out on the internet).

I can 'fix' the problem by doing:

ifconfig re0.300 down; ifconfig re0.300 up

and then everything continues to work as usual, until the next time my Starlink dish is rebooted.

I understand that people are going to suggest that the problem the FreeBSD 're' driver, etc but L2 is working fine and has been working fine for ages and I don't think this is related to the network drivers.

I also have the suggested 'Reject leases from' for Starlink handing out bogus leases while it's rebooting, etc.

For now I'm mitigating the problem with a cron job on one of my LAN hosts that checks for this situation and does the 'ifconfig down up' dance via an ssh connection, but I'd like to figure out what the real cause is.