Found a quirk w/ pfSense on EC2.. Hope this helps someone else
It appears that the DNS servers assigned by DHCP in EC2 get an entry in the route table that puts them on the local interface (the mac address) as shown by 'netstat -r'. These servers are in a different subnet than pfSense and as such are accessible by the default route. Bad behavior only manifested itself when routing traffic through an IPSEC tunnel terminated on the EC2 end by pfSense. Any traffic destined for those DNS server IP addresses (which, in my case, also happen to be Active Directory DCs) coming over the IPSEC tunnel never exits the single interface of the EC2 instance. Instead, all I see is pfSense ARP requests for the IP of DNS server.
Local access is fine; pfSense itself can ping and otherwise access the DNS servers in question; it's only traffic coming from the other side of the tunnel that has an issue.
Fix is to delete those routes. After that, pfSense routes traffic from the other side of the tunnel to those specific IPs just fine. For a fix that survives a reboot, manually assign the DNS servers and assign them the default route in the pfSense GUI.
IMO this seems like a bug.
If there's a better way to report this, let me know.
Hmm, so the DNS servers supplied are in a different subnet?
Can we see any example netstat output?
@stephenw10 Sure. Here's the current state of a second one I just deployed in another region that has the same VPC layout:
Output of of
Internet: Destination Gateway Flags Netif Expire default 10.251.253.33 UGS xn0 10.251.251.67 06:06:92:93:60:92 UHS xn0 10.251.251.252 06:06:92:93:60:92 UHS xn0 10.251.253.32/27 link#5 U xn0 10.251.253.55 link#5 UHS lo0 10.252.252.116 10.251.253.33 UGHS xn0 10.252.252.245 10.251.253.33 UGHS xn0 22.214.171.124 10.251.253.33 UGHS xn0 localhost link#2 UH lo0 172.19.0.1 link#2 UH lo0
10.251.251.67 and 10.251.251.252 are the DNS servers.
xn0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=3<RXCSUM,TXCSUM> ether 06:06:92:93:60:92 hwaddr 06:06:92:93:60:92 inet6 fe80::406:92ff:fe93:6092%xn0 prefixlen 64 scopeid 0x5 inet 10.251.253.55 netmask 0xffffffe0 broadcast 10.251.253.63 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> media: Ethernet manual status: active
And here's what the routes look like after manually assiging the DNS servers in the GUI, assigning them the default gateway, and unchecking "Allow DNS server list to be overridden by DHCP/PPP on WAN"
Internet: Destination Gateway Flags Netif Expire default 10.251.253.33 UGS xn0 10.251.251.67 10.251.253.33 UGHS xn0 10.251.251.252 10.251.253.33 UGHS xn0 10.251.253.32/27 link#5 U xn0 10.251.253.55 link#5 UHS lo0 10.252.252.245 10.251.253.33 UGHS xn0 126.96.36.199 10.251.253.33 UGHS xn0 localhost link#2 UH lo0 172.19.0.1 link#2 UH lo0
Now traffic from the other side of an IPSEC tunnel can reach the DNS server IP addresses.