Found a quirk w/ pfSense on EC2.. Hope this helps someone else

[whosmatt]

It appears that the DNS servers assigned by DHCP in EC2 get an entry in the route table that puts them on the local interface (the mac address) as shown by 'netstat -r'. These servers are in a different subnet than pfSense and as such are accessible by the default route. Bad behavior only manifested itself when routing traffic through an IPSEC tunnel terminated on the EC2 end by pfSense. Any traffic destined for those DNS server IP addresses (which, in my case, also happen to be Active Directory DCs) coming over the IPSEC tunnel never exits the single interface of the EC2 instance. Instead, all I see is pfSense ARP requests for the IP of DNS server.

Local access is fine; pfSense itself can ping and otherwise access the DNS servers in question; it's only traffic coming from the other side of the tunnel that has an issue.

Fix is to delete those routes. After that, pfSense routes traffic from the other side of the tunnel to those specific IPs just fine. For a fix that survives a reboot, manually assign the DNS servers and assign them the default route in the pfSense GUI.

IMO this seems like a bug.

If there's a better way to report this, let me know.

[stephenw10]

Hmm, so the DNS servers supplied are in a different subnet?

Can we see any example netstat output?

Steve

[whosmatt]

@stephenw10 Sure. Here's the current state of a second one I just deployed in another region that has the same VPC layout:

Output of of netstat -r

Internet:
Destination        Gateway            Flags     Netif Expire
default            10.251.253.33      UGS         xn0
10.251.251.67      06:06:92:93:60:92  UHS         xn0
10.251.251.252     06:06:92:93:60:92  UHS         xn0
10.251.253.32/27   link#5             U           xn0
10.251.253.55      link#5             UHS         lo0
10.252.252.116     10.251.253.33      UGHS        xn0
10.252.252.245     10.251.253.33      UGHS        xn0
104.43.216.101     10.251.253.33      UGHS        xn0
localhost          link#2             UH          lo0
172.19.0.1         link#2             UH          lo0

10.251.251.67 and 10.251.251.252 are the DNS servers.
For reference:

xn0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=3<RXCSUM,TXCSUM>
        ether 06:06:92:93:60:92
        hwaddr 06:06:92:93:60:92
        inet6 fe80::406:92ff:fe93:6092%xn0 prefixlen 64 scopeid 0x5
        inet 10.251.253.55 netmask 0xffffffe0 broadcast 10.251.253.63
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet manual
        status: active

[whosmatt]

And here's what the routes look like after manually assiging the DNS servers in the GUI, assigning them the default gateway, and unchecking "Allow DNS server list to be overridden by DHCP/PPP on WAN"

Internet:
Destination        Gateway            Flags     Netif Expire
default            10.251.253.33      UGS         xn0
10.251.251.67      10.251.253.33      UGHS        xn0
10.251.251.252     10.251.253.33      UGHS        xn0
10.251.253.32/27   link#5             U           xn0
10.251.253.55      link#5             UHS         lo0
10.252.252.245     10.251.253.33      UGHS        xn0
104.43.216.101     10.251.253.33      UGHS        xn0
localhost          link#2             UH          lo0
172.19.0.1         link#2             UH          lo0

Now traffic from the other side of an IPSEC tunnel can reach the DNS server IP addresses.