DNS forwarder and resolution stopped working?
I'm running pfSense 2.1-RELEASE with a single LAN and two WANs. I have DNS forwarder enabled on the pfSense and DNS servers configured as follows (in System –> General):
ISP1 Primary DNS - Use WAN1 gateway
ISP2 Primary DNS - Use WAN2 gateway
18.104.22.168 (Google Primary) - no gateway
22.214.171.124 (Google Secondary) - no gateway
For some reason, DNS resolution in my LAN has stopped working… it was on and off for a while, but now it's completely "broken", and I'm not sure why.
I can reach all 4 DNS servers from both inside the LAN and also from the pfSense itself - i.e. they are responding to ICMP ping.
But any domain lookups, whether from my LAN (using pfSense as the DNS resolver) or from the pfSense itself no longer work. I tried disabling the DNS Forwarder as a DNS server for the firewall (again in System --> General) and this still didn't change anything.
Here's the even stranger thing: Even if I set DNS servers on a PC on the LAN to Google public DNS it still fails with a timeout, as if the pfSense is blocking it!
> server 126.96.36.199 DNS request timed out. timeout was 2 seconds. Default Server: [188.8.131.52] Address: 184.108.40.206 > www.google.com Server: [220.127.116.11] Address: 18.104.22.168 DNS request timed out. timeout was 2 seconds. DNS request timed out. timeout was 2 seconds. DNS request timed out. timeout was 2 seconds. DNS request timed out. timeout was 2 seconds. *** Request to [22.214.171.124] timed-out
Ping to the above IP is fine from inside the LAN, as is access to DNS itself from WAN, as I have checked from a server that is running "outside" the pfSense and connected directly to the ADSL modem/router. This is the case for all 4 DNS servers on both WAN connections.
I of course tried the usual, stop/start DNS forwarder, disable it completely, restarted pfSense etc… also, I checked and I'm not blocking (as far as I can tell) DNS IPs/ports in the firewall rules. Nothing has changed there for quite some time...
I'm stumped, any help would be appreciated!!!
And what are you lan rules? You sure your not blocking 53 udp.. If you changed default rule which any to say tcp only dns your going to have problems with dns ;)
LAN rules are fine, I'm not blocking anything outbound.
I should have mentioned - when DNS resolution works via the pfSense forwarder, it also works if I use external nameservers directly from my LAN too. When it stops working via forwarder, it stops working everywhere…
Strange as usual, everything is working fine again... for the time being!
To add to this, since it's happening again… I've confirmed that pfSense itself cannot connect to the DNS servers at all, even if I explicitly set it in "nslookup" in the shell:
[2.1-RELEASE][email@example.com]/root(1): nslookup > server 126.96.36.199 Default server: 188.8.131.52 Address: 184.108.40.206#53 > www.google.com ;; connection timed out; no servers could be reached > ^C [2.1-RELEASE][firstname.lastname@example.org]/root(2): ping 220.127.116.11 PING 18.104.22.168 (22.214.171.124): 56 data bytes 64 bytes from 126.96.36.199: icmp_seq=0 ttl=50 time=25.249 ms 64 bytes from 188.8.131.52: icmp_seq=1 ttl=50 time=25.323 ms 64 bytes from 184.108.40.206: icmp_seq=2 ttl=50 time=25.262 ms ^C --- 220.127.116.11 ping statistics --- 3 packets transmitted, 3 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 25.249/25.278/25.323/0.032 ms
Though you can see the DNS server is reachable via ping (and traceroute) from the same shell session…
From a server sitting just "outside" the pfSense (directly connected to the ADSL modem, and in the same public subnet as pfSense WAN1), everything is fine. So the problem is definitely with pfSense itself...
For now I have disabled WAN2 altogether, to eliminate that as an issue, but it hasn't changed anything...
So question for you on this "modem" is it bridged or is it also doing nat? What is the IP address pfsense gets on its wan? Public or rfc1918?
I would sniff on your wan of pfsense when you do this check you just did above.. What do you see in that? If pfsense sends out the query and doesn't get an answer then its not a pfsense problem - but something between pfsense and the server 18.104.22.168
So example - see attached. I did a packet capture on the wan interface of pfsense, set host to 22.214.171.124, udp and port 53 so I would only see this traffic I was interested in. On pfsense using nslookup with server set to 126.96.36.199 did query – You can see pfsense send out correct/valid query and get a response and its valid. If in your capture you don't see the query go out or its mangled somehow, then sure something wrong with pfsense - if you see the query go out and valid and no response then not pfsense.
Maybe its just slow to respond? Did you up your wait time? Do you see a response in the capture but pfsense didn't take it because it was too late?
WAN1 is a public range, I have a /28 addressable subnet. One of the IPs is statically assigned to the pfSense, one is the modem/router itself (default gateway) and a couple of the remaining ones are assigned to devices sitting "outside" the firewall - these are the ones I tested DNS lookups from whenever I get timeouts from pfSense itself.
WAN2 is slightly different, I get an RFC1918 address, but have a 1:1 NAT set so I can configure port forwarding etc on the pfSense directly. No other devices between that modem/router and the pfSense WAN port.
However, as I mentioned above, I disabled WAN2 altogether last time I saw the issue, and it was still happening after that, all the while DNS queries outside the pfSense were fine.
I'll try your suggestion of running a packet capture on pfSense next time this happens, and will report back…