nslookup: Got SERVFAIL reply from 127.0.0.1
-
I have a box having some weird DNS issues. Not sure when they started but I believe that it was either over the weekend or today. Resolution just fails sporadically. I've tried running unbound as both resolver and forwarder but they both have the same problems. There is no Suricata, Snort, or pfBlocker on this network to interfere. Running nslookup on the pfSense server itself gives a variety of answers. Here are 3 attempts I tried back to back within a span of a minute:
[root@fw.site1]/: nslookup anandtech.com Server: 127.0.0.1 Address: 127.0.0.1#53 Non-authoritative answer: ;; Got SERVFAIL reply from 127.0.0.1, trying next server *** Can't find anandtech.com: No answer [root@fw.site1]/: nslookup anandtech.com ;; Got SERVFAIL reply from 127.0.0.1, trying next server Server: 9.9.9.9 Address: 9.9.9.9#53 Non-authoritative answer: Name: anandtech.com Address: 192.65.241.100 ;; Got SERVFAIL reply from 127.0.0.1, trying next server [root@fw.site1]/: nslookup anandtech.com Server: 127.0.0.1 Address: 127.0.0.1#53 Non-authoritative answer: Name: anandtech.com Address: 192.65.241.100 ;; Got SERVFAIL reply from 127.0.0.1, trying next serverHere's the result of dig:
[root@fw.site1]/: dig anandtech.com ; <<>> DiG 9.14.12 <<>> anandtech.com ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 120 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4096 ;; QUESTION SECTION: ;anandtech.com. IN A ;; Query time: 684 msec ;; SERVER: 127.0.0.1#53(127.0.0.1) ;; WHEN: Mon Sep 27 13:36:40 EDT 2021 ;; MSG SIZE rcvd: 42Dig +trace seems to work OK
[root@fw.site1]/: dig anandtech.com +trace ; <<>> DiG 9.14.12 <<>> anandtech.com +trace ;; global options: +cmd . 78685 IN NS m.root-servers.net. . 78685 IN NS b.root-servers.net. . 78685 IN NS c.root-servers.net. . 78685 IN NS d.root-servers.net. . 78685 IN NS e.root-servers.net. . 78685 IN NS f.root-servers.net. . 78685 IN NS g.root-servers.net. . 78685 IN NS h.root-servers.net. . 78685 IN NS i.root-servers.net. . 78685 IN NS a.root-servers.net. . 78685 IN NS j.root-servers.net. . 78685 IN NS k.root-servers.net. . 78685 IN NS l.root-servers.net. . 78685 IN RRSIG NS 8 0 518400 20211010050000 20210927040000 26838 . FuhPJ1858hCoSQPXHEiODBaykNCm7q76FsRcwNsPtxNWpWBvEuSdCvMs iZKykxzCOCZ/cUFBlzbE8EJQDYBgEGhzfg1tmwVLsqK5o6JgFi/J9K5E xehdqRLJ7DiCv683DxznsKBUaRww/btYiZwrMwL6hWSYq++H8kbyhFlO 80Qx+nxoD/W94ZZbq0vf8ifS9laScDm4iNOrcUYvMpP8Vha6h9Ufn53H ndN19K48Xo9ZPlITKlHEx7b3raehJfikzq90hr9VISnGofsN2ASNqfFl HdJldZrHOfPrCLgQgVr08akSv+sIUit/qHOaTLdTziUqLq0iSsXuObiK OJMinA== ;; Received 525 bytes from 127.0.0.1#53(127.0.0.1) in 0 ms com. 172800 IN NS a.gtld-servers.net. com. 172800 IN NS b.gtld-servers.net. com. 172800 IN NS c.gtld-servers.net. com. 172800 IN NS d.gtld-servers.net. com. 172800 IN NS e.gtld-servers.net. com. 172800 IN NS f.gtld-servers.net. com. 172800 IN NS g.gtld-servers.net. com. 172800 IN NS h.gtld-servers.net. com. 172800 IN NS i.gtld-servers.net. com. 172800 IN NS j.gtld-servers.net. com. 172800 IN NS k.gtld-servers.net. com. 172800 IN NS l.gtld-servers.net. com. 172800 IN NS m.gtld-servers.net. com. 86400 IN DS 30909 8 2 E2D3C916F6DEEAC73294E8268FB5885044A833FC5459588F4A9184CF C41A5766 com. 86400 IN RRSIG DS 8 1 86400 20211010170000 20210927160000 26838 . b2NXDWIkyTYvEJ4IL5doWmrLmBKg8EnFDcaEbUM79m2+oSEQYRxzhfUM OFQYEGBamIe9r63+I2py3J3BCstahP667q82dvMWwOaP4JB6T+S3LU8I Wrihmr3FKMIubGZvm5Aju+Ep2T+HyB9ek2lSS2XUyQGe0qoJ77q4PbCr KzTF9YlXkpSw065WaloRpMvTgPrd2NLQhmkzVPrybYYOlP94eazdWi3S headYqVi7b8tBCk+8t7Zuh11LOThs7NN7zwPzsBdyyz1y0/xivdLvSt8 p/BwNWJw95uRsneQAoHMqWLkwpAnC5RgYebntkLClVyKnahgyT/4waoC iBYakw== ;; Received 1173 bytes from 192.58.128.30#53(j.root-servers.net) in 81 ms anandtech.com. 172800 IN NS ns-534.awsdns-02.net. anandtech.com. 172800 IN NS ns-334.awsdns-41.com. anandtech.com. 172800 IN NS ns-1685.awsdns-18.co.uk. anandtech.com. 172800 IN NS ns-1496.awsdns-59.org. CK0POJMG874LJREF7EFN8430QVIT8BSM.com. 86400 IN NSEC3 1 1 0 - CK0Q1GIN43N1ARRC9OSM6QPQR81H5M9A NS SOA RRSIG DNSKEY NSEC3PARAM CK0POJMG874LJREF7EFN8430QVIT8BSM.com. 86400 IN RRSIG NSEC3 8 2 86400 20211003042352 20210926031352 39343 com. fdKztlvTFtfJm793/ER3F5+kzv7I1SUP3+JqqHCbk5HotOwsZeqzU4U4 DqkanSWZkzl2rC4qL8Q02oo6F53HSZgaVXEzCAcRRj2gRIlhCsL+0iQx X7SLPer7CVmSf1Fm9DL82E1Y4RBemd8K9+8xeRd+VoPA0x+0Z/afPO7B Em6wmLHmxBoYEdG6HAuWvLnNresmCnk8YvKPsc3wElk2yQ== 4DD4IAAB12F1OJ1HKA22OV8V5IQ1QFEO.com. 86400 IN NSEC3 1 1 0 - 4DD4NKQBPBN71MOSJEK6GCJQBI4CS34R NS DS RRSIG 4DD4IAAB12F1OJ1HKA22OV8V5IQ1QFEO.com. 86400 IN RRSIG NSEC3 8 2 86400 20211004052426 20210927041426 39343 com. uighLoCULJWTKiaWwaEmHaNZL/3mN+URLaUX/TSD7a24mPv5I74A8daO E9gx1T2h5azp8lkWdoYNdiiKRTIidOvoAxJhOxNcMpCNYc2kkuRePPcd e+ifiXmYH4Fwy0nL51BwSZBQSP9fbGPirjHNOugm85EblMG0xztEQIzC GBACwCghqiIEnPiFLe4UlSYLcR0pb+JtpZPz+PwOaC95ag== ;; Received 744 bytes from 192.26.92.30#53(c.gtld-servers.net) in 47 ms anandtech.com. 300 IN A 192.65.241.100 anandtech.com. 172800 IN NS ns-1496.awsdns-59.org. anandtech.com. 172800 IN NS ns-1685.awsdns-18.co.uk. anandtech.com. 172800 IN NS ns-334.awsdns-41.com. anandtech.com. 172800 IN NS ns-534.awsdns-02.net. ;; Received 195 bytes from 205.251.194.22#53(ns-534.awsdns-02.net) in 28 msAny idea where to start troubleshooting? It's been in place about 2 years and the problem just suddenly started happening.
-
@stewart said in nslookup: Got SERVFAIL reply from 127.0.0.1:
DiG 9.14.12
What version of pfsense are you running..
My dig on my pfsense is 9.16.16
[21.05.1-RELEASE][admin@sg4860.local.lan]/root: dig ; <<>> DiG 9.16.16 <<>>it's been in place about 2 years
Have you not updated this site in like 2 years?
-
2.4.4-Release. I thought I had all of them at 2.4.5-p1 (and some at 2.5.2) but I must have missed this one.
-
@stewart what does the unbound log say - is it restarting a lot?
I have some boxes on older 2.4.4p3 - because of covid and no access to the site they have not been updated as of yet. But have had no reports of any issues.. And only dns they can use is the resolver..
You could up your logging level, and even log queries and responses - this might give you some clue to what is going on.
-
@johnpoz Initially I thought it was restarting but the logs don't show that. I've upped the logging level to Level 3: Query. I'll post back what I find.
-
@stewart sometimes adding these to your options box can help in troubleshooting
log-queries: yes log-replies: yesThey would go under the server: line in your options box
-
@johnpoz This is a small snippet:
Sep 27 15:22:08 unbound 17240:1 info: processQueryTargets: ns-1685.awsdns-18.co.uk. A IN Sep 27 15:22:08 unbound 17240:1 info: error sending query to auth server 2600:9000:5301:5200::1 port 53 Sep 27 15:22:08 unbound 17240:1 debug: Need to send query but have no outgoing interfaces of that family Sep 27 15:22:08 unbound 17240:1 debug: sending to target: <awsdns-18.co.uk.> 2600:9000:5301:5200::1#53 Sep 27 15:22:08 unbound 17240:1 info: sending query: ns-1685.awsdns-18.co.uk. A IN Sep 27 15:22:08 unbound 17240:1 info: processQueryTargets: ns-1685.awsdns-18.co.uk. A IN Sep 27 15:22:08 unbound 17240:1 info: query response was nodata ANSWER Sep 27 15:22:08 unbound 17240:1 info: reply from <awsdns-18.co.uk.> 205.251.193.82#53 Sep 27 15:22:08 unbound 17240:1 info: response for ns-1685.awsdns-18.co.uk. A IN Sep 27 15:22:08 unbound 17240:1 debug: sanitize: removing public name with private address <ns-1685.awsdns-18.co.uk.> 192.168.1.254#53 Sep 27 15:22:08 unbound 17240:1 info: iterator operate: query ns-1685.awsdns-18.co.uk. A IN Sep 27 15:22:08 unbound 17240:1 debug: iterator[module 0] operate: extstate:module_wait_reply event:module_event_reply Sep 27 15:22:08 unbound 17240:1 debug: cache memory msg=126880 rrset=207186 infra=120437 val=0 Sep 27 15:22:08 unbound 17240:1 info: finishing processing for anandtech.com. A IN Sep 27 15:22:08 unbound 17240:1 info: query response was nodata ANSWER Sep 27 15:22:08 unbound 17240:1 info: reply from <anandtech.com.> 205.251.194.22#53 Sep 27 15:22:08 unbound 17240:1 info: response for anandtech.com. A IN Sep 27 15:22:08 unbound 17240:1 debug: sanitize: removing public name with private address <anandtech.com.> 192.168.1.254#53 Sep 27 15:22:08 unbound 17240:1 info: iterator operate: query anandtech.com. A IN Sep 27 15:22:08 unbound 17240:1 debug: iterator[module 0] operate: extstate:module_wait_reply event:module_event_reply Sep 27 15:22:08 unbound 17240:1 debug: cache memory msg=126880 rrset=207186 infra=120437 val=0 Sep 27 15:22:08 unbound 17240:1 debug: sending to target: <com.> 192.35.51.30#53 Sep 27 15:22:08 unbound 17240:1 info: sending query: awsdns-41.com. A INBut then seeing 192.168.1.254 gave me an idea. While trying to get the failover between WAN (Cable Modem) and OPT1WAN2 (DSL) working I entered this information:
While one of my guys was on site today we found that the AT&T Modem was on but had no upstream link (surprise, surprise). I'm guessing that even though Forwarding is turned off it's still querying those servers over AT&T?
-
@stewart said in nslookup: Got SERVFAIL reply from 127.0.0.1:
I'm guessing that even though Forwarding is turned off it's still querying those servers over AT&T?
So you got it sorted? Yeah a link being actually down, but pfsense thinking it up could cause issues if your telling to pfsense to query via a specific gateway.
I personally only ever resolve.. So I have no dns set at all.. I can not try and duplicate the problem either because I only have 1 internet connection.. I could simulate multiple connections and see what happens.
But I take it you have it working again?
-
@johnpoz Right now I have them routed directly to upstream via DHCP to bypass Unbound entirely. I'll need to work on it in the morning. I thought if those were in there they only got used IF forwarding is set up. Apparently it affects even if set to resolve.
-
@stewart it shouldn't - resolver would have nothing to do with those.
