What am I missing here? Google and CF DNS empty response?
-
I've been having DNS issues for a few days now and looking for some advice since I'm ready to pull my hair out.
Here's some info:
- Satellite internet in a forced double NAT and is not configurable
- ISP allows public DNS use (can confirm, was working fine for years)
- ISP has "Confirmed" there are no issues on their side, or any DNS filtering (not sure if i believe them)
- Simple setup, modem > pfsense (SG-1000) > dumb switch
The Pfsense is using DNS resolver in forward mode. When I dig from the LAN side, I get empty responses from both the pfsense box, and directly from the public dns server
Example (not using pfsense resolver):
dig @8.8.8.8 ubuntu.com Fri Jan 31 11:40:41 2020 ; <<>> DiG 9.10.3-P4-Ubuntu <<>> @8.8.8.8 ubuntu.com ; (1 server found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 58847 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0 ;; QUESTION SECTION: ;ubuntu.com. IN A ;; Query time: 754 msec ;; SERVER: 8.8.8.8#53(8.8.8.8) ;; WHEN: Fri Jan 31 11:40:41 EST 2020 ;; MSG SIZE rcvd: 28
And an example WAN capture:
x.x.x.x.65008 > 8.8.8.8.53: [udp sum ok] 29338+ [1au] A? ubuntu.com. ar: . OPT UDPsize=4096 (39) 21:30:51.130717 00:80:ae:b2:ff:07 > 0c:b2:b7:af:44:37, ethertype IPv4 (0x0800), length 70: (tos 0x0, ttl 63, id 13895, offset 0, flags [none], proto UDP (17), length 56) 8.8.8.8.53 > x.x.x.x.65008: [udp sum ok] 29338 q: A? ubuntu.com. 0/0/0 (28) 21:30:53.146432 0c:b2:b7:af:44:37 > 00:80:ae:b2:ff:07, ethertype IPv4 (0x0800), length 81: (tos 0x0, ttl 63, id 14032, offset 0, flags [none], proto UDP (17), length 67)
This same response happens when using the "DNS lookup" in the PFSense UI, or using DNS Resolver in forward mode.
Other pieces of info that are interesting:
Any consecutive digs against any other DNS server (bypassing pfsense cache) for the same domain name will respond as if cached upstream, 2ms vs the 500-800ms I usually get on this connection. This leads me to believe its the double nat thats messing with me, although I cannot confirm. However, that doesn't explain the initial empty response.
When using DNS over TLS I get 4000ms+ response times, so I typically get timeouts before a valid response returns. The good news is that I dont think I get any empty responses. I'm curious if the delay on those requests are due to the SG-1000 performance, or if its something else? If upgrading the hardware to quickly resolve DNS that way is the fix, I'm fine with that!
Any calls with the ISP to try and troubleshoot together or ask questions about their hardware just ends in "we dont support netgate devices nor pfsense firewalls....very frustrating
Any feedback or ideas would be awesome!
-
System Advanced Networking , try to disable
Hardware Checksum Offloading / Hardware TCP Segmentation Offloading and Hardware Large Receive Offloading
or there is a bad connection/misconfiguration between the router/modem and the firewall -
Satellite Internet, due to the nature of the signal path, is going to already have a very high latency as compared to land-based Internet technologies. Negotiating a TLS connection over that slow pathway is going to take a while due to the inherent link latency. I seriously doubt the SG-1000 hardware has anything to do with the performance issue.
My guess is your ISP is in fact perhaps redirecting DNS lookups to their own server (most likely to take advantage of local caching on their end of the link).
As a test to see if their statement about allowing public DNS servers is true, reconfigure pfSense to use unbound in resolver mode (not forwarder), restart unbound or reboot your pfSense box, and then attempt DNS lookups. Also drop the use of DNS over TLS for now. Just test simple DNS lookups. If those don't succeed either, then that would point to your ISP not really allowing true public DNS settings.
You also state you are in a forced double-NAT situation. I assume by that you mean you must use the ISP-supplied satellite modem in your setup. Maybe the ISP's modem is doing something funky with DNS?
-
@kiokoman Thanks for the reply. I've disabled all the above mentioned offloading already after reading a few posts here about the sg-1000 performance.
-
@bmeeks said in What am I missing here? Google and CF DNS empty response?:
Satellite Internet, due to the nature of the signal path, is going to already have a very high latency as compared to land-based Internet technologies. Negotiating a TLS connection over that slow pathway is going to take a while due to the inherent link latency. I seriously doubt the SG-1000 hardware has anything to do with the performance issue.
My guess is your ISP is in fact perhaps redirecting DNS lookups to their own server (most likely to take advantage of local caching on their end of the link).
As a test to see if their statement about allowing public DNS servers is true, reconfigure pfSense to use unbound in resolver mode (not forwarder), restart unbound or reboot your pfSense box, and then attempt DNS lookups. Also drop the use of DNS over TLS for now. Just test simple DNS lookups. If those don't succeed either, then that would point to your ISP not really allowing true public DNS settings.
You also state you are in a forced double-NAT situation. I assume by that you mean you must use the ISP-supplied satellite modem in your setup. Maybe the ISP's modem is doing something funky with DNS?
Thanks for the reply! I realize satellite internet is by nature high latency, however for the last 2-3 years we have been managing to get decent DNS response times in the 500-800ms range. Its just a recent issue where I'm getting these empty responses.
I should also note I've used multiple routers and laptops to troubleshoot this issue initially, but I keep coming back to PFSense since I'm most comfortable with it for troubleshooting network issues.
I tried using unbound without forwarding, restarted the service (stopped and started), I don't see any indication of errors or problems in the unbound startup log.
I see pf pulling the root server records as expected, then still get:
x.x.x.x.48431 > 199.7.91.13.53: [udp sum ok] 35213% [1au] A? ubuntu.com. ar: . OPT UDPsize=4096 DO (39) 00:24:10.931919 00:80:ae:b2:ff:07 > 0c:b2:b7:af:44:37, ethertype IPv4 (0x0800), length 70: (tos 0x0, ttl 64, id 16579, offset 0, flags [none], proto UDP (17), length 56) 199.7.91.13.53 > x.x.x.x.48431: [udp sum ok] 35213 q: A? ubuntu.com. 0/0/0 (28) 00:24:10.932759 0c:b2:b7:af:44:37 > 00:80:ae:b2:ff:07, ethertype IPv4 (0x0800), length 81: (tos 0x0, ttl 64, id 19857, offset 0, flags [none], proto UDP (17), length 67)
root@server:~# dig ubuntu.com ; <<>> DiG 9.10.3-P4-Ubuntu <<>> ubuntu.com ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 35641 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4096 ;; QUESTION SECTION: ;ubuntu.com. IN A ;; Query time: 12 msec ;; SERVER: 192.168.12.1#53(192.168.12.1) ;; WHEN: Fri Jan 31 14:24:22 EST 2020 ;; MSG SIZE rcvd: 39
I've also set up a virtual environment with the same configuration (same DNS config at least) so I can test at another location with a different internet connection and all works as expected.
Considering this all appears like unencrypted DNS is being manipulated upstream, do you have any suggestions to get around it aside from the DNS over TLS that has a 4000ms response?
thanks!
-
@casperette said in What am I missing here? Google and CF DNS empty response?:
@bmeeks said in What am I missing here? Google and CF DNS empty response?:
Satellite Internet, due to the nature of the signal path, is going to already have a very high latency as compared to land-based Internet technologies. Negotiating a TLS connection over that slow pathway is going to take a while due to the inherent link latency. I seriously doubt the SG-1000 hardware has anything to do with the performance issue.
My guess is your ISP is in fact perhaps redirecting DNS lookups to their own server (most likely to take advantage of local caching on their end of the link).
As a test to see if their statement about allowing public DNS servers is true, reconfigure pfSense to use unbound in resolver mode (not forwarder), restart unbound or reboot your pfSense box, and then attempt DNS lookups. Also drop the use of DNS over TLS for now. Just test simple DNS lookups. If those don't succeed either, then that would point to your ISP not really allowing true public DNS settings.
You also state you are in a forced double-NAT situation. I assume by that you mean you must use the ISP-supplied satellite modem in your setup. Maybe the ISP's modem is doing something funky with DNS?
Thanks for the reply! I realize satellite internet is by nature high latency, however for the last 2-3 years we have been managing to get decent DNS response times in the 500-800ms range. Its just a recent issue where I'm getting these empty responses.
I should also note I've used multiple routers and laptops to troubleshoot this issue initially, but I keep coming back to PFSense since I'm most comfortable with it for troubleshooting network issues.
I tried using unbound without forwarding, restarted the service (stopped and started), I don't see any indication of errors or problems in the unbound startup log.
I see pf pulling the root server records as expected, then still get:
x.x.x.x.48431 > 199.7.91.13.53: [udp sum ok] 35213% [1au] A? ubuntu.com. ar: . OPT UDPsize=4096 DO (39) 00:24:10.931919 00:80:ae:b2:ff:07 > 0c:b2:b7:af:44:37, ethertype IPv4 (0x0800), length 70: (tos 0x0, ttl 64, id 16579, offset 0, flags [none], proto UDP (17), length 56) 199.7.91.13.53 > x.x.x.x.48431: [udp sum ok] 35213 q: A? ubuntu.com. 0/0/0 (28) 00:24:10.932759 0c:b2:b7:af:44:37 > 00:80:ae:b2:ff:07, ethertype IPv4 (0x0800), length 81: (tos 0x0, ttl 64, id 19857, offset 0, flags [none], proto UDP (17), length 67)
root@server:~# dig ubuntu.com ; <<>> DiG 9.10.3-P4-Ubuntu <<>> ubuntu.com ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 35641 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4096 ;; QUESTION SECTION: ;ubuntu.com. IN A ;; Query time: 12 msec ;; SERVER: 192.168.12.1#53(192.168.12.1) ;; WHEN: Fri Jan 31 14:24:22 EST 2020 ;; MSG SIZE rcvd: 39
I've also set up a virtual environment with the same configuration (same DNS config at least) so I can test at another location with a different internet connection and all works as expected.
Considering this all appears like unencrypted DNS is being manipulated upstream, do you have any suggestions to get around it aside from the DNS over TLS that has a 4000ms response?
thanks!
Sorry, but no I don't. When providers manipulate DNS you are somewhat screwed. Sure you can sometimes switch to something like DNS over TLS or even DNS over HTTPS (DOH), but that has its own hazards over certain types of links.
I assume the answer is "no", but do you maybe have a choice of another terrestrial Internet service provider?
Are things OK when you use the DNS server suggested or provided by the ISP? If so, you might just be stuck with using their configuration.
-
@bmeeks said in What am I missing here? Google and CF DNS empty response?:
@casperette said in What am I missing here? Google and CF DNS empty response?:
@bmeeks said in What am I missing here? Google and CF DNS empty response?:
Satellite Internet, due to the nature of the signal path, is going to already have a very high latency as compared to land-based Internet technologies. Negotiating a TLS connection over that slow pathway is going to take a while due to the inherent link latency. I seriously doubt the SG-1000 hardware has anything to do with the performance issue.
My guess is your ISP is in fact perhaps redirecting DNS lookups to their own server (most likely to take advantage of local caching on their end of the link).
As a test to see if their statement about allowing public DNS servers is true, reconfigure pfSense to use unbound in resolver mode (not forwarder), restart unbound or reboot your pfSense box, and then attempt DNS lookups. Also drop the use of DNS over TLS for now. Just test simple DNS lookups. If those don't succeed either, then that would point to your ISP not really allowing true public DNS settings.
You also state you are in a forced double-NAT situation. I assume by that you mean you must use the ISP-supplied satellite modem in your setup. Maybe the ISP's modem is doing something funky with DNS?
Thanks for the reply! I realize satellite internet is by nature high latency, however for the last 2-3 years we have been managing to get decent DNS response times in the 500-800ms range. Its just a recent issue where I'm getting these empty responses.
I should also note I've used multiple routers and laptops to troubleshoot this issue initially, but I keep coming back to PFSense since I'm most comfortable with it for troubleshooting network issues.
I tried using unbound without forwarding, restarted the service (stopped and started), I don't see any indication of errors or problems in the unbound startup log.
I see pf pulling the root server records as expected, then still get:
x.x.x.x.48431 > 199.7.91.13.53: [udp sum ok] 35213% [1au] A? ubuntu.com. ar: . OPT UDPsize=4096 DO (39) 00:24:10.931919 00:80:ae:b2:ff:07 > 0c:b2:b7:af:44:37, ethertype IPv4 (0x0800), length 70: (tos 0x0, ttl 64, id 16579, offset 0, flags [none], proto UDP (17), length 56) 199.7.91.13.53 > x.x.x.x.48431: [udp sum ok] 35213 q: A? ubuntu.com. 0/0/0 (28) 00:24:10.932759 0c:b2:b7:af:44:37 > 00:80:ae:b2:ff:07, ethertype IPv4 (0x0800), length 81: (tos 0x0, ttl 64, id 19857, offset 0, flags [none], proto UDP (17), length 67)
root@server:~# dig ubuntu.com ; <<>> DiG 9.10.3-P4-Ubuntu <<>> ubuntu.com ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 35641 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4096 ;; QUESTION SECTION: ;ubuntu.com. IN A ;; Query time: 12 msec ;; SERVER: 192.168.12.1#53(192.168.12.1) ;; WHEN: Fri Jan 31 14:24:22 EST 2020 ;; MSG SIZE rcvd: 39
I've also set up a virtual environment with the same configuration (same DNS config at least) so I can test at another location with a different internet connection and all works as expected.
Considering this all appears like unencrypted DNS is being manipulated upstream, do you have any suggestions to get around it aside from the DNS over TLS that has a 4000ms response?
thanks!
Sorry, but no I don't. When providers manipulate DNS you are somewhat screwed. Sure you can sometimes switch to something like DNS over TLS or even DNS over HTTPS (DOH), but that has its own hazards over certain types of links.
I assume the answer is "no", but do you maybe have a choice of another terrestrial Internet service provider?
Are things OK when you use the DNS server suggested or provided by the ISP? If so, you might just be stuck with using their configuration.
There are other resellers, but I don't believe there's another distinct provider.
I am currently providing DNS over an IPSec VPN to another physical location, and it appears to be stable. I cant remember if I tried the default DNS servers I get over DHCP, but this whole incident has me not wanting to use it out of principle. I'll probably test with other hardware this weekend and see what happens.
Thanks for your help!