AT&T blocking outbound DNS Resolver traffic
-
This is mainly an AT&T issue, but am hoping someone in the community might have encountered and overcome it.
Summary: the AT&T residential gateway on the WAN side of pfSense seems to think the outbound traffic from DNS Resolver is some kind of attack, and it responds by periodically shutting down internet access for 1 to 5 minutes.
Detail: I have AT&T gigabit fiber feeding an Arris NVG589 residential gateway, which is set to pass the public IP address through to a downstream pfSense box that runs my home network.
95% of the time everything works perfectly, but periodically -- perhaps 5 to 10 times per day -- internet access stops for 1 to 5 minutes.
Every outage can be matched with entries in the AT&T firewall log related to DNS queries, eg. here is one from a couple of days ago:2020-12-09T15:50:00-06:00 45.25.xxx.xxx 202.12.27.33 UDP Policy (filtersets, etc.)
The 45.25.xxx.xxx address is the "Source IP", which is the public IP address assigned to my pfSense WAN port. The destination IP, 202.12.27.33, by reverse IP lookup, is M.ROOT-SERVERS.NET. Every time the internet goes down, I can find one or more entries like this, all with UDP traffic going to a DNS server of some sort.
Note that when these events occur, the AT&T gateway blocks ALL traffic, not just the DNS packets, and not just traffic from the pfSense box. I was surprised that even a computer connected directly to the AT&T gateway in parallel to the pfSense box also lost internet connectivity during these events. So the DNS queries appear to trigger a complete time-out on all internet access through the gateway, as if it is trying to shut down a perceived attack.
There does not appear to be any way to completely disable the AT&T firewall. I tried disabling everything that could be disabled, but none of those settings changed the behavior.
Then I tried "Enable Forwarding Mode" in DNS Resolver (with DNS servers set to Cloudflare and Google) and that immediately solved the problem. Subsequently I have toggled that setting on and off a couple of times and proven that it is cause & effect: the internet shutdowns and firewall log entries only occur when pfSense is operating in DNS resolver mode.
Has anyone else encountered similar issues with AT&T (or other providers) objecting to outbound DNS resolver traffic? Any recommendations for fixing, other than giving up on DNS resolver and switching to DNS forwarding mode as I have done?
Thanks
Bob -
@bunkerbob that sounds crazy but it is an ISP we're talking about after all. These steps may have already been taken, but the first step I would take is to contact AT&T and ask them to solve this. Does your agreement with them state somewhere in fine print you are not allowed to perform your own DNS lookups? ISP's sometimes don't like home users running servers but it's not like you're hosting a server with enormous traffic hitting it. They shouldn't be blocking DNS. There might be consumer/privacy protection laws against this in your area.
The next step I would take is to contact another ISP if a viable alternative is available.
-
@raffi_ thanks for your commnets. I can't say I disagree with you in principle about contacting AT&T, but I've been around the block enough with ISP technical support to expect it to be a fruitless and frustrating experience. It will take hours just to get through all the preliminary steps of rebooting/resetting/etc before they escalate the issue to anyone with the technical chops to understand and troubleshoot it. And I expect the end result will be that the AT&T firewall behavior is hard-coded into the firmware, so there is nothing they can do in the short term, and given that my use-case probably only applies to 0.01% of their customer base, they won't have much incentive to change their firmware.
In terms of switching to a different ISP, my only other option is Spectrum cable which doesn't come close to the speed and low latency of gigabit AT&T fiber. I've had AT&T for about 5 years and it has been rock solid; best internet I've ever had at home or office. Until a couple of months ago I used a consumer ASUS router for my home network and it coexisted peacefully with the AT&T gateway. It was only when I replaced the ASUS box with pfSense that my problems started. And now that I have switched pfSense to DNS forwarding mode, everything is working great again, so there is really no reason for me to switch to a new provider.
-
Interesting to hear. I have AT&T U-verse (DSL) at home (not my choice) and have been using forwarding in the resolver in order to use Quad9 DNS, so would not have run into this (if it even applies to DSL). I would be curious to poke around in the AT&T router settings to see if there was anything there that could be turned off, for instance all security/firewall. I don't recall specific security features in my router but yours is surely different.
