ATT Internet AIr
-
Back again lol
Trying to get ready for the holidays so my side line IT job has sufferedDuring the last week- both of the ATT moderns gateways have remained online ( one with the direct interface connection and the other with the VLAN connection ) - that is with the static IP configuration.
Got around to a few fail over tests. Latest one tonight.
Upon unplugging the WAN cable from the fiber modem fail over is successful but 2 issues remain1 ) the ATT modem just doesn’t seem to behave as far as DNS resolution (will ping IP addresses but can’t resolve DNS) - unless I check Enable DNS Forward Mode in the DNS Resolver settings. We explored this before but never figured out why
- possibly bigger issues - when I plug the fiber modem cable back in the Frontier fiber internet cable just won’t come back online. Even if I “reset all states” it won’t come back online. The only way I can get the fiber modem Tier 1 gateway back online is to full reboot pfsense.
Ideas?
Thank you
-
@ahole4sure Hmm, how have you set up your resolver? Which interfaces have you selected in the list?
-
Hmm, odd. Unbound should be able to resolve directly via any outbound interface as long as it's selected. By default it tries to use all interfaces.
Do you see any blocked traffic? Do you see states on port 53 when failed over to the AT&T modem?The Frontier gateway shows as off line? Or the NIC never links?
-
I did not check for "blocked traffic". If I test again this PM would I just fail the WAN and then check the logs for the firewall? And where / how is best way to monitor or look for "states" on port 53?
As far as the Frontier WAN interface -- the gateway on the status page shows red or "offliine" for much longer than expected (like a couple minutes), then it flashes to blue "unknown", then back to offline, then back to blue unknown -- ultimately after 4 or 5 minutes it settles in on blue unknown and just never links back up. I then tried reset states --- Ulitmately the only thing that lets it reconnect is just go to Diagnostics and choose "normal reboot". Back online , like normal, after the reboot.
-
@ahole4sure I guess the two problems are separate and not necessarily related. So on the first one about the DNS issue, which I responded to, have you checked the settings for the resolver? Which interfaces have you selected for it to use?
In the forwarder, there is no such setting and I assume it's automatic as @stephenw10 suggests. -
Go to Diag > States and filter by
:53
What is logged in the System and Gateway logs when you reconnect the Frontier modem?
How are you running the failover test? Physically unplugging something? -
@stephenw10 I'll have to retest and check the logs
And yes to fail - I just unplug the ethernet cord from the back of the frontier modem
-
@stephenw10
Here are the screen shots of the info
Apologies and thanks for help!!They start with states during failover
Show the DNS error on web browser
(the is with Forwarder Mode off under DNS Resolver settings)Then shows what happens after waiting for Frontier WAN to come back up
Ultimately would not fully connect and come back up untill full reboot of pfsense
🔒 Log in to view 🔒 Log in to view 🔒 Log in to view 🔒 Log in to view
🔒 Log in to view 🔒 Log in to view 🔒 Log in to view 🔒 Log in to view
🔒 Log in to view -
-
@Gblenn
I have checked ALL for bothIs that correct?
-
Hmm, no DNS states shown there on states with ports starting with 53. You might need to filter differently to see that in the states.
I don't see the link state change in the logs there. Does it actually show the that when you reconnect the modem?
-
@ahole4sure said in ATT Internet AIr:
@Gblenn
I have checked ALL for bothIs that correct?
Yes that is correct, and I'm starting to think that ATT is actually blocking access to authoritative servers... If this is true you either have to stick to Forwarder mode, or ask ATT to stop blocking. I wonder if there is some setting you can change for your account perhaps?
On the fiber issue, there are a few things that stand out to me.
-
Gateway shows an IP but status is pending. So that IP is probably cached data in pfsense, the link is up (eth to frontier box) but pfsense has not verified the DHCP lease?
-
I notice you have IPv6 enabled on the fiber WAN. Could that cause issues? Try disabling it and see if that makes any difference when testing?
-
Have you tried pulling the fiber instead of the ethernet cable to pfsense? If that is possible to do?
Keeping the physical link up on pfsense interface thoughout the test... If there is something on that side that isn't really working as it should?
If you look at the dashboard instead of Status / Gateways, you will also see the negotiated settings for the interface. Does that seem ok, and is it the same before and after pulling and reconnecting the cable?
-
-
@stephenw10
Yes it does show this when I reconnect the Frontier fiber modem -- I will have to try it again this PM to try to get the states again
As far as ATT - wonder if @Gblenn is correct - I have been reading that ATT has some sort of DNS assist that could be hijacking my DNS queries -- not sure how to determine that.https://gist.github.com/CollinChaffin/24f6c9652efb3d6d5ef2f5502720ef00
@Gblenn
I will retest with IPv6 disableld tonight ! I see (after some ressearch) that ATT could be hijacking the DNS. I will look into this. Thanks -
Yes if they're redirecting DNS that would break the resolver.
-
@ahole4sure Why can't ISP's just leave things alone... I'm assuming, since you run the modem(s) in passthrough mode, that it's not in the device that is being blocked, or is it?
I'm thinking you need to log in to their portal (your accounts) to disable that "helper feature"... -
So am trying to leave things in situ as they have been setting and working (ie the ATT modems)
They are supposoed to be accesible from my LAN (and they were right after I re setup my pfsense) - via 192.168.2.1 and 192.168.3.1 (set in the device as the configuration addresses)
Through VIPsFor some reason I can't access either of those VIPs any longer --
I was trying to see if I could change the DNS settings we were discussing.
I can ping both 192.168.2.1 and 192.168.3.1 but can't access via web browser
-
Do you have outbound NAT rules in place for those?
-
Just auto
But looks like it’s there ????🔒 Log in to view 🔒 Log in to view -
Well it looks like you have conflict with the 192.168.2.10 VIP. You have that subnet on an internal interface too it appears since it's included in the auto rules.
You should also have manual rules to translate to the VIPs for traffic from LAN to the modem.
-
I think it is where I set up a static route trying to do what @Gblenn had suggested back in this thread previously
So the static route must have done the auto rule
Just so I don’t screw it up, after I have now deleted the static route, how should I set up the the outbound rule?
-
@stephenw10
So I got rid of the static route and then created the NAT rule ( ?? correct )Still get the error trying to reach 192.168.2.1
-
-
Will out seems that maybe disabling IPv6 might have fixed the problem with the frontier modem coming back online
After disabling. Came back online in secondsThere had been a support request sent to ATT about their DNS
After fixing NAT and being able to access the ATT modems config site - even after manually setting the DNS servers ( to non ATT servers) on fail over or attempt to utilize the ATT modem for internet access it just didn’t work unless DNS Forward Mode is checked in DNS Resolver
-
Nice. Yes those outbound NAT rules look correct now and no conflict is shown.
I would still try to capture the DNS traffic with the resolver enabled just to check it's actually trying to use the AT&T connection. A pcap filtered for port 53 would show that.
-
@ahole4sure Ok great that it works as intended when IPv6 is disabled. I have now idea why this is an issue other than I'm guessing something is not initializing as it should on IPv6 so it gets stuck. Perhaps @stephenw10 has some idea as to why this is and how to make it work fully, also with IPv6.
And on the DNS issue, I hope ATT comes back with a resolution that allows you to chose resolver mode as well.
-
Thank you guys for your continued help and putting up with me.
@Gblenn - I know it is semantics , but I have Resolver enabled but just have to check to "use Forwarding Moding" on that resolver settings page --- not sure what the difference is ??Also one other issue -- in my prior pfsense setup I was able to connect to 192.168.2.1 and 192.168.3.1 when connected remotely via my Wireguard VPN -- but now I cannot. Any addnl NAT settings for that to work? I have a Wireguard any rule in place and I have 192.168.2.0/24 and 192.168.3.0/24 as allowed IP's in my settings for my WG client
-
@ahole4sure said in ATT Internet AIr:
I know it is semantics , but I have Resolver enabled but just have to check to "use Forwarding Moding" on that resolver settings page --- not sure what the difference is ??
In that mode, you are using unbound instead of dnsmasq but you target the same DNS servers you specified in the System / General. These are different from the Root or Authoritative servers used in resolver mode.
So it's not that ATT are trying to force you to use their own DNS servers, only not use the top level one's (root and authoritative)...
About your issue not being able to access those subnets over VPN... I guess you need to add your VPN to the Source of those rules for it to work. Now you only have LAN subnets?? Or rather create a copy of the rule where you specify the WG subnet...
-
The AT&T router could be passing DNS servers via DHCP and if pfSense is set to use those it will work in forwarding mode if they are blocking all others.
Look in the router config for anything named 'DNS helper' or similar.
-
I added addnl NAT rule for the WG -- initially uisng WG network as the source -- that did not work
I changed the rule to have "any " as source and I was then able to access 192.168.2.1Is it bad to leave the source "any" as a NAT rule for that ?
I would assume my WG network should be considered secure??
-
Usually having 'any' as source for outbound NAT rule is bad because it can over-match. But here it's limited by the destination which you're unlikely to use for anything except modem access.
But the fact that works implies your client is not in the LAN subnet since it was not matching the other rule. So you should find out where hat client is actually connecting from and change the source from any to match just that source.
-
@stephenw10 I guess the challenge here is that it's incoming from e.g. a phone or some roaming device using wireguard. And since it didn't work using the Wireguard network (from the dropdown I suppose) should it be the peer IP instead?
As per the post above...
@ahole4sure said in ATT Internet AIr:
I added addnl NAT rule for the WG -- initially uisng WG network as the source -- that did not work
I changed the rule to have "any " as source and I was then able to access 192.168.2.1 -
Ok I changed to the connecting network and it worked -- I didn't have to do this before I trashed my pfsense setup
Thanks
-
Ah, yes if you're connecting via VPN then it would need to be using the tunnel subnet as source. No idea how it worked before then!
-
@stephenw10 I suppose it was static routing before wasn't it, so no outgoing NAT rules...
-
Well finally the test work as they should.
Setup a 3 Tier failover and failed the fiber - tested, and then failed one of the ATT Modems - tested
Finally worked as it should - the failover is almost immediate (less than 20 sec).
The recovery after correcting the fail is very quick now , with IPv6 disabled on the fiber. The recovery (coming back online) takes about a minute or 2 with the ATT modems.The ATT modems require - "Enabling Forwarding Mode" under the Reslover settings (apparently a known issue with various ATT moodems). I do not have a setting anywhere to disable DNS error assist. Haven't heard back from backend support at ATT yet.
Also to re-cap -- apparently if ATT gives you a static IP on one of thier 5G gateway modems they hobble or disable the dhclient (dhcp for a WAN) so it requires entering the static IP if you are using IP passthrough.Thanks for ALL you guys help so far!!
The graphical tests show the change to the 2 different ATT modems static IP and a speedtest use of data up and down.
🔒 Log in to view 🔒 Log in to view 🔒 Log in to view 🔒 Log in to view
-
@ahole4sure Cool that you got it all up and running as intended. I really hope ATT will come back to you about the DNS issue and that they can unblock the servers.
The time it takes for the failover to happen depends a bit on your settings, so you can finetune by changing the percentages needed for it to decide. Also when recovering you might want to think about what setting is best for you. I have it set to NOT flush any states which means that if I'm on a Teams call during failover and the fiber recovers. I stay on the LTE modem until I end the Teams call. Any new sessions end up on the fiber though.
I did try having it set to flush states (since the LTE connection is metered), but that will generate a drop during a call first at failover and then at recovery. And if the fiber fails again for some reason, maintenance perhaps, you drop the call again, and again... -
@Gblenn
Thank you
That’s great advice
Merry (or Happy) Christmas or Happy Holidays