ATT Internet AIr

[stephenw10]

Do you have outbound NAT rules in place for those?

[ahole4sure]

@stephenw10

Just auto
But looks like it’s there ????

[stephenw10]

Well it looks like you have conflict with the 192.168.2.10 VIP. You have that subnet on an internal interface too it appears since it's included in the auto rules.

You should also have manual rules to translate to the VIPs for traffic from LAN to the modem.

[ahole4sure]

@stephenw10

I think it is where I set up a static route trying to do what @Gblenn had suggested back in this thread previously

So the static route must have done the auto rule

Just so I don’t screw it up, after I have now deleted the static route, how should I set up the the outbound rule?

[ahole4sure]

@stephenw10
So I got rid of the static route and then created the NAT rule ( ?? correct )

Still get the error trying to reach 192.168.2.1

[ahole4sure]

@stephenw10

Correction == waited for a bit

This worked
Thanks

[ahole4sure]

@stephenw10 @Gblenn

Will out seems that maybe disabling IPv6 might have fixed the problem with the frontier modem coming back online
After disabling. Came back online in seconds

There had been a support request sent to ATT about their DNS

After fixing NAT and being able to access the ATT modems config site - even after manually setting the DNS servers ( to non ATT servers) on fail over or attempt to utilize the ATT modem for internet access it just didn’t work unless DNS Forward Mode is checked in DNS Resolver

[stephenw10]

Nice. Yes those outbound NAT rules look correct now and no conflict is shown.

I would still try to capture the DNS traffic with the resolver enabled just to check it's actually trying to use the AT&T connection. A pcap filtered for port 53 would show that.

[Gblenn]

@ahole4sure Ok great that it works as intended when IPv6 is disabled. I have now idea why this is an issue other than I'm guessing something is not initializing as it should on IPv6 so it gets stuck. Perhaps @stephenw10 has some idea as to why this is and how to make it work fully, also with IPv6.

And on the DNS issue, I hope ATT comes back with a resolution that allows you to chose resolver mode as well.

[ahole4sure]

@Gblenn @stephenw10

Thank you guys for your continued help and putting up with me.
@Gblenn - I know it is semantics , but I have Resolver enabled but just have to check to "use Forwarding Moding" on that resolver settings page --- not sure what the difference is ??

Also one other issue -- in my prior pfsense setup I was able to connect to 192.168.2.1 and 192.168.3.1 when connected remotely via my Wireguard VPN -- but now I cannot. Any addnl NAT settings for that to work? I have a Wireguard any rule in place and I have 192.168.2.0/24 and 192.168.3.0/24 as allowed IP's in my settings for my WG client

[Gblenn]

@ahole4sure said in ATT Internet AIr:

I know it is semantics , but I have Resolver enabled but just have to check to "use Forwarding Moding" on that resolver settings page --- not sure what the difference is ??

In that mode, you are using unbound instead of dnsmasq but you target the same DNS servers you specified in the System / General. These are different from the Root or Authoritative servers used in resolver mode.

So it's not that ATT are trying to force you to use their own DNS servers, only not use the top level one's (root and authoritative)...

About your issue not being able to access those subnets over VPN... I guess you need to add your VPN to the Source of those rules for it to work. Now you only have LAN subnets?? Or rather create a copy of the rule where you specify the WG subnet...

[stephenw10]

The AT&T router could be passing DNS servers via DHCP and if pfSense is set to use those it will work in forwarding mode if they are blocking all others.

Look in the router config for anything named 'DNS helper' or similar.

[ahole4sure]

@Gblenn @stephenw10

I added addnl NAT rule for the WG -- initially uisng WG network as the source -- that did not work
I changed the rule to have "any " as source and I was then able to access 192.168.2.1

Is it bad to leave the source "any" as a NAT rule for that ?

I would assume my WG network should be considered secure??

[stephenw10]

Usually having 'any' as source for outbound NAT rule is bad because it can over-match. But here it's limited by the destination which you're unlikely to use for anything except modem access.

But the fact that works implies your client is not in the LAN subnet since it was not matching the other rule. So you should find out where hat client is actually connecting from and change the source from any to match just that source.

[Gblenn]

@stephenw10 I guess the challenge here is that it's incoming from e.g. a phone or some roaming device using wireguard. And since it didn't work using the Wireguard network (from the dropdown I suppose) should it be the peer IP instead?

As per the post above...

@ahole4sure said in ATT Internet AIr:

I added addnl NAT rule for the WG -- initially uisng WG network as the source -- that did not work
I changed the rule to have "any " as source and I was then able to access 192.168.2.1

[ahole4sure]

@Gblenn @stephenw10

Ok I changed to the connecting network and it worked -- I didn't have to do this before I trashed my pfsense setup

Thanks

[stephenw10]

Ah, yes if you're connecting via VPN then it would need to be using the tunnel subnet as source. No idea how it worked before then!

[Gblenn]

@stephenw10 I suppose it was static routing before wasn't it, so no outgoing NAT rules...

[ahole4sure]

@Gblenn @stephenw10

Well finally the test work as they should.

Setup a 3 Tier failover and failed the fiber - tested, and then failed one of the ATT Modems - tested
Finally worked as it should - the failover is almost immediate (less than 20 sec).
The recovery after correcting the fail is very quick now , with IPv6 disabled on the fiber. The recovery (coming back online) takes about a minute or 2 with the ATT modems.

The ATT modems require - "Enabling Forwarding Mode" under the Reslover settings (apparently a known issue with various ATT moodems). I do not have a setting anywhere to disable DNS error assist. Haven't heard back from backend support at ATT yet.
Also to re-cap -- apparently if ATT gives you a static IP on one of thier 5G gateway modems they hobble or disable the dhclient (dhcp for a WAN) so it requires entering the static IP if you are using IP passthrough.

Thanks for ALL you guys help so far!!

The graphical tests show the change to the 2 different ATT modems static IP and a speedtest use of data up and down.

[Gblenn]

@ahole4sure Cool that you got it all up and running as intended. I really hope ATT will come back to you about the DNS issue and that they can unblock the servers.

The time it takes for the failover to happen depends a bit on your settings, so you can finetune by changing the percentages needed for it to decide. Also when recovering you might want to think about what setting is best for you. I have it set to NOT flush any states which means that if I'm on a Teams call during failover and the fiber recovers. I stay on the LTE modem until I end the Teams call. Any new sessions end up on the fiber though.
I did try having it set to flush states (since the LTE connection is metered), but that will generate a drop during a call first at failover and then at recovery. And if the fiber fails again for some reason, maintenance perhaps, you drop the call again, and again...