AT&T Gateway bypass/true bridge using new authbridge

cmillets · Jun 8, 2023, 12:02 AM

@cmillets said in AT&T Gateway bypass/true bridge using new authbridge:

@stephenw10 I'm not 100% sure. According to the PFSense logs the modem link goes down. I would assume that means something is happing on the AT&T Gateway. This is my first experience on PFSense so I not sure if I'm reading the logs correctly.

I've had the Gateway for 6 years and have never once had an issue with it. That leads me to believe the Gateway is not liking something to do with the authentication. The DHCP address that get handed off hasn't changed.

Any other ideas on how to troubleshoot this?

@stephenw10 After connect directly to the Gateway I've learned a few things:

The Gateway is indeed restarting when I'm having these issues
The Gateway states it does not have any service despite everything running well on PFSense

From my understanding, following the authbridge instructions should make the Gateway think it's connected to AT&T right? After reflecting, since doing the 'authbridge" the status light for "Service" hasn't been green.

It's also important to note, I haven't setup IPv6.

stephenw10 · Jun 8, 2023, 11:04 AM

The gateway is only involved for authentication. It will show as down because it doesn't have an IP itself so any link monitoring it's doing will fail.

When the gateway is rebooting like that does it actually disrupt the connection from pfSense?

Putting a switch in between pfSense and the gateway as a test will eliminate a lot of the scripts that gets fired in pfSense because it won't lose link on igb2 when the gateway reboots.

Steve

cmillets · Jun 8, 2023, 11:41 PM

@stephenw10 said in AT&T Gateway bypass/true bridge using new authbridge:

The gateway is only involved for authentication. It will show as down because it doesn't have an IP itself so any link monitoring it's doing will fail.

When the gateway is rebooting like that does it actually disrupt the connection from pfSense?

Putting a switch in between pfSense and the gateway as a test will eliminate a lot of the scripts that gets fired in pfSense because it won't lose link on igb2 when the gateway reboots.

Steve

I've placed a switch in-between and will update with my findings tomorrow. Is there a way to trick the Gateway so it doesn't reboot every 24 hours when it can't find a connection? Is that only a known issue with the Pace 5268ac?

stephenw10 · Jun 9, 2023, 2:14 PM

Potentially it could be. I spoke to some of my colleagues who have AT&T and they report the gateway device will reboot as it tries to connect to AT&T. However it doesn't lose the link causing pfSense to reload services etc.
When you see that does it disrupt the connection on the WAN? Do LAN client lose internet?

Do you have a gateway defined on the modem interface somehow? It's odd that pfSense is running the newwanip script for that.

Steve

cmillets · Jun 9, 2023, 2:14 PM

@stephenw10 said in AT&T Gateway bypass/true bridge using new authbridge:

Potentially it could be. I spoke to some of my colleagues who have AT&T and they report the gateway device will reboot as it tries to connect to AT&T. However it doesn't lose the link causing pfSense to reload services etc.
When you see that does it disrupt the connection on the WAN? Do LAN client lose internet?

Do you have a gateway defined on the modem interface somehow? It's odd that pfSense is running the newwanip script for that.

Steve

Last night was the first time with the switch and as you've mentioned PFSense no longer detects a down link. (which is progress)

When the gateway restarts the DNS server drops. I don't understand why it goes crazy since it's done through PFSense. So when this issue is happening the internet connection to my lan does drop. A couple days ago I did change the Monitor IP so when the AT&T Gateway goes down it does not effect the signal loss. I change the ip to my local AT&T Center Office (Which is hop #2 for me)

The Gateway setup is exactly how the documentation said to do it. I'll include some screenshots taken this morning.
!

stephenw10 · Jun 9, 2023, 3:38 PM

Hmm, curiously specific. I guess it reboots after 10mins and takes 5mins to reauth....

As far as I know no other AT&T users have reported that. So it may be something specific to that device. Or even to how it's configured.

DefenderLLC · Jun 28, 2023, 4:41 PM

Slightly related topic... If you currently have residential AT&T Fiber with any static IP block(s) attached and want to change speed (example: 1 gig to 2 gigs or 5 gigs), make sure to call the AT&T Loyalty department to have them do the order for you. If you do it yourself from the website or mobile app, you will lose your status IP block(s) and you will have to call in to receive a new block.

If you want to keep your existing IP block(s), call the number below. If not, you will lose what you have now and have to call back in to get new ones.

AT&T Loyalty (direct no.): 877-999-1083

GPz1100 · Jun 29, 2023, 5:09 PM

@cmillets Did you ever get this figured out?

In general once an eapol auth session takes place, only two things will break it.

link between ont and firewall going down (ie cable unplug, ont or firewall reboot)
Att doing maint that reboots the OLT

This means that the gateway itself can reboot indefinitely without actually affecting your lan/wan link unless either 1 or 2 above happens.

RichardR · Jul 12, 2023, 8:34 PM

I also have the same symptoms as OP with the same Pace modem. I have identified that during the 5 minutes of downtime that happens several times during a one hour period, the pfSense is unable to get a WAN DHCP IP address from the ONT. I see multiple outgoing requests and then after attempt ~7 it finally gets a DHCP reply with my customary IP.

I'm not sure how to troubleshoot this any further, but in my case, I'm not sure that the modem is the culprit and it feels like the ONT is not getting what it needs, but perhaps that's because it's needing something from the modem.

I had to revert back to the inline setup but I might try it again to see whether there is evidence of the modem rebooting.

GPz1100 · Jul 12, 2023, 8:34 PM

@RichardR The pace is a rather old modem, maybe see if yo can something newer like the bgw210 or 320.

Which ONT do you have.

AiC0315 · Jul 13, 2023, 8:52 PM

@GPz1100
I have the same modem as OP and had no issues with the auth bypass. As stated earlier in this thread all the modem does is authenticate the line. I have since switched to the wpa_supplicant bypass and don't use my modem.

RichardR · Jul 13, 2023, 5:51 PM

@GPz1100 My ONT is an Alcatel Lucent Intertek G-240G-A Optical Network terminal from 2015

GPz1100 · Jul 13, 2023, 8:52 PM

@AiC0315 It's possible there's a difference in firmware or some other setting causing one not to work.

@RichardR I would see about extracting certs from your gateway so you can eliminate it entirely.

There's a newer method out that may work - https://github.com/mozzarellathicc/attcerts/

Basically it's a brute force attempt to grab the file during the bootup cycle of the modem. Given how old yours is, chances are its not been patched for this exploit.

See if step #6 works. If it does it's worth a try.

I was able to do this successfully on a bgw210 with 3.18.2 fw. Did requiring launching the script in about 6 or 7 separate tabs (and folders), and took a number of retries to get a hit.

AiC0315 · Jul 13, 2023, 9:16 PM

@GPz1100 He could do a factory reset. There hasn't been an update with the Pace firmware in quite some time, I'm sure he's on the newest.

pokrifchakd · Jul 13, 2023, 9:39 PM

Has anyone gotten this working on the BGW320-505 (Nokia version). I'm looking to make the configuration changes, but would like to know if there are any "gotchas" with this particular gateway.

DefenderLLC · Jul 14, 2023, 2:25 PM

@pokrifchakd Not that I have heard of. AT&T hss really locked those down. I have the same one too.

pokrifchakd · Jul 14, 2023, 2:25 PM

@DefenderLLC Dang, I was hoping to do this as my weekend project.

DefenderLLC · Jul 14, 2023, 2:33 PM

@pokrifchakd It it was easy, I would probably do it too, but I honestly don't see the point. Even with the 5gig fiber service, I never experience any type of packet loss using the 320 gateway (all wireless radios disabled) in IP passthrough mode to my 6100 MAX. I don't see how eliminating the device is going to improve anything - BUT AGAIN - I probably would if I could.

eldog · Aug 17, 2023, 6:29 PM

This behaviour is happening for me as well with the PACE modem. There are several of these cycles, these are the logs from 'System' during one cycle. Replaced my public IP with x.x.x.x.

Aug 16 20:28:58 check_reload_status 462 Reloading filter
Aug 16 20:28:58 check_reload_status 462 Restarting OpenVPN tunnels/interfaces
Aug 16 20:28:58 check_reload_status 462 Restarting IPsec tunnels
Aug 16 20:28:58 check_reload_status 462 updating dyndns WAN_DHCP
Aug 16 20:28:58 rc.gateway_alarm 14055 >>> Gateway alarm: WAN_DHCP (Addr:x.x.x.x Alarm:0 RTT:.452ms RTTsd:.067ms Loss:5%)
Aug 16 20:25:50 kernel igb1: promiscuous mode enabled
Aug 16 20:25:50 arpwatch 36560 listening on igb1
Aug 16 20:25:47 kernel igb1: promiscuous mode disabled
Aug 16 20:25:47 php-fpm 399 /rc.start_packages: Restarting/Starting all packages.
Aug 16 20:25:46 check_reload_status 462 Reloading filter
Aug 16 20:25:46 check_reload_status 462 Starting packages
Aug 16 20:25:46 php-fpm 36316 /rc.newwanip: Netgate pfSense Plus package system has detected an IP change or dynamic WAN reconnection - 0.0.0.0 -> - Restarting packages.
Aug 16 20:25:45 php-fpm 54442 /rc.openvpn: Gateway, NONE AVAILABLE
Aug 16 20:25:45 php-fpm 54442 /rc.openvpn: Gateway, none 'available' for inet, use the first one configured. 'WAN_DHCP'
Aug 16 20:25:43 check_reload_status 462 Reloading filter
Aug 16 20:25:43 check_reload_status 462 Restarting OpenVPN tunnels/interfaces
Aug 16 20:25:43 check_reload_status 462 Restarting IPsec tunnels
Aug 16 20:25:43 check_reload_status 462 updating dyndns WAN_DHCP
Aug 16 20:25:43 rc.gateway_alarm 45819 >>> Gateway alarm: WAN_DHCP (Addr:x.x.x.x Alarm:1 RTT:0ms RTTsd:0ms Loss:100%)
Aug 16 20:25:43 php-fpm 36316 /rc.newwanip: Creating rrd update script
Aug 16 20:25:43 php-fpm 36316 /rc.newwanip: Resyncing OpenVPN instances for interface ATTMODEM.
Aug 16 20:25:43 php-fpm 36316 /rc.newwanip: IP Address has changed, killing states on former IP Address 0.0.0.0.
Aug 16 20:25:43 php-fpm 36316 /rc.newwanip: Gateway, NONE AVAILABLE
Aug 16 20:25:43 php-fpm 36316 /rc.newwanip: Gateway, none 'available' for inet, use the first one configured. 'WAN_DHCP'
Aug 16 20:25:37 kernel igb1: promiscuous mode enabled
Aug 16 20:25:37 arpwatch 83423 listening on igb1
Aug 16 20:25:37 php-fpm 36316 /rc.newwanip: rc.newwanip: on (IP address: ) (interface: ATTMODEM[opt1]) (real interface: igb2).
Aug 16 20:25:37 php-fpm 36316 /rc.newwanip: rc.newwanip: Info: starting on igb2.
Aug 16 20:25:36 check_reload_status 462 Reloading filter
Aug 16 20:25:36 check_reload_status 462 rc.newwanip starting igb2
Aug 16 20:25:36 php-fpm 400 /rc.linkup: HOTPLUG: Triggering address refresh on opt1 (igb2)
Aug 16 20:25:36 php-fpm 400 /rc.linkup: DEVD Ethernet attached event for opt1
Aug 16 20:25:36 php-fpm 400 /rc.linkup: Hotplug event detected for ATTMODEM(opt1) static IP address ()
Aug 16 20:25:35 kernel igb2: link state changed to UP
Aug 16 20:25:35 check_reload_status 462 Linkup starting igb2
Aug 16 20:25:34 kernel igb1: promiscuous mode disabled
Aug 16 20:25:34 php-fpm 54442 /rc.start_packages: Restarting/Starting all packages.
Aug 16 20:25:33 check_reload_status 462 Starting packages
Aug 16 20:25:33 php-fpm 400 /rc.newwanip: Netgate pfSense Plus package system has detected an IP change or dynamic WAN reconnection - 0.0.0.0 -> - Restarting packages.
Aug 16 20:25:33 check_reload_status 462 Reloading filter
Aug 16 20:25:33 php-fpm 399 /rc.linkup: DEVD Ethernet detached event for opt1
Aug 16 20:25:33 php-fpm 399 /rc.linkup: Hotplug event detected for ATTMODEM(opt1) static IP address ()
Aug 16 20:25:32 php-fpm 54442 /rc.openvpn: Gateway, NONE AVAILABLE
Aug 16 20:25:32 php-fpm 54442 /rc.openvpn: Gateway, none 'available' for inet, use the first one configured. 'WAN_DHCP'
Aug 16 20:25:32 kernel igb2: link state changed to DOWN
Aug 16 20:25:32 check_reload_status 462 Linkup starting igb2
Aug 16 20:25:31 check_reload_status 462 Reloading filter
Aug 16 20:25:31 check_reload_status 462 Restarting OpenVPN tunnels/interfaces
Aug 16 20:25:31 check_reload_status 462 Restarting IPsec tunnels
Aug 16 20:25:31 check_reload_status 462 updating dyndns WAN_DHCP
Aug 16 20:25:31 rc.gateway_alarm 93025 >>> Gateway alarm: WAN_DHCP (Addr:x.x.x.x Alarm:1 RTT:0ms RTTsd:0ms Loss:100%)
Aug 16 20:25:31 php-fpm 400 /rc.newwanip: Creating rrd update script
Aug 16 20:25:31 php-fpm 400 /rc.newwanip: Resyncing OpenVPN instances for interface ATTMODEM.
Aug 16 20:25:31 php-fpm 400 /rc.newwanip: IP Address has changed, killing states on former IP Address 0.0.0.0.
Aug 16 20:25:31 php-fpm 400 /rc.newwanip: Gateway, NONE AVAILABLE
Aug 16 20:25:31 php-fpm 400 /rc.newwanip: Gateway, none 'available' for inet, use the first one configured. 'WAN_DHCP'
Aug 16 20:25:26 php-fpm 400 /rc.newwanip: rc.newwanip: on (IP address: ) (interface: ATTMODEM[opt1]) (real interface: igb2).
Aug 16 20:25:26 php-fpm 400 /rc.newwanip: rc.newwanip: Info: starting on igb2.
Aug 16 20:25:24 check_reload_status 462 Reloading filter
Aug 16 20:25:24 check_reload_status 462 rc.newwanip starting igb2
Aug 16 20:25:24 php-fpm 400 /rc.linkup: HOTPLUG: Triggering address refresh on opt1 (igb2)
Aug 16 20:25:24 php-fpm 400 /rc.linkup: DEVD Ethernet attached event for opt1
Aug 16 20:25:24 php-fpm 400 /rc.linkup: Hotplug event detected for ATTMODEM(opt1) static IP address ()
Aug 16 20:25:23 kernel igb2: link state changed to UP
Aug 16 20:25:23 check_reload_status 462 Linkup starting igb2
Aug 16 20:25:22 check_reload_status 462 Reloading filter
Aug 16 20:25:22 php-fpm 54442 /rc.linkup: DEVD Ethernet detached event for opt1
Aug 16 20:25:22 php-fpm 54442 /rc.linkup: Hotplug event detected for ATTMODEM(opt1) static IP address ()
Aug 16 20:25:21 kernel igb2: link state changed to DOWN
Aug 16 20:25:21 check_reload_status 462 Linkup starting igb2
Aug 16 20:25:09 kernel igb1: promiscuous mode enabled
Aug 16 20:25:09 arpwatch 11720 listening on igb1
Aug 16 20:25:06 kernel igb1: promiscuous mode disabled
Aug 16 20:25:06 php-fpm 54442 /rc.start_packages: Restarting/Starting all packages.
Aug 16 20:25:05 check_reload_status 462 Reloading filter
Aug 16 20:25:05 check_reload_status 462 Starting packages
Aug 16 20:25:05 php-fpm 400 /rc.newwanip: Netgate pfSense Plus package system has detected an IP change or dynamic WAN reconnection - 0.0.0.0 -> - Restarting packages.
Aug 16 20:25:04 php-fpm 399 /rc.openvpn: Gateway, NONE AVAILABLE
Aug 16 20:25:04 php-fpm 399 /rc.openvpn: Gateway, none 'available' for inet, use the first one configured. 'WAN_DHCP'
Aug 16 20:25:03 php-fpm 400 /rc.newwanip: Creating rrd update script
Aug 16 20:25:03 check_reload_status 462 Reloading filter
Aug 16 20:25:03 check_reload_status 462 Restarting OpenVPN tunnels/interfaces
Aug 16 20:25:03 check_reload_status 462 Restarting IPsec tunnels
Aug 16 20:25:03 check_reload_status 462 updating dyndns WAN_DHCP
Aug 16 20:25:03 rc.gateway_alarm 24185 >>> Gateway alarm: WAN_DHCP (Addr:x.x.x.x Alarm:1 RTT:0ms RTTsd:0ms Loss:100%)
Aug 16 20:25:03 php-fpm 400 /rc.newwanip: Resyncing OpenVPN instances for interface ATTMODEM.
Aug 16 20:25:03 php-fpm 400 /rc.newwanip: IP Address has changed, killing states on former IP Address 0.0.0.0.
Aug 16 20:25:03 php-fpm 400 /rc.newwanip: Gateway, NONE AVAILABLE
Aug 16 20:25:03 php-fpm 400 /rc.newwanip: Gateway, none 'available' for inet, use the first one configured. 'WAN_DHCP'
Aug 16 20:24:59 php-fpm 400 /rc.newwanip: rc.newwanip: on (IP address: ) (interface: ATTMODEM[opt1]) (real interface: igb2).
Aug 16 20:24:59 php-fpm 400 /rc.newwanip: rc.newwanip: Info: starting on igb2.
Aug 16 20:24:58 check_reload_status 462 Reloading filter
Aug 16 20:24:58 check_reload_status 462 rc.newwanip starting igb2
Aug 16 20:24:58 php-fpm 400 /rc.linkup: HOTPLUG: Triggering address refresh on opt1 (igb2)
Aug 16 20:24:58 php-fpm 400 /rc.linkup: DEVD Ethernet attached event for opt1
Aug 16 20:24:58 php-fpm 400 /rc.linkup: Hotplug event detected for ATTMODEM(opt1) static IP address ()
Aug 16 20:24:57 kernel igb2: link state changed to UP
Aug 16 20:24:57 check_reload_status 462 Linkup starting igb2
Aug 16 20:24:56 php-fpm 54442 /rc.openvpn: Gateway, NONE AVAILABLE
Aug 16 20:24:56 php-fpm 54442 /rc.openvpn: Gateway, none 'available' for inet, use the first one configured. 'WAN_DHCP'
Aug 16 20:24:55 check_reload_status 462 Reloading filter
Aug 16 20:24:55 check_reload_status 462 Restarting OpenVPN tunnels/interfaces
Aug 16 20:24:55 check_reload_status 462 Restarting IPsec tunnels
Aug 16 20:24:55 check_reload_status 462 updating dyndns WAN_DHCP
Aug 16 20:24:55 rc.gateway_alarm 28082 >>> Gateway alarm: WAN_DHCP (Addr:x.x.x.x Alarm:1 RTT:.462ms RTTsd:.050ms Loss:21%)
Aug 16 20:24:55 check_reload_status 462 Reloading filter
Aug 16 20:24:55 php-fpm 36316 /rc.linkup: DEVD Ethernet detached event for opt1
Aug 16 20:24:55 php-fpm 36316 /rc.linkup: Hotplug event detected for ATTMODEM(opt1) static IP address ()
Aug 16 20:24:54 kernel igb2: link state changed to DOWN
Aug 16 20:24:54 check_reload_status 462 Linkup starting igb2

GPz1100 · Aug 17, 2023, 6:31 PM

@eldog This is just a shot in the dark. What happens if you disable the gateway monitoring option (assuming it's enabled)?