AT&T Gateway bypass/true bridge using new authbridge
-
This behaviour is happening for me as well with the PACE modem. There are several of these cycles, these are the logs from 'System' during one cycle. Replaced my public IP with x.x.x.x.
Aug 16 20:28:58 check_reload_status 462 Reloading filter
Aug 16 20:28:58 check_reload_status 462 Restarting OpenVPN tunnels/interfaces
Aug 16 20:28:58 check_reload_status 462 Restarting IPsec tunnels
Aug 16 20:28:58 check_reload_status 462 updating dyndns WAN_DHCP
Aug 16 20:28:58 rc.gateway_alarm 14055 >>> Gateway alarm: WAN_DHCP (Addr:x.x.x.x Alarm:0 RTT:.452ms RTTsd:.067ms Loss:5%)
Aug 16 20:25:50 kernel igb1: promiscuous mode enabled
Aug 16 20:25:50 arpwatch 36560 listening on igb1
Aug 16 20:25:47 kernel igb1: promiscuous mode disabled
Aug 16 20:25:47 php-fpm 399 /rc.start_packages: Restarting/Starting all packages.
Aug 16 20:25:46 check_reload_status 462 Reloading filter
Aug 16 20:25:46 check_reload_status 462 Starting packages
Aug 16 20:25:46 php-fpm 36316 /rc.newwanip: Netgate pfSense Plus package system has detected an IP change or dynamic WAN reconnection - 0.0.0.0 -> - Restarting packages.
Aug 16 20:25:45 php-fpm 54442 /rc.openvpn: Gateway, NONE AVAILABLE
Aug 16 20:25:45 php-fpm 54442 /rc.openvpn: Gateway, none 'available' for inet, use the first one configured. 'WAN_DHCP'
Aug 16 20:25:43 check_reload_status 462 Reloading filter
Aug 16 20:25:43 check_reload_status 462 Restarting OpenVPN tunnels/interfaces
Aug 16 20:25:43 check_reload_status 462 Restarting IPsec tunnels
Aug 16 20:25:43 check_reload_status 462 updating dyndns WAN_DHCP
Aug 16 20:25:43 rc.gateway_alarm 45819 >>> Gateway alarm: WAN_DHCP (Addr:x.x.x.x Alarm:1 RTT:0ms RTTsd:0ms Loss:100%)
Aug 16 20:25:43 php-fpm 36316 /rc.newwanip: Creating rrd update script
Aug 16 20:25:43 php-fpm 36316 /rc.newwanip: Resyncing OpenVPN instances for interface ATTMODEM.
Aug 16 20:25:43 php-fpm 36316 /rc.newwanip: IP Address has changed, killing states on former IP Address 0.0.0.0.
Aug 16 20:25:43 php-fpm 36316 /rc.newwanip: Gateway, NONE AVAILABLE
Aug 16 20:25:43 php-fpm 36316 /rc.newwanip: Gateway, none 'available' for inet, use the first one configured. 'WAN_DHCP'
Aug 16 20:25:37 kernel igb1: promiscuous mode enabled
Aug 16 20:25:37 arpwatch 83423 listening on igb1
Aug 16 20:25:37 php-fpm 36316 /rc.newwanip: rc.newwanip: on (IP address: ) (interface: ATTMODEM[opt1]) (real interface: igb2).
Aug 16 20:25:37 php-fpm 36316 /rc.newwanip: rc.newwanip: Info: starting on igb2.
Aug 16 20:25:36 check_reload_status 462 Reloading filter
Aug 16 20:25:36 check_reload_status 462 rc.newwanip starting igb2
Aug 16 20:25:36 php-fpm 400 /rc.linkup: HOTPLUG: Triggering address refresh on opt1 (igb2)
Aug 16 20:25:36 php-fpm 400 /rc.linkup: DEVD Ethernet attached event for opt1
Aug 16 20:25:36 php-fpm 400 /rc.linkup: Hotplug event detected for ATTMODEM(opt1) static IP address ()
Aug 16 20:25:35 kernel igb2: link state changed to UP
Aug 16 20:25:35 check_reload_status 462 Linkup starting igb2
Aug 16 20:25:34 kernel igb1: promiscuous mode disabled
Aug 16 20:25:34 php-fpm 54442 /rc.start_packages: Restarting/Starting all packages.
Aug 16 20:25:33 check_reload_status 462 Starting packages
Aug 16 20:25:33 php-fpm 400 /rc.newwanip: Netgate pfSense Plus package system has detected an IP change or dynamic WAN reconnection - 0.0.0.0 -> - Restarting packages.
Aug 16 20:25:33 check_reload_status 462 Reloading filter
Aug 16 20:25:33 php-fpm 399 /rc.linkup: DEVD Ethernet detached event for opt1
Aug 16 20:25:33 php-fpm 399 /rc.linkup: Hotplug event detected for ATTMODEM(opt1) static IP address ()
Aug 16 20:25:32 php-fpm 54442 /rc.openvpn: Gateway, NONE AVAILABLE
Aug 16 20:25:32 php-fpm 54442 /rc.openvpn: Gateway, none 'available' for inet, use the first one configured. 'WAN_DHCP'
Aug 16 20:25:32 kernel igb2: link state changed to DOWN
Aug 16 20:25:32 check_reload_status 462 Linkup starting igb2
Aug 16 20:25:31 check_reload_status 462 Reloading filter
Aug 16 20:25:31 check_reload_status 462 Restarting OpenVPN tunnels/interfaces
Aug 16 20:25:31 check_reload_status 462 Restarting IPsec tunnels
Aug 16 20:25:31 check_reload_status 462 updating dyndns WAN_DHCP
Aug 16 20:25:31 rc.gateway_alarm 93025 >>> Gateway alarm: WAN_DHCP (Addr:x.x.x.x Alarm:1 RTT:0ms RTTsd:0ms Loss:100%)
Aug 16 20:25:31 php-fpm 400 /rc.newwanip: Creating rrd update script
Aug 16 20:25:31 php-fpm 400 /rc.newwanip: Resyncing OpenVPN instances for interface ATTMODEM.
Aug 16 20:25:31 php-fpm 400 /rc.newwanip: IP Address has changed, killing states on former IP Address 0.0.0.0.
Aug 16 20:25:31 php-fpm 400 /rc.newwanip: Gateway, NONE AVAILABLE
Aug 16 20:25:31 php-fpm 400 /rc.newwanip: Gateway, none 'available' for inet, use the first one configured. 'WAN_DHCP'
Aug 16 20:25:26 php-fpm 400 /rc.newwanip: rc.newwanip: on (IP address: ) (interface: ATTMODEM[opt1]) (real interface: igb2).
Aug 16 20:25:26 php-fpm 400 /rc.newwanip: rc.newwanip: Info: starting on igb2.
Aug 16 20:25:24 check_reload_status 462 Reloading filter
Aug 16 20:25:24 check_reload_status 462 rc.newwanip starting igb2
Aug 16 20:25:24 php-fpm 400 /rc.linkup: HOTPLUG: Triggering address refresh on opt1 (igb2)
Aug 16 20:25:24 php-fpm 400 /rc.linkup: DEVD Ethernet attached event for opt1
Aug 16 20:25:24 php-fpm 400 /rc.linkup: Hotplug event detected for ATTMODEM(opt1) static IP address ()
Aug 16 20:25:23 kernel igb2: link state changed to UP
Aug 16 20:25:23 check_reload_status 462 Linkup starting igb2
Aug 16 20:25:22 check_reload_status 462 Reloading filter
Aug 16 20:25:22 php-fpm 54442 /rc.linkup: DEVD Ethernet detached event for opt1
Aug 16 20:25:22 php-fpm 54442 /rc.linkup: Hotplug event detected for ATTMODEM(opt1) static IP address ()
Aug 16 20:25:21 kernel igb2: link state changed to DOWN
Aug 16 20:25:21 check_reload_status 462 Linkup starting igb2
Aug 16 20:25:09 kernel igb1: promiscuous mode enabled
Aug 16 20:25:09 arpwatch 11720 listening on igb1
Aug 16 20:25:06 kernel igb1: promiscuous mode disabled
Aug 16 20:25:06 php-fpm 54442 /rc.start_packages: Restarting/Starting all packages.
Aug 16 20:25:05 check_reload_status 462 Reloading filter
Aug 16 20:25:05 check_reload_status 462 Starting packages
Aug 16 20:25:05 php-fpm 400 /rc.newwanip: Netgate pfSense Plus package system has detected an IP change or dynamic WAN reconnection - 0.0.0.0 -> - Restarting packages.
Aug 16 20:25:04 php-fpm 399 /rc.openvpn: Gateway, NONE AVAILABLE
Aug 16 20:25:04 php-fpm 399 /rc.openvpn: Gateway, none 'available' for inet, use the first one configured. 'WAN_DHCP'
Aug 16 20:25:03 php-fpm 400 /rc.newwanip: Creating rrd update script
Aug 16 20:25:03 check_reload_status 462 Reloading filter
Aug 16 20:25:03 check_reload_status 462 Restarting OpenVPN tunnels/interfaces
Aug 16 20:25:03 check_reload_status 462 Restarting IPsec tunnels
Aug 16 20:25:03 check_reload_status 462 updating dyndns WAN_DHCP
Aug 16 20:25:03 rc.gateway_alarm 24185 >>> Gateway alarm: WAN_DHCP (Addr:x.x.x.x Alarm:1 RTT:0ms RTTsd:0ms Loss:100%)
Aug 16 20:25:03 php-fpm 400 /rc.newwanip: Resyncing OpenVPN instances for interface ATTMODEM.
Aug 16 20:25:03 php-fpm 400 /rc.newwanip: IP Address has changed, killing states on former IP Address 0.0.0.0.
Aug 16 20:25:03 php-fpm 400 /rc.newwanip: Gateway, NONE AVAILABLE
Aug 16 20:25:03 php-fpm 400 /rc.newwanip: Gateway, none 'available' for inet, use the first one configured. 'WAN_DHCP'
Aug 16 20:24:59 php-fpm 400 /rc.newwanip: rc.newwanip: on (IP address: ) (interface: ATTMODEM[opt1]) (real interface: igb2).
Aug 16 20:24:59 php-fpm 400 /rc.newwanip: rc.newwanip: Info: starting on igb2.
Aug 16 20:24:58 check_reload_status 462 Reloading filter
Aug 16 20:24:58 check_reload_status 462 rc.newwanip starting igb2
Aug 16 20:24:58 php-fpm 400 /rc.linkup: HOTPLUG: Triggering address refresh on opt1 (igb2)
Aug 16 20:24:58 php-fpm 400 /rc.linkup: DEVD Ethernet attached event for opt1
Aug 16 20:24:58 php-fpm 400 /rc.linkup: Hotplug event detected for ATTMODEM(opt1) static IP address ()
Aug 16 20:24:57 kernel igb2: link state changed to UP
Aug 16 20:24:57 check_reload_status 462 Linkup starting igb2
Aug 16 20:24:56 php-fpm 54442 /rc.openvpn: Gateway, NONE AVAILABLE
Aug 16 20:24:56 php-fpm 54442 /rc.openvpn: Gateway, none 'available' for inet, use the first one configured. 'WAN_DHCP'
Aug 16 20:24:55 check_reload_status 462 Reloading filter
Aug 16 20:24:55 check_reload_status 462 Restarting OpenVPN tunnels/interfaces
Aug 16 20:24:55 check_reload_status 462 Restarting IPsec tunnels
Aug 16 20:24:55 check_reload_status 462 updating dyndns WAN_DHCP
Aug 16 20:24:55 rc.gateway_alarm 28082 >>> Gateway alarm: WAN_DHCP (Addr:x.x.x.x Alarm:1 RTT:.462ms RTTsd:.050ms Loss:21%)
Aug 16 20:24:55 check_reload_status 462 Reloading filter
Aug 16 20:24:55 php-fpm 36316 /rc.linkup: DEVD Ethernet detached event for opt1
Aug 16 20:24:55 php-fpm 36316 /rc.linkup: Hotplug event detected for ATTMODEM(opt1) static IP address ()
Aug 16 20:24:54 kernel igb2: link state changed to DOWN
Aug 16 20:24:54 check_reload_status 462 Linkup starting igb2 -
@eldog This is just a shot in the dark. What happens if you disable the gateway monitoring option (assuming it's enabled)?
-
It was monitoring. I'll let you know what happens.
-
@GPz1100 No change in behaviour
-
@eldog Which specific modem is this, the 5268ac too?
When this happens, are you losing service/connectivity to the internet?
In theory, once the ont is authorized, you don't need to reauthorize unless the ethernet cable is unplugged (from it), or att does some sort of maintenance that resets the olt requiring a reauth from the ONT. Both of these are quite rare.
What you could do is disconnect the gateway entirely for a day or two. Assuming neither of the above are occurring, you should have uninterrupted service.
The gateway likely expects to be able to phone home/have internet connectivity. In this configuration it does not. It's there strictly to respond to eapol traffic and nothing else. After x many attempts, it shits the bed and reboots, cycle continues.
Given the age of the rg, I'd try to pull off certs and have pf handle the entire eapol sequence.
-
Well, found another problem in this config. If I enable Suricata all the interfaces go haywire, even the LAN, which eventually comes back up, but the WAN is never able to get a DHCP lease from AT&T. I don't think this setup is ready for primetime, and I have to get work done, headed back to hide behind the crappy gateway.
-
@eldog You could contact att to request a newer gateway (bgw210? or the 320 if they'll send it). The 5268ac is quite old.
Edit: Try disconnecting the gateway once the connection is authorized. I used to run this set up for nearly a year some years ago --- aka dumb switch method. Connect rg and ONT via switch to authorize. Once authorized, remove rg and connect wan. It all stayed working until either the ont link was broken or att reset the olt. This set up worked for weeks to months at a time.
-
H HLPPC referenced this topic on
-
I misposted my reply here. I was having issues on a BGW-210 until I made DNS ACLs
https://forum.netgate.com/topic/182852/at-t-bypass/2"I downgraded from the BGW 320 to the BGW 210-700 with a Nokia ONT. I noticed you didn't mention what sort of ONT your setup has. Here are some things which may help you get the setup to stop dropping connectivity:
Upgrade from the pace to a BGW 210-700 (NOT the BGW 320 because it doesn't have a separate ONT).
These other recommendations are here whether your use a BGW or pace:
2. Disable all IPV6 on the pfSense. Do not enable or try to mess with it until you get your system stable.
3. Make sure your PCP value is actually "1" (you can try to change it to "2-5" later for "better" connectivity if you live in an apartment complex aka FTTH. It is commonly accepted to omit 6-7 values from actually doing anything)-
Create DNS ACLs. Use pfTop to find the AT&T DHCP server's IP address. It is likely different than your ONT, and communicates as DHCP server on ports 67 and 68. Add the ONT's IP address AND the AT&T DHCP/DNS server to your DNS server's blacklist. Only allow localhost and your custom subnets. Do not allow the ISP to override your DNS. You are constantly exposing port 53 anyways, and their systems are incompatible beyond issuing a DHCP lease to your WAN. Their DNS servers commonly crash unbound and even the DNS forwarder. As a side note, you do not have to add your chosen DNS servers to a white list. I personally use DoT to 1.1.1.2 with the DNS server.
On that note, also go into /var/unbound/unbound.conf and remove anything associated with attlocal.net. Additionally, you want to block all LAN devices from TCP to DNS port 53 or 853. And set up port forwarding to either localhost or the LAN interface for DNS requests. Note, some devices do not support having their DNS queries forwarded to localhost. Windows does, Nintendo does not. So you have to instead port forward to the LAN interface. -
Make sure to set your domain name under System > General Setup. Don't use the default pfSense domain, and don't try and use an AT&T domain.
-
Try creating floating rules that pass communications between WAN and the ONT AND the AT&T DHCP server, on UDP ports 67 and 68 (aka DHCP), and ICMP. Eventually, the system will figure itself out, and you can delete these rules; ICMP is allowed out of the firewall by default.
-
Try setting your "Modem" interface on a different NIC than your WAN. My modem NIC is on a cheap realtek NIC built into my motherboard, and my WAN is on my igb card.
Also disable flow control on the Modem interface, and make sure the speed is not "auto-negotiated." 10, 100 or 1000 mbps full duplex should be sufficient. Create system and loader tunables:
dev.igb.2.fc="0"- If your WAN NIC is capable of speeds higher than your AT&T router, you'll need to auto-negotiate the speed and duplex on the WAN. Additionally you MUST have eee enabled on the WAN NIC's port. My igc card, for instance, is capable of 2.5Gbps, and won't work with the ONT unless EEE is enabled. Create system tunables and/or loader tunables:
dev.igb.0.eee_control="1"
hw.igb.eee_setting="1"
Some NICs have sysctl "hw.xxx" tunables and some don't. I don't think any igb ones do, which is frustrating when you have an igb card. I have an "em" card, "re" card "igb" card and "igc" card.
-For the Empire"
-
-
The DNS ACL is enabled and enforced by going to Services > DNS Resolver > Advanced settings.
Checking "Disable Auto-added Access Control"
Then go to Services > DNS Resolver > Access Lists
Create a pass rule for your LANs
Create a deny rule for the ONT and the AT&T DHCP server.
If you want to get super particular, you can find the actual AT&T DNS server on the AT&T router and add that to the deny rules too.Also under System > General Setup
Uncheck DNS Server Override and Select either "Use Local DNS, fall back to remote DNS Servers" or "Use Local DNS, ignore remote DNS servers." And add your favorite DNS Servers under DNS Server Settings.AT&T does a bunch of weird crap with their DNS, such as DNS "Error Assist". You are pre-enrolled in them selling your DNS data, and can "opt-out" on their website.
"ATTHelp
Sign in to your AT&T account.
Once signed in, click on your name and select View Profile.
Now select Communication Preferences, then Privacy Settings.
From here, you should be able to select DNS Error Assist, and see an option to opt out."https://www.reddit.com/r/homelab/comments/wkypc3/att_users_dont_forget_to_turn_off_dns_error_assist/?rdt=42954
-
You could also try disabling EEE on the igb NIC. I disable it on mine anyways, and it works fine over WAN. My igb NIC is an 82576.
dev.igb.0.eee_control="0"
My em NIC works fine as a WAN interface too in the bypass. It is an 82573.
dev.em.0.eee_control="0"
hw.em.eee_setting="0" -
Having this same issue with my pace gateway, have you figured out a solution to get it to work yet?
-
What issue exactly? Quite a few things in this thread.
-
As Stephen indicated, more details needed.
If you have an external ONT, it's possible to rather easily pull certs from a 210 (possibly others).
https://github.com/mozzarellathicc/attcerts
The 210 can be found for cheap on ebay. Going this route eliminates a useless piece of a equipment remaining connected, not to mention gaining a port.
With the newer pf (CE 2.7x) and plus (23.09?) - which ever versions they introduced openssl3, there are additional steps needed to make eapol authentication work using wpa_supplicant. You can refer to this lovely reddit post which details the requirements quite nicely. - https://www.reddit.com/r/PFSENSE/comments/18jz0uc/installing_att_bypass_on_a_clean_install_of/ .
If you have a 320 with no external ONT, there's still ways of bypassing but more involved. You will need to obtain an sfp ont module and have a switch w/port to accept such modules.
-
@stephenw10 I follow the configuration recipe, and internet connectivity seems to work for a little bit, but then in the gateway monitor packet loss slowly starts to increase and then internet connectivity is gone. I haven't been able to see if connectivity comes back after a while but restarting pfsense brings it back for about another 5 minutes and then it is gone.
Definition of internet connectivity (being able to ping 1.1.1.1 for example). both from pfsense and client device.
-
@GPz1100 I'm not sure if with the Pace 5268ac this is possible as I haven't found anything about someone being able to get the certificates off of it without some hardware hacking.
-
Is it only rebooting that brings it back? Does reconnecting the WAN or rebooting the modem bring it back for example? Or resaving the WAN perhaps?
-
@matthewgcampbell I recommend following the path of least resistance. Contact att, tell 'em your gateway is disconnecting and request they send you a newer one. Ask if you can get the bgw210. Once in hand, attempt to get certs from it and forgo the whole L2 proxy business. You'll eliminate a useless piece of hardware and gain stability.
Or just buy a 210 on ebay or the certs.
I have no experience with the proxy method, while it may work, ultimately keeps the useless turd (att gateway) still in the loop to some extent. This may cause weird behavior depending on what the mothership may be trying to do.
In general, once eapol authenticated, you stay authenticated until the link is severed (wan cable disconnected), or att reboots the OLT requiring another auth. Otherwise, sessions can last weeks and months. As for wan dhcp; att dhcp servers issue leases good for 1 hr. They start renewing around the 30 min mark. As far as I know, ipv4 dhcp lease is needed before any data starts to flow, or ipv6 is available. Meaning you can't just set wan to the same static values and have it work.
I've been using wpa_supplicant bypass with external ont going on 5 years now, att knows im bypassed but they've never raised an issue about it. They know because i've had other issues (routing/peering) that required filing fcc complaints to get any attention. Upper level tech support I spoke with never mentioned bypass in any manner. Hope this helps.
-
@stephenw10 Resaving the WAN would bring it back, I haven't tried unplugging, but rebooting also fixed it.
-
@GPz1100 said in AT&T Gateway bypass/true bridge using new authbridge:
In general, once eapol authenticated, you stay authenticated until the link is severed (wan cable disconnected), or att reboots the OLT requiring another auth. Otherwise, sessions can last weeks and months. As for wan dhcp; att dhcp servers issue leases good for 1 hr. They start renewing around the 30 min mark. As far as I know, ipv4 dhcp lease is needed before any data starts to flow, or ipv6 is available. Meaning you can't just set wan to the same static values and have it work.
This is the part that has me confused, it seems to pass traffic for a little bit (5 minutes) and then stops. But according to this that shouldn't be possible.
-
@matthewgcampbell I have never experienced a scenario where it passes traffic for a short amount of time then stops, at least not in the context of eapol auth related. It either passes or it doesn't. Then again I've never done any proxy bypasses either, can't really comment on odd behavior as a result.
I assume you're following this - https://docs.netgate.com/pfsense/en/latest/recipes/authbridge.html ?
You might want to give a try to one of the proxy scripts here - https://github.com/MonkWho/pfatt/tree/master . This is what we used before vlan0 compliant wpa_supplicant and dhclient.
Edit, one other idea to try is the old dumb switch bypass method.
I can't find a good write up but in essence you connect ethernet from ont and gateway to a dumb switch (preferably not netgear). Wait until the lights on the modem are all green and stop flashing. Disconnect gateway cable while leaving ONT/switch connected. Connect cable from the modem to your pfsense wan port (again, you're not touching the ONT/switch cable). Pfsense should be configured for dhcp on wan.
See if you experience the same disconnect issues after x amount of time. If you do, try a release /renew on the wan. If it doesn't pull an ip, try rebooting pfsense only. This whole time, the link between the ONT and switch should remain connect and as far as ONT concerned, remain authenticated.
