DHCPv6 with AT&T Uverse issue

Spy Alelo · Sep 24, 2023, 9:38 AM

I have been having trouble getting a lease from AT&T with pfSense+ 23.05.1, here are some details:

The AT&T GW has been completely bypassed with the new bridge function for the 802.1X requests
All settings are done per Netgate guidelines, and DUID matches the one from the AT&T GW (got a pcap and WireShark to analyze it and find it)

I have enabled DHCPv6 debugging already and these are the DHCP logs (my DUID has been obscured):

Sep 24 03:44:38	dhcp6c	18418	XID mismatch
Sep 24 03:44:38	dhcp6c	18418	status code: not on-link
Sep 24 03:44:38	dhcp6c	18418	get DHCP option status code, len 2
Sep 24 03:44:38	dhcp6c	18418	DUID: 00:02:00:00:0d:e9:XXXXXXXXXXXXXXXXXXXXXXXX
Sep 24 03:44:38	dhcp6c	18418	get DHCP option client ID, len 28
Sep 24 03:44:38	dhcp6c	18418	DUID: 00:03:00:01:e4:81:84:49:fc:0f
Sep 24 03:44:38	dhcp6c	18418	get DHCP option server ID, len 10
Sep 24 03:44:38	dhcp6c	18418	receive reply from fe80::e681:84ff:fe49:fc0f%igc0 on igc0
Sep 24 03:44:35	dhcp6c	18418	XID mismatch
Sep 24 03:44:35	dhcp6c	18418	status code: not on-link
Sep 24 03:44:35	dhcp6c	18418	get DHCP option status code, len 2
Sep 24 03:44:35	dhcp6c	18418	DUID: 00:02:00:00:0d:e9:XXXXXXXXXXXXXXXXXXXXXXXX
Sep 24 03:44:35	dhcp6c	18418	get DHCP option client ID, len 28
Sep 24 03:44:35	dhcp6c	18418	DUID: 00:03:00:01:e4:81:84:49:fc:0f
Sep 24 03:44:35	dhcp6c	18418	get DHCP option server ID, len 10
Sep 24 03:44:35	dhcp6c	18418	receive reply from fe80::e681:84ff:fe49:fc0f%igc0 on igc0
Sep 24 03:44:32	dhcpleases	30495	Sending HUP signal to dns daemon(59292)
Sep 24 03:44:26	dhcpleases	95167	Sending HUP signal to dns daemon(59292)
Sep 24 03:44:25	dhclient	21618	bound to 104.189.70.126 -- renewal in 1800 seconds.
Sep 24 03:44:25	dhclient	83270	Creating resolv.conf
Sep 24 03:44:25	dhclient	82698	/sbin/route add default 104.189.68.1
Sep 24 03:44:25	dhclient	82168	/sbin/route add -host 104.189.68.1 -iface igc0
Sep 24 03:44:25	dhclient	81381	Adding new routes to interface: igc0
Sep 24 03:44:25	dhclient	81074	New Routers (igc0): 104.189.68.1
Sep 24 03:44:25	dhclient	80943	New Broadcast Address (igc0): 104.189.71.255
Sep 24 03:44:25	dhclient	80716	New Subnet Mask (igc0): 255.255.252.0
Sep 24 03:44:25	dhclient	80396	New IP Address (igc0): 104.189.70.126
Sep 24 03:44:25	dhclient	79864	ifconfig igc0 inet 104.189.70.126 netmask 255.255.252.0 broadcast 104.189.71.255
Sep 24 03:44:25	dhclient	79804	Starting add_new_address()
Sep 24 03:44:25	dhclient	79259	BOUND
Sep 24 03:44:25	dhclient	21618	DHCPACK from 104.189.68.1
Sep 24 03:44:25	dhclient	21618	DHCPREQUEST on igc0 to 255.255.255.255 port 67
Sep 24 03:44:25	dhclient	78312	ARPCHECK
Sep 24 03:44:24	dhcp6c		18418	reset a timer on igc0, state=SOLICIT, timeo=4, retrans=16326
Sep 24 03:44:24	dhcp6c		18418	send solicit to ff02::1:2%igc0
Sep 24 03:44:24	dhcp6c		18418	set IA_PD
Sep 24 03:44:24	dhcp6c		18418	set IA_PD prefix
Sep 24 03:44:24	dhcp6c		18418	set option request (len 4)
Sep 24 03:44:24	dhcp6c		18418	set elapsed time (len 2)
Sep 24 03:44:24	dhcp6c		18418	set identity association
Sep 24 03:44:24	dhcp6c		18418	set client ID (len 28)
Sep 24 03:44:24	dhcp6c		18418	Sending Solicit
Sep 24 03:44:23	dhclient	63018	ARPSEND
Sep 24 03:44:23	dhclient	21618	DHCPOFFER from 104.189.68.1
Sep 24 03:44:23	dhclient	21618	DHCPDISCOVER on igc0 to 255.255.255.255 port 67 interval 1
Sep 24 03:44:16	dhcp6c	18418	reset a timer on igc0, state=SOLICIT, timeo=3, retrans=8065
Sep 24 03:44:16	dhcp6c	18418	send solicit to ff02::1:2%igc0
Sep 24 03:44:16	dhcp6c	18418	set IA_PD
Sep 24 03:44:16	dhcp6c	18418	set IA_PD prefix
Sep 24 03:44:16	dhcp6c	18418	set option request (len 4)
Sep 24 03:44:16	dhcp6c	18418	set elapsed time (len 2)
Sep 24 03:44:16	dhcp6c	18418	set identity association
Sep 24 03:44:16	dhcp6c	18418	set client ID (len 28)
Sep 24 03:44:16	dhcp6c	18418	Sending Solicit
Sep 24 03:44:16	dhclient	21618	DHCPREQUEST on igc0 to 255.255.255.255 port 67
Sep 24 03:44:12	dhcp6c	18418	reset a timer on igc0, state=SOLICIT, timeo=2, retrans=3982
Sep 24 03:44:12	dhcp6c	18418	send solicit to ff02::1:2%igc0
Sep 24 03:44:12	dhcp6c	18418	set IA_PD
Sep 24 03:44:12	dhcp6c	18418	set IA_PD prefix
Sep 24 03:44:12	dhcp6c	18418	set option request (len 4)
Sep 24 03:44:12	dhcp6c	18418	set elapsed time (len 2)
Sep 24 03:44:12	dhcp6c	18418	set identity association
Sep 24 03:44:12	dhcp6c	18418	set client ID (len 28)
Sep 24 03:44:12	dhcp6c	18418	Sending Solicit
Sep 24 03:44:11	dhclient	21618	DHCPREQUEST on igc0 to 255.255.255.255 port 67
Sep 24 03:44:10	dhcp6c	18418	reset a timer on igc0, state=SOLICIT, timeo=1, retrans=2083
Sep 24 03:44:10	dhcp6c	18418	send solicit to ff02::1:2%igc0
Sep 24 03:44:10	dhcp6c	18418	set IA_PD
Sep 24 03:44:10	dhcp6c	18418	set IA_PD prefix
Sep 24 03:44:10	dhcp6c	18418	set option request (len 4)
Sep 24 03:44:10	dhcp6c	18418	set elapsed time (len 2)
Sep 24 03:44:10	dhcp6c	18418	set identity association
Sep 24 03:44:10	dhcp6c	18418	set client ID (len 28)
Sep 24 03:44:10	dhcp6c	18418	Sending Solicit
Sep 24 03:44:09	dhclient	21618	DHCPREQUEST on igc0 to 255.255.255.255 port 67
Sep 24 03:44:09	dhclient	21662	PREINIT
Sep 24 03:44:09	dhcp6c	18418	reset a timer on igc0, state=SOLICIT, timeo=0, retrans=1091
Sep 24 03:44:09	dhcp6c	18418	send solicit to ff02::1:2%igc0
Sep 24 03:44:09	dhcp6c	18418	set IA_PD
Sep 24 03:44:09	dhcp6c	18418	set IA_PD prefix
Sep 24 03:44:09	dhcp6c	18418	set option request (len 4)
Sep 24 03:44:09	dhcp6c	18418	set elapsed time (len 2)
Sep 24 03:44:09	dhcp6c	18418	set identity association
Sep 24 03:44:09	dhcp6c	18418	set client ID (len 28)
Sep 24 03:44:09	dhcp6c	18418	a new XID (8fefcb) is generated
Sep 24 03:44:09	dhcp6c	18418	Sending Solicit
Sep 24 03:44:08	dhcp6c	18418	reset a timer on igc0, state=INIT, timeo=0, retrans=891

It just seems to give up right after XID Mismatch and repeats the process all over again, even though from the logs it appears that it has been offered a /60 prefix already as expected. I hope I have enough information here.

Thanks!

Spy Alelo · Sep 24, 2023, 8:59 PM

Here's some more logs which show the prefix being offered, but it ends on XID mismatch again:

Sep 24 04:20:52	dhcp6c	18418	XID mismatch
Sep 24 04:20:52	dhcp6c	18418	IA_NA address: 2001:506:7112:4ad::1 pltime=3600 vltime=3600
Sep 24 04:20:52	dhcp6c	18418	get DHCP option IA address, len 24
Sep 24 04:20:52	dhcp6c	18418	IA_NA: ID=1, T1=1800, T2=2880
Sep 24 04:20:52	dhcp6c	18418	get DHCP option identity association, len 40
Sep 24 04:20:52	dhcp6c	18418	DUID: 00:02:00:00:0d:e9:XXXXXXXXXXXXXXXXXXXXXX
Sep 24 04:20:52	dhcp6c	18418	get DHCP option client ID, len 28
Sep 24 04:20:52	dhcp6c	18418	DUID: 00:03:00:01:e4:81:84:49:fc:0f
Sep 24 04:20:52	dhcp6c	18418	get DHCP option server ID, len 10
Sep 24 04:20:52	dhcp6c	18418	receive advertise from fe80::e681:84ff:fe49:fc0f%igc0 on igc0

Spy Alelo · Sep 24, 2023, 9:41 PM

Sorry to keep adding, but after doing another pcap, I noticed that pfSense is soliciting two XIDs, one right after the other. I did verify that it doesn't have two instances of it running either, very confusing:

🔒 Log in to view

ps uxawww | grep dhcp6c
root     9838   0.0  0.0  12820   2644  -  Is   16:10       0:00.01 /usr/local/sbin/dhcp6c -D -c /var/etc/dhcp6c.conf -p /var/run/dhcp6c.pid igc0
root     6350   0.0  0.0  12768   2432  0  S+   16:37       0:00.00 grep dhcp6c

JKnott · Sep 25, 2023, 1:02 AM

@Spy-Alelo

Do a packet capture of the full DHCPv6 sequence and post the capture file here.

Here's what it looks like with my ISP (Rogers):

🔒 Log in to view

Spy Alelo · Sep 30, 2023, 8:20 PM

@JKnott here's my packet capture packetcapture-igc0-20230930151238.zip

JKnott · Sep 30, 2023, 9:56 PM

@Spy-Alelo

Is that from the beginning, following my instructions?

Spy Alelo · Sep 30, 2023, 10:20 PM

@JKnot no reboot, just link/unlink of the interfaces.

I can do the reboot and re-link the interfaces, just give me a few minutes.

Spy Alelo · Oct 1, 2023, 12:06 AM

@JKnott here ya go, sorry about that packetcapture-igc0-20230930172601.zip

JKnott · Oct 1, 2023, 12:06 AM

@Spy-Alelo

Why is it so different from what mine looks like? DHCPv6 works pretty much the same as on IPv4, with 4 steps to getting the address.

Spy Alelo · Oct 1, 2023, 12:19 AM

@JKnott I don't know what to tell ya! I wish I knew, I am still trying to figure out what the issue is with my setup.

Spy Alelo · Oct 1, 2023, 1:11 PM

I see you do get a solicit, advertise, request and a reply between their DHCP servers and your pfSense box.

Mine gets stuck between replies, rebinds and solicit/advertise.

There's something either not listening to a reply, or gets ignored. AT&T's IPv6 implementation is a pain in the butt.

JKnott · Oct 1, 2023, 1:11 PM

@Spy-Alelo

Try connecting a computer to the modem and see if it gets an IPv6 address. You can also install Wireshark on the computer to do the captures.

Is there anyone else here who's on AT&T that can help? I'm on Rogers, in Canada, and they work fine.

Spy Alelo · Oct 1, 2023, 7:27 PM

@JKnott IPv6 works fine with their gateway.

AT&T goes out of their way to make sure that you only use their gateway and pay for it for as long as you have their service, even though is absolutely terrible and unreliable.

I’ve been able to bypass it entirely for over 2 years now, but only over IPv4. From what I read, some folks have been able to get IPv6 leases with some additional work, but I’m stuck.

jmmm · Jun 8, 2024, 12:23 AM

Same problem. Any update?

Spy Alelo · Jun 8, 2024, 11:44 AM

@jmmm I moved on with OPNsense and it worked on the first try. Full IPv6 routing with no issues. I never figured out why it wouldn’t work with pfSense.

jmmm · Jun 9, 2024, 2:12 PM

Has anyone using any of the uATT Fiber gateway bypass methods gotten IPV6 working with ATT Fiber?

jmmm · Jun 11, 2024, 8:37 PM

after a bunch of screwing around, I have it working. I wrote up what I found here:
https://forum.netgate.com/topic/188676/ipv6-dhcp-client-with-att-fiber-without-gateway-working

In particular:
In the instructions at [https://docs.netgate.com/pfsense/en/latest/recipes/authbridge.html], section Add Modem-WAN Bridge Rule, the instructions say to set Protocol
Any. If you do this, the DHCP6 requests from the modem will be forwarded through PFSENSE to the ONT and cause XID mismatch errors. This should instead be set for Protocol IEEE 802.1X.

and a lot of rebooting or it will not work.