Multiple IPv6 Prefix Delegation over AT&T Residential Gateway for pfSense 2.4.5

ttmcmurry

I've been working on this one for a while. This is the result of others posting their work across various forums, reading BSD docs, and plenty of testing as a result of needing something to do while being stuck at home. :)

The purpose of this is to make it easier for AT&T customers who wish to assign more than one IPv6 prefix delegation inside their pfSense firewall to more than one internal network interface. I am providing an example dhcp.conf script and explaining what's needed step-by-step. AT&T customers must have been furnished a Residential Gateway (Pace 5268AC / Arris BGW210-700, possibly others) and have configured the RG in DMZ+/IP Passthrough mode. This has been written with pfSense 2.4.5 in mind.

Why do this? In short, AT&T U-Verse & Fiber customer equipment is assigned a /60 and can only hand out eight /64 prefix delegations. It is not possible to request a larger PD, however it is possible to request multiple /64 PDs from pfSense's WAN interface. Since the pfSense UI does not expose this functionality directly, it is possible to take advantage of it by supplying a dhcp.conf to override pfSense DHCP6 behavior available from the UI.

I welcome improvements and feedback and will be happy to update this to help make other's lives easier and work within pfSense native functionality as much as possible.

Once this script is in place, if you need to reassign interfaces & prefix delegations, the script has to be updated. You will need to edit the IPv6 Track Interface Prefix ID on the LAN/OPT interfaces with the IA-PD you specify in the .conf file.

Assumptions:

The WAN interface IPv6 Configuration type is configured for "none" or,
The WAN interface IPv6 Configuration type is configured for DHCP6 and IPv6 Prefix Delegation is set /60
The WAN interface IPv6 DHCP6 Client Option "Do not allow PD/Address release" is UNCHECKED
The LAN/OPT interfaces' DHCP6 option is set to "none"
DHCPv6 Server & RA -> DHCP6 Server -> Disabled
DHCPv6 Server & RA -> Router Advertisements -> Defaults (Router Mode: Assisted)

Step one: Update the "interface" stanza

Make a local copy of the code below
Look at Interfaces -> Assignments -> Network Port for the adapter associated to the WAN interface
Replace the adapter in the interface stanza below with the WAN adapter network port name below, e.g. hn0, igb0, vmx0, eth0, etc
If using VLANs, remember to use numerical subinterface number e.g. hn0.10 for VLAN 10
IA-NA Note: The IA-NA is an arbitrary number. A unique number must be chosen for each device connected to the AT&T residential gateway (RG) which will request a prefix delegation from the RG. If only one device will be requesting PDs from the RG (i.e. this pfSense firewall), then "ia-na 0" is fine.

Step two: Update the "ia-pd" stanzas

Look at Interfaces -> Assignments -> Network Port for the adapter associated to each LAN/OPT interface(s)
Replace the prefix-interface "hn1" with LAN/OPT adapter network port name
If using VLANs, remember to use numerical subinterface number e.g. hn0.10 for VLAN 10
You can arbitrarily assign adapters to PDs, staring with PD 0 and working down the list
- SLA-ID is always zero (0)
- SLA-LEN is always zero (0)
Formatting is specific. Each new PD stanza needs to be formatted exactly as PD0; only update the adapter name
If you are not assigning an adapter to a PD, leave it blank as shown below
Note: You do not need to request all 8 PDs if you don't need them. You may remove any "send ia-pd" & "id-assoc pd x" statements where you aren't assigning them to interfaces.
Note: Assigned PDs will result in numerically different networks, depending on the RG
- Pace 5268AC first assigns F then decrements to 8 to PD 0-7, i.e. PD0 = ::xxxF::/64
- Arris BGW210-700 first assigns 8 then increments to F to PD 0-7, i.e. PD0 = ::xxx8::/64

Step three: Add the script to pfSense

Create this file on pfSense under Diagnostics -> Edit File
Copy and paste your edited script into the text window
In the grey filename box, enter /usr/local/etc/rc.d/att-rg-dhcpv6-pd.conf
Click on Save

Step four: Edit the WAN interface

Set IPv6 Configuration Type to "DHCP6" (it may already be set, see "assumptions" above)
Under DHCP6 client configuration, select Configuration Override
Enter the following in Configuration File Override: /usr/local/etc/rc.d/att-rg-dhcpv6-pd.conf
Click on Save and Apply the changes

Step five: Edit the LAN/OPT interface(s), one at a time

Set the IPv6 Configuration Type to "Track Interface"
Set the Track IPv6 Interface -> IPv6 Interface to the WAN's interface name ("WAN" is the default name)
Set the IPv6 Prefix ID to the PD number configured in the .conf file
Click on Save and Apply the changes

Step six: Enable pfSense DHCPv6 Server & Test

For each configured interface..
DHCPv6 Server & RA -> DHCPv6 Server -> Enable -> Save
DHCPv6 Server & RA -> Router Advertisements -> Router Mode -> Assisted (Default)
Test a client in each configured, connected network

---ONLY COPY/PASTE THE CODE YOU EDIT BELOW HERE INTO PFSENSE---

interface hn0 {
	send ia-na 0;
	send ia-pd 0;
	send ia-pd 1;
	send ia-pd 2;
	send ia-pd 3;
	send ia-pd 4;
	send ia-pd 5;
        send ia-pd 6;
	send ia-pd 7;
	request domain-name-servers;
	request domain-name;
	script "/var/etc/dhcp6c_wan_dhcp6withoutra_script.sh";
};
id-assoc na 0 { };
id-assoc pd 0 {
	prefix-interface hn1 {
		sla-id 0;
		sla-len 0;
	};
};
id-assoc pd 1 { };
id-assoc pd 2 { };
id-assoc pd 3 { };
id-assoc pd 4 { };
id-assoc pd 5 { };
id-assoc pd 6 { };
id-assoc pd 7 { };

ttmcmurry

Reserved for future use. :)

State limits

AT&T Residential gateways have a state table that is far smaller than pfSense's defaults, which can result in problems once the RG begins tracking more states than available. pfSense should be set to never go above that limit. pfSense will adjust how states are managed based on its default adaptive algorithm from "Firewall Adaptive Timeouts." There is no need to adjust pfSense default Adaptive Timeout behavior, only the maximum number of states pfSesnse can use.

The values below are from known hardware & firmware capabilities. Depending on the # of devices directly plugged into the RG, like U-Verse set-top-boxes and devices NOT behind pfSense, you may need to adjust pfSense's maximum states downward. This information can be found on the RG under Settings -> Diagnostics -> NAT.

Pace 5268AC Firmware v11.5.1.532678-att - 15460 states max - Set pfSense to 15000 states
Arris NVG599 - Firmware v9.2.2h0d79 - 4096 states max - Set pfSense to 3500 states
Arris BGW210-700 - Firmware 1.9.16 - 8000 states max - Set pfSense to 7500 states
Motorola NVG589 - Firmware ? - 8192 states max - Set pfSense to 7600 states

Set the pfSense state limit in Advanced -> Firewall & NAT -> Firewall Maximum States

Note: If anyone has more up-to-date information about RG firmware and state capabilities, let me know and I'll update this table.

JKnott

@ttmcmurry said in Multiple IPv6 Prefix Delegation over AT&T Residential Gateway for pfSense 2.4.5:

is assigned a /60 and can only hand out eight /64 prefix delegations

Shouldn't that be 16 /64s?

ttmcmurry

@JKnott - No. The RG reserves 8 /64s for itself. Only 8 are able to be claimed by a router.

JKnott

@ttmcmurry said in Multiple IPv6 Prefix Delegation over AT&T Residential Gateway for pfSense 2.4.5:

No. The RG reserves 8 /64s for itself.

WOW!!! That device must have a lot of stuff to require 8 /64s. I wonder why they don't use Unique Local Addresses. Then they'd have an entire /7 to play with, without taking anything from the customer space. ULAs, like RFC 1918 addresses, on IPv4, are routeable within a private network.

Also, there's no reason for them to be so stingy with addresses. There are enough public addresses to give every single person on earth over 4000 /48s.

ttmcmurry

Can't say I fully understand AT&T's design choices. The reality is the RG is a multipurpose device.

Integrated DSL/ADSL modem
Runs at full line speed with GigaFiber service
Voice gateway for POTS service over IP
Native inside and isolated guest networks over WiFi on 2 frequencies (up to 4 SSIDs)
U-Verse TV service requires the RG
- If the U-Verse STB is not directly connected to the RG, it doesn't work right, if at all
- Said another way, U-Verse STBs must use the RG's DNS service
MoCA adapter is built-in on some models (inside network only)
Communicates in real time to AT&T so the AT&T U-Verse app can manage the RG
It totally logs where you go on the interwebs .. this is completely true

The /60 is enough for the device to fulfill its own purposes; it doesn't use all 8 /64 PDs reserved for itself yet it leaves 8 leftover PDs for power users like us. While I agree with you in principle, the average user is oblivious to these things and it just needs to work. Within that structure the RG must work a specific way for their support staff to keep costs down.

I plan on using the reserved space (1st post) for pfSense configuration tips. There are some quirks that should be accounted for in pfSense so undesired operation doesn't occur.

jkfritcher

Thanks for writing up these instructions, I've been wondering how to do this for a long time.

@ttmcmurry said in Multiple IPv6 Prefix Delegation over AT&T Residential Gateway for pfSense 2.4.5:

Step five: Edit the LAN/OPT interface(s), one at a time

Set the IPv6 Configuration Type to "Track Interface"

Set the Track IPv6 Interface -> IPv6 Interface to the WAN's interface name ("WAN" is the default name)

Leave the IPv6 Prefix ID at default "0" (zero)

Click on Save and Apply the changes

This step is being problematic for me.

I have two VLANs on the same physical interface that I want to assign PDs to. I can see through Status -> Interfaces that each VLAN is getting a different PD assigned to it like I configured in the custom config. I was also able to set the first VLAN's IPv6 configuration type to 'Track interface' and set it to WAN and Prefix ID 0. But when I try to configure the second VLAN to 'Track interface', with WAN and Prefix ID 0, I get the following error.

This track6 prefix ID is already being used in LAN

How were you able to successfully configure multiple interfaces with the same settings for the instructions above? Below is my custom config for dhcp6c. I did intentionally leave off the ia-na request, as on my WAN interface, I let the system auto config the address from RA.

Thanks!

interface igb0 {
	send ia-pd 0;
	send ia-pd 1;
	request domain-name-servers;
	request domain-name;
	script "/var/etc/dhcp6c_wan_dhcp6withoutra_script.sh";
};
id-assoc pd 0 {
	prefix-interface igb1.10 {
		sla-id 0;
		sla-len 0;
	};
};
id-assoc pd 1 {
	prefix-interface igb1.20 {
		sla-id 0;
		sla-len 0;
	};
};

ttmcmurry

Good observation. I have been meaning to build out my second internal interface for my work-at-home network so the company I work for can't sniff my home network. Could you try setting your 2nd interface to 1 and verify it is working proper? The configuration file on the WAN would be responsible for assigning the PDs. I can update the article once there's confirmation.

Glad you found this helpful. :)

jkfritcher

@ttmcmurry,

I've tried using 1 for the Prefix ID and I get an out of range error when saving.

I was looking through the code to see what its doing, but to be honest, PHP isn't even close to one of my favorite languages, so didn't make it far into figuring out what was going on.

SimpsomRJ

@ttmcmurry said in Multiple IPv6 Prefix Delegation over AT&T Residential Gateway for pfSense 2.4.5:

script "/var/etc/dhcp6c_wan_dhcp6withoutra_script.sh";

Using this script I could not set it up properly. Dhcp6c was restarting all the time and IPs were not holding. I had to go back to the previous script (/var/etc/dhcp6c_wan_script.sh) to make it work.

@jkfritcher said in Multiple IPv6 Prefix Delegation over AT&T Residential Gateway for pfSense 2.4.5:

@ttmcmurry,

I've tried using 1 for the Prefix ID and I get an out of range error when saving.

I was looking through the code to see what its doing, but to be honest, PHP isn't even close to one of my favorite languages, so didn't make it far into figuring out what was going on.

I was able to set it up properly. To be able to change the prefix ID you need to change the DHCPv6 Prefix Delegation size to 60 BEFORE you check the Configuration Override checkbox.

SimpsomRJ

@ttmcmurry said in Multiple IPv6 Prefix Delegation over AT&T Residential Gateway for pfSense 2.4.5:

Step four: Edit the WAN interface

Set IPv6 Configuration Type to "DHCP6" (it may already be set, see "assumptions" above)
Under DHCP6 client configuration, select Configuration Override
Enter the following in Configuration File Override: /usr/local/etc/rc.d/att-rg-dhcpv6-pd.conf
Click on Save and Apply the changes

This step should actually be:

Step four: Edit the WAN interface

Set IPv6 Configuration Type to "DHCP6" (it may already be set, see "assumptions" above)
Under DHCP6 client configuration, change DHCPv6 Prefix Delegation size from 64 to 60
Select Configuration Override
Enter the following in Configuration File Override: /usr/local/etc/rc.d/att-rg-dhcpv6-pd.conf
Click on Save and Apply the changes

My final conf file with 2 networks looks like this:

interface igb0 {
	send ia-na 0;
	send ia-pd 0;
	send ia-pd 1;
	send ia-pd 2;
	send ia-pd 3;
	send ia-pd 4;
	send ia-pd 5;
        send ia-pd 6;
	send ia-pd 7;
	request domain-name-servers;
	request domain-name;
	script "/var/etc/dhcp6c_wan_script.sh";
};
id-assoc na 0 { };
id-assoc pd 0 {
	prefix-interface vtnet0 {
		sla-id 0;
		sla-len 0;
	};
};
id-assoc pd 1 {
	prefix-interface vtnet1 {
		sla-id 0;
		sla-len 0;
	};
};
id-assoc pd 2 { };
id-assoc pd 3 { };
id-assoc pd 4 { };
id-assoc pd 5 { };
id-assoc pd 6 { };
id-assoc pd 7 { };

ttmcmurry

Hello @SimpsomRJ - I reviewed what you found and how things are working on mine. I've made some changes, but it was minor. This also addresses what @jkfritcher said.

SimpsomRJ is correct, on the WAN interface, before selecting configuration override, the PD size needs to be set to 60. The irony is the option to do so is hidden once configuration override is selected. Nonetheless, pfSense accepts the value. Once this is set, each lan-side interface needs to be set to IPv6 type Track Interface. The IA-PD as specified in the .conf file should be assigned to the interface's Track IPv6 Interface statement. I.e. LAN -> IPv6 Configuration Type -> Track Interface; IPv6 Prefix ID -> 0, 1, 2, etc.

I'll update the main article. I have this working with all other configuration unchanged. Thank you for your insight!

stevebion

Hello
I have two WAN interfaces: WAN1 (Comcast), WAN2 (ATT). I get ipv6 addresses on the LAN side by tracking WAN1.
WAN2 gets ipv6 however "IPv6 delegated Prefix Subnet" is empty in ATT gateway Arris BGW210 , regardless of what I do.

I have tried the tutorial above but no luck. Any suggestions hugely appreciated
Config :

interface mvneta0 {
	send ia-na 0;
	send ia-pd 0;
	send ia-pd 1;
	send ia-pd 2;
	send ia-pd 3;
	send ia-pd 4;
	send ia-pd 5;
        send ia-pd 6;
	send ia-pd 7;
	request domain-name-servers;
	request domain-name;
	script "/var/etc/dhcp6c_wan_script.sh";
};
id-assoc na 0 { };
id-assoc pd 0 {
	prefix-interface mvneta1 {
		sla-id 0;
		sla-len 0;
	};
};

id-assoc pd 2 { };
id-assoc pd 3 { };
id-assoc pd 4 { };
id-assoc pd 5 { };
id-assoc pd 6 { };
id-assoc pd 7 { };

ttmcmurry

@stevebion - I would pick one interface or the other for IPv6 unless you have static IPv6 addresses from both providers. The problem you'll run into is since both IPv6 networks are dynamically assigned, you won't be able to use NPT to translate between the interfaces if one or the other goes down. My article wasn't written to account for dynamically assigned IPv6 translation failover.

If you had static networks, then this article doesn't begin to address how to properly set that up. :)

stevebion

Sigh..Comcast Ipv6 will have to do.
ATT ipv6 was actually working after getting the line. Was able to do "Gateway Groups" and failover seemed to behave normally.
Currently, "IPv6 delegated Prefix Subnet" is empty, which makes me think, ATT not delegating Ipv6 at all, but that's an ATT issue, not pfsense :/

ttmcmurry

Seems netgate changed forum rules and it's no possible to edit a post if it's older than an hour...

There's an error in the DHCPv6 config. It needs to be Managed, not assisted. If it is set to assisted, one of two things happens - AT&T gateway may reply with an IPv6 address outside of your defined scope (SLAAC), or any device after the first will not get an IPv6 address, or both.

I believe I may need to look into a firewall rule to block slaac on the WAN interface.. but need more time to research and test.

JKnott

@ttmcmurry said in Multiple IPv6 Prefix Delegation over AT&T Residential Gateway for pfSense 2.4.5:

I would pick one interface or the other for IPv6 unless you have static IPv6 addresses from both providers. The problem you'll run into is since both IPv6 networks are dynamically assigned, you won't be able to use NPT to translate between the interfaces if one or the other goes down.

Actually, IPv6 is designed to support multiple connections and both prefixes will appear on the clients. I have done that with ULA and GUA addresses, but not 2 GUA. If one fails, then the other is still there. However, you'd need some mechanism, perhaps the metric, to favour one over the other when both are up. IPv6 also supports a priority, but I haven't done anything with that.

ttmcmurry

Overall, the end-user experience is what I would rely upon as the measurement for success. Keep in mind this article's context is a home environment where AT&T is the primary ISP and likely the only ISP.

If there's a working example of how to handle 2 GUAs which are dynamically allocated from both ISPs, on pfSense - and the failover mechanism is no worse than / as reliable as IPv4 gateway groups - then I'd love to see how that works.

In the IPv4 world, there are a few good reasons why multiple gateway IPs with assigned metrics are ultimately an unused feature in virtually all corporate & home networks. pfSense doesn't even directly support this in its DHCP server - though it could be manually overridden with Options. My guess is similar concerns would exist for IPv6. In the end, it always boils down to user experience. Still, I'd enjoy seeing a working theory in action. :)

JKnott

@ttmcmurry

There's a router priority setting on the Router Advertisement page and the help says:

If multiple IPv6 routers exist on the same network segment, they can indicate to clients in which order they should be used. If a high priority router becomes unavailable, clients will try a normal priority router, and finally a low priority router. Select either Low, Normal, or High from the list. If there is only one router on the network, use Normal.

So, you'd use 2 or 3 routers, configured for different ISPs. You'd normally use the highest priority one. However, if you are getting different address ranges, then you will have different addresses used when one fails. This isn't an issue for outgoing connections but would mess up anything for which there's a DNS entry. I use the ULA for that reason. My local DNS points to the ULA address.

A more appropriate would be in a business setting, where you "own" your address block, instead of being assigned one by an ISP. In this case, you'd have your own autonomous system and carriers, not ISPs, would route to your address, as advertised by a routing protocol such as OSPF.
In this case, you'd have 2 connections to 1 or more carriers and you'd select one to be your main connection. Of course, home and small business users are unlikely to do this.

There are several improvements in IPv6 v IPv4, beyond addresses. The concept of multiple addresses on a network is one of them. As I mentioned, I have both ULA and GUA prefixes on mine.

ttmcmurry

If that design were implemented in a LAN, it would potentially bypass pfSense. The firewall needs to be the only routing interface in a subnet with end-user devices to guarantee security as configured. Theoretically, sure, I get it, doesn't seem practical in the real world.

Indulging the idea - I'd attempt multi-metric routing on one or more routers ahead of pfSense's WAN interface to set proper expectations of pfSense's routing behavior, and to simplify pfSense's configuration while maintaining security integrity on the LAN nets.

Diving deeper in the rabbit hole - if pfSense were in transparent mode and was between each layer-1 connection, then sure, I can dig this. But that's way off this article's main topic.

I wholly agree with you on static addressing, that's the most ideal scenario and perfect for failover configurations, not to mention ease of configuration & maintenance over time. AT&T still doesn't allow static IPv6 address allocation for residential customers.