Multiple IPv6 Prefix Delegation over AT&T Residential Gateway for pfSense 2.4.5

[stevebion]

Sigh..Comcast Ipv6 will have to do.
ATT ipv6 was actually working after getting the line. Was able to do "Gateway Groups" and failover seemed to behave normally.
Currently, "IPv6 delegated Prefix Subnet" is empty, which makes me think, ATT not delegating Ipv6 at all, but that's an ATT issue, not pfsense :/

[ttmcmurry]

Seems netgate changed forum rules and it's no possible to edit a post if it's older than an hour...

There's an error in the DHCPv6 config. It needs to be Managed, not assisted. If it is set to assisted, one of two things happens - AT&T gateway may reply with an IPv6 address outside of your defined scope (SLAAC), or any device after the first will not get an IPv6 address, or both.

I believe I may need to look into a firewall rule to block slaac on the WAN interface.. but need more time to research and test.

[JKnott]

@ttmcmurry said in Multiple IPv6 Prefix Delegation over AT&T Residential Gateway for pfSense 2.4.5:

I would pick one interface or the other for IPv6 unless you have static IPv6 addresses from both providers. The problem you'll run into is since both IPv6 networks are dynamically assigned, you won't be able to use NPT to translate between the interfaces if one or the other goes down.

Actually, IPv6 is designed to support multiple connections and both prefixes will appear on the clients. I have done that with ULA and GUA addresses, but not 2 GUA. If one fails, then the other is still there. However, you'd need some mechanism, perhaps the metric, to favour one over the other when both are up. IPv6 also supports a priority, but I haven't done anything with that.

[ttmcmurry]

Overall, the end-user experience is what I would rely upon as the measurement for success. Keep in mind this article's context is a home environment where AT&T is the primary ISP and likely the only ISP.

If there's a working example of how to handle 2 GUAs which are dynamically allocated from both ISPs, on pfSense - and the failover mechanism is no worse than / as reliable as IPv4 gateway groups - then I'd love to see how that works.

In the IPv4 world, there are a few good reasons why multiple gateway IPs with assigned metrics are ultimately an unused feature in virtually all corporate & home networks. pfSense doesn't even directly support this in its DHCP server - though it could be manually overridden with Options. My guess is similar concerns would exist for IPv6. In the end, it always boils down to user experience. Still, I'd enjoy seeing a working theory in action. :)

[JKnott]

@ttmcmurry

There's a router priority setting on the Router Advertisement page and the help says:

If multiple IPv6 routers exist on the same network segment, they can indicate to clients in which order they should be used. If a high priority router becomes unavailable, clients will try a normal priority router, and finally a low priority router. Select either Low, Normal, or High from the list. If there is only one router on the network, use Normal.

So, you'd use 2 or 3 routers, configured for different ISPs. You'd normally use the highest priority one. However, if you are getting different address ranges, then you will have different addresses used when one fails. This isn't an issue for outgoing connections but would mess up anything for which there's a DNS entry. I use the ULA for that reason. My local DNS points to the ULA address.

A more appropriate would be in a business setting, where you "own" your address block, instead of being assigned one by an ISP. In this case, you'd have your own autonomous system and carriers, not ISPs, would route to your address, as advertised by a routing protocol such as OSPF.
In this case, you'd have 2 connections to 1 or more carriers and you'd select one to be your main connection. Of course, home and small business users are unlikely to do this.

There are several improvements in IPv6 v IPv4, beyond addresses. The concept of multiple addresses on a network is one of them. As I mentioned, I have both ULA and GUA prefixes on mine.

[ttmcmurry]

If that design were implemented in a LAN, it would potentially bypass pfSense. The firewall needs to be the only routing interface in a subnet with end-user devices to guarantee security as configured. Theoretically, sure, I get it, doesn't seem practical in the real world.

Indulging the idea - I'd attempt multi-metric routing on one or more routers ahead of pfSense's WAN interface to set proper expectations of pfSense's routing behavior, and to simplify pfSense's configuration while maintaining security integrity on the LAN nets.

Diving deeper in the rabbit hole - if pfSense were in transparent mode and was between each layer-1 connection, then sure, I can dig this. But that's way off this article's main topic.

I wholly agree with you on static addressing, that's the most ideal scenario and perfect for failover configurations, not to mention ease of configuration & maintenance over time. AT&T still doesn't allow static IPv6 address allocation for residential customers.

[chriv]

Hi!

I'm not new to pfSense, but out of practice.

I'm struggling trying to get IPv6 working correctly with pfSense on AT&T Fiber with a BGW200-710. While all I really need is Stateless config with working IPv6 addresses through RAs on both the WAN interface and all devices on my LAN, I do have pfSense running on a device with four Ethernet ports and I recognize that using prefix delegation for each port (LAN, OPT1, OPT2) is good design.

I'd really like to throw the BGW200-710 RG in the trash (I only have gigabit Internet; no other services from AT&T), but I've been unable so far to make my devices work with a direct connection to the ONT, so I'm stuck with it (for now).

My WAN adapter correctly gets an IPv6 address. My LAN adapter does not. Devices on my LAN do not. I decided to try your solution to see if that improved things. It did not (for me). I have nothing on the OPT1 or OPT2 ports because I haven't decided how I'm going to use them yet.

Here is the version of your script I customized:

interface igb0 {
	send ia-na 0;
	send ia-pd 0;
	send ia-pd 1;
	send ia-pd 2;
	request domain-name-servers;
	request domain-name;
	script "/var/etc/dhcp6c_wan_dhcp6withoutra_script.sh";
};
id-assoc na 0 { };
id-assoc pd 0 {
	prefix-interface igb1 {
		sla-id 0;
		sla-len 0;
	};
};
id-assoc pd 1 {
	prefix-interface igb2 {
		sla-id 0;
		sla-len 0;
	};
};
id-assoc pd 2 {
	prefix-interface igb3 {
		sla-id 0;
		sla-len 0;
	};
};

I think I followed your instructions to the letter. I was a little confused with why I didn't have to set Prefix Delegation options on the DHCPv6 Server used to assist the RAs, but I didn't mess with those settings as your instructions did not include that.

I don't know how to troubleshoot this because I don't know where the IPv6 and DHCPv6 debugging is saved/logged.

Any help is appreciated.

EDIT: Never Mind. Your script worked perfectly once I fixed a configuration error on my side (I sloppily applied it to the DHCP IPv4 configuration instead of the DHCPv6 client). Once I fixed the issue, both my LAN interface and my LAN devices correctly configured IPv6 addresses in the properly delegated subnet.

Thanks!

[thomas.munn]

I seem to be getting somewhere. I have almost the exact configuration as above. I am getting the following in my logs:

Oct 24 01:35:15 firewall dhcp6c[20706]: IA_PD: ID=1, T1=1800, T2=2880
Oct 24 01:35:15 firewall dhcp6c[20706]: get DHCP option IA_PD prefix, len 25
Oct 24 01:35:15 firewall dhcp6c[20706]: IA_PD prefix: 2600:1700:1852:88de::/64 pltime=3600 vltime=3600
Oct 24 01:35:15 firewall dhcp6c[20706]: get DHCP option IA_PD, len 41
Oct 24 01:35:15 firewall dhcp6c[20706]: IA_PD: ID=2, T1=1800, T2=2880
Oct 24 01:35:15 firewall dhcp6c[20706]: get DHCP option IA_PD prefix, len 25
Oct 24 01:35:15 firewall dhcp6c[20706]: IA_PD prefix: 2600:1700:1852:88dd::/64 pltime=3600 vltime=3600
Oct 24 01:35:15 firewall dhcp6c[20706]: get DHCP option client ID, len 14
Oct 24 01:35:15 firewall dhcp6c[20706]: DUID: 00:01:00:01:25:f0:a5:cc:00:25:90:b9:e6:6c
Oct 24 01:35:15 firewall dhcp6c[20706]: get DHCP option server ID, len 14
Oct 24 01:35:15 firewall dhcp6c[20706]: DUID: 00:01:00:01:26:f3:80:30:e8:b2:fe:47:f8:b1
Oct 24 01:35:15 firewall dhcp6c[20706]: get DHCP option preference, len 1
Oct 24 01:35:15 firewall dhcp6c[20706]: preference: 255
Oct 24 01:35:15 firewall dhcp6c[20706]: get DHCP option DNS, len 16
Oct 24 01:35:15 firewall dhcp6c[20706]: get DHCP option domain search list, len 14
Oct 24 01:35:15 firewall dhcp6c[20706]: XID mismatch
Oct 24 01:35:15 firewall dhcp6c[20706]: receive reply from fe80::eab2:feff:fe47:f8b1%igb1 on igb1
Oct 24 01:35:15 firewall dhcp6c[20706]: get DHCP option client ID, len 14
Oct 24 01:35:15 firewall dhcp6c[20706]: DUID: 00:01:00:01:25:f0:a5:cc:00:25:90:b9:e6:6c
Oct 24 01:35:15 firewall dhcp6c[20706]: get DHCP option server ID, len 14
Oct 24 01:35:15 firewall dhcp6c[20706]: DUID: 00:01:00:01:26:f3:80:30:e8:b2:fe:47:f8:b1
Oct 24 01:35:15 firewall dhcp6c[20706]: get DHCP option status code, len 19

I don't seem to have any ipv6 on any addresses(except on wan!) . I used the same as above, but with slightly different interface assignments. The system really doesn't tell me why it isn't working even in advanced debug mode.

If I go to 'defaults' I do get working ipv6 on lan only. I did notice another script dhcp6c_wan_script.sh should I try that?

[thomas.munn]

Never Mind, like the other guy, I had to use the right script. Seems to be working now. Why does it take a bit for an ipv6 address to get assigned? I see the same behaviour on some of my vm's in a different isp.

[thomas.munn]

How to make netgate work with ipv6 on multiple interfaces (redo with table)

I've been working on this one for a while. This is the result of others posting their work across various forums, reading BSD docs, and plenty of testing as a result of needing something to do while being stuck at home. :)

The purpose of this is to make it easier for AT&T customers who wish to assign more than one IPv6 prefix delegation inside their pfSense firewall to more than one internal network interface. I am providing an example dhcp.conf script and explaining what's needed step-by-step. AT&T customers must have been furnished a Residential Gateway (Pace 5268AC / Arris BGW210-700, possibly others) and have configured the RG in DMZ+/IP Passthrough mode. This has been written with pfSense 2.4.5 in mind.

Why do this? In short, AT&T U-Verse & Fiber customer equipment is assigned a /60 and can only hand out eight /64 prefix delegations. It is not possible to request a larger PD, however it is possible to request multiple /64 PDs from pfSense's WAN interface. Since the pfSense UI does not expose this functionality directly, it is possible to take advantage of it by supplying a dhcp.conf to override pfSense DHCP6 behavior available from the UI.

I welcome improvements and feedback and will be happy to update this to help make other's lives easier and work within pfSense native functionality as much as possible.

Once this script is in place, if you need to reassign interfaces & prefix delegations, the script has to be updated. You will need to edit the IPv6 Track Interface Prefix ID on the LAN/OPT interfaces with the IA-PD you specify in the .conf file.

Assumptions:

• The WAN interface IPv6 Configuration type is configured for "none" or,
• The WAN interface IPv6 Configuration type is configured for DHCP6 and IPv6 Prefix Delegation is set /60
• The WAN interface IPv6 DHCP6 Client Option "Do not allow PD/Address release" is UNCHECKED
• The LAN/OPT interfaces' DHCP6 option is set to "none"
• DHCPv6 Server & RA -> DHCP6 Server -> Disabled
• DHCPv6 Server & RA -> Router Advertisements -> Defaults (Router Mode: Assisted)

Step one: Update the "interface" stanza

• Make a local copy of the code below
• Look at Interfaces -> Assignments -> Network Port for the adapter associated to the WAN interface
• Replace the adapter in the interface stanza below with the WAN adapter network port name below, e.g. hn0, igb0, vmx0, eth0, etc
• If using VLANs, remember to use numerical subinterface number e.g. hn0.10 for VLAN 10
• IA-NA Note: The IA-NA is an arbitrary number. A unique number must be chosen for each device connected to the AT&T residential gateway (RG) which will request a prefix delegation from the RG. If only one device will be requesting PDs from the RG (i.e. this pfSense firewall), then "ia-na 0" is fine.

Step two: Update the "ia-pd" stanzas

• Look at Interfaces -> Assignments -> Network Port for the adapter associated to each LAN/OPT interface(s)

• Replace the prefix-interface "hn1" with LAN/OPT adapter network port name

• If using VLANs, remember to use numerical subinterface number e.g. hn0.10 for VLAN 10

• You can arbitrarily assign adapters to PDs, staring with PD 0 and working down the list

  • SLA-ID is always zero (0)
  • SLA-LEN is always zero (0)

• Formatting is specific. Each new PD stanza needs to be formatted exactly as PD0; only update the adapter name

• If you are not assigning an adapter to a PD, leave it blank as shown below

• Note: You do not need to request all 8 PDs if you don't need them. You may remove any "send ia-pd" & "id-assoc pd x" statements where you aren't assigning them to interfaces.

• Note

  : Assigned PDs will result in numerically different networks, depending on the RG

  • Pace 5268AC first assigns F then decrements to 8 to PD 0-7, i.e. PD0 = ::xxxF::/64
  • Arris BGW210-700 first assigns 8 then increments to F to PD 0-7, i.e. PD0 = ::xxx8::/64

Step three: Add the script to pfSense

• Create this file on pfSense under Diagnostics -> Edit File
• Copy and paste your edited script into the text window
• In the grey filename box, enter /usr/local/etc/rc.d/att-rg-dhcpv6-pd.conf
• Click on Save

Step four: Edit the WAN interface

• Set IPv6 Configuration Type to "DHCP6" (it may already be set, see "assumptions" above)
• Under DHCP6 client configuration, select Configuration Override
• Enter the following in Configuration File Override: /usr/local/etc/rc.d/att-rg-dhcpv6-pd.conf
• Click on Save and Apply the changes

Step five: Edit the LAN/OPT interface(s), one at a time

• Set the IPv6 Configuration Type to "Track Interface"
• Set the Track IPv6 Interface -> IPv6 Interface to the WAN's interface name ("WAN" is the default name)
• Set the IPv6 Prefix ID to the PD number configured in the .conf file
• Click on Save and Apply the changes

Step six: Enable pfSense DHCPv6 Server & Test

• For each configured interface..
• DHCPv6 Server & RA -> DHCPv6 Server -> Enable -> Save
• DHCPv6 Server & RA -> Router Advertisements -> Router Mode -> Assisted (Default)
• Test a client in each configured, connected network

Step Seven: FOLLOW THE DIRECTIONS! Easy to foobar the typing. Be patient, slow, and it will eventually work. I did add a completed configuration, and a table which helps a little in doing the work.

interface igb1 {
	send ia-na 0;
	send ia-pd 0;
	send ia-pd 1;
	send ia-pd 2;
	request domain-name-servers;
	request domain-name;
	script "/var/etc/dhcp6c_wan_script.sh";
};
id-assoc na 0 { };
id-assoc pd 0 {
	prefix-interface igb3 {
		sla-id 0;
		sla-len 0;
	};
};
id-assoc pd 1 {
	prefix-interface igb0 {
		sla-id 0;
		sla-len 0;
	};
};
id-assoc pd 2 {
	prefix-interface igb2 {
		sla-id 0;
		sla-len 0;
	};
};

Interface delegations

Interface	Name	Association
igb1	WAN
igb3	LAN	0
igb0	IOTVIDEO	1
DMZ	igb2	2

I highly recommend that you backup your configuration and save. At&t can be very annoying since they don't hand out the whole /60 Oh well.

[ttmcmurry]

Glad to see that y'all got it working. :)

Regarding why does it take so long to get an address .. My best guess is more at the AT&T RG's IPv6 stack is simply slow. The RG already knows what /60 is assigned. Delegating that out should be in the tenths of a second to a second to work, end-to-end.

The reality is IPv6 is treated like a second class citizen by AT&T and if I'm being truthful, they have zero reason to prioritize developing a better, faster stack. Like chriv, I would be happy to interface with the ONT and be done with it.

One of these days I wanted to do a repost of everything for a 'newer' version of pfSense .. since the forum moderators started locking thread edits.

[thomas.munn]

@ttmcmurry I added the table just because it makes the whole thing a little simpler. It’s just crazy that AT&T does all this nonsense trying to prevent me from doing multiple segments with IPv6. Honestly I think SLAAC is a lot easier than stupid DHCPv6

[ttmcmurry]

It's important to remember SLAAC doesn't register with DNS or DHCP. This is one of the reasons why it isn't the best solution for all situations. It might be fine for home use if the apps you use don't need DNS to find other devices. SLAAC also doesn't support features like DHCP options, which are commonly used in businesses and especially with VOIP systems.

Between lack of configurability, no ability to perform DNS registration, thus making life very difficult for anyone running an Active Directory domain, SLAAC has a place, but it doesn't work great for corporate environments.

Also, be careful when using SLAAC if you're using pfSense for DNS filtering. When I had it enabled, devices were being configured by the AT&T router as the DNS source and was bypassing pfSense.

[JKnott]

@ttmcmurry

With SLAAC, you point your DNS at the consistent address, which is often based on the MAC address. On the other hand, if you use DHCPv6, you won't be able to use Android devices, as for some stupid reason it's not supported. I have never been able to determine any reason for it not to be supported, other than the developer doesn't like it. His arguments against it could also apply to SLAAC.

[ttmcmurry]

@JKnott
I suppose I have 2 observations:

1. Android devices will still work on the IPv4 stack. If Managed DHCPv6 mode is enabled, you're correct, those will not get an IPv6 address.

2. With respect to using pfSense to perform DNS filtering while using stateless autoconfig/Assisted mode/SLAAC.
   When using DHCPv6 in this manner, when pfSense's IPv6 WAN IP is dynamic, problems become apparent quickly. Since a static IPv6 WAN address is not available to configure in pfSense's DHCPv6 RA DNS Server list, what happens is the RA is passed on from the AT&T RG. Therefore in this configuration, DNS will bypass pfSense..

If you're not using pfSense to filter/route DNS toward a filtering provider, then this isn't a concern and go stateless and you're probably fine with allowing AT&T to monitor where you go on the internet.

Yes, a configuration paradox exists because Google won't support DHCPv6 on Android. The addressing problems are resolved when using DHCPv6 in Managed mode. However, Android devices cannot be allocated IPv6 addresses as a result. In the end the same underlying goal is achievable, albeit without direct IPv6 support.

It's about compromising & determining which compromise covers the majority of your needs.

Of course I would love to be proven wrong. The moment I set DHCPv6 for Stateless or Assisted, my subnets get addresses from the AT&T RG. If that's a configuration error on my side (firewall rule on wan interface??) I'd love to fix that and update this doc!

[lilchancep]

To anyone that comes to this thread I have made a GitHub to have all the changes that have been made throughout this thread. Also for easier reading. [all input is welcome]

Link

@ttmcmurry I hope this is okay, I made sure to give you 190% credit as this is your hard work.

[deet]

A couple of notes:

• On the WAN interface settings page, setting the prefix size to /60 has no effect on dhcp6c or what the dhcp6c client requests from the ISP. The reason for doing that is to tell the web UI what range of prefix IDs to allow on the LAN configuration pages. If you leave it at /64, it will only allow '0', and it won't let you set that on more than one interface. You're setting it to /60 to trick the web UI validator, not because you want a /60 from your ISP.
• With this in mind, although you can use arbitrary values for ia_pd in your config file, you will want to stick to 0–8 for prefix IDs so that they will validate in the UI.

[ttmcmurry]

@deet have you tested this effect?

in 2.4.5, the UI doesn't allow other interfaces to track a wan interface to use a metric higher than 0 when the wan is set to /64.

It was necessary to set both the script and the corresponding interface in the UI to use the same prefix id.

I believe there's a DHCP Service dependency on the interface in the UI to use the tracked interface prefix specified there. Said another way, I do not believe the DHCP v6 Service evaluates a dhcp.conf file if the WAN's DHCP Override contains a conf; it has to use the value specified in the interface's UI.

If this behavior has changed in "p1" - could you provide confirmation and the updated steps for others to follow?

[ttmcmurry]

@lilchancep - thank you. Ever since the netgate forum rules change, it became difficult to keep this up to date. I have been waiting for 2.5 to come out to make any significant overhaul, but would end up in the same place, unable to update it after a few hours.

I've starred the page you created in github. :)

[deet]

@ttmcmurry said in Multiple IPv6 Prefix Delegation over AT&T Residential Gateway for pfSense 2.4.5:

@deet have you tested this effect?

I am describing my own findings, so yes.

I was initially confused about the need to set the prefix to /60 in the WAN settings before setting the override file path. "Why would it matter what the prefix length is set to, if we're just going to override it in the .conf file?" I wondered. So I tried setting the WAN prefix length to different values to see what would happen.

I found that the .conf override file is what governs the behavior of dhcp6c, not the value in the field, "overriding" indeed.

Setting the WAN prefix length to /60 does not, therefore, have any bearing on what prefix dhcp6c requests.

But the pfsense web GUI does use that value when validating the prefix IDs on the LAN pages. If you leave it at /64, the only valid prefix ID is 0. The validator looks at /64 and calculates only one available prefix ID. By setting the WAN prefix length to /60, however, the validator calculates a range of up to 16 IDs. (Of which 8 are actually available, but that's a gateway limitation, not an IPv6 thing.)

So like I say, setting it to /60 matters not because it tells dhcp6c to do anything relevant, but because without it, the pfsense web GUI validator won't let you set the values you need to set in the LAN pages.

in 2.4.5, the UI doesn't allow other interfaces to track a wan interface to use a metric higher than 0 when the wan is set to /64.

Right.

It was necessary to set both the script and the corresponding interface in the UI to use the same prefix id.

With this method, yes. But ordinarily, no; "ia_pd" in the .conf file doesn't have to be anything in particular; it is purely an index for configuration purposes. And, ordinarily, the prefix ID in the GUI doesn't have to be anything particular, either, except that it needs to be within a range valid for the delegated prefix, because ordinarily it would end up appended to the delegated prefix being tracked. These are, ordinarily, separate values unrelated to each other.

For example, if this were Comcast, and we'd been properly delegated a /60, we could put 'f' as the ID in one interface and '9' as the ID in another, and each network would get a prefix ending in 'f' and '9' respectively. Thus can this field be set to anything we want as long as it results in a valid prefix.

But because of the way we're fitting square pegs into round holes, we need the ia_pd in the .conf file to match the "prefix ID" in the GUI. Not because it will be appended to the delegated prefix as usual — because the delegated prefix we get will go straight to the tracking interface. Rather, we are using that field to tell pfSense which /64 prefix (referring to the ia_pd set in the conf file) goes to which interface (which is also indicated in the conf file).

In practice, the prefix we get will not be the usual <delegated /60 prefix> + <"prefix suffix" to form a complete /64 prefix>, but rather <whatever /64 prefix is assigned to us by the gateway>.

Apologies if I am trying to dig myself out of a hole with this explanation. It makes sense the way I'm reading it but I get that it's confusing and I am probably not being clear. It's just, after wondering why setting /60 would matter in the GUI when we were using a conf override file (and the answer is, for GUI-related reasons only), my next question was why the prefix ID set in the GUI would have any bearing, since the /64 being delegated has no more bits available for any suffixes to be appended. The answer is, the ID is not actually being appended as usual. Instead, through this method, we are setting not a "prefix suffix" but rather telling pfsense which ia_pd dhpc6c should associate with the interface.

I believe there's a DHCP Service dependency on the interface in the UI to use the tracked interface prefix specified there.

The dependency is in how the pfsense GUI works, not in the DHCP service.

Said another way, I do not believe the DHCP v6 Service evaluates a dhcp.conf file if the WAN's DHCP Override contains a conf; it has to use the value specified in the interface's UI.

Sort of. The .conf file does indeed take full control of what actually happens to all interfaces. But we have to do this GUI stuff to make sure the GUI matches what dhcp6c is doing.

If this behavior has changed in "p1" - could you provide confirmation and the updated steps for others to follow?

The steps are the same! We're doing all the right things. My comments are as much for my own future reference as for anything. "Wait why does it matter what we do in the GUI if we're overriding everything in the conf file? Oh, it's because we need the GUI to match what the conf file is doing, and we have to fudge some things in order for that to happen." That's all.

Clear as mud?