Multiple IPv6 Prefix Delegation over AT&T Residential Gateway for pfSense 2.4.5

[lolipoplo]

@ttmcmurry I am using the IP Passthru (pfS taking the WAN IP of RG)

I was just doing nmap syn scan on a /64 prefix, which is of course likely just SYN_SENT:CLOSED so I didn't read too much into pf/RG's state tables.

I have no idea of how RG handles state table either. Would CLOSED/NO_TRAFFIC be prioritzed for deletion? then the least recently matched states? I was making this guess because it did not impact my ESTABLISHED. (downloading huge file or running speedtest.net)

I have no devices other than pfS connected to RG.

Maybe I should try again with with more than 8k ESTABLISHED and then see the behavior

[JKnott]

@lolipoplo said in Multiple IPv6 Prefix Delegation over AT&T Residential Gateway for pfSense 2.4.5:

I was just doing nmap syn scan on a /64 prefix

Ouch!!! With 18.4 billion, billion addresses to scan, that's going to take a while!

One of the security benefits of IPv6 is the address space is so sparse it's virtually impossible to scan even a single /64 for targets.

[ttmcmurry]

@jknott

Nothing one of those fancy Threadripper 5000s with 64 cores & 128 threads can't handle.

[JKnott]

@ttmcmurry

It doesn't matter how fast your CPU is, you still have to work through whatever the Internet will allow.

[lolipoplo]

@jknott of course I didn't finish scanning, just enough to keep pf's state table blown up for a good duration

[djdawson]

@lolipoplo I think the biggest issue is how the two devices clear out states when their respective tables get full. The pfsense docs describe the process reasonably well, and the gist of it is as the table fills the various state lifetimes are reduced in a liner fashion between the min and max thresholds, with the entire state table getting flushed if the max threshold is crossed. That table flush would seem to be a traumatic event, since all existing connections would likely be dropped, but that max threshold defaults to being like 120% of the state table size so it should only get hit if there's a significant increase in the number of simultaneous connections. Since you can also tweak the individual state timers in pfsense you also have some control from that side of things, and this whole approach seems intended to be a more intelligent way of dropping the oldest, least important states first when the state table fills.

I couldn't find any docs that describe how the AT&T RG deals with a full state table, but I'm assuming it's also an oldest-first based approach and likely not as fancy as the pfsense methodology, though it appears to run a version of Linux so it could also be pretty full-featured.

The easiest way to know which table causes the least trouble when it fills would be to test it as @lolipoplo has been doing, but in my experience such testing is never as easy as it seems like it should be.

Thanks for the great discussion everybody!

[ttmcmurry]

@jknott that was an attempt at humor. You're absolutely right, there's no realistic way for a full /64 scan to complete in a reasonable amount of time due to the address space and bandwidth limitations.

Perhaps a more realistic approach is to track MAC/IPv6 addresses over a period of time and then port scan those IPs. I've done similar things when dealing with IPv4 /8s.

[lolipoplo]

@djdawson some people have rooted their RGs which I cannot on my model. They can check if it's using netfilter for traffic between WAN/LAN and other settings related to state limit and eviction policy

EDIT:
I see conntrack, netfilter, iptables in RG/legal.txt. High likelihood it's using netfilter. I don't understand why ATT made it in such a bizarre way of assigning its own WAN IPv4 to LAN in passthru mode, must have created lots of ugly, mangled rules and violated tons of RFCs.

If they really insist customers use their RGs, just assign at least 2 IPv4s so that one of them can be delegated to LAN

[JKnott]

@ttmcmurry said in Multiple IPv6 Prefix Delegation over AT&T Residential Gateway for pfSense 2.4.5:

Perhaps a more realistic approach is to track MAC/IPv6 addresses over a period of time and then port scan those IPs

MAC addresses where? If they're not on the local LAN, you'll never see them.

[lilchancep]

Just to give a n=1 sample size. I have never set "State Table" settings and this stuff has been working great for me. I have been using this on opnsense for the past year or so (pfsense prior). I should probably add notes that this works on opnsense as well on the github repo.

[bplein]

@lilchancep I tried out your Github repo and it worked flawlessly. Thank you for the efforts.

pfSense 2.6.0-RELEASE
NVG599
Software version 11.6.0h0d48

Now the bigger question is why can't we have this "just work" from the pfSense GUI (and not have to have end users work around it as you have?)

In any event, this is awesome, thank you!

[lolipoplo]

@bplein I did tcpdump on WAN and saw as many PD packets as I configured in v6. I remember when it was comcast, I only had 1 PD request and 1 response with many prefixes. likely the RG is configured to only hand out /64 prefixes with a range of /62.

you'll see similar config options in pf's dhcpv6 server, the PD range, and PD prefix

pfsense probably just didn't expect such an ugly config by ATT thus not having this option built into web ui

[bplein]

@ttmcmurry Here's some data points:

NVG599
IP Passthrough: Disabled
Cascaded Router Option: Enabled (I have 5 static IPs from AT&T, plus my single dynamic)

pfSense operates on a private IP from the RG's LAN DHCP range.

The 5 private IPs are set up as IP Alias Virtual IPs.

The NAT table from the RG shows it is tracking outbound state for anything coming from the pfSense box (192.168.1.65), plus anything coming from any of my 5 private IPs, plus all ipV6 traffic.

So (reading your last post), if I reconfigured to use one of my 5 static IPs as the pfSense WAN IP, then I'd bypass NAT and the state table? Is this known to work?

[ziggo0]

@bplein said in Multiple IPv6 Prefix Delegation over AT&T Residential Gateway for pfSense 2.4.5:

So (reading your last post), if I reconfigured to use one of my 5 static IPs as the pfSense WAN IP, then I'd bypass NAT and the state table? Is this known to work?

Curious about this also. I recently got AT&T 1Gbps with a BGW320-505 and would like to avoid it's state table limit restriction with my pfSense box.

[ttmcmurry]

@bplein

Your last sentence is accurate. For reasons that may only make sense to AT&T, once the static public IP in the RG is associated to pfSense's WAN MAC, it enters the closest thing to IP Passthrough that it's capable of.

The only downside is pfSense gets 1 public IP.

The RG's state table drops off to a few hundred states, which I assume is the RG itself talking to AT&T. Meanwhile pfSense is tracking everything it has open independently of the RG. Assuming enough traffic is being generated behind pfSense, you can compare both state tables and see that both are entirely different.

[SirSilentBob]

Hello everyone!

Is something like this script adaptable for Verizon FIOS, by chance?

I'm in the similar situation, where I get a /56 from them, and set my interfaces to Track Interface, give each interface it's own IPv6 Prefix ID, etc. But it only seems to work on a single LAN interface.

FIOS is a little different it seems from AT&T, in that the WAN IPv6 settings must have "Use IPv4 connectivity as parent interface" checked, "Request only an IPv6 prefix" checked and "Send IPv6 prefix hint" checked.

I attempted to basically adapt the file here and use it, but I think the lack of those options is maybe the holdup.

The custom file I created looks like this:

interface igb0 {
	send ia-na 0;
	send ia-pd 0;
	send ia-pd 1;
	send ia-pd 2;
	script "/var/etc/dhcp6c_wan_script.sh";
};
id-assoc na 0 { };
id-assoc pd 0 {
	prefix-interface igb1 {
		sla-id 0;
		sla-len 0;
	};
};
id-assoc pd 1 {
	prefix-interface igb2 {
		sla-id 0;
		sla-len 0;
	};
};
id-assoc pd 2 { 
	prefix-interface igb3 {
		sla-id 0;
		sla-len 0;
	};
};

With IPv6 working on just a single interface, my pfSense dhcp6c_wan.conf looks like this:

interface igb0 {
	send ia-pd 0;	# request prefix delegation
	request domain-name-servers;
	request domain-name;
	script "/var/etc/dhcp6c_wan_script.sh"; # we'd like some nameservers please
};
id-assoc pd 0 {
	prefix ::/56 infinity;
	prefix-interface igb1 {
		sla-id 0;
		sla-len 8;
	};
};

Contents of radvd.conf with the single LAN interface working looks like:

# Automatically Generated, do not edit
# Generated for DHCPv6 Server lan
interface igb1 {
	AdvSendAdvert on;
	MinRtrAdvInterval 200;
	MaxRtrAdvInterval 600;
	AdvDefaultLifetime 1800;
	AdvLinkMTU 1500;
	AdvDefaultPreference high;
	AdvManagedFlag on;
	AdvOtherConfigFlag on;
	prefix 2600:4040:XXXX:XXXX::/64 {
		DeprecatePrefix on;
		AdvOnLink on;
		AdvAutonomous on;
		AdvValidLifetime 86400;
		AdvPreferredLifetime 14400;
	};
	route ::/0 {
		AdvRoutePreference high;
		RemoveRoute on;
	};
	RDNSS 2600:4040:XXXX:XXXX:XXX:XXXX:XXXX:XXXX {
		AdvRDNSSLifetime 1800;
	};
	DNSSL Removed.US  {
		AdvDNSSLLifetime 1800;
	};
};

Any suggestions/possibility of implementing the same thing so I can put IPv6 on all my LANs which are igb1, igb2 and igb3, instead of it only working on igb1?

I see there's a lot of posts (on a lot of other sites, a lot on reddit, etc.) where it seems for others who are also trying to take an ISP supplied /56 and put individual /64's on multiple LANs and seemingly having the same issue, where it just doesn't work.

I know in the advanced DHCP6 client configuration, the "Prefix Interface" drop-down is set for the first LAN, and there's no way to select multiple LANs there.

It just seems like there's gotta be a way to break these large IPv6 blocks into individual /64s that could help out everyone!

Thanks for any thoughts/suggestions!

[styxl]

@sirsilentbob

This is how i did it (on an SG-5100)

interface igb0 {
        send ia-na 0;
        send ia-pd 0;
        send ia-pd 1;
        send ia-pd 2;
        send ia-pd 3;
        send ia-pd 4;
        send ia-pd 5;
        send ia-pd 6;
        request domain-name-servers;
        request domain-name;
        script "/var/etc/dhcp6c_wan_script.sh";
};
id-assoc na 0 { };
id-assoc pd 0 {
        prefix-interface ix0 {
                sla-id 0;
                sla-len 0;
        };
};
id-assoc pd 1 {
        prefix-interface ix1.101 {
                sla-id 0;
                sla-len 0;
        };
};
id-assoc pd 2 { };
id-assoc pd 3 { };
id-assoc pd 4 { };
id-assoc pd 5 { };
id-assoc pd 6 { };
id-assoc pd 7 { };

Then my LAN interfaces as setup as such;

For LAN:

For LAN2:

[SirSilentBob]

@styxl Thanks! It turns out there was another configuration oddity that was the culprit. I've been able to get it going fine on FIOS without this script. I'll keep it in mind if anything ends up changing though.

[AiC0315]

I am able to get an IPV6 address on WAN and LAN But when I go to a site to test IPV6 it fails. When I plug directly into the RG it passes the test. Am I missing something?

TIA

[JKnott]

@aic0315

What failure do you see at the test site?
Which test site are you using?