Multiple IPv6 Prefix Delegation over AT&T Residential Gateway for pfSense 2.4.5

lilchancep

@thekorn Updated the repo, let me know if you have anything else you think I should add.

thekorn

@lilchancep Now that you mention it, I suppose now is a good a time as any to add that it works with a Nokia BGW320-505 (fw 3.15.7).

(States are same as before, 8k, set pf to 7.5k)

lolipoplo

I see discussions on ATT's state table limit.

I tested it using nmap and it doesn't seem to be a concern really

I have once blown the states to almost 100 k on pfsense and the network still seemed working fine browsing the web, while the entries on BGW320 stayed at 8k.

since the RG is set to FW/NAT passthru (maybe has only one rule that's similar to allow from any to any), any packet can effective create a state, removing an old one.

I think it's a bad idea to limit the state table on pfsense, because if you are maxing out on RG's state table and pf removes your state, your mid-flight connections all have to reset

ttmcmurry

@lolipoplo - (edit) I'm going to attempt answering this without knowing the use case for your pfSense installation or the composition of the devices & traffic in your network.

If pfSense is taking on the WAN IP of the RG (meaning you aren't using static IPs from AT&T), and if the states table in the RG has 8k entries in it, then its state table is full and has been depleted. The RG is unaware of the states pfSense is tracking and vice versa. What you'd need to do is look at pfSense's state table and examine the state flags. If the RG's state table is depleted, you should see a high number of syn_sent, no_traffic (stateful/tcp), or single:no_traffic (stateless/udp). Another way of looking at it is there could be more than 8k pfSense states in established/single/multiple:established/single/multiple state - depending on how fast the RG is terminating those connections and pfSense gives up/times out; that number would be nowhere near 100K established states. You'd need to dump both state tables and compare the two - neither table is telling the full truth on its own.

If you have AT&T static IPs, and have assigned a Static IP to pfSense's WAN MAC, that configuration bypasses the RG's state table. pfSense is then in full control of its states without the RG being the middleman.

What I'm unsure about is what the behavior of "Cascaded Router" does as I've never gotten that to work .. the fee for static IPs was more than affordable considering how much time I sank into trying to get Cascaded Router to work when the benefits of using it were also unclear.

The idea expressed in this forum thread is if pfSense shares the WAN IP with the AT&T RG, then pfSense must respect the RG's state table limitations for overall network reliability. Keep in mind the RG needs states to do other things beyond internet: TV (uVerse), Voice/VoIP, MoCA, built-in WiFi, and so on. Exhausting this limit with pfSense would likely cause adverse performance for anything connected to the RG, that's not behind pfSense, but also anything connected to pfSense where the RG is likely killing states or not accepting new connections from pfSense, due to exhaustion.

lolipoplo

@ttmcmurry I am using the IP Passthru (pfS taking the WAN IP of RG)

I was just doing nmap syn scan on a /64 prefix, which is of course likely just SYN_SENT:CLOSED so I didn't read too much into pf/RG's state tables.

I have no idea of how RG handles state table either. Would CLOSED/NO_TRAFFIC be prioritzed for deletion? then the least recently matched states? I was making this guess because it did not impact my ESTABLISHED. (downloading huge file or running speedtest.net)

I have no devices other than pfS connected to RG.

Maybe I should try again with with more than 8k ESTABLISHED and then see the behavior

JKnott

@lolipoplo said in Multiple IPv6 Prefix Delegation over AT&T Residential Gateway for pfSense 2.4.5:

I was just doing nmap syn scan on a /64 prefix

Ouch!!! With 18.4 billion, billion addresses to scan, that's going to take a while!

One of the security benefits of IPv6 is the address space is so sparse it's virtually impossible to scan even a single /64 for targets.

ttmcmurry

@jknott

Nothing one of those fancy Threadripper 5000s with 64 cores & 128 threads can't handle.

JKnott

@ttmcmurry

It doesn't matter how fast your CPU is, you still have to work through whatever the Internet will allow.

lolipoplo

@jknott of course I didn't finish scanning, just enough to keep pf's state table blown up for a good duration

djdawson

@lolipoplo I think the biggest issue is how the two devices clear out states when their respective tables get full. The pfsense docs describe the process reasonably well, and the gist of it is as the table fills the various state lifetimes are reduced in a liner fashion between the min and max thresholds, with the entire state table getting flushed if the max threshold is crossed. That table flush would seem to be a traumatic event, since all existing connections would likely be dropped, but that max threshold defaults to being like 120% of the state table size so it should only get hit if there's a significant increase in the number of simultaneous connections. Since you can also tweak the individual state timers in pfsense you also have some control from that side of things, and this whole approach seems intended to be a more intelligent way of dropping the oldest, least important states first when the state table fills.

I couldn't find any docs that describe how the AT&T RG deals with a full state table, but I'm assuming it's also an oldest-first based approach and likely not as fancy as the pfsense methodology, though it appears to run a version of Linux so it could also be pretty full-featured.

The easiest way to know which table causes the least trouble when it fills would be to test it as @lolipoplo has been doing, but in my experience such testing is never as easy as it seems like it should be.

Thanks for the great discussion everybody!

ttmcmurry

@jknott that was an attempt at humor. You're absolutely right, there's no realistic way for a full /64 scan to complete in a reasonable amount of time due to the address space and bandwidth limitations.

Perhaps a more realistic approach is to track MAC/IPv6 addresses over a period of time and then port scan those IPs. I've done similar things when dealing with IPv4 /8s.

lolipoplo

@djdawson some people have rooted their RGs which I cannot on my model. They can check if it's using netfilter for traffic between WAN/LAN and other settings related to state limit and eviction policy

EDIT:
I see conntrack, netfilter, iptables in RG/legal.txt. High likelihood it's using netfilter. I don't understand why ATT made it in such a bizarre way of assigning its own WAN IPv4 to LAN in passthru mode, must have created lots of ugly, mangled rules and violated tons of RFCs.

If they really insist customers use their RGs, just assign at least 2 IPv4s so that one of them can be delegated to LAN

JKnott

@ttmcmurry said in Multiple IPv6 Prefix Delegation over AT&T Residential Gateway for pfSense 2.4.5:

Perhaps a more realistic approach is to track MAC/IPv6 addresses over a period of time and then port scan those IPs

MAC addresses where? If they're not on the local LAN, you'll never see them.

lilchancep

Just to give a n=1 sample size. I have never set "State Table" settings and this stuff has been working great for me. I have been using this on opnsense for the past year or so (pfsense prior). I should probably add notes that this works on opnsense as well on the github repo.

bplein

@lilchancep I tried out your Github repo and it worked flawlessly. Thank you for the efforts.

pfSense 2.6.0-RELEASE
NVG599
Software version 11.6.0h0d48

Now the bigger question is why can't we have this "just work" from the pfSense GUI (and not have to have end users work around it as you have?)

In any event, this is awesome, thank you!

lolipoplo

@bplein I did tcpdump on WAN and saw as many PD packets as I configured in v6. I remember when it was comcast, I only had 1 PD request and 1 response with many prefixes. likely the RG is configured to only hand out /64 prefixes with a range of /62.

you'll see similar config options in pf's dhcpv6 server, the PD range, and PD prefix

pfsense probably just didn't expect such an ugly config by ATT thus not having this option built into web ui

bplein

@ttmcmurry Here's some data points:

NVG599
IP Passthrough: Disabled
Cascaded Router Option: Enabled (I have 5 static IPs from AT&T, plus my single dynamic)

pfSense operates on a private IP from the RG's LAN DHCP range.

The 5 private IPs are set up as IP Alias Virtual IPs.

The NAT table from the RG shows it is tracking outbound state for anything coming from the pfSense box (192.168.1.65), plus anything coming from any of my 5 private IPs, plus all ipV6 traffic.

So (reading your last post), if I reconfigured to use one of my 5 static IPs as the pfSense WAN IP, then I'd bypass NAT and the state table? Is this known to work?

ziggo0

@bplein said in Multiple IPv6 Prefix Delegation over AT&T Residential Gateway for pfSense 2.4.5:

So (reading your last post), if I reconfigured to use one of my 5 static IPs as the pfSense WAN IP, then I'd bypass NAT and the state table? Is this known to work?

Curious about this also. I recently got AT&T 1Gbps with a BGW320-505 and would like to avoid it's state table limit restriction with my pfSense box.

ttmcmurry

@bplein

Your last sentence is accurate. For reasons that may only make sense to AT&T, once the static public IP in the RG is associated to pfSense's WAN MAC, it enters the closest thing to IP Passthrough that it's capable of.

The only downside is pfSense gets 1 public IP.

The RG's state table drops off to a few hundred states, which I assume is the RG itself talking to AT&T. Meanwhile pfSense is tracking everything it has open independently of the RG. Assuming enough traffic is being generated behind pfSense, you can compare both state tables and see that both are entirely different.

SirSilentBob

Hello everyone!

Is something like this script adaptable for Verizon FIOS, by chance?

I'm in the similar situation, where I get a /56 from them, and set my interfaces to Track Interface, give each interface it's own IPv6 Prefix ID, etc. But it only seems to work on a single LAN interface.

FIOS is a little different it seems from AT&T, in that the WAN IPv6 settings must have "Use IPv4 connectivity as parent interface" checked, "Request only an IPv6 prefix" checked and "Send IPv6 prefix hint" checked.

I attempted to basically adapt the file here and use it, but I think the lack of those options is maybe the holdup.

The custom file I created looks like this:

interface igb0 {
	send ia-na 0;
	send ia-pd 0;
	send ia-pd 1;
	send ia-pd 2;
	script "/var/etc/dhcp6c_wan_script.sh";
};
id-assoc na 0 { };
id-assoc pd 0 {
	prefix-interface igb1 {
		sla-id 0;
		sla-len 0;
	};
};
id-assoc pd 1 {
	prefix-interface igb2 {
		sla-id 0;
		sla-len 0;
	};
};
id-assoc pd 2 { 
	prefix-interface igb3 {
		sla-id 0;
		sla-len 0;
	};
};

With IPv6 working on just a single interface, my pfSense dhcp6c_wan.conf looks like this:

interface igb0 {
	send ia-pd 0;	# request prefix delegation
	request domain-name-servers;
	request domain-name;
	script "/var/etc/dhcp6c_wan_script.sh"; # we'd like some nameservers please
};
id-assoc pd 0 {
	prefix ::/56 infinity;
	prefix-interface igb1 {
		sla-id 0;
		sla-len 8;
	};
};

Contents of radvd.conf with the single LAN interface working looks like:

# Automatically Generated, do not edit
# Generated for DHCPv6 Server lan
interface igb1 {
	AdvSendAdvert on;
	MinRtrAdvInterval 200;
	MaxRtrAdvInterval 600;
	AdvDefaultLifetime 1800;
	AdvLinkMTU 1500;
	AdvDefaultPreference high;
	AdvManagedFlag on;
	AdvOtherConfigFlag on;
	prefix 2600:4040:XXXX:XXXX::/64 {
		DeprecatePrefix on;
		AdvOnLink on;
		AdvAutonomous on;
		AdvValidLifetime 86400;
		AdvPreferredLifetime 14400;
	};
	route ::/0 {
		AdvRoutePreference high;
		RemoveRoute on;
	};
	RDNSS 2600:4040:XXXX:XXXX:XXX:XXXX:XXXX:XXXX {
		AdvRDNSSLifetime 1800;
	};
	DNSSL Removed.US  {
		AdvDNSSLLifetime 1800;
	};
};

Any suggestions/possibility of implementing the same thing so I can put IPv6 on all my LANs which are igb1, igb2 and igb3, instead of it only working on igb1?

I see there's a lot of posts (on a lot of other sites, a lot on reddit, etc.) where it seems for others who are also trying to take an ISP supplied /56 and put individual /64's on multiple LANs and seemingly having the same issue, where it just doesn't work.

I know in the advanced DHCP6 client configuration, the "Prefix Interface" drop-down is set for the first LAN, and there's no way to select multiple LANs there.

It just seems like there's gotta be a way to break these large IPv6 blocks into individual /64s that could help out everyone!

Thanks for any thoughts/suggestions!