Multiple IPv6 Prefix Delegation over AT&T Residential Gateway for pfSense 2.4.5

ttmcmurry

I think we're also highlighting some bugs - see redmine #11187 and #11454

It might be time to start a 2.5 thread. :)

slab_bulkhead

@ttmcmurry thank you so much for your work on this! One of my biggest irritations with AT&T was the inability to pull more than one /64, while on Spectrum I can get a /56 PD with no issues at all. I have this working on 2.5 -- I had some issues at first and then discovered it was because things do not behave well with IPv6 enabled on multiple WAN interfaces at the same time (I still have the Spectrum modem connected until service cancels out at the end of the month).

I am on VDSL and therefore am unable to attempt bypassing the gateway.

djdawson

@ttmcmurry I've got a Humax BGW320-500 with my symmetric 1G service from AT&T and its NAT State Table size is 8192. Here are my RG details, copied from the device status page:

Manufacturer: HUMAX
Model Number: BGW320-500
Software Version: 2.14.4
Hardware Version: 02001F0046005

styxl

I have attempted to do this on "21.05.2-RELEASE" to no avail; the script simply doesnt seem to work anymore.

this is the error i get:

Dec  2 20:40:00 Scimitar dhcp6c[58269]: /var/etc/dhcp6c_wan.conf 20: syntax error
Dec  2 20:40:00 Scimitar dhcp6c[58269]: /var/etc/dhcp6c_wan.conf 20: fatal parse failure: exiting (1 errors)
Dec  2 20:40:00 Scimitar dhcp6c[58269]: failed to parse configuration file
Dec  2 20:40:00 Scimitar rtsold[58345]: Starting dhcp6 client for interface wan(igb0)
Dec  2 20:40:01 Scimitar reboot[98400]: rebooted by root
Dec  2 20:40:01 Scimitar syslogd: exiting on signal 15

styxl

my config is as such:

interface igb0 {
	send ia-na 0;
	send ia-pd 0;
	send ia-pd 1;
	request domain-name-servers;
	request domain-name;
	script "/var/etc/dhcp6c_wan_script.sh";
};
id-assoc na 0 { };
id-assoc pd 0 {
	prefix-interface ix0 {
		sla-id 0;
		sla-len 0;
	};
};
id-assoc pd 1 { 
	prefix-interface ix1 {
		sla-id 0;
		sla-len 0;
	};
};

dmac1418

I wanted to chime in and thank you for all your info on this thread.

I followed your guide via the GitHub adaptation and it worked no problem on 2.5.2-RELEASE (amd64) and the AT&T BGW320 modem/gateway.

interface em0 {
	send ia-na 0;
	send ia-pd 0;
	send ia-pd 1;
	send ia-pd 2;
	send ia-pd 3;
	request domain-name-servers;
	request domain-name;
	script "/var/etc/dhcp6c_wan_script.sh";
};
id-assoc na 0 { };
id-assoc pd 0 {
	prefix-interface igb0 {
		sla-id 0;
		sla-len 0;
	};
};
id-assoc pd 1 { 
	prefix-interface igb0.11 {
		sla-id 0;
		sla-len 0;
	};
};
id-assoc pd 2 { 
	prefix-interface igb0.12 {
		sla-id 0;
		sla-len 0;
	};
};
id-assoc pd 3 { };

Thanks & Happy New Year!

styxl

@dmac1418 lucky you; i gave up on making it work on the SG-5100

thekorn

Hello!

I'm running pfsense 2.6.0-release, and am trying to get ipv6 to work with multiple VLANs.

I've been able to implement all steps except step 7, enabling the DHCPv6 server and testing. When I go to dhcpv6 server & ra, dhcpv6 server, enable, save it kicks back "A valid range must be specified for any mode except Stateless DHCP."

I put in a range of :: to ::ffff:ffff:ffff:ffff, and that made it happy. Might want to update the... github.

Note that I was able to retireve an ipv6 address without this. I assume this is because SLAAC is being passed through to the residential gateway, and it's assigning the IPv6 address instead?

Thanks for all the hard work documenting this! I certainly wasn't going to figure this all out on my own!

styxl

@styxl i finally got mine to work, my config was wrong;

interface igb0 {
        send ia-na 0;
        send ia-pd 0;
        send ia-pd 1;
        send ia-pd 2;
        send ia-pd 3;
        send ia-pd 4;
        send ia-pd 5;
        send ia-pd 6;
        request domain-name-servers;
        request domain-name;
        script "/var/etc/dhcp6c_wan_script.sh";
};
id-assoc na 0 { };
id-assoc pd 0 {
        prefix-interface ix0 {
                sla-id 0;
                sla-len 0;
        };
};
id-assoc pd 1 {
        prefix-interface ix1.101 {
                sla-id 0;
                sla-len 0;
        };
};
id-assoc pd 2 { };
id-assoc pd 3 { };
id-assoc pd 4 { };
id-assoc pd 5 { };
id-assoc pd 6 { };
id-assoc pd 7 { };

lilchancep

@thekorn Updated the repo, let me know if you have anything else you think I should add.

thekorn

@lilchancep Now that you mention it, I suppose now is a good a time as any to add that it works with a Nokia BGW320-505 (fw 3.15.7).

(States are same as before, 8k, set pf to 7.5k)

lolipoplo

I see discussions on ATT's state table limit.

I tested it using nmap and it doesn't seem to be a concern really

I have once blown the states to almost 100 k on pfsense and the network still seemed working fine browsing the web, while the entries on BGW320 stayed at 8k.

since the RG is set to FW/NAT passthru (maybe has only one rule that's similar to allow from any to any), any packet can effective create a state, removing an old one.

I think it's a bad idea to limit the state table on pfsense, because if you are maxing out on RG's state table and pf removes your state, your mid-flight connections all have to reset

ttmcmurry

@lolipoplo - (edit) I'm going to attempt answering this without knowing the use case for your pfSense installation or the composition of the devices & traffic in your network.

If pfSense is taking on the WAN IP of the RG (meaning you aren't using static IPs from AT&T), and if the states table in the RG has 8k entries in it, then its state table is full and has been depleted. The RG is unaware of the states pfSense is tracking and vice versa. What you'd need to do is look at pfSense's state table and examine the state flags. If the RG's state table is depleted, you should see a high number of syn_sent, no_traffic (stateful/tcp), or single:no_traffic (stateless/udp). Another way of looking at it is there could be more than 8k pfSense states in established/single/multiple:established/single/multiple state - depending on how fast the RG is terminating those connections and pfSense gives up/times out; that number would be nowhere near 100K established states. You'd need to dump both state tables and compare the two - neither table is telling the full truth on its own.

If you have AT&T static IPs, and have assigned a Static IP to pfSense's WAN MAC, that configuration bypasses the RG's state table. pfSense is then in full control of its states without the RG being the middleman.

What I'm unsure about is what the behavior of "Cascaded Router" does as I've never gotten that to work .. the fee for static IPs was more than affordable considering how much time I sank into trying to get Cascaded Router to work when the benefits of using it were also unclear.

The idea expressed in this forum thread is if pfSense shares the WAN IP with the AT&T RG, then pfSense must respect the RG's state table limitations for overall network reliability. Keep in mind the RG needs states to do other things beyond internet: TV (uVerse), Voice/VoIP, MoCA, built-in WiFi, and so on. Exhausting this limit with pfSense would likely cause adverse performance for anything connected to the RG, that's not behind pfSense, but also anything connected to pfSense where the RG is likely killing states or not accepting new connections from pfSense, due to exhaustion.

lolipoplo

@ttmcmurry I am using the IP Passthru (pfS taking the WAN IP of RG)

I was just doing nmap syn scan on a /64 prefix, which is of course likely just SYN_SENT:CLOSED so I didn't read too much into pf/RG's state tables.

I have no idea of how RG handles state table either. Would CLOSED/NO_TRAFFIC be prioritzed for deletion? then the least recently matched states? I was making this guess because it did not impact my ESTABLISHED. (downloading huge file or running speedtest.net)

I have no devices other than pfS connected to RG.

Maybe I should try again with with more than 8k ESTABLISHED and then see the behavior

JKnott

@lolipoplo said in Multiple IPv6 Prefix Delegation over AT&T Residential Gateway for pfSense 2.4.5:

I was just doing nmap syn scan on a /64 prefix

Ouch!!! With 18.4 billion, billion addresses to scan, that's going to take a while!

One of the security benefits of IPv6 is the address space is so sparse it's virtually impossible to scan even a single /64 for targets.

ttmcmurry

@jknott

Nothing one of those fancy Threadripper 5000s with 64 cores & 128 threads can't handle.

JKnott

@ttmcmurry

It doesn't matter how fast your CPU is, you still have to work through whatever the Internet will allow.

lolipoplo

@jknott of course I didn't finish scanning, just enough to keep pf's state table blown up for a good duration

djdawson

@lolipoplo I think the biggest issue is how the two devices clear out states when their respective tables get full. The pfsense docs describe the process reasonably well, and the gist of it is as the table fills the various state lifetimes are reduced in a liner fashion between the min and max thresholds, with the entire state table getting flushed if the max threshold is crossed. That table flush would seem to be a traumatic event, since all existing connections would likely be dropped, but that max threshold defaults to being like 120% of the state table size so it should only get hit if there's a significant increase in the number of simultaneous connections. Since you can also tweak the individual state timers in pfsense you also have some control from that side of things, and this whole approach seems intended to be a more intelligent way of dropping the oldest, least important states first when the state table fills.

I couldn't find any docs that describe how the AT&T RG deals with a full state table, but I'm assuming it's also an oldest-first based approach and likely not as fancy as the pfsense methodology, though it appears to run a version of Linux so it could also be pretty full-featured.

The easiest way to know which table causes the least trouble when it fills would be to test it as @lolipoplo has been doing, but in my experience such testing is never as easy as it seems like it should be.

Thanks for the great discussion everybody!

ttmcmurry

@jknott that was an attempt at humor. You're absolutely right, there's no realistic way for a full /64 scan to complete in a reasonable amount of time due to the address space and bandwidth limitations.

Perhaps a more realistic approach is to track MAC/IPv6 addresses over a period of time and then port scan those IPs. I've done similar things when dealing with IPv4 /8s.