PPPoE: Problems getting an IPv6 address on reconnection and other problems

[stephenw10]

Ha, OK. Definitely more coffee required! Somehow read straight past those.

[w0w]

pppoe0: flags=1008851<UP,POINTOPOINT,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1492
	description: WAN
	options=0
	inet xx.52.22.101 --> yyy.7.29.248 netmask 0xffffffff
	inet6 fe80::aab8:e0ff:fe02:6zz9%pppoe0 prefixlen 64 scopeid 0x11
	inet6 2001:1b28:b248:e39:NNNN:e0ff:fe02:6zz9 prefixlen 64 autoconf pltime 604800 vltime 2592000
	groups: pppoec
	nd6 options=123<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL,NO_DAD>

This is what I see when I get IPv6 only on WAN. Sometimes it shows also "detached" in Ipv6 options

If I run rtsol pppoe0 I am getting

rtsol: pppoe0 does not accept Router Advertisement.

So if I force it by issuing

ifconfig pppoe0 inet6 -ifdisabled -no_radr accept_rtadv
rtsol -F pppoe0

This results in the same outcome — I still receive an IPv6 address marked detached or not and after some time it disappears.

On reboot of firewall I am getting

pppoe0: flags=1008851<UP,POINTOPOINT,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1492
	description: WAN
	options=0
	inet xx.52.5.227 --> yyy.7.29.248 netmask 0xffffffff
	inet6 fe80::aab8:e0ff:fe02:6zz9%pppoe0 prefixlen 64 scopeid 0x10
	inet6 2001:1234:b248:e39:NNNN:e0ff:fe02:6zz9 prefixlen 64 detached autoconf pltime 604800 vltime 2592000
	inet6 2001:4321:d248:ffff:f8b3:abab:1435:769f prefixlen 128 pltime 86400 vltime 172800
	inet6 2001:4321:d248:ffff:49e6:baba:d204:50b8 prefixlen 128 pltime 86400 vltime 172800
	groups: pppoec
	nd6 options=123<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL,NO_DAD>

IPv6 works fine then.

I am not quite sure what it means, but…

EDIT:
It turns out that when I enable the “Do Not Wait for RA” option, the interface receives all addresses just fine.
Curiously, I recall having trouble obtaining addresses even with that option enabled. I probably need to run some more tests—maybe the other settings I tried in combination are now doing the trick.

[stephenw10]

Ah, OK. So, yes, if your ISP requires that then it should now work correctly with the new patch.

So with that set is it working as expected for you?

[w0w]

@stephenw10 said in PPPoE: Problems getting an IPv6 address on reconnection and other problems:

So, yes, if your ISP requires that then it should

It was not required until 23.05.
Of course, this doesn’t necessarily mean that it worked without that option in version 24.11, but it definitely works without it in 23.09. In fact, the need to enable this option in earlier alpha versions of 23.05 is what originally led to the creation of a patch, which was supposed to fix the issue that occurred without this option enabled. And logically, the problem—namely, log spam and endless restarts of services triggered by each RA received from the ISP—did stop, so I assumed I no longer needed the option.

Apologies for the slight confusion... But again, how is one supposed to make sense of all these modern connection and address assignment algorithms, when in one version of pfSense I need to enable this option, and in another, everything works fine without it?

Definitely will test later and report back with this option enabled.

[stephenw10]

Yup, this has been an interesting learning experience. Basically that mpd5/netgraph is doing a bunch of things I never thought it was so if_pppoe handling had to be improved.

[w0w]

I’m not sure what did the trick this time — whether it was rebooting the firewall or just the stars aligning — but everything’s back to how it was: getting an IPv6 address is still unreliable.

I ran an experiment. If I skip the GUI’s “WAN disconnect/connect” button and instead run ifconfig pppoe0 down / up, I have no issues getting an IPv6 address on either the WAN or LAN. During a reboot it stays that way too — still no address-assignment problems.

Another thing I’ve noticed: for as long as I can remember (definitely more than a couple of years), any PPPoE reconnect with my ISP via the same mpd5 used to require a handful of connection attempts before the link came up — say ten tries, sometimes more. Now I mostly see that behavior only on a reboot. If I break the connection through the GUI, wait until the “Connect” button is clickable again, the link comes up almost instantly — but IPv6 addressing becomes flaky. I don’t know what this means; maybe it’s unrelated and pfSense is simply closing the session cleanly when the GUI disconnects, which it might not have done before… No idea.

EDIT: Additional testing showed that if the link drops on the provider’s side or the WAN-side switch loses power, the connection comes back up normally and all addresses are obtained. For now it looks like the problem is only with the GUI button—and maybe with the rc.linkup script—and that’s perfectly fine by me since I almost never use that button anyway. Hopefully that’s the end of it.

[stephenw10]

Hmm, well I guess that's a good result? But I hate unexplained behaviour like that.

The connection now still loos like the connection log above?

That clearly shows the ISP does send an RA so I wouldn't expect to need 'do not wait' to be set:

2025-06-20 05:42:59.290097+03:00 	rtsold 	75589 	Received RA specifying route fe80::669e::IPV68 for interface wan(pppoe0)

I also note that appears to be part of an HA pair. Is that LAN side only?

As far as I know if_pppoe cannot work in an HA setup on the WAN side. The previous workaround setup no longer works.

[w0w]

@stephenw10 said in PPPoE: Problems getting an IPv6 address on reconnection and other problems:

Hmm, well I guess that's a good result? But I hate unexplained behaviour like that.

I can live with that

@stephenw10 said in PPPoE: Problems getting an IPv6 address on reconnection and other problems:

The connection now still loos like the connection log above?

Something like that, yes

@stephenw10 said in PPPoE: Problems getting an IPv6 address on reconnection and other problems:

I also note that appears to be part of an HA pair. Is that LAN side only?

The HA VIPs are configured only on WAN 2 (which has a static address) and on two LANs. However, even after I disabled HA, the same behavior persists.

@stephenw10 said in PPPoE: Problems getting an IPv6 address on reconnection and other problems:

As far as I know if_pppoe cannot work in an HA setup on the WAN side. The previous workaround setup no longer works.

Overall, the setup—HA only on one WAN (the non-PPPoE link) and on the LAN networks, plus active fail-over between the WANs—does work. The only thing I don’t quite understand is why HA triggers specifically when the PPPoE link drops: the master flips to backup and back again for a few seconds, basically flapping. I’m not sure whether it always behaved like that. In any case, I’ve reached the point where, even if the switchover isn’t completely seamless, the downtime is only about five seconds when, say, I power off one firewall—and that’s good enough for me.

[stephenw10]

Hmm, interesting. I agree, I wouldn't expect to see CARP bounce if there's no CARP VIP on the pppoe interface.

[w0w]

@stephenw10 said in PPPoE: Problems getting an IPv6 address on reconnection and other problems:

I wouldn't expect to see CARP bounce

me too

CARP bounce if there's no CARP VIP on the pppoe interface

No VIP on the PPPoE interface and no VIP on the parent interface.
Don’t you have anything like that in your test configurations? It would be nice to check, just out of curiosity. I’m not sure it hasn’t always been there.

[stephenw10]

We don't test that by default because 'officially' pppoe is not a supported interface type in HA. But I know there were/are people using it in various unsupported ways.

I'll try to set something up.

[w0w]

Some other bug or feature found, or maybe I did something wrong

For example, I put pppoe0 down by issuing ifconfig pppoe0 down:

pppoe0: flags=8810<POINTOPOINT,SIMPLEX,MULTICAST> metric 0 mtu 1492
	description: WAN
	options=0
	inet6 fe80::d250:99ff:fec1:71b%pppoe0 prefixlen 64 scopeid 0x11
	groups: pppoec
	nd6 options=121<PERFORMNUD,AUTO_LINKLOCAL,NO_DAD>

And then I see default route still the same

IPv6 Routes
::/0	fe80::669e:f3ff:fe94:dd00%pppoe0	UGS	29	1492	pppoe0	

By the way, I don’t see fe80::669e:f3ff:fe94:dd00 anywhere.
Expected behavior is switching to another gateway on the WAN2, ex tier2

[stephenw10]

Hmm, does the gateway show as down? Anything logged?

[w0w]

@stephenw10
Pending. Shouldn’t it be in a pending state only when PPPoE is up but not yet connected?

@stephenw10 said in PPPoE: Problems getting an IPv6 address on reconnection and other problems:

Anything logged?

Unfortunately I've already destroyed pppoe0 so no logs currently, but I'll try to reproduce it later and see what do we have in the logs.

[stephenw10]

It depends how the gaterway is defined but if it's dynamic like that then, yes, I'd expect it to not show as pending.

Do you see the same thing of you disconnect the pppoe via the Interfaces Status page? Downing the interface manually is not something that would usually happen.

[w0w]

@stephenw10 said in PPPoE: Problems getting an IPv6 address on reconnection and other problems:

Do you see the same thing of you disconnect the pppoe via the Interfaces Status page?

Yes, it is showing "pending" even if I disconnect it via GUI.

[w0w]

Something tells me that all the issues with obtaining IPv6 addresses come from the fact that I’ve got a multi-WAN setup, and I’m using IPv6 on both links (different subnets). Because, if we look at the scenario where the WAN PPPoE interface doesn’t get an IPv6, I simply open the WAN2 interface—where both IPv4 and IPv6 addresses are picked up automatically by DHCP—click “Save” and “Apply” without changing anything (which effectively just runs the reconfigure script), and after that I suddenly get an IPv6 on the WAN PPPoE… Very strange, isn’t it?

All right, so I go back into WAN2’s settings again, disable IPv6 completely, set the default route to “Automatic,” delete the failover gateway, and remove the remaining DHCPv6 gateway.

Then I flip CARP into maintenance mode and end up with the following picture (note the WAN2 interface).

Why am I getting an IPv6 address on WAN2 when it’s clearly set to “None”?
I think someone else needs to test a multi-WAN setup with IPv6 enabled on at least two WAN interfaces and one is PPPoE, and I don't think CARP is needed.

[stephenw10]

Hmm. Well that's interesting.

Is that IPv6 address you're getting on WAN2 correct for WAN2? Or is it somehow applying the PPPoE v6 address there?

[w0w]

@stephenw10
No, that’s the correct address for WAN2 when DHCPv6 is enabled. It has only happened once in my tests so far, and I haven’t run long-term tests yet—I need the network up right now. The next time I can experiment will be tomorrow morning.

[w0w]

The glitch where an IPv6 address appeared on WAN2 even though IPv6 was disabled on that interface hasn’t happened again after I simply hit Save & Apply with no changes.

I’ve now booted into version 23.09.1. From what I can see, the key difference in the GUI is how interface status is reported: in 23.09.1 the interface stays down until the PPPoE session is actually established by mpd5, whereas in 23.05 the interface flips to UP as soon as the virtual pppoe0 device gets the UP flag—even though the PPP session hasn’t finished negotiating. I don’t think this behaviour affects anything functional, but it is worth noting.

At this point my working theory is that the problem only occurs when both WAN links use DHCPv6. If I switch WAN2 to SLAAC (which is also an option), the disconnect/connect cycle works fine and I don’t see any issues.