PPPoE: Problems getting an IPv6 address on reconnection and other problems

[stephenw10]

Yup, this has been an interesting learning experience. Basically that mpd5/netgraph is doing a bunch of things I never thought it was so if_pppoe handling had to be improved.

[w0w]

I’m not sure what did the trick this time — whether it was rebooting the firewall or just the stars aligning — but everything’s back to how it was: getting an IPv6 address is still unreliable.

I ran an experiment. If I skip the GUI’s “WAN disconnect/connect” button and instead run ifconfig pppoe0 down / up, I have no issues getting an IPv6 address on either the WAN or LAN. During a reboot it stays that way too — still no address-assignment problems.

Another thing I’ve noticed: for as long as I can remember (definitely more than a couple of years), any PPPoE reconnect with my ISP via the same mpd5 used to require a handful of connection attempts before the link came up — say ten tries, sometimes more. Now I mostly see that behavior only on a reboot. If I break the connection through the GUI, wait until the “Connect” button is clickable again, the link comes up almost instantly — but IPv6 addressing becomes flaky. I don’t know what this means; maybe it’s unrelated and pfSense is simply closing the session cleanly when the GUI disconnects, which it might not have done before… No idea.

EDIT: Additional testing showed that if the link drops on the provider’s side or the WAN-side switch loses power, the connection comes back up normally and all addresses are obtained. For now it looks like the problem is only with the GUI button—and maybe with the rc.linkup script—and that’s perfectly fine by me since I almost never use that button anyway. Hopefully that’s the end of it.

[stephenw10]

Hmm, well I guess that's a good result? But I hate unexplained behaviour like that.

The connection now still loos like the connection log above?

That clearly shows the ISP does send an RA so I wouldn't expect to need 'do not wait' to be set:

2025-06-20 05:42:59.290097+03:00 	rtsold 	75589 	Received RA specifying route fe80::669e::IPV68 for interface wan(pppoe0)

I also note that appears to be part of an HA pair. Is that LAN side only?

As far as I know if_pppoe cannot work in an HA setup on the WAN side. The previous workaround setup no longer works.

[w0w]

@stephenw10 said in PPPoE: Problems getting an IPv6 address on reconnection and other problems:

Hmm, well I guess that's a good result? But I hate unexplained behaviour like that.

I can live with that

@stephenw10 said in PPPoE: Problems getting an IPv6 address on reconnection and other problems:

The connection now still loos like the connection log above?

Something like that, yes

@stephenw10 said in PPPoE: Problems getting an IPv6 address on reconnection and other problems:

I also note that appears to be part of an HA pair. Is that LAN side only?

The HA VIPs are configured only on WAN 2 (which has a static address) and on two LANs. However, even after I disabled HA, the same behavior persists.

@stephenw10 said in PPPoE: Problems getting an IPv6 address on reconnection and other problems:

As far as I know if_pppoe cannot work in an HA setup on the WAN side. The previous workaround setup no longer works.

Overall, the setup—HA only on one WAN (the non-PPPoE link) and on the LAN networks, plus active fail-over between the WANs—does work. The only thing I don’t quite understand is why HA triggers specifically when the PPPoE link drops: the master flips to backup and back again for a few seconds, basically flapping. I’m not sure whether it always behaved like that. In any case, I’ve reached the point where, even if the switchover isn’t completely seamless, the downtime is only about five seconds when, say, I power off one firewall—and that’s good enough for me.

[stephenw10]

Hmm, interesting. I agree, I wouldn't expect to see CARP bounce if there's no CARP VIP on the pppoe interface.

[w0w]

@stephenw10 said in PPPoE: Problems getting an IPv6 address on reconnection and other problems:

I wouldn't expect to see CARP bounce

me too

CARP bounce if there's no CARP VIP on the pppoe interface

No VIP on the PPPoE interface and no VIP on the parent interface.
Don’t you have anything like that in your test configurations? It would be nice to check, just out of curiosity. I’m not sure it hasn’t always been there.

[stephenw10]

We don't test that by default because 'officially' pppoe is not a supported interface type in HA. But I know there were/are people using it in various unsupported ways.

I'll try to set something up.

[w0w]

Some other bug or feature found, or maybe I did something wrong

For example, I put pppoe0 down by issuing ifconfig pppoe0 down:

pppoe0: flags=8810<POINTOPOINT,SIMPLEX,MULTICAST> metric 0 mtu 1492
	description: WAN
	options=0
	inet6 fe80::d250:99ff:fec1:71b%pppoe0 prefixlen 64 scopeid 0x11
	groups: pppoec
	nd6 options=121<PERFORMNUD,AUTO_LINKLOCAL,NO_DAD>

And then I see default route still the same

IPv6 Routes
::/0	fe80::669e:f3ff:fe94:dd00%pppoe0	UGS	29	1492	pppoe0	

By the way, I don’t see fe80::669e:f3ff:fe94:dd00 anywhere.
Expected behavior is switching to another gateway on the WAN2, ex tier2

[stephenw10]

Hmm, does the gateway show as down? Anything logged?

[w0w]

@stephenw10
Pending. Shouldn’t it be in a pending state only when PPPoE is up but not yet connected?

@stephenw10 said in PPPoE: Problems getting an IPv6 address on reconnection and other problems:

Anything logged?

Unfortunately I've already destroyed pppoe0 so no logs currently, but I'll try to reproduce it later and see what do we have in the logs.

[stephenw10]

It depends how the gaterway is defined but if it's dynamic like that then, yes, I'd expect it to not show as pending.

Do you see the same thing of you disconnect the pppoe via the Interfaces Status page? Downing the interface manually is not something that would usually happen.

[w0w]

@stephenw10 said in PPPoE: Problems getting an IPv6 address on reconnection and other problems:

Do you see the same thing of you disconnect the pppoe via the Interfaces Status page?

Yes, it is showing "pending" even if I disconnect it via GUI.

[w0w]

Something tells me that all the issues with obtaining IPv6 addresses come from the fact that I’ve got a multi-WAN setup, and I’m using IPv6 on both links (different subnets). Because, if we look at the scenario where the WAN PPPoE interface doesn’t get an IPv6, I simply open the WAN2 interface—where both IPv4 and IPv6 addresses are picked up automatically by DHCP—click “Save” and “Apply” without changing anything (which effectively just runs the reconfigure script), and after that I suddenly get an IPv6 on the WAN PPPoE… Very strange, isn’t it?

All right, so I go back into WAN2’s settings again, disable IPv6 completely, set the default route to “Automatic,” delete the failover gateway, and remove the remaining DHCPv6 gateway.

Then I flip CARP into maintenance mode and end up with the following picture (note the WAN2 interface).

Why am I getting an IPv6 address on WAN2 when it’s clearly set to “None”?
I think someone else needs to test a multi-WAN setup with IPv6 enabled on at least two WAN interfaces and one is PPPoE, and I don't think CARP is needed.

[stephenw10]

Hmm. Well that's interesting.

Is that IPv6 address you're getting on WAN2 correct for WAN2? Or is it somehow applying the PPPoE v6 address there?

[w0w]

@stephenw10
No, that’s the correct address for WAN2 when DHCPv6 is enabled. It has only happened once in my tests so far, and I haven’t run long-term tests yet—I need the network up right now. The next time I can experiment will be tomorrow morning.

[w0w]

The glitch where an IPv6 address appeared on WAN2 even though IPv6 was disabled on that interface hasn’t happened again after I simply hit Save & Apply with no changes.

I’ve now booted into version 23.09.1. From what I can see, the key difference in the GUI is how interface status is reported: in 23.09.1 the interface stays down until the PPPoE session is actually established by mpd5, whereas in 23.05 the interface flips to UP as soon as the virtual pppoe0 device gets the UP flag—even though the PPP session hasn’t finished negotiating. I don’t think this behaviour affects anything functional, but it is worth noting.

At this point my working theory is that the problem only occurs when both WAN links use DHCPv6. If I switch WAN2 to SLAAC (which is also an option), the disconnect/connect cycle works fine and I don’t see any issues.

[stephenw10]

Hmm, yes if_pppoe shows as up as soon as it starts to try to connect. If that is the case you should also not see it when running mpd5 in 25.03.

[w0w]

@stephenw10
No, I'm not sure whether the up/down state affects the receiving of IPv6 addresses or not. I can certainly test it with mpd5 just in case, but later. If no one else is having issues in a multi-WAN configuration where both WANs receive IPv6 via DHCP, then I'm fine with switching the second WAN to an alternative configuration using SLAAC only.

[stephenw10]

It's not something I've heard from anyone else. I know at least one of our devs has dual pppoe WANs with IPv6 on both.

[w0w]

@stephenw10 said in PPPoE: Problems getting an IPv6 address on reconnection and other problems:

dual pppoe WANs with IPv6 on both.
DHCP?