PPPoE: Problems getting an IPv6 address on reconnection and other problems

[w0w]

Disconnection log disconnect_masked.txt
Connection log connect_masked.txt
I don’t see anything unusual—perhaps only the CARP storming. I use CARP only on WAN2 and the LAN subnets, not on the PPPoE WAN.

@stephenw10 said in PPPoE: Problems getting an IPv6 address on reconnection and other problems:

I forget if you were previously seeing RAs from your ISP?

Every 3 seconds I receive RA packet from ISP. Before the early patch, I was getting a massive storm of log entries and all services kept restarting continuously, and so on, because of that. Now I have problem with receiving Ipv6 but only on reconnection. A clean boot simply works, and in rare cases, just saving the WAN interface settings and applying the changes works too. The same behavior first time occurred on the first 25.03 beta available, without new backend active.

[stephenw10]

Hmm, so if you disconnect the pppoe and reconnect it in Status > Interfaces what do you see logged?

[w0w]

@stephenw10
Yes

[stephenw10]

Um I was hoping for an actual log to look through.

[w0w]

@stephenw10
I just wasn’t fully awake when I read the message — for some reason, it seemed like the question was worded differently.
The last log I posted was actually from when I pressed disconnect and then arter it disconnected, connect.

[stephenw10]

Could be my turn to find more coffee then! Where is that log?

[w0w]

@w0w said in PPPoE: Problems getting an IPv6 address on reconnection and other problems:

Disconnection log disconnect_masked.txt
Connection log connect_masked.txt

My FF opens those attachments just fine.

[stephenw10]

Ha, OK. Definitely more coffee required! Somehow read straight past those.

[w0w]

pppoe0: flags=1008851<UP,POINTOPOINT,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1492
	description: WAN
	options=0
	inet xx.52.22.101 --> yyy.7.29.248 netmask 0xffffffff
	inet6 fe80::aab8:e0ff:fe02:6zz9%pppoe0 prefixlen 64 scopeid 0x11
	inet6 2001:1b28:b248:e39:NNNN:e0ff:fe02:6zz9 prefixlen 64 autoconf pltime 604800 vltime 2592000
	groups: pppoec
	nd6 options=123<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL,NO_DAD>

This is what I see when I get IPv6 only on WAN. Sometimes it shows also "detached" in Ipv6 options

If I run rtsol pppoe0 I am getting

rtsol: pppoe0 does not accept Router Advertisement.

So if I force it by issuing

ifconfig pppoe0 inet6 -ifdisabled -no_radr accept_rtadv
rtsol -F pppoe0

This results in the same outcome — I still receive an IPv6 address marked detached or not and after some time it disappears.

On reboot of firewall I am getting

pppoe0: flags=1008851<UP,POINTOPOINT,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1492
	description: WAN
	options=0
	inet xx.52.5.227 --> yyy.7.29.248 netmask 0xffffffff
	inet6 fe80::aab8:e0ff:fe02:6zz9%pppoe0 prefixlen 64 scopeid 0x10
	inet6 2001:1234:b248:e39:NNNN:e0ff:fe02:6zz9 prefixlen 64 detached autoconf pltime 604800 vltime 2592000
	inet6 2001:4321:d248:ffff:f8b3:abab:1435:769f prefixlen 128 pltime 86400 vltime 172800
	inet6 2001:4321:d248:ffff:49e6:baba:d204:50b8 prefixlen 128 pltime 86400 vltime 172800
	groups: pppoec
	nd6 options=123<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL,NO_DAD>

IPv6 works fine then.

I am not quite sure what it means, but…

EDIT:
It turns out that when I enable the “Do Not Wait for RA” option, the interface receives all addresses just fine.
Curiously, I recall having trouble obtaining addresses even with that option enabled. I probably need to run some more tests—maybe the other settings I tried in combination are now doing the trick.

[stephenw10]

Ah, OK. So, yes, if your ISP requires that then it should now work correctly with the new patch.

So with that set is it working as expected for you?

[w0w]

@stephenw10 said in PPPoE: Problems getting an IPv6 address on reconnection and other problems:

So, yes, if your ISP requires that then it should

It was not required until 23.05.
Of course, this doesn’t necessarily mean that it worked without that option in version 24.11, but it definitely works without it in 23.09. In fact, the need to enable this option in earlier alpha versions of 23.05 is what originally led to the creation of a patch, which was supposed to fix the issue that occurred without this option enabled. And logically, the problem—namely, log spam and endless restarts of services triggered by each RA received from the ISP—did stop, so I assumed I no longer needed the option.

Apologies for the slight confusion... But again, how is one supposed to make sense of all these modern connection and address assignment algorithms, when in one version of pfSense I need to enable this option, and in another, everything works fine without it?

Definitely will test later and report back with this option enabled.

[stephenw10]

Yup, this has been an interesting learning experience. Basically that mpd5/netgraph is doing a bunch of things I never thought it was so if_pppoe handling had to be improved.

[w0w]

I’m not sure what did the trick this time — whether it was rebooting the firewall or just the stars aligning — but everything’s back to how it was: getting an IPv6 address is still unreliable.

I ran an experiment. If I skip the GUI’s “WAN disconnect/connect” button and instead run ifconfig pppoe0 down / up, I have no issues getting an IPv6 address on either the WAN or LAN. During a reboot it stays that way too — still no address-assignment problems.

Another thing I’ve noticed: for as long as I can remember (definitely more than a couple of years), any PPPoE reconnect with my ISP via the same mpd5 used to require a handful of connection attempts before the link came up — say ten tries, sometimes more. Now I mostly see that behavior only on a reboot. If I break the connection through the GUI, wait until the “Connect” button is clickable again, the link comes up almost instantly — but IPv6 addressing becomes flaky. I don’t know what this means; maybe it’s unrelated and pfSense is simply closing the session cleanly when the GUI disconnects, which it might not have done before… No idea.

EDIT: Additional testing showed that if the link drops on the provider’s side or the WAN-side switch loses power, the connection comes back up normally and all addresses are obtained. For now it looks like the problem is only with the GUI button—and maybe with the rc.linkup script—and that’s perfectly fine by me since I almost never use that button anyway. Hopefully that’s the end of it.

[stephenw10]

Hmm, well I guess that's a good result? But I hate unexplained behaviour like that.

The connection now still loos like the connection log above?

That clearly shows the ISP does send an RA so I wouldn't expect to need 'do not wait' to be set:

2025-06-20 05:42:59.290097+03:00 	rtsold 	75589 	Received RA specifying route fe80::669e::IPV68 for interface wan(pppoe0)

I also note that appears to be part of an HA pair. Is that LAN side only?

As far as I know if_pppoe cannot work in an HA setup on the WAN side. The previous workaround setup no longer works.

[w0w]

@stephenw10 said in PPPoE: Problems getting an IPv6 address on reconnection and other problems:

Hmm, well I guess that's a good result? But I hate unexplained behaviour like that.

I can live with that

@stephenw10 said in PPPoE: Problems getting an IPv6 address on reconnection and other problems:

The connection now still loos like the connection log above?

Something like that, yes

@stephenw10 said in PPPoE: Problems getting an IPv6 address on reconnection and other problems:

I also note that appears to be part of an HA pair. Is that LAN side only?

The HA VIPs are configured only on WAN 2 (which has a static address) and on two LANs. However, even after I disabled HA, the same behavior persists.

@stephenw10 said in PPPoE: Problems getting an IPv6 address on reconnection and other problems:

As far as I know if_pppoe cannot work in an HA setup on the WAN side. The previous workaround setup no longer works.

Overall, the setup—HA only on one WAN (the non-PPPoE link) and on the LAN networks, plus active fail-over between the WANs—does work. The only thing I don’t quite understand is why HA triggers specifically when the PPPoE link drops: the master flips to backup and back again for a few seconds, basically flapping. I’m not sure whether it always behaved like that. In any case, I’ve reached the point where, even if the switchover isn’t completely seamless, the downtime is only about five seconds when, say, I power off one firewall—and that’s good enough for me.

[stephenw10]

Hmm, interesting. I agree, I wouldn't expect to see CARP bounce if there's no CARP VIP on the pppoe interface.

[w0w]

@stephenw10 said in PPPoE: Problems getting an IPv6 address on reconnection and other problems:

I wouldn't expect to see CARP bounce

me too

CARP bounce if there's no CARP VIP on the pppoe interface

No VIP on the PPPoE interface and no VIP on the parent interface.
Don’t you have anything like that in your test configurations? It would be nice to check, just out of curiosity. I’m not sure it hasn’t always been there.

[stephenw10]

We don't test that by default because 'officially' pppoe is not a supported interface type in HA. But I know there were/are people using it in various unsupported ways.

I'll try to set something up.

[w0w]

Some other bug or feature found, or maybe I did something wrong

For example, I put pppoe0 down by issuing ifconfig pppoe0 down:

pppoe0: flags=8810<POINTOPOINT,SIMPLEX,MULTICAST> metric 0 mtu 1492
	description: WAN
	options=0
	inet6 fe80::d250:99ff:fec1:71b%pppoe0 prefixlen 64 scopeid 0x11
	groups: pppoec
	nd6 options=121<PERFORMNUD,AUTO_LINKLOCAL,NO_DAD>

And then I see default route still the same

IPv6 Routes
::/0	fe80::669e:f3ff:fe94:dd00%pppoe0	UGS	29	1492	pppoe0	

By the way, I don’t see fe80::669e:f3ff:fe94:dd00 anywhere.
Expected behavior is switching to another gateway on the WAN2, ex tier2

[stephenw10]

Hmm, does the gateway show as down? Anything logged?