I tried it this moring before working hours but i could not reliably reproduce the stale state.
Didn't have much time to test but i tried a few enable/disable interfaces and pause/resume of the virtual machine of the primary firewall so that the failover occurs and pppoe needs to reconnect.
This morning everything worked as expected.
A few notes on my side: Both firewalls only had an uptime of less than a day. so it might be that the issue only occurs on longer uptime, maybe also if something happens on the provider side in regards to the pppoe session.
Also i did reconfigure the dhcp v6 settings accoring to the post of @wow which seemed to make the IPv6 stuff (more) stable than before.
The only things i've seen from the (debug) logs which kind of are hinting towards an error are the following lines:
Sep 19 07:55:53 kernel if_pppoe: pppoe0: failed to set default route 17
Sep 19 07:55:42 kernel if_pppoe: pppoe0: pap failure
Sep 19 07:55:41 kernel if_pppoe: pppoe0: failed to clear IP address: 49
we can safely ignore the pap failure in my case as this must be due to the failover i was testing. in this case the provider thinks the pppoe session for my user is still active (on the other firewall) so i get a few pap failures in failover until the provider closes the stale session of the other firewall.
I've seen the other errors (default route/ip address) previously though and those might be related to some error.
anyway I will do another test in a couple of days and keep you posted...
edit: i extracted the relevant parts of the pppoe debug log. don't know if that is any help, but i am posting it anyway:
[25.07.1-RELEASE][root@pfSense.xxxxxxxxxx]/root: bzcat /var/log/ppp.log.* | grep "Sep 19" | grep -v output
Sep 19 07:43:00 pfSense newsyslog[73829]: logfile turned over due to size>500K
Sep 19 07:43:05 pfSense kernel: if_pppoe: pppoe0: lcp input(opened): <echo-req id=0x3 len=8
Sep 19 07:43:05 pfSense kernel: if_pppoe: pppoe0: got lcp echo req, sending echo rep
Sep 19 07:43:15 pfSense kernel: if_pppoe: pppoe0: lcp input(opened): <echo-req id=0x4 len=8
Sep 19 07:43:15 pfSense kernel: if_pppoe: pppoe0: got lcp echo req, sending echo rep
Sep 19 07:43:25 pfSense kernel: if_pppoe: pppoe0: lcp input(opened): <echo-req id=0x5 len=8
Sep 19 07:43:25 pfSense kernel: if_pppoe: pppoe0: got lcp echo req, sending echo rep
Sep 19 07:43:35 pfSense kernel: if_pppoe: pppoe0: lcp input(opened): <echo-req id=0x6 len=8
Sep 19 07:43:35 pfSense kernel: if_pppoe: pppoe0: got lcp echo req, sending echo rep
Sep 19 07:43:45 pfSense kernel: if_pppoe: pppoe0: lcp input(opened): <echo-req id=0x7 len=8
Sep 19 07:43:45 pfSense kernel: if_pppoe: pppoe0: got lcp echo req, sending echo rep
Sep 19 07:43:46 pfSense kernel: if_pppoe: pppoe0: lcp close(opened)
Sep 19 07:43:46 pfSense kernel: if_pppoe: pppoe0: lcp opened->closing
Sep 19 07:43:46 pfSense kernel: if_pppoe: pppoe0: phase terminate
Sep 19 07:43:46 pfSense kernel: if_pppoe: pppoe0: ipcp down(opened)
Sep 19 07:43:46 pfSense kernel: if_pppoe: pppoe0: ipcp opened->starting
Sep 19 07:43:46 pfSense kernel: if_pppoe: pppoe0: ipcp close(starting)
Sep 19 07:43:46 pfSense kernel: if_pppoe: pppoe0: ipcp starting->initial
Sep 19 07:43:46 pfSense kernel: if_pppoe: pppoe0: ipv6cp down(opened)
Sep 19 07:43:46 pfSense kernel: if_pppoe: pppoe0: ipv6cp opened->starting
Sep 19 07:43:46 pfSense kernel: if_pppoe: pppoe0: ipv6cp close(starting)
Sep 19 07:43:46 pfSense kernel: if_pppoe: pppoe0: ipv6cp starting->initial
Sep 19 07:43:48 pfSense kernel: if_pppoe: pppoe0: pap failure
Sep 19 07:43:58 pfSense kernel: if_pppoe: pppoe0: failed to set default route 17
Sep 19 07:44:00 pfSense newsyslog[82494]: logfile turned over due to size>500K
Sep 19 07:35:30 pfSense newsyslog[52425]: logfile turned over due to size>500K
Sep 19 07:35:48 pfSense kernel: if_pppoe: pppoe0: LCP keepalive timeout
Sep 19 07:35:48 pfSense kernel: if_pppoe: pppoe0: lcp close(opened)
Sep 19 07:35:48 pfSense kernel: if_pppoe: pppoe0: lcp opened->closing
Sep 19 07:35:48 pfSense kernel: if_pppoe: pppoe0: phase terminate
Sep 19 07:35:48 pfSense kernel: if_pppoe: pppoe0: ipcp down(opened)
Sep 19 07:35:48 pfSense kernel: if_pppoe: pppoe0: ipcp opened->starting
Sep 19 07:35:48 pfSense kernel: if_pppoe: pppoe0: ipcp close(starting)
Sep 19 07:35:48 pfSense kernel: if_pppoe: pppoe0: ipcp starting->initial
Sep 19 07:35:48 pfSense kernel: if_pppoe: pppoe0: ipv6cp down(opened)
Sep 19 07:35:48 pfSense kernel: if_pppoe: pppoe0: ipv6cp opened->starting
Sep 19 07:35:48 pfSense kernel: if_pppoe: pppoe0: ipv6cp close(starting)
Sep 19 07:35:48 pfSense kernel: if_pppoe: pppoe0: ipv6cp starting->initial
Sep 19 07:35:48 pfSense kernel: if_pppoe: pppoe0: lcp closing->stopped
Sep 19 07:35:48 pfSense kernel: if_pppoe: pppoe0: timeout
Sep 19 07:35:48 pfSense kernel: if_pppoe: pppoe0: disconnecting
Sep 19 07:35:48 pfSense kernel: if_pppoe: pppoe0: lcp down(stopped)
Sep 19 07:35:48 pfSense kernel: if_pppoe: pppoe0: lcp stopped->starting
Sep 19 07:35:48 pfSense kernel: if_pppoe: pppoe0: phase establish
Sep 19 07:35:48 pfSense kernel: if_pppoe: pppoe0: Down event (carrier loss), taking interface down.if_pppoe: pppoe0: lcp close(starting)
Sep 19 07:35:48 pfSense kernel: if_pppoe: pppoe0: lcp starting->initial
Sep 19 07:35:48 pfSense kernel: if_pppoe: pppoe0: phase dead
Sep 19 07:35:48 pfSense kernel: if_pppoe: pppoe0: session 0x1bd connected
Sep 19 07:35:48 pfSense kernel: if_pppoe: pppoe0: lcp open(initial)
Sep 19 07:35:48 pfSense kernel: if_pppoe: pppoe0: lcp initial->starting
Sep 19 07:35:48 pfSense kernel: if_pppoe: pppoe0: phase establish
Sep 19 07:35:48 pfSense kernel: if_pppoe: pppoe0: lcp up(starting)
Sep 19 07:35:48 pfSense kernel: if_pppoe: pppoe0: lcp starting->req-sent
Sep 19 07:35:48 pfSense kernel: if_pppoe: pppoe0: lcp input(req-sent): <conf-req id=0xb7 len=18
Sep 19 07:35:48 pfSense kernel: if_pppoe: pppoe0: lcp parse opts:
Sep 19 07:35:48 pfSense kernel: if_pppoe: pppoe0: lcp parse opt values:
Sep 19 07:35:48 pfSense kernel: if_pppoe: pppoe0: lcp req-sent->ack-sent
Sep 19 07:35:48 pfSense kernel: if_pppoe: pppoe0: lcp input(ack-sent): <conf-ack id=0x16 len=14
Sep 19 07:35:48 pfSense kernel: if_pppoe: pppoe0: lcp ack-sent->opened
Sep 19 07:35:48 pfSense kernel: if_pppoe: pppoe0: lcp tlu
Sep 19 07:35:48 pfSense kernel: if_pppoe: pppoe0: up
Sep 19 07:35:48 pfSense kernel: if_pppoe: pppoe0: phase authenticate
Sep 19 07:35:49 pfSense kernel: if_pppoe: pppoe0: pap success
Sep 19 07:35:49 pfSense kernel: if_pppoe: pppoe0: phase network
Sep 19 07:35:49 pfSense kernel: if_pppoe: pppoe0: ipcp open(initial)
Sep 19 07:35:49 pfSense kernel: if_pppoe: pppoe0: ipcp initial->starting
Sep 19 07:35:49 pfSense kernel: if_pppoe: pppoe0: ipv6cp open(initial)
Sep 19 07:35:49 pfSense kernel: if_pppoe: pppoe0: ipv6cp initial->starting
Sep 19 07:35:49 pfSense kernel: if_pppoe: pppoe0: ipcp up(starting)
Sep 19 07:35:49 pfSense kernel: if_pppoe: pppoe0: ipcp starting->req-sent
Sep 19 07:35:49 pfSense kernel: if_pppoe: pppoe0: ipv6cp up(starting)
Sep 19 07:35:49 pfSense kernel: if_pppoe: pppoe0: ipv6cp starting->req-sent
Sep 19 07:35:49 pfSense kernel: if_pppoe: pppoe0: ipcp input(req-sent): <conf-req id=0x99 len=10
Sep 19 07:35:49 pfSense kernel: if_pppoe: pppoe0: ipcp parse opts:
Sep 19 07:35:49 pfSense kernel: if_pppoe: pppoe0: ipcp parse opt values:
Sep 19 07:35:49 pfSense kernel: if_pppoe: pppoe0: ipcp req-sent->ack-sent
Sep 19 07:35:49 pfSense kernel: if_pppoe: pppoe0: ipcp input(ack-sent): <conf-nak id=0x18 len=22
Sep 19 07:35:49 pfSense kernel: if_pppoe: pppoe0: ipcp nak opts:
Sep 19 07:35:49 pfSense kernel: if_pppoe: pppoe0: ipv6cp input(req-sent): <conf-req id=0x40 len=14
Sep 19 07:35:49 pfSense kernel: if_pppoe: pppoe0: ipv6cp parse opts:
Sep 19 07:35:49 pfSense kernel: if_pppoe: pppoe0: ipv6cp parse opt values:
Sep 19 07:35:49 pfSense kernel: if_pppoe: pppoe0: ipv6cp req-sent->ack-sent
Sep 19 07:35:49 pfSense kernel: if_pppoe: pppoe0: failed to set default route 17
Sep 19 07:35:49 pfSense kernel: if_pppoe: pppoe0: ipcp input(ack-sent): <conf-ack id=0x1a len=22
Sep 19 07:35:49 pfSense kernel: if_pppoe: pppoe0: ipcp ack-sent->opened
Sep 19 07:35:49 pfSense kernel: if_pppoe: pppoe0: ipcp tlu
Sep 19 07:35:49 pfSense kernel: if_pppoe: pppoe0: failed to set default route 17
Sep 19 07:35:49 pfSense kernel: if_pppoe: pppoe0: ipv6cp input(ack-sent): <conf-ack id=0x19 len=14
Sep 19 07:35:49 pfSense kernel: if_pppoe: pppoe0: ipv6cp ack-sent->opened
Sep 19 07:35:49 pfSense kernel: if_pppoe: pppoe0: ipv6cp tlu
Sep 19 07:35:55 pfSense kernel: if_pppoe: pppoe0: lcp input(opened): <echo-req id=0x0 len=8
Sep 19 07:35:55 pfSense kernel: if_pppoe: pppoe0: got lcp echo req, sending echo rep
Sep 19 07:36:06 pfSense kernel: if_pppoe: pppoe0: lcp input(opened): <echo-req id=0x1 len=8
Sep 19 07:36:06 pfSense kernel: if_pppoe: pppoe0: got lcp echo req, sending echo rep
Sep 19 07:36:15 pfSense kernel: if_pppoe: pppoe0: lcp input(opened): <echo-req id=0x2 len=8
Sep 19 07:36:15 pfSense kernel: if_pppoe: pppoe0: got lcp echo req, sending echo rep
Sep 19 07:36:25 pfSense kernel: if_pppoe: pppoe0: lcp input(opened): <echo-req id=0x3 len=8
Sep 19 07:36:25 pfSense kernel: if_pppoe: pppoe0: got lcp echo req, sending echo rep
Sep 19 07:36:35 pfSense kernel: if_pppoe: pppoe0: lcp input(opened): <echo-req id=0x4 len=8
Sep 19 07:36:35 pfSense kernel: if_pppoe: pppoe0: got lcp echo req, sending echo rep
Sep 19 07:42:28 pfSense kernel: if_pppoe: pppoe0: LCP keepalive timeout
Sep 19 07:42:28 pfSense kernel: if_pppoe: pppoe0: lcp close(opened)
Sep 19 07:42:28 pfSense kernel: if_pppoe: pppoe0: lcp opened->closing
Sep 19 07:42:28 pfSense kernel: if_pppoe: pppoe0: phase terminate
Sep 19 07:42:28 pfSense kernel: if_pppoe: pppoe0: ipcp down(opened)
Sep 19 07:42:28 pfSense kernel: if_pppoe: pppoe0: ipcp opened->starting
Sep 19 07:42:28 pfSense kernel: if_pppoe: pppoe0: ipcp close(starting)
Sep 19 07:42:28 pfSense kernel: if_pppoe: pppoe0: ipcp starting->initial
Sep 19 07:42:28 pfSense kernel: if_pppoe: pppoe0: ipv6cp down(opened)
Sep 19 07:42:28 pfSense kernel: if_pppoe: pppoe0: ipv6cp opened->starting
Sep 19 07:42:28 pfSense kernel: if_pppoe: pppoe0: ipv6cp close(starting)
Sep 19 07:42:28 pfSense kernel: if_pppoe: pppoe0: ipv6cp starting->initial
Sep 19 07:42:28 pfSense kernel: if_pppoe: pppoe0: lcp closing->stopped
Sep 19 07:42:28 pfSense kernel: if_pppoe: pppoe0: timeout
Sep 19 07:42:28 pfSense kernel: if_pppoe: pppoe0: disconnecting
Sep 19 07:42:28 pfSense kernel: if_pppoe: pppoe0: lcp down(stopped)
Sep 19 07:42:28 pfSense kernel: if_pppoe: pppoe0: lcp stopped->starting
Sep 19 07:42:28 pfSense kernel: if_pppoe: pppoe0: phase establish
Sep 19 07:42:28 pfSense kernel: if_pppoe: pppoe0: Down event (carrier loss), taking interface down.if_pppoe: pppoe0: lcp close(starting)
Sep 19 07:42:28 pfSense kernel: if_pppoe: pppoe0: lcp starting->initial
Sep 19 07:42:28 pfSense kernel: if_pppoe: pppoe0: phase dead
Sep 19 07:42:28 pfSense kernel: if_pppoe: pppoe0: session 0x1bd connected
Sep 19 07:42:28 pfSense kernel: if_pppoe: pppoe0: lcp open(initial)
Sep 19 07:42:28 pfSense kernel: if_pppoe: pppoe0: lcp initial->starting
Sep 19 07:42:28 pfSense kernel: if_pppoe: pppoe0: phase establish
Sep 19 07:42:28 pfSense kernel: if_pppoe: pppoe0: lcp up(starting)
Sep 19 07:42:28 pfSense kernel: if_pppoe: pppoe0: lcp starting->req-sent
Sep 19 07:42:28 pfSense kernel: if_pppoe: pppoe0: lcp input(req-sent): <conf-req id=0x36 len=18
Sep 19 07:42:28 pfSense kernel: if_pppoe: pppoe0: lcp parse opts:
Sep 19 07:42:28 pfSense kernel: if_pppoe: pppoe0: lcp parse opt values:
Sep 19 07:42:28 pfSense kernel: if_pppoe: pppoe0: lcp req-sent->ack-sent
Sep 19 07:42:28 pfSense kernel: if_pppoe: pppoe0: lcp input(ack-sent): <conf-ack id=0x1f len=14
Sep 19 07:42:28 pfSense kernel: if_pppoe: pppoe0: lcp ack-sent->opened
Sep 19 07:42:28 pfSense kernel: if_pppoe: pppoe0: lcp tlu
Sep 19 07:42:28 pfSense kernel: if_pppoe: pppoe0: up
Sep 19 07:42:28 pfSense kernel: if_pppoe: pppoe0: phase authenticate
Sep 19 07:42:29 pfSense kernel: if_pppoe: pppoe0: pap success
Sep 19 07:42:29 pfSense kernel: if_pppoe: pppoe0: phase network
Sep 19 07:42:29 pfSense kernel: if_pppoe: pppoe0: ipcp open(initial)
Sep 19 07:42:29 pfSense kernel: if_pppoe: pppoe0: ipcp initial->starting
Sep 19 07:42:29 pfSense kernel: if_pppoe: pppoe0: ipv6cp open(initial)
Sep 19 07:42:29 pfSense kernel: if_pppoe: pppoe0: ipv6cp initial->starting
Sep 19 07:42:29 pfSense kernel: if_pppoe: pppoe0: ipcp up(starting)
Sep 19 07:42:29 pfSense kernel: if_pppoe: pppoe0: ipcp starting->req-sent
Sep 19 07:42:29 pfSense kernel: if_pppoe: pppoe0: ipv6cp up(starting)
Sep 19 07:42:29 pfSense kernel: if_pppoe: pppoe0: ipv6cp starting->req-sent
Sep 19 07:42:29 pfSense kernel: if_pppoe: pppoe0: ipcp input(req-sent): <conf-req id=0x22 len=10
Sep 19 07:42:29 pfSense kernel: if_pppoe: pppoe0: ipcp parse opts:
Sep 19 07:42:29 pfSense kernel: if_pppoe: pppoe0: ipcp parse opt values:
Sep 19 07:42:29 pfSense kernel: if_pppoe: pppoe0: ipcp req-sent->ack-sent
Sep 19 07:42:29 pfSense kernel: if_pppoe: pppoe0: ipcp input(ack-sent): <conf-nak id=0x21 len=22
Sep 19 07:42:29 pfSense kernel: if_pppoe: pppoe0: ipcp nak opts:
Sep 19 07:42:29 pfSense kernel: if_pppoe: pppoe0: ipv6cp input(req-sent): <conf-req id=0x36 len=14
Sep 19 07:42:29 pfSense kernel: if_pppoe: pppoe0: ipv6cp parse opts:
Sep 19 07:42:29 pfSense kernel: if_pppoe: pppoe0: ipv6cp parse opt values:
Sep 19 07:42:29 pfSense kernel: if_pppoe: pppoe0: ipv6cp req-sent->ack-sent
Sep 19 07:42:29 pfSense kernel: if_pppoe: pppoe0: failed to set default route 17
Sep 19 07:42:29 pfSense kernel: if_pppoe: pppoe0: ipcp input(ack-sent): <conf-ack id=0x23 len=22
Sep 19 07:42:29 pfSense kernel: if_pppoe: pppoe0: ipcp ack-sent->opened
Sep 19 07:42:29 pfSense kernel: if_pppoe: pppoe0: ipcp tlu
Sep 19 07:42:29 pfSense kernel: if_pppoe: pppoe0: failed to set default route 17
Sep 19 07:42:29 pfSense kernel: if_pppoe: pppoe0: ipv6cp input(ack-sent): <conf-ack id=0x22 len=14
Sep 19 07:42:29 pfSense kernel: if_pppoe: pppoe0: ipv6cp ack-sent->opened
Sep 19 07:42:29 pfSense kernel: if_pppoe: pppoe0: ipv6cp tlu
Sep 19 07:42:35 pfSense kernel: if_pppoe: pppoe0: lcp input(opened): <echo-req id=0x0 len=8
Sep 19 07:42:35 pfSense kernel: if_pppoe: pppoe0: got lcp echo req, sending echo rep
Sep 19 07:42:45 pfSense kernel: if_pppoe: pppoe0: lcp input(opened): <echo-req id=0x1 len=8
Sep 19 07:42:45 pfSense kernel: if_pppoe: pppoe0: got lcp echo req, sending echo rep
Sep 19 07:42:55 pfSense kernel: if_pppoe: pppoe0: lcp input(opened): <echo-req id=0x2 len=8
Sep 19 07:42:55 pfSense kernel: if_pppoe: pppoe0: got lcp echo req, sending echo rep
Sep 19 07:43:00 pfSense newsyslog[73829]: logfile turned over due to size>500K
Sep 19 07:25:28 pfSense kernel: if_pppoe: pppoe0: failed to set default route 17
Sep 19 07:27:28 pfSense kernel: if_pppoe: pppoe0: failed to clear IP address: 49
Sep 19 07:27:29 pfSense kernel: if_pppoe: pppoe0: pap failure
Sep 19 07:31:18 pfSense kernel: if_pppoe: pppoe0: LCP keepalive timeout
Sep 19 07:31:18 pfSense kernel: if_pppoe: pppoe0: failed to set default route 17
Sep 19 07:31:18 pfSense kernel: if_pppoe: pppoe0: failed to set default route 17
Sep 19 07:34:59 pfSense kernel: if_pppoe: pppoe0: lcp input(opened): <echo-req id=0x7 len=8
Sep 19 07:34:59 pfSense kernel: if_pppoe: pppoe0: got lcp echo req, sending echo rep
Sep 19 07:35:30 pfSense newsyslog[52425]: logfile turned over due to size>500K