New PPPoE backend, some feedback

stephenw10

Yup there are more IPv6 fixes coming.

Phil2025

@louis2 Yes several of us are having this issue. A fix seems to be to go to: Status - Gateways - then click the recycle button to restart the Gateway service, it should then switch to online.

azalea

In my first report, I reported "IPv6 Address is set but not reachable", but found a condition to obtain global reachability.
(I checked the global response in pfSense's "Diagnostics > Ping".)

My ISP has a limit on the number of PPPoE sessions available. My ISP won't allow me to open a new PPPoE session until it knows the previous one has been closed. Also, if I disconnect a PPPoE session without manually closing it, my ISP takes about 5 minutes to recognize that the PPPoE session has been closed.

These specifications make the following difference:

(a) If I reboot pfSense with manually closing the PPPoE session, pppoe0 links up and the IPv6 address is set before "Bootup complete".
(b) If I reboot pfSense without manually closing the PPPoE session, pppoe0 links up and the IPv6 address is set after "Bootup complete".

In case (a), the ping passed, but in case (b), the ping failed.

When I checked "Use if_pppoe kernel module for PPPoE client" to enable if_pppoe, this fell under case (b). As a result, the ping failed.

Here are some other facts I have confirmed:
[PPP log]
Each time a PPPoE session is established, the message "failed to set default route 17" may or may not be displayed.

The message is not displayed after bootup.
The message is displayed after executing the "ifconfig pppoe0 down/up" command.
The message is not displayed after clicking the "Disconnect/Connect" button in Interfaces Status.

Every time a PPPoE session is established, the message "failed to clear IP address: 49" is displayed.

[Interfaces Status]
The "Gateway IPv6" item is not always displayed. It is not blank. The item itself does not exist.
The "Gateway IPv6" undisplayed issue is unrelated to the message "failed to set default route 17" displayed/undisplayed issue.
The "Gateway IPv6" undisplayed issue is unrelated to Ping pass/fail. In case (a), Ping passes.

[DHCP log]
Basically, the message "add an address [IPv6 Address]/64 on em1" is displayed each time a PPPoE session is established.

The message is displayed before or after "Bootup complete" and the IPv6 address is set.
The message is displayed after executing the "ifconfig pppoe0 down/up" command and the IPv6 address is set.
However, after clicking the "Disconnect/Connect" button in Interfaces Status, "Sending Solicit" is displayed repeatedly, no message is displayed, and the IPv6 address is not set. After that, even after executing the "ifconfig pppoe0 down/up" command, "Sending Solicit" is displayed repeatedly, no message is displayed, and the IPv6 address is not set. If I reboot again, the message is displayed and the IPv6 address is set.

[Ping]
In case (a), the ping passes regardless of whether the "failed to set default route 17" message is displayed or not.
In case (a), the ping passes regardless of the dpinger Status (Running/Stopped).
In case (a), the ping passes regardless of the Gateways Status (Online/Pending).
In case (a), the ping passes even after executing the "ifconfig pppoe0 down/up" command.

[Gateways/dpinger]
When dpinger Status is Stopped, nothing is displayed in the Gateways Log.
If I set Gateway's IPv6 Link Local to "Monitor IP", the dpinger Status is displayed as Running. Then, "send_interval" and so on is displayed in the Gateways Log.
The dpinger status (Running/Stopped) is unrelated to Ping pass/fail. Ping pass/fail is determined by case (a)/case (b).
The Gateways Status (Online/Pending) is unrelated to Ping pass/fail. Ping pass/fail is determined by case (a)/case (b).

Based on the facts presented, I assume that the issue of "Gateway IPv6" not being set and the issue of ping failure are independent issues.

stephenw10

So you also see failures in IPv6 when using mpd5/netgraph?

Do IPv4 pings (general connectivity) succeed in both cases? Jusr varies the IPv6 response?

perrin

@azalea said in New PPPoE backend, some feedback:

failed to set default route 17

i am having the exact same issue with the if_pppoe failed setting the default route and failed getting an IPv6 config. Never had any problems with this using MPD.

The issue is occurring when you disconnect, reconnect the interface or disable/enable it. upon a clean boot, everything works as expected

the only solution to regain connectivity is to reboot the firewall. it will not work otherwise. did not try to unload/reload the kernel module due to lack of knowledge on BSD.

w0w

In my case, enabling "Request only an IPv6 prefix" gives me a more stable IPv6 connection, and the WAN interface now consistently gets an IPv6 address as well.
There have definitely been some changes on the ISP’s side too, because it suddenly started losing the IPv6 connection while IPv4 was still working fine. Enabling this option fixed the issue — my IPv6 connection is now stable.

And yes, as I’ve mentioned before, I had the same problem: it wouldn’t get an IPv6 address after a reconnection, but everything worked fine after rebooting the firewall.

Here is a screenshot of my final settings.

stephenw10

@perrin said in New PPPoE backend, some feedback:

i am having the exact same issue with the if_pppoe failed setting the default route and failed getting an IPv6 config.

This happens when it first connects? Just at random during runtime?

perrin

@stephenw10 it always happens whenever you manually reconnect (via status/interfaces) the interface or disable / enable the interface. Upon reboot / first connect everything works fine

To me it seems that there is a case where if_pppoe gets in some sort of stale state which can only be solved by rebooting

it came to my attention when i was developing the pppoe_ha package. in that process I was disabling/enabling the interface often and noted the connectivity loss.

stephenw10

What does pppcfg show in that situation?

If you enable debugging (ifconfig pppoe0 debug) what does it show?

If you manually reconnect the parent NIC dies that allow it to connect?

perrin

@stephenw10 I will try that as soon as possible when I can afford some downtime. Was not aware of these debug options. Will keep you posted and update the thread asap.

stephenw10

Be aware that the debug option currently will spam the console. So be ready to run ifconfig pppoe0 -debug

perrin

I tried it this moring before working hours but i could not reliably reproduce the stale state.

Didn't have much time to test but i tried a few enable/disable interfaces and pause/resume of the virtual machine of the primary firewall so that the failover occurs and pppoe needs to reconnect.
This morning everything worked as expected.

A few notes on my side: Both firewalls only had an uptime of less than a day. so it might be that the issue only occurs on longer uptime, maybe also if something happens on the provider side in regards to the pppoe session.

Also i did reconfigure the dhcp v6 settings accoring to the post of @wow which seemed to make the IPv6 stuff (more) stable than before.

The only things i've seen from the (debug) logs which kind of are hinting towards an error are the following lines:

Sep 19 07:55:53 	kernel 		if_pppoe: pppoe0: failed to set default route 17
Sep 19 07:55:42 	kernel 		if_pppoe: pppoe0: pap failure
Sep 19 07:55:41 	kernel 		if_pppoe: pppoe0: failed to clear IP address: 49

we can safely ignore the pap failure in my case as this must be due to the failover i was testing. in this case the provider thinks the pppoe session for my user is still active (on the other firewall) so i get a few pap failures in failover until the provider closes the stale session of the other firewall.
I've seen the other errors (default route/ip address) previously though and those might be related to some error.

anyway I will do another test in a couple of days and keep you posted...

edit: i extracted the relevant parts of the pppoe debug log. don't know if that is any help, but i am posting it anyway:

[25.07.1-RELEASE][root@pfSense.xxxxxxxxxx]/root: bzcat /var/log/ppp.log.* | grep "Sep 19" | grep -v output
Sep 19 07:43:00 pfSense newsyslog[73829]: logfile turned over due to size>500K
Sep 19 07:43:05 pfSense kernel: if_pppoe: pppoe0: lcp input(opened): <echo-req id=0x3 len=8
Sep 19 07:43:05 pfSense kernel: if_pppoe: pppoe0: got lcp echo req, sending echo rep
Sep 19 07:43:15 pfSense kernel: if_pppoe: pppoe0: lcp input(opened): <echo-req id=0x4 len=8
Sep 19 07:43:15 pfSense kernel: if_pppoe: pppoe0: got lcp echo req, sending echo rep
Sep 19 07:43:25 pfSense kernel: if_pppoe: pppoe0: lcp input(opened): <echo-req id=0x5 len=8
Sep 19 07:43:25 pfSense kernel: if_pppoe: pppoe0: got lcp echo req, sending echo rep
Sep 19 07:43:35 pfSense kernel: if_pppoe: pppoe0: lcp input(opened): <echo-req id=0x6 len=8
Sep 19 07:43:35 pfSense kernel: if_pppoe: pppoe0: got lcp echo req, sending echo rep
Sep 19 07:43:45 pfSense kernel: if_pppoe: pppoe0: lcp input(opened): <echo-req id=0x7 len=8
Sep 19 07:43:45 pfSense kernel: if_pppoe: pppoe0: got lcp echo req, sending echo rep
Sep 19 07:43:46 pfSense kernel: if_pppoe: pppoe0: lcp close(opened)
Sep 19 07:43:46 pfSense kernel: if_pppoe: pppoe0: lcp opened->closing
Sep 19 07:43:46 pfSense kernel: if_pppoe: pppoe0: phase terminate
Sep 19 07:43:46 pfSense kernel: if_pppoe: pppoe0: ipcp down(opened)
Sep 19 07:43:46 pfSense kernel: if_pppoe: pppoe0: ipcp opened->starting
Sep 19 07:43:46 pfSense kernel: if_pppoe: pppoe0: ipcp close(starting)
Sep 19 07:43:46 pfSense kernel: if_pppoe: pppoe0: ipcp starting->initial
Sep 19 07:43:46 pfSense kernel: if_pppoe: pppoe0: ipv6cp down(opened)
Sep 19 07:43:46 pfSense kernel: if_pppoe: pppoe0: ipv6cp opened->starting
Sep 19 07:43:46 pfSense kernel: if_pppoe: pppoe0: ipv6cp close(starting)
Sep 19 07:43:46 pfSense kernel: if_pppoe: pppoe0: ipv6cp starting->initial
Sep 19 07:43:48 pfSense kernel: if_pppoe: pppoe0: pap failure
Sep 19 07:43:58 pfSense kernel: if_pppoe: pppoe0: failed to set default route 17
Sep 19 07:44:00 pfSense newsyslog[82494]: logfile turned over due to size>500K
Sep 19 07:35:30 pfSense newsyslog[52425]: logfile turned over due to size>500K
Sep 19 07:35:48 pfSense kernel: if_pppoe: pppoe0: LCP keepalive timeout
Sep 19 07:35:48 pfSense kernel: if_pppoe: pppoe0: lcp close(opened)
Sep 19 07:35:48 pfSense kernel: if_pppoe: pppoe0: lcp opened->closing
Sep 19 07:35:48 pfSense kernel: if_pppoe: pppoe0: phase terminate
Sep 19 07:35:48 pfSense kernel: if_pppoe: pppoe0: ipcp down(opened)
Sep 19 07:35:48 pfSense kernel: if_pppoe: pppoe0: ipcp opened->starting
Sep 19 07:35:48 pfSense kernel: if_pppoe: pppoe0: ipcp close(starting)
Sep 19 07:35:48 pfSense kernel: if_pppoe: pppoe0: ipcp starting->initial
Sep 19 07:35:48 pfSense kernel: if_pppoe: pppoe0: ipv6cp down(opened)
Sep 19 07:35:48 pfSense kernel: if_pppoe: pppoe0: ipv6cp opened->starting
Sep 19 07:35:48 pfSense kernel: if_pppoe: pppoe0: ipv6cp close(starting)
Sep 19 07:35:48 pfSense kernel: if_pppoe: pppoe0: ipv6cp starting->initial
Sep 19 07:35:48 pfSense kernel: if_pppoe: pppoe0: lcp closing->stopped
Sep 19 07:35:48 pfSense kernel: if_pppoe: pppoe0: timeout
Sep 19 07:35:48 pfSense kernel: if_pppoe: pppoe0: disconnecting
Sep 19 07:35:48 pfSense kernel: if_pppoe: pppoe0: lcp down(stopped)
Sep 19 07:35:48 pfSense kernel: if_pppoe: pppoe0: lcp stopped->starting
Sep 19 07:35:48 pfSense kernel: if_pppoe: pppoe0: phase establish
Sep 19 07:35:48 pfSense kernel: if_pppoe: pppoe0: Down event (carrier loss), taking interface down.if_pppoe: pppoe0: lcp close(starting)
Sep 19 07:35:48 pfSense kernel: if_pppoe: pppoe0: lcp starting->initial
Sep 19 07:35:48 pfSense kernel: if_pppoe: pppoe0: phase dead
Sep 19 07:35:48 pfSense kernel: if_pppoe: pppoe0: session 0x1bd connected
Sep 19 07:35:48 pfSense kernel: if_pppoe: pppoe0: lcp open(initial)
Sep 19 07:35:48 pfSense kernel: if_pppoe: pppoe0: lcp initial->starting
Sep 19 07:35:48 pfSense kernel: if_pppoe: pppoe0: phase establish
Sep 19 07:35:48 pfSense kernel: if_pppoe: pppoe0: lcp up(starting)
Sep 19 07:35:48 pfSense kernel: if_pppoe: pppoe0: lcp starting->req-sent
Sep 19 07:35:48 pfSense kernel: if_pppoe: pppoe0: lcp input(req-sent): <conf-req id=0xb7 len=18
Sep 19 07:35:48 pfSense kernel: if_pppoe: pppoe0: lcp parse opts:
Sep 19 07:35:48 pfSense kernel: if_pppoe: pppoe0: lcp parse opt values:
Sep 19 07:35:48 pfSense kernel: if_pppoe: pppoe0: lcp req-sent->ack-sent
Sep 19 07:35:48 pfSense kernel: if_pppoe: pppoe0: lcp input(ack-sent): <conf-ack id=0x16 len=14
Sep 19 07:35:48 pfSense kernel: if_pppoe: pppoe0: lcp ack-sent->opened
Sep 19 07:35:48 pfSense kernel: if_pppoe: pppoe0: lcp tlu
Sep 19 07:35:48 pfSense kernel: if_pppoe: pppoe0: up
Sep 19 07:35:48 pfSense kernel: if_pppoe: pppoe0: phase authenticate
Sep 19 07:35:49 pfSense kernel: if_pppoe: pppoe0: pap success
Sep 19 07:35:49 pfSense kernel: if_pppoe: pppoe0: phase network
Sep 19 07:35:49 pfSense kernel: if_pppoe: pppoe0: ipcp open(initial)
Sep 19 07:35:49 pfSense kernel: if_pppoe: pppoe0: ipcp initial->starting
Sep 19 07:35:49 pfSense kernel: if_pppoe: pppoe0: ipv6cp open(initial)
Sep 19 07:35:49 pfSense kernel: if_pppoe: pppoe0: ipv6cp initial->starting
Sep 19 07:35:49 pfSense kernel: if_pppoe: pppoe0: ipcp up(starting)
Sep 19 07:35:49 pfSense kernel: if_pppoe: pppoe0: ipcp starting->req-sent
Sep 19 07:35:49 pfSense kernel: if_pppoe: pppoe0: ipv6cp up(starting)
Sep 19 07:35:49 pfSense kernel: if_pppoe: pppoe0: ipv6cp starting->req-sent
Sep 19 07:35:49 pfSense kernel: if_pppoe: pppoe0: ipcp input(req-sent): <conf-req id=0x99 len=10
Sep 19 07:35:49 pfSense kernel: if_pppoe: pppoe0: ipcp parse opts:
Sep 19 07:35:49 pfSense kernel: if_pppoe: pppoe0: ipcp parse opt values:
Sep 19 07:35:49 pfSense kernel: if_pppoe: pppoe0: ipcp req-sent->ack-sent
Sep 19 07:35:49 pfSense kernel: if_pppoe: pppoe0: ipcp input(ack-sent): <conf-nak id=0x18 len=22
Sep 19 07:35:49 pfSense kernel: if_pppoe: pppoe0: ipcp nak opts:
Sep 19 07:35:49 pfSense kernel: if_pppoe: pppoe0: ipv6cp input(req-sent): <conf-req id=0x40 len=14
Sep 19 07:35:49 pfSense kernel: if_pppoe: pppoe0: ipv6cp parse opts:
Sep 19 07:35:49 pfSense kernel: if_pppoe: pppoe0: ipv6cp parse opt values:
Sep 19 07:35:49 pfSense kernel: if_pppoe: pppoe0: ipv6cp req-sent->ack-sent
Sep 19 07:35:49 pfSense kernel: if_pppoe: pppoe0: failed to set default route 17
Sep 19 07:35:49 pfSense kernel: if_pppoe: pppoe0: ipcp input(ack-sent): <conf-ack id=0x1a len=22
Sep 19 07:35:49 pfSense kernel: if_pppoe: pppoe0: ipcp ack-sent->opened
Sep 19 07:35:49 pfSense kernel: if_pppoe: pppoe0: ipcp tlu
Sep 19 07:35:49 pfSense kernel: if_pppoe: pppoe0: failed to set default route 17
Sep 19 07:35:49 pfSense kernel: if_pppoe: pppoe0: ipv6cp input(ack-sent): <conf-ack id=0x19 len=14
Sep 19 07:35:49 pfSense kernel: if_pppoe: pppoe0: ipv6cp ack-sent->opened
Sep 19 07:35:49 pfSense kernel: if_pppoe: pppoe0: ipv6cp tlu
Sep 19 07:35:55 pfSense kernel: if_pppoe: pppoe0: lcp input(opened): <echo-req id=0x0 len=8
Sep 19 07:35:55 pfSense kernel: if_pppoe: pppoe0: got lcp echo req, sending echo rep
Sep 19 07:36:06 pfSense kernel: if_pppoe: pppoe0: lcp input(opened): <echo-req id=0x1 len=8
Sep 19 07:36:06 pfSense kernel: if_pppoe: pppoe0: got lcp echo req, sending echo rep
Sep 19 07:36:15 pfSense kernel: if_pppoe: pppoe0: lcp input(opened): <echo-req id=0x2 len=8
Sep 19 07:36:15 pfSense kernel: if_pppoe: pppoe0: got lcp echo req, sending echo rep
Sep 19 07:36:25 pfSense kernel: if_pppoe: pppoe0: lcp input(opened): <echo-req id=0x3 len=8
Sep 19 07:36:25 pfSense kernel: if_pppoe: pppoe0: got lcp echo req, sending echo rep
Sep 19 07:36:35 pfSense kernel: if_pppoe: pppoe0: lcp input(opened): <echo-req id=0x4 len=8
Sep 19 07:36:35 pfSense kernel: if_pppoe: pppoe0: got lcp echo req, sending echo rep
Sep 19 07:42:28 pfSense kernel: if_pppoe: pppoe0: LCP keepalive timeout
Sep 19 07:42:28 pfSense kernel: if_pppoe: pppoe0: lcp close(opened)
Sep 19 07:42:28 pfSense kernel: if_pppoe: pppoe0: lcp opened->closing
Sep 19 07:42:28 pfSense kernel: if_pppoe: pppoe0: phase terminate
Sep 19 07:42:28 pfSense kernel: if_pppoe: pppoe0: ipcp down(opened)
Sep 19 07:42:28 pfSense kernel: if_pppoe: pppoe0: ipcp opened->starting
Sep 19 07:42:28 pfSense kernel: if_pppoe: pppoe0: ipcp close(starting)
Sep 19 07:42:28 pfSense kernel: if_pppoe: pppoe0: ipcp starting->initial
Sep 19 07:42:28 pfSense kernel: if_pppoe: pppoe0: ipv6cp down(opened)
Sep 19 07:42:28 pfSense kernel: if_pppoe: pppoe0: ipv6cp opened->starting
Sep 19 07:42:28 pfSense kernel: if_pppoe: pppoe0: ipv6cp close(starting)
Sep 19 07:42:28 pfSense kernel: if_pppoe: pppoe0: ipv6cp starting->initial
Sep 19 07:42:28 pfSense kernel: if_pppoe: pppoe0: lcp closing->stopped
Sep 19 07:42:28 pfSense kernel: if_pppoe: pppoe0: timeout
Sep 19 07:42:28 pfSense kernel: if_pppoe: pppoe0: disconnecting
Sep 19 07:42:28 pfSense kernel: if_pppoe: pppoe0: lcp down(stopped)
Sep 19 07:42:28 pfSense kernel: if_pppoe: pppoe0: lcp stopped->starting
Sep 19 07:42:28 pfSense kernel: if_pppoe: pppoe0: phase establish
Sep 19 07:42:28 pfSense kernel: if_pppoe: pppoe0: Down event (carrier loss), taking interface down.if_pppoe: pppoe0: lcp close(starting)
Sep 19 07:42:28 pfSense kernel: if_pppoe: pppoe0: lcp starting->initial
Sep 19 07:42:28 pfSense kernel: if_pppoe: pppoe0: phase dead
Sep 19 07:42:28 pfSense kernel: if_pppoe: pppoe0: session 0x1bd connected
Sep 19 07:42:28 pfSense kernel: if_pppoe: pppoe0: lcp open(initial)
Sep 19 07:42:28 pfSense kernel: if_pppoe: pppoe0: lcp initial->starting
Sep 19 07:42:28 pfSense kernel: if_pppoe: pppoe0: phase establish
Sep 19 07:42:28 pfSense kernel: if_pppoe: pppoe0: lcp up(starting)
Sep 19 07:42:28 pfSense kernel: if_pppoe: pppoe0: lcp starting->req-sent
Sep 19 07:42:28 pfSense kernel: if_pppoe: pppoe0: lcp input(req-sent): <conf-req id=0x36 len=18
Sep 19 07:42:28 pfSense kernel: if_pppoe: pppoe0: lcp parse opts:
Sep 19 07:42:28 pfSense kernel: if_pppoe: pppoe0: lcp parse opt values:
Sep 19 07:42:28 pfSense kernel: if_pppoe: pppoe0: lcp req-sent->ack-sent
Sep 19 07:42:28 pfSense kernel: if_pppoe: pppoe0: lcp input(ack-sent): <conf-ack id=0x1f len=14
Sep 19 07:42:28 pfSense kernel: if_pppoe: pppoe0: lcp ack-sent->opened
Sep 19 07:42:28 pfSense kernel: if_pppoe: pppoe0: lcp tlu
Sep 19 07:42:28 pfSense kernel: if_pppoe: pppoe0: up
Sep 19 07:42:28 pfSense kernel: if_pppoe: pppoe0: phase authenticate
Sep 19 07:42:29 pfSense kernel: if_pppoe: pppoe0: pap success
Sep 19 07:42:29 pfSense kernel: if_pppoe: pppoe0: phase network
Sep 19 07:42:29 pfSense kernel: if_pppoe: pppoe0: ipcp open(initial)
Sep 19 07:42:29 pfSense kernel: if_pppoe: pppoe0: ipcp initial->starting
Sep 19 07:42:29 pfSense kernel: if_pppoe: pppoe0: ipv6cp open(initial)
Sep 19 07:42:29 pfSense kernel: if_pppoe: pppoe0: ipv6cp initial->starting
Sep 19 07:42:29 pfSense kernel: if_pppoe: pppoe0: ipcp up(starting)
Sep 19 07:42:29 pfSense kernel: if_pppoe: pppoe0: ipcp starting->req-sent
Sep 19 07:42:29 pfSense kernel: if_pppoe: pppoe0: ipv6cp up(starting)
Sep 19 07:42:29 pfSense kernel: if_pppoe: pppoe0: ipv6cp starting->req-sent
Sep 19 07:42:29 pfSense kernel: if_pppoe: pppoe0: ipcp input(req-sent): <conf-req id=0x22 len=10
Sep 19 07:42:29 pfSense kernel: if_pppoe: pppoe0: ipcp parse opts:
Sep 19 07:42:29 pfSense kernel: if_pppoe: pppoe0: ipcp parse opt values:
Sep 19 07:42:29 pfSense kernel: if_pppoe: pppoe0: ipcp req-sent->ack-sent
Sep 19 07:42:29 pfSense kernel: if_pppoe: pppoe0: ipcp input(ack-sent): <conf-nak id=0x21 len=22
Sep 19 07:42:29 pfSense kernel: if_pppoe: pppoe0: ipcp nak opts:
Sep 19 07:42:29 pfSense kernel: if_pppoe: pppoe0: ipv6cp input(req-sent): <conf-req id=0x36 len=14
Sep 19 07:42:29 pfSense kernel: if_pppoe: pppoe0: ipv6cp parse opts:
Sep 19 07:42:29 pfSense kernel: if_pppoe: pppoe0: ipv6cp parse opt values:
Sep 19 07:42:29 pfSense kernel: if_pppoe: pppoe0: ipv6cp req-sent->ack-sent
Sep 19 07:42:29 pfSense kernel: if_pppoe: pppoe0: failed to set default route 17
Sep 19 07:42:29 pfSense kernel: if_pppoe: pppoe0: ipcp input(ack-sent): <conf-ack id=0x23 len=22
Sep 19 07:42:29 pfSense kernel: if_pppoe: pppoe0: ipcp ack-sent->opened
Sep 19 07:42:29 pfSense kernel: if_pppoe: pppoe0: ipcp tlu
Sep 19 07:42:29 pfSense kernel: if_pppoe: pppoe0: failed to set default route 17
Sep 19 07:42:29 pfSense kernel: if_pppoe: pppoe0: ipv6cp input(ack-sent): <conf-ack id=0x22 len=14
Sep 19 07:42:29 pfSense kernel: if_pppoe: pppoe0: ipv6cp ack-sent->opened
Sep 19 07:42:29 pfSense kernel: if_pppoe: pppoe0: ipv6cp tlu
Sep 19 07:42:35 pfSense kernel: if_pppoe: pppoe0: lcp input(opened): <echo-req id=0x0 len=8
Sep 19 07:42:35 pfSense kernel: if_pppoe: pppoe0: got lcp echo req, sending echo rep
Sep 19 07:42:45 pfSense kernel: if_pppoe: pppoe0: lcp input(opened): <echo-req id=0x1 len=8
Sep 19 07:42:45 pfSense kernel: if_pppoe: pppoe0: got lcp echo req, sending echo rep
Sep 19 07:42:55 pfSense kernel: if_pppoe: pppoe0: lcp input(opened): <echo-req id=0x2 len=8
Sep 19 07:42:55 pfSense kernel: if_pppoe: pppoe0: got lcp echo req, sending echo rep
Sep 19 07:43:00 pfSense newsyslog[73829]: logfile turned over due to size>500K
Sep 19 07:25:28 pfSense kernel: if_pppoe: pppoe0: failed to set default route 17
Sep 19 07:27:28 pfSense kernel: if_pppoe: pppoe0: failed to clear IP address: 49
Sep 19 07:27:29 pfSense kernel: if_pppoe: pppoe0: pap failure
Sep 19 07:31:18 pfSense kernel: if_pppoe: pppoe0: LCP keepalive timeout
Sep 19 07:31:18 pfSense kernel: if_pppoe: pppoe0: failed to set default route 17
Sep 19 07:31:18 pfSense kernel: if_pppoe: pppoe0: failed to set default route 17
Sep 19 07:34:59 pfSense kernel: if_pppoe: pppoe0: lcp input(opened): <echo-req id=0x7 len=8
Sep 19 07:34:59 pfSense kernel: if_pppoe: pppoe0: got lcp echo req, sending echo rep
Sep 19 07:35:30 pfSense newsyslog[52425]: logfile turned over due to size>500K

w0w

Possible that another problem found and maybe solved :)
I want to note that the problem shows only with the new if_pppoe; with mpd5 all VIP addresses come up successfully.
I tried to study why on my HA primary sometimes the system boots with VIP addresses on LAN not initialized. It looks like we have again a race condition.

Even if PPPoE is configured at the end using the function

/* setup PPPoE and PPTP (this will bring up if_pppoe) -edited by me */
vpn_setup();

called from /etc/rc.bootup, in my case interfaces often do not have time to get all VIPs. I have many ports, they flap several times (down/up), and this coincides with the moment when PPPoE brings the link up successfully. At this moment it seems the VIP configuration stays unfinished. From experiments I noticed usually only one VIP gets configurated.

I am not sure if this is the correct solution, but I added into /etc/rc.bootup the line

interfaces_vips_configure("");

before

/* setup PPPoE and PPTP (this will bring up if_pppoe) */
vpn_setup();

This tends system to wait interfaces_vips_configure is completed.
After that it looks OK. All VIPs are present and CARP works too.

azalea

When I checked "Use if_pppoe kernel module for PPPoE client" to enable if_pppoe, this fell under case (b). As a result, the ping failed.

I have a correction. When I checked again, I found that, as in the case (a), pppoe0 had linked up and the IPv6 address had been set before "Bootup complete". (ping passed)

I am prompted to reboot after checking "Use if_pppoe kernel module for PPPoE client" to enable if_pppoe. I canceled here and checked the WAN Interfaces Status, and found that the PPPoE session had been automatically closed. After that I rebooted manually and the result was the same as in case (a). The General log also recorded "pppoe0: link state changed to DOWN" just before the reboot began.

Do IPv4 pings (general connectivity) succeed in both cases? Just varies the IPv6 response?

IPv4 ping passes in both cases. Ping results differ for IPv6 only.

So you also see failures in IPv6 when using mpd5/netgraph?

(c) If I reboot pfSense with manually closing the PPPoE session, pppoe0 links up and the IPv6 address is set before "Bootup complete".
(d) If I reboot pfSense without manually closing the PPPoE session, pppoe0 links up and the IPv6 address is set before "Bootup complete".
(e) If I hard reset pfSense without manually closing the PPPoE session, pppoe0 links up and the IPv6 address is set after "Bootup complete".

In case (c), the ping passes.
In case (d), Ping passes most of the time. In rare cases, ping fails because dpinger is stopped. After I transition from step 1 to step 3, ping passes.
In case (e), the ping fails. After I transition from step 1 to step 3, the ping fails.

[step 1]

Gateway address is displayed.
Monitor address is not displayed.
Gateway status is Pending.
"(default)" under "WAN_DHCP6" is not displayed.
Gateway IPv6 is displayed.
dpinger is stopped.
Ping fails.

[step 2]
I start dpinger.

Monitor address is displayed.
Gateway status is Online.
"(default)" under "WAN_DHCP6" is not displayed.
Ping fails.

[step 3]
I save the Default gateway configuration (Automatic) without any changes.

"(default)" under "WAN_DHCP6" is displayed.
In case (d), the ping passes.
In case (e), the ping fails.

(d) is a surprising result. In the case of mpd5, pfSense may automatically close the PPPoE session before rebooting. If so, I assume that "automatically closing of PPPoE sessions before rebooting" would be an effective solution for if_pppoe as well.

Regarding (e), I assumed that mpd5 had automatically closed the PPPoE session before rebooting, so I tried a hard reset to avoid this.

From the ping results for (b) and (e), it seems that ping fails when the IPv6 address is set after "Bootup complete" whether using mpd5 or if_pppoe.

When I look at (d), in the case of mdp5, the Gateway/dpinger status and the ping result change in tandem with the transition from step 1 to step 3. This is natural. On the other hand, when I look at (a), in the case of if_pppoe, the Gateway/dpinger status and the ping result do not have the correct relationship. This is strange.

azalea

In addition to the ping issue, there is also an issue where the Gateway address (Gateway IPv6) is not set.

Am I the only one for whom the Gateway address (Gateway IPv6) is not set when using if_pppoe? If so, I assume it's due to the uniqueness of setting only IPv6 for one PPPoE session.

Specifically, if_pppoe assumes that IPv4 is configured. However, since there is no IPv4 configuration, if_pppoe cannot set the IPv4 Gateway (WAN_PPPOE). It is determined that an error has occurred in the IPv4 Gateway setting, and the IPv6 Gateway (WAN_DHCP6) setting is canceled. Is this guess correct?