New PPPoE backend, some feedback

stephenw10

@RobbieTT said in New PPPoE backend, some feedback:

No, until the upstream PPPoE assigns my static IP via DHCP the firewall has no idea of my IP address.

Hmm, your ISP is doing DHCP over PPPoE? For IPv4? I don't think I've ever seen that if so.

RobbieTT

@stephenw10 said in New PPPoE backend, some feedback:

Hmm, your ISP is doing DHCP over PPPoE? For IPv4? I don't think I've ever seen that if so.

Probably my poor choice of terms Steve. The ISP dynamically sets my WAN IP address (albeit my IPv4 address is static) as part of the PPPoE connection process. For most UK residential customers they will have a dynamic WAN IP address assigned by the ISP on connection (PPPoE or DHCP) and this will periodically change, including on reboot etc.

Whilst we use different terms on routers between DHCP and PPPoE connections a PPPoE connection can still (and usually does in the UK) have a form of dynamic host configuration. This is sent by the upstream PPPoE server down to the router, which is effectively now a PPPoE client, including the dynamically assigned IP address(es) as well as the other network parameters.

For PPPoE connection the dynamic host configuration comes immediately after the CHAP, triggered by the Configuration Request with the dynamic host configuration coming back on the Ack (dynamic options, IP address to use, DNS info etc. Sometimes it is referred to as the PPP IP Control measures but it is a bit broader than IP address control alone.

You and I both use PPPoE over the BT/Openreach infrastructure so presumably there are few (if any) significant differences between our connections, even though I have a different ISP.

There are many ways to describe things but this one is mine so only subject to my errors!

️

stephenw10

Ah OK, good. For a minute there I thought you had some weird edge case connection.

Yes, AFAIK almost all UK ISPs use the same low level infrastructure. Certainly if they are not LLU providers, like Sky for example.

Then it seems like I should be able to hit it given enough poking....

chrcoluk

I think this is awesome this has been developed, thank you to everyone who has contributed to it. I think in the UK it will be especially appreciated as almost all ISPs are wanting to keep using PPPoE instead of IPoE.

stephenw10

@RobbieTT
Try this. Run: ifconfig pppoe debug

Then try to reproduce the connection issue. Check dmesg for any additional error logs whilst is fails to connect.

RobbieTT

@stephenw10 said in New PPPoE backend, some feedback:

@RobbieTT
Try this. Run: ifconfig pppoe debug

Then try to reproduce the connection issue. Check dmesg for any additional error logs whilst is fails to connect.

All done. I will email you a clean connection from a reboot and a 'failed' but ultimately reconnected pppoe0 interface down/up event.

Nothing of note in the reboot logs (to my eyes at lease) but disconnecting / reconnecting from the interface page produced lots of this (plus GUI hangs, server not responding etc):

Apr 30 17:02:32	php	29393	/usr/local/sbin/ppp-ipv6: The command '/sbin/ifconfig 'pppoe0' inet6 -accept_rtadv' returned exit code '1', the output was 'ifconfig: interface pppoe0 does not exist'
Apr 30 17:02:31	php-cgi	28665	pppoe-handler: HOTPLUG event: Invalid IP address
Apr 30 17:02:31	kernel		if_pppoe: pppoe0 (8864) state=3, session=0x1425 output -> f8:xx:xx:xx:xx:ea, len=37
Apr 30 17:02:31	kernel		if_pppoe: pppoe0 (8864) state=3, session=0x1425 output -> f8:xx:xx:xx:xx:ea, len=80
Apr 30 17:02:31	kernel		if_pppoe: pppoe0 (8864) state=3, session=0x1425 output -> f8:xx:xx:xx:xx:ea, len=111
Apr 30 17:02:31	kernel		if_pppoe: pppoe0 (8864) state=3, session=0x1425 output -> f8:xx:xx:xx:xx:ea, len=60
Apr 30 17:02:31	kernel		if_pppoe: pppoe0 (8864) state=3, session=0x1425 output -> f8:xx:xx:xx:xx:ea, len=787

️

RobbieTT

@stephenw10
Updated the latest public beta (25.03.b.20250429.1329) but no change in the symptoms or the GUI presentation of the true interface status.

I captured system logs and dmesg but nothing notable against the previous ones. It took a few retries to get itself going again.

[25.03-BETA][@Router-7.me]/root: pppcfg pppoe0
	dev: igc0 state: session
	sid: 0x1552 PADI retries: 5 PADR retries: 0 time: 02:57:46
	sppp: phase network authproto auto authname "xxxxxxxxxx@idnet" peerproto auto 
	dns: 212.69.40.23 212.69.36.23
[25.03-BETA][@Router-7.me]/root:

️

RobbieTT

@stephenw10

On 25.03.b.20250429.1329

With the “Do not wait for a RA” box unchecked (my usual config is to have this box checked for reasons long since forgotten but sure to bite me at some point) the PPPoE interface symptoms, when selecting disconnect / reconnect, appear to have gone.

Not really sure why this box makes a difference but I have yet to see pfSense trip over itself since testing it with it unchecked.

I can see the additional PPP logging that has been added (rather than just reflecting when I last used the old PPPoE backend). Not sure what value it has added just yet as the logs are just filled with this:

if_pppoe: pppoe0 (8864) state=3, session=0x16bb output -> f8:13:08:xx:xx:ea, len=92
if_pppoe: pppoe0 (8864) state=3, session=0x16bb output -> f8:13:08:xx:xx:ea, len=347
if_pppoe: pppoe0 (8864) state=3, session=0x16bb output -> f8:13:08:xx:xx:ea, len=424
if_pppoe: pppoe0 (8864) state=3, session=0x16bb output -> f8:13:08:xx:xx:ea, len=37
if_pppoe: pppoe0 (8864) state=3, session=0x16bb output -> f8:13:08:xx:xx:ea, len=64
etc...

A further observation, with a layer of the onion now removed, is that additional services (eg Avahi, pfBlocker, VPNs etc) spend considerable time in the PPPoE interface starting session tying pfSense in knots, trying to re-initialise themselves for each and every stage of the PPPoE connection process.

Rather than waiting for the PPPoE interface to be fully up they clutter up the process with each (very short lived) up/down, port open/closed or re-numeration of the interface. This is somewhat similar to how the GUI seems to think the interface is 'up' for WAN / PPPoE when it is in the middle of restarting the session. It is like everything expects the PPPoE to be up and running before if_pppoe has signalled that it has completed the task.

ifconfig pppoe0 debug, dmesg -a and system log available on request.

️

RobbieTT

@stephenw10

With the “Do not wait for a RA” box unchecked (my usual config is to have this box checked for reasons ~~long since forgotten~~ but sure to bite me at some point)

Ok, un-forgotten quite quickly. Whilst leaving the RA box unchecked works for taking the PPPoE interface down and up again it screws-up a full reboot instead.

Without the “Do not wait for a RA” box checked, on a full reboot the interface and the PPPoE appear to be up and running on the GUI but no actual internet traffic is passed for a further 4 or 5 minutes or more.

Start Time:

May 3 17:06:55	kernel		---<<BOOT>>---
May 3 17:06:55	syslogd		kernel boot file is /boot/kernel/kernel
May 3 17:05:09	syslogd		exiting on signal 15
May 3 17:05:09	reboot	97088	rebooted by root

To this point when pfSense thinks it is ready (and normally where it should be up and running) but cannot reach outside:

May 3 17:07:50	kernel		done.
May 3 17:07:48	php-cgi	68067	notify_monitor.php: Could not send the message to xxxxxxx@xxxxxxx.me -- Error: Failed to connect to mail.haveworx.co.uk:587 [SMTP: Failed to connect socket: php_network_getaddresses: getaddrinfo for mail.haveworx.co.uk failed: Name does not resolve (code: -1, response: )]

To this point, where traffic does actually flow:

May 3 17:11:44	php-fpm	44318	/rc.newwanipv6: Resyncing OpenVPN instances for interface WAN.
May 3 17:11:44	check_reload_status	680	Reloading filter
May 3 17:11:35	php_pfb	5699	[pfBlockerNG] filterlog daemon started
May 3 17:11:35	php_pfb	4074	[pfBlockerNG] filterlog daemon started
May 3 17:11:35	php-fpm	44318	/rc.newwanipv6: rc.newwanipv6: on (IP address: 2a02:xxx:feed:xxxx:xxxx:xxxx:xxxx:xx06) (interface: wan) (real interface: pppoe0).
May 3 17:11:35	php-fpm	44318	/rc.newwanipv6: rc.newwanipv6: Info: starting on pppoe0 due to REQUEST.

So I guess we still have a problem but we can move the problem somewhere else.

️

stephenw10

Hmm, interesting.

I expect to not have that checked because the dhcp is set to go over PPPoE. It should only try to pull a lease once the PPPoE is up and remote server sends an RA over it. But that does depend on the frequency the ISP sends at. One of the other issues we are seeing is with ISPs that send RAs at high frequency, like 10s intervals, and trigger events at each.

But I suspect the difference here is that the old backend only marks the interface up once it's actually connected and if_pppoe is seen as UP as soon as it's created. If dhcp6c doesn't wait for an RA it will immediately try and fail and then.... get stuck in some fail-loop!

We are changing that behaviour now so it may be fixed in the next build anyway.

RobbieTT

@stephenw10 said in New PPPoE backend, some feedback:

Hmm, interesting.

I expect to not have that checked because the dhcp is set to go over PPPoE. It should only try to pull a lease once the PPPoE is up and remote server sends an RA over it.

Looking forward to the changes.

My ISP RA's are sent reasonably infrequently so once the PPPoE session is up the client router (pfSense) should send an RS upstream and get the RA straight back. Occasionally an RA is captured first but typically the RA used will be triggered by the RS.

The days of waiting obediently for an RA should be confined to history (well, whenever the replacement RFC came out, which is a number of years ago now). ISPs that deliberately machine-gun out unsolicited RAs should be sent a burning copy of the standards.

️