New PPPoE backend, some feedback
-
@stephenw10 said in New PPPoE backend, some feedback:
@RobbieTT said in New PPPoE backend, some feedback:
Is there a definitive list of PPPoE performance tweaks that should be removed before using if_pppoe?
There isn't. The testing we did though was without any of the previous pppoe tweaks AFAIK. So the default sysctls only.
I know, I just need a nudge on what I now need to remove (eg deleting the top entry for a start):
️ -
I think only the net.isr.dispatch value there is non-default.
Default values I see here are:
Tunable Name Description Value net.inet.ip.portrange.first 1024 net.inet.tcp.blackhole Do not send RST on segments to closed ports 2 net.inet.udp.blackhole Do not send port unreachables for refused connects 1 net.inet.ip.random_id Assign random ip_id values 1 net.inet.tcp.drop_synfin Drop TCP packets with SYN+FIN set 1 net.inet.ip.redirect Enable sending IP redirects 1 net.inet6.ip6.redirect Send ICMPv6 redirects for unforwardable IPv6 packets 1 net.inet6.ip6.use_tempaddr Create RFC3041 temporary addresses for autoconfigured addresses 0 net.inet6.ip6.prefer_tempaddr Prefer RFC3041 temporary addresses in source address selection 0 net.inet.tcp.syncookies Use TCP SYN cookies if the syncache overflows 1 net.inet.tcp.recvspace Initial receive socket buffer size 65228 net.inet.tcp.sendspace Initial send socket buffer size 65228 net.inet.tcp.delayed_ack Delay ACK to try and piggyback it onto a data packet 0 net.inet.udp.maxdgram Maximum outgoing UDP datagram size 57344 net.link.bridge.pfil_onlyip Only pass IP packets when pfil is enabled 0 net.link.bridge.pfil_member Packet filter on the member interface 1 net.link.bridge.pfil_bridge Packet filter on the bridge interface 0 net.link.tap.user_open Enable legacy devfs interface creation for all users 1 net.link.vlan.mtag_pcp Retain VLAN PCP information as packets are passed up the stack 1 kern.randompid Random PID modulus. Special values: 0: disable, 1: choose random value 347 net.inet.ip.intr_queue_maxlen Maximum size of the IP input queue 1000 hw.syscons.kbd_reboot enable keyboard reboot 0 net.inet.tcp.log_debug Log errors caused by incoming TCP segments 0 net.inet.tcp.tso Enable TCP Segmentation Offload 1 net.inet.icmp.icmplim Maximum number of ICMP responses per second 0 vfs.read_max Cluster read-ahead max block count 32 kern.ipc.maxsockbuf Maximum socket buffer size 4262144 net.inet.ip.process_options Enable IP options processing ([LS]SRR, RR, TS) 0 kern.random.harvest.mask Entropy harvesting mask 351 net.route.netisr_maxqlen maximum routing socket dispatch queue length 1024 net.inet.udp.checksum compute udp checksum 1 net.inet.icmp.reply_from_interface ICMP reply from incoming interface for non-local packets 1 net.inet6.ip6.rfc6204w3 Accept the default router list from ICMPv6 RA messages even when packet forwarding is enabled 1 net.key.preferred_oldsa 0 net.inet.carp.senderr_demotion_factor Send error demotion factor adjustment 0 net.pfsync.carp_demotion_factor pfsync's CARP demotion factor adjustment 0 net.raw.recvspace 65536 net.raw.sendspace 65536 net.inet.raw.recvspace Maximum space for incoming raw IP datagrams 131072 net.inet.raw.maxdgram Maximum outgoing raw IP datagram size 131072 kern.corefile Process corefile name format string /root/%N.core kern.crypto.iimb.enable_aescbc 1 kern.crypto.iimb.enable_multiq 1 kern.crypto.iimb.use_task 0 kern.crypto.iimb.arch auto kern.crypto.iimb.prefetch 1 kern.crypto.iimb.max_jobs 256Urgh, formatting fail!
-
Well I'm up and running on if_pppoe and I can see the pppoe load being spread across multiple cores. It works!
UK / Openreach FTTP / 1500MTU (baby-jumbos) / Xeon D-1736NT CPU @ 2.70GHz / HyperThreading Off / IPV4 & IPV6 / bidirectional fq_codel / 10 GbE LANs & VLANs
I think I will need to get the Netgate 6100 out and try this. Running pfSense+ on my Xeon provided no issues for single-core PPPoE anyway but it does seem to run at a slightly lower CPU load with the latest config.
No PPP logs or entries in the System logs, which I think has been mentioned already.
️ -
Ooo you got full size packets over pppoe working. For some reason I can't seem to set that. I'm wondering if I'm hitting some hardware restriction.
Did you have to do anything special?
-
@stephenw10 said in New PPPoE backend, some feedback:
Ooo you got full size packets over pppoe working. For some reason I can't seem to set that. I'm wondering if I'm hitting some hardware restriction.
Did you have to do anything special?
Didn't change anything as I did an in-place update so my config was unchanged from before:
@Smaug ~ % ping -D -s 1472 8.8.8.8 PING 8.8.8.8 (8.8.8.8): 1472 data bytes 1480 bytes from 8.8.8.8: icmp_seq=0 ttl=119 time=8.628 ms 1480 bytes from 8.8.8.8: icmp_seq=1 ttl=119 time=8.637 ms 1480 bytes from 8.8.8.8: icmp_seq=2 ttl=119 time=8.457 ms 1480 bytes from 8.8.8.8: icmp_seq=3 ttl=119 time=8.303 ms 1480 bytes from 8.8.8.8: icmp_seq=4 ttl=119 time=8.383 ms 1480 bytes from 8.8.8.8: icmp_seq=5 ttl=119 time=8.600 ms 1480 bytes from 8.8.8.8: icmp_seq=6 ttl=119 time=8.442 ms 1480 bytes from 8.8.8.8: icmp_seq=7 ttl=119 time=8.582 ms 1480 bytes from 8.8.8.8: icmp_seq=8 ttl=119 time=8.845 ms 1480 bytes from 8.8.8.8: icmp_seq=9 ttl=119 time=8.423 msSo everything worked for me out of the gate; I only checked it 'just to be sure' as it is a bit of a UK oddity.
I can compare a few things for you tomorrow if you like.
️ -
Yup, something odd Openreach are doing. I'll recheck....
-
Hmm, I have that set but the actual interface MTU is 1492 still:
pppoe1: flags=1008851<UP,POINTOPOINT,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1492 description: BT options=0 inet 86.191.X.X --> 172.16.13.252 netmask 0xffffffff inet6 fe80::201:21ff:fe01:6775%pppoe1 prefixlen 64 scopeid 0x10 groups: pppoec nd6 options=123<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL,NO_DAD>Do you have the parent set to 1508?
-
Ah forgot I'd added it to a bridge.
That worked once I set the parent to 1508.
[2.8.0-BETA][admin@pfsense.fire.box]/root: ifconfig igb1 igb1: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1508 description: BT_MODEM options=48100b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,HWSTATS,MEXTPG> ether 00:01:21:01:67:76 inet 192.168.102.10 netmask 0xffffff00 broadcast 192.168.102.255 inet6 fe80::201:21ff:fe01:6776%igb1 prefixlen 64 scopeid 0x2 media: Ethernet autoselect (1000baseT <full-duplex>) status: active nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> [2.8.0-BETA][admin@pfsense.fire.box]/root: pppcfg pppoe1 dev: igb1 svc: BTInfinity state: session sid: 0x1a99 PADI retries: 5 PADR retries: 0 time: 00:01:24 sppp: phase network authproto auto authname "bthomehub@btbroadband.com" peerproto auto dns: 81.139.56.100 81.139.57.100 [2.8.0-BETA][admin@pfsense.fire.box]/root: ifconfig pppoe1 pppoe1: flags=1008851<UP,POINTOPOINT,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500 description: BT options=0 inet 86.191.X.X --> 172.16.13.252 netmask 0xffffffff inet6 fe80::201:21ff:fe01:6775%pppoe1 prefixlen 64 scopeid 0xf groups: pppoec nd6 options=123<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL,NO_DAD> -
@stephenw10 said in New PPPoE backend, some feedback:
Hmm, I have that set but the actual interface MTU is 1492 still:
pppoe1: flags=1008851<UP,POINTOPOINT,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1492 description: BT options=0 inet 86.191.X.X --> 172.16.13.252 netmask 0xffffffff inet6 fe80::201:21ff:fe01:6775%pppoe1 prefixlen 64 scopeid 0x10 groups: pppoec nd6 options=123<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL,NO_DAD>Do you have the parent set to 1508?
The link to the ONT? It is set to 1508 as you would expect - the final pipe needs to be a bit bigger to carry the PPPoE overhead and I am sure you used to have it set that way:
️ -
Yeah, that would do it for sure!
Mine:
igc0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1508 description: ONT options=4e020bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_MAGIC,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG> ether 02:76:xx:xx:xx:81 inet 10.0.0.1 netmask 0xffffff00 broadcast 10.0.0.255 inet6 fe80::76:xxxx:fe00:xxxx%igc0 prefixlen 64 scopeid 0x5 media: Ethernet autoselect (1000baseT <full-duplex>) status: active nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>️ -
Yup and in fact you don't have to set the parent to 1508. If you set the pppoe interface to 1500 the parent will inherit that and be set to 1508.
I had the parent interface as part of an unassigned bridge and forgot. In that situation the bridge interface controls the MTU of all the members and was forcing it to 1500.
All good now though.
-
@stephenw10 said in New PPPoE backend, some feedback:
Yup and in fact you don't have to set the parent to 1508. If you set the pppoe interface to 1500 the parent will inherit that and be set to 1508.
All good now though.
More things I didn't know!
I'll probably leave it set as 1508 just to help remind me what I am doing when I am tired and forgetting what I am supposed to be doing...
️ -
@RobbieTT I've posted my issues over in the reddit forum but I'm not sure its been picked up.
If you go to Status-> Interfaces - then Disconnect the WAN interface to drop the PPPoE, then refresh the page to get the button to change to Connect, then Connect, does it reconnect? I only get every other connection attempt work, every other one stalls saying its "UP" but no Gateways or connectivity is established, I need to drop and then reconnect again. It's fine switching back to the original PPPoE.
Also the IPv6 Gateway doesn't always start monitoring correctly, and shows status Unknown, but there is IPv6 connectivity okay. Going into Gateways and into the IPv6 gateway and disabling monitoring, then saving that, then re-enabling monitoring and it starts monitoring and switches back to online.
With regards to the MTU, I find a great way to check is to use Speedguide.net and from the left hand menu select their TCP/IP Analzyer, snapshot of mine below. I would think you are only seeing 1492 MTU and losing the 8 bytes due to PPPoE. Your MTU can be set to 1508 (1508 being what they call a Baby Jumbo Frame) which is supported via Openreach and most others, so that holds the extra overhead for PPPoE, and when that's stripped of you are left then with the full 1500 bytes.
-
@Phil2025 said in New PPPoE backend, some feedback:
Speedguide.net
Hi Phil, perhaps lost in the noise but I didn't say I had an issue with the PPPoE MTU, only that I checked it to be sure it was still ok, demonstrated my settings and displayed the test results to prove it. @stephenw10 did have a hiccup with his settings due to his role in testing multiple configurations - all now resolved.
I did try the speedguide.net link and I note that it does not run on Safari and seems to require a Chromium-based browser. Using Brave browser my results are:
« SpeedGuide.net TCP Analyzer Results » Tested on: 2025.04.25 04:13 IP address: 93.8x.xxx.xx Client OS/browser: Mac OS (Chrome 135.0.0.0) TCP options string: 020405b4010303060101080a26e64ea30000000004020000 MSS: 1460 MTU: 1500 TCP Window: 131776 (not multiple of MSS) RWIN Scaling: 6 bits (2^6=64) Unscaled RWIN : 2059 Recommended RWINs: 64240, 128480, 256960, 513920, 1027840 BDP limit (200ms): 527 Mbps (53 Megabytes/s) BDP limit (500ms): 211 Mbps (21 Megabytes/s) MTU Discovery: ON TTL: 50 Timestamps: ON SACKs: ON IP ToS: 00000000 (0)
Nothing leaps out at me but I am unfamiliar with this tool so feel free to run your eyes over it for me.
Regarding the Status / Interfaces / PPPoE interface 'Disconnect' I do experience the exact same issue you describe and pfSense seems to hang after the first state change command (eg Disconnect WAN) and if left alone it fails to recover IPv6 properly. If you repeat the command it seems to all work fine again, including IPv6.
So that looks like a bug that needs resolving. It may even be linked to the PPP status logs not working, as noted by others.
️ -
@RobbieTT Many thanks for trying the connect and disconnect, good to know its not just me, I hope they pick it up and fix it as I think it could mean if the Internet goes down for some reason, it may not come back up again by itself.
Yes your MTU looks spot on, usually with 1500 set under the WAN it would mean you would only see 1492 as MTU, so I expect your ISP is negotiating 1508 when you connect and overriding the 1500 MTU you have set.
With other overheads you end up with 1448 bytes of useable data in each packet. You can tweak things to get rid of those overheads and get a bit more throughput, but things like timestamps are useful for retransmission and recovering from errors, so what you lose in maximum throughput you gain in another way, so its all good as it is I would say.
-
I have 1508 MTU set on the interface carrying the PPPoE encapsulation (listed as ONT in my examples above).
The un-molested WAN connection inside this encapsulated and slightly larger MTU pipe is set at the standard 1500 MTU (albeit note @stephenw10's comments above).
I don't think the ISP or Openreach can negotiate a 1508 MTU if you have a smaller one set. Indeed, there are many users out there with either no appreciation for baby jumbo frames or have inflexible hardware that does not allow for an MTU above 1500 to be set.
The BT SINs make no mention of being able to negotiate a higher MTU than requested; the exact opposite is stated though, with connectivity warnings. From memory the BT SINs give a maximum MTU of 1530 (previously 1520) on the Openreach network.
I'm not sure what overheads can be tweaked or removed on my PPPoE connection to get additional throughput though - can you expand on these for me please?
️ -
I think the connection status you see is actually an artifact of the new interface type. I can sort of replicate what you see but in fact it does connect it's just not instant. You can see what the actual state is using the new
pppcfgcommand.When you disconnect the interface in the gui it is removed entirely:
[2.8.0-BETA][admin@pfsense.fire.box]/root: pppcfg pppoe1 [2.8.0-BETA][admin@pfsense.fire.box]/root: ifconfig pppoe1 ifconfig: interface pppoe1 does not existThen as soon as you click connect the interface is created and is shown as UP but the PPP link it still connecting:
[2.8.0-BETA][admin@pfsense.fire.box]/root: pppcfg pppoe1 dev: igb1 svc: BTInfinity state: PADI sent sid: 0x0 PADI retries: 3 PADR retries: 0 sppp: phase establish authproto auto authname "bthomehub@btbroadband.com" peerproto auto [2.8.0-BETA][admin@pfsense.fire.box]/root: ifconfig pppoe1 pppoe1: flags=1008851<UP,POINTOPOINT,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500 description: BT options=0 inet6 fe80::201:21ff:fe01:6775%pppoe1 prefixlen 64 tentative scopeid 0x13 groups: pppoec nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>Then after some time it completes the connection:
[2.8.0-BETA][admin@pfsense.fire.box]/root: pppcfg pppoe1 dev: igb1 svc: BTInfinity state: session sid: 0x1b6d PADI retries: 4 PADR retries: 0 time: 00:00:12 sppp: phase network authproto auto authname "bthomehub@btbroadband.com" peerproto auto dns: 81.139.56.100 81.139.57.100 [2.8.0-BETA][admin@pfsense.fire.box]/root: ifconfig pppoe1 pppoe1: flags=1008851<UP,POINTOPOINT,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500 description: BT options=0 inet 217.45.X.X --> 172.16.13.252 netmask 0xffffffff inet6 fe80::201:21ff:fe01:6775%pppoe1 prefixlen 64 scopeid 0x13 groups: pppoec nd6 options=123<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL,NO_DAD>The actual time taken depends how fast the server responds. For be that be almost instant or it can be several retries and 20-30s.
If you just refresh the Status page without clicking connect again it should just connect.
We saw your post on Reddit yesterday and were discussing options to make the age clearer for if_pppoe. Not sure what will happen yet.
-
@Phil2025 said in New PPPoE backend, some feedback:
Many thanks for trying the connect and disconnect, good to know its not just me, I hope they pick it up and fix it as I think it could mean if the Internet goes down for some reason, it may not come back up again by itself.
I have the same problem. I thought it was just me. Please report it on Redmine.
@RobbieTT said in New PPPoE backend, some feedback:
It may even be linked to the PPP status logs not working, as noted by others
I don't think so, this is some kind of "GUI switching between backends" problem.
As for MTU....mine is set to 1492 but I am using MSS also custom
« SpeedGuide.net TCP Analyzer Results » Tested on: 2025.04.25 05:40 IP address: 84.52.xx.xx Client OS/browser: Windows 10 (Firefox 137.0) TCP options string: 020405200103030801010402 MSS: 1312 MTU: 1352 TCP Window: 65280 (not multiple of MSS) RWIN Scaling: 8 bits (2^8=256) Unscaled RWIN : 255 Recommended RWINs: 62976, 125952, 251904, 503808, 1007616 BDP limit (200ms): 2611 kbps (261 Kilobytes/s) BDP limit (500ms): 1044 kbps (104 Kilobytes/s) MTU Discovery: OFF TTL: 54 Timestamps: OFF SACKs: ON IP ToS: 00000010 (2) Precedence: 000 (routine) Delay: 0 (normal delay) Throughput: 0 (normal throughput) Reliability: 0 (normal reliability) Cost: 1 (low cost) Check bit: 0 (correct) DSCP (DiffServ): CS0 000000 (0) - class 0, default traffic (RFC 2474).And Empty fields in WAN configuration gives me:
« SpeedGuide.net TCP Analyzer Results » Tested on: 2025.04.25 05:55 IP address: 84.52.xx.xxx Client OS/browser: Windows 10 (Firefox 137.0) TCP options string: 020405ac0103030801010402 MSS: 1452 MTU: 1492 TCP Window: 65280 (not multiple of MSS) RWIN Scaling: 8 bits (2^8=256) Unscaled RWIN : 255 Recommended RWINs: 63888, 127776, 255552, 511104, 1022208 BDP limit (200ms): 2611 kbps (261 Kilobytes/s) BDP limit (500ms): 1044 kbps (104 Kilobytes/s) MTU Discovery: OFF TTL: 54 Timestamps: OFF SACKs: ON IP ToS: 00000010 (2) Precedence: 000 (routine) Delay: 0 (normal delay) Throughput: 0 (normal throughput) Reliability: 0 (normal reliability) Cost: 1 (low cost) Check bit: 0 (correct) DSCP (DiffServ): CS0 000000 (0) - class 0, default traffic (RFC 2474).I'm pretty sure my ISP doesn't support any RFCs that would allow pushing over 1492, because the connection just fails when I set the MTU to 1500.
-
@stephenw10 said in New PPPoE backend, some feedback:
pppcfg pppoe1
Ok, that makes sense but slightly odd that GUI seems to hang for a while.
New pppcfg commands noted too:
[25.03-BETA][admin@Router-7.xxxx.me]/root: pppcfg pppoe0 dev: igc0 state: session sid: 0x1155 PADI retries: 0 PADR retries: 0 time: 14:30:28 sppp: phase network authproto auto authname "FTTP.xxxx@idnet" peerproto auto dns: 212.69.40.23 212.69.36.23 [25.03-BETA][admin@Router-7.xxxx.me]/root: ifconfig pppoe0 pppoe0: flags=1008851<UP,POINTOPOINT,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500 description: WAN options=0 inet 93.xx.xxx.xx --> 212.xx.xx.xx netmask 0xffffffff inet6 fe80::3eec:efff:xxxx:xxxx%pppoe0 prefixlen 64 scopeid 0xf inet6 2axx:xxxx:feed:xxxx:3eec:efff:xxxx:xxxx prefixlen 64 autoconf pltime 604800 vltime 2592000 groups: pppoec nd6 options=123<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL,NO_DAD> [25.03-BETA][admin@Router-7.xxxx.me]/root:️ -
Just to be clear if you actually have to click connect twice on the Status page to get connectivity that's a bug. But I don't think that's the case, it just shows UP with no IP when it initially reloads but is still connecting in the background and will connect without further interaction.
