25.03.b.20250306.0140 - if_pppoe kernel module decreases performance on SG 6100

mfld

No custom tunables, limiters, buffer-bloat fixes.

PowerD on hi-adaptive.

WAN is a 10G fiber PON with a slightly over-provisioned 2000/500 account.

25.03.b.20250306.0140 with if_pppoe kernel module disabled:

Download:  1551.54 Mbps (data used: 1.0 GB)
                 19.94 ms   (jitter: 3.56ms, low: 4.80ms, high: 70.39ms)

      Upload:   680.27 Mbps (data used: 728.5 MB)
                  3.89 ms   (jitter: 6.28ms, low: 1.56ms, high: 233.49ms)
 Packet Loss:     0.0%

System / CPU load are high as would be expected. Load average in the 4s and CPU util close to 90%

25.03.b.20250306.0140 with if_pppoe kernel module enabled:

Download:   724.45 Mbps (data used: 487.4 MB)
                 31.30 ms   (jitter: 1.17ms, low: 5.50ms, high: 34.69ms)

      Upload:   695.23 Mbps (data used: 564.6 MB)
                  4.24 ms   (jitter: 11.69ms, low: 1.53ms, high: 237.66ms)
 Packet Loss:     0.0%

System / CPU load are nice and low. Load average around 1.8 and CPU util under 70% but we take a huge hit.

mfld

Side-note:

Toggling this option kills WAN until a reboot is done. It should prompt the user and schedule a reboot similar to how "Disable hardware checksum offload" is done or print stern warnings about remote lockout risk when being applied. Y'all know how I found out Was saved by a */10 min /sbin/reboot cron I placed before playing with this.

kprovost

@mfld That's not what we'd expect to see, of course.

Is there anything obviously different? E.g. a different MTU or something like that?
Does this use compression or multilink?

rpotter28

@mfld I find this interesting as I admin a couple of 6100's. BUT I am testing at home with a whitebox (Dell R230) on a 940/940 pppoe WAN and I can fully saturate it. Same as with previous beta.

rpotter28

@mfld said in 25.03.b.20250306.0140 - if_pppoe kernel module decreases performance on SG 6100:

Was saved by a */10 min /sbin/reboot cron I placed before playing with this.

Good job, and burned into my memory for remote :-)

mfld

@kprovost

Only thing custom is that MTU is set 1492.

Only change made between speed testing was to toggle if_pppoe kernel module and reboot.

mfld

@rpotter28 said in 25.03.b.20250306.0140 - if_pppoe kernel module decreases performance on SG 6100:

@mfld I find this interesting as I admin a couple of 6100's. BUT I am testing at home with a whitebox (Dell R230) on a 940/940 pppoe WAN and I can fully saturate it. Same as with previous beta.

I did not test on white label hardware as they are specced for achieving line speed on these 2Gbit pppoe links. The feature was of interest only on this 6100 as that's the only site that has hardware that cannot currently achieve line speed. The other sites that I have access to that use pppoe for fast fiber connections are i3, i7, i9 CPU that don't struggle. I could only compare system load there.

kprovost

@mfld said in 25.03.b.20250306.0140 - if_pppoe kernel module decreases performance on SG 6100:

Only thing custom is that MTU is set 1492.

I was insufficiently clear: I meant check what MTU ends up being negotiated. Confirm that it is the same for both mpd5 and if_pppoe.

The same applies to the question about compression and multilink.

Clearly there must be a difference between them to explain the lower throughput, and we would like to figure out why.

rpotter28

@mfld said in 25.03.b.20250306.0140 - if_pppoe kernel module decreases performance on SG 6100:

The feature was of interest only on this 6100 as that's the only site that has hardware that cannot currently achieve line speed.

Understood. And yeah, I'm doing the 940/940 on the Dell with a Intel(R) Xeon(R) CPU E3-1280 v5.

mfld

@kprovost

Vanilla pppoe:

COMPPROTO VJCOMP, 16 comp. channels, no comp-cid

with if_pppoe I am not yet able to see where it is logging to. It is not in /status_logs.php?logfile=ppp

Edit: I see

2025-03-28 01:31:19.912146+00:00 	kernel 	- 	pppoe: received PADO but could not find request for it
2025-03-28 01:31:19.912127+00:00 	kernel 	- 	pppoe0: host unique tag found, but it belongs to a connection in state 3
2025-03-28 01:31:19.912107+00:00 	kernel 	- 	pppoe: received PADO but could not find request for it
2025-03-28 01:31:19.912087+00:00 	kernel 	- 	pppoe0: host unique tag found, but it belongs to a connection in state 3
2025-03-28 01:31:19.912066+00:00 	kernel 	- 	pppoe: received PADO but could not find request for it
2025-03-28 01:31:19.911973+00:00 	kernel 	- 	pppoe0: host unique tag found, but it belongs to a connection in state 3
2025-03-28 01:31:13.007984+00:00 	kernel 	- 	pppoe0: link state changed to UP

stephenw10

OK I think I've replicated this locally. We are digging into it....

stephenw10

Just to confirm, you are using ix0/ix1 for this connection?

mfld

@stephenw10
Yes pppoe0 is bound to ix0