New PPPoE backend, some feedback
-
@stephenw10 Yup yup
dev.ixl.0.pf.txq07.packets: 8401098
dev.ixl.0.pf.txq06.packets: 7316577
dev.ixl.0.pf.txq05.packets: 3704542
dev.ixl.0.pf.txq04.packets: 4271522
dev.ixl.0.pf.txq03.packets: 3784653
dev.ixl.0.pf.txq02.packets: 7219027
dev.ixl.0.pf.txq01.packets: 4471821
dev.ixl.0.pf.txq00.packets: 2694355
dev.ixl.0.pf.rxq07.packets: 0
dev.ixl.0.pf.rxq06.packets: 0
dev.ixl.0.pf.rxq05.packets: 0
dev.ixl.0.pf.rxq04.packets: 0
dev.ixl.0.pf.rxq03.packets: 0
dev.ixl.0.pf.rxq02.packets: 0
dev.ixl.0.pf.rxq01.packets: 0
dev.ixl.0.pf.rxq00.packets: 47313613 -
@w0w Thanks for this I'll do some comparisons and see if there is something different.
What I might do this weekend, is push the pppoe back up to the provider modem and just run it as dhcp from there. I know it will be double NAT but if it's something between the PPPOE and the queuing that should then clear up. If I get better results and the queues are better then it narrows it down, and proves the rest of my network. I had already ran iperf between other devices and the firewall when I upgraded servers and switches to 10gig to try and tune everything, so it shouldn't be but I don't want to discount it without some proof either.
-
@w0w Taking into account that I have 4 more queues than you did here's what I see are differences.
dev.ixl.0.eee.enable: 0
dev.ixl.0.unallocated_queues: 760 <- makes sense I have more queues enabled.
dev.ixl.0.fw_version: fw 6.81.49447 api 1.7 nvm 6.80 etid 80003d72 oem 18.4608.9
dev.ixl.0.supported_speeds: 6 <- probably due to the firmware difference.
dev.ixl.0.advertise_speed: 6 <- probably due to the firmware difference.
dev.ixl.0.iflib.override_nrxds: 4096 <-these are in my overrides so expected.
dev.ixl.0.iflib.override_ntxds: 4096 <-these are in my overrides so expected.
dev.ixl.0.iflib.allocated_msix_vectors: 9
dev.ixl.0.iflib.override_qs_enable: 1
dev.ixl.0.iflib.override_nrxqs: 8 <-related to my extra queue tests.
dev.ixl.0.iflib.override_ntxqs: 8 <-related to my extra queue tests.so
a) I could try updating the firmware, never a bad thing
b) unsure about the
dev.ixl.0.eee.enable: 0
dev.ixl.0.iflib.allocated_msix_vectors: 9
dev.ixl.0.iflib.override_qs_enable: 1
and
c)Your queue outputs show that on the RX side only the one queue was active on yours too. -
@stephenw10 said in New PPPoE backend, some feedback:
What CPU usage are you seeing when you test? What about per core usage? I one core still pegged at 100%
Netgate 6100
Internet: 1G(1200 Mb/s) Down / 1G Up
3x Stream 4K 60FPS on 3 device
2x Download using Steam on 2 device
1x OpenVPN session open on the router but not very busy
2x Speed test (spot where it reach 1 gig) on 1 device
This is what it's look like in Grafana Cloud , it's not the most accurate I've seen but it's really close (I saw some spike at 80% for cpu 0 system on top "top -P" during the speed test ). What we can observe is that cpu 0 still very busy compare to the other cpu.
-
@dsl-ottawa said in New PPPoE backend, some feedback:
a) I could try updating the firmware, never a bad thing
b) unsure about the
dev.ixl.0.eee.enable: 0
dev.ixl.0.iflib.allocated_msix_vectors: 9
dev.ixl.0.iflib.override_qs_enable: 1
and
c)Your queue outputs show that on the RX side only the one queue was active on yours too.I don't think those parameters are related to the problem — they seem like some OEM firmware/CPU settings, maybe.
And about the queue... That’s a good find! I didn’t pay attention to it before. Hmm, interesting...
I have a gigabit PPPoE connection, and the new backend gives me slightly better speeds compared to MPD5 with the deferred option. It almost saturates my ISP’s Ethernet link — over 900 Mbps — which I think is a great result for 20-year-old Cat5 cabling. -
@w0w I would agree nothing there in the settings..
BUT I found some more info.. and I should have kept to my gut feeling and done the further tests before pinpointing the PPPOE.
I disabled my pppoe and ran straight IP, to my provider modem, so double nat'd.
- All the queues were used. as expected.
- my desktop speed tests DIDNT CHANGE, still slow downloads
- I found a copy of the cli speedtest package for pfsense and installed it.
- download test from the pfsense box came back at full 3gig
- upload was slower than normal but I'm less worried about that since my desktop tests were fine - put PPPOE back, and ran the same tests.
- download 3gig from pfsense using speedtest-cli
- and normal 1.7-1.9 on my desktop
This to me proves that it's not the PPPOE or even the wan side of pfsense but something on my bleepin LAN, or at a minimum the LAN side of pfsense.
On the up side it does seem to prove that the single queue on the download isn't an issue :)
-
Ah that's interesting!
What did the per-core CPU usage look like when downloading at 3G over PPPoE? Though the speedtest itself will be using a lot.
-
@stephenw10 it's a pretty quick test so I'm not sure it's a good capture
CPU 0: 11.8% user, 0.0% nice, 28.6% system, 0.0% interrupt, 59.6% idle
CPU 1: 3.9% user, 0.0% nice, 6.3% system, 10.2% interrupt, 79.6% idle
CPU 2: 12.2% user, 0.0% nice, 7.8% system, 7.1% interrupt, 72.9% idle
CPU 3: 5.9% user, 0.0% nice, 6.3% system, 11.0% interrupt, 76.9% idle
CPU 4: 10.6% user, 0.0% nice, 7.8% system, 8.2% interrupt, 73.3% idle
CPU 5: 4.7% user, 0.0% nice, 8.6% system, 12.9% interrupt, 73.7% idle
CPU 6: 5.1% user, 0.0% nice, 6.7% system, 17.3% interrupt, 71.0% idle
CPU 7: 4.3% user, 0.0% nice, 7.1% system, 11.0% interrupt, 77.6% idlebut it certainly looks clean
-
@dsl-ottawa said in New PPPoE backend, some feedback:
put PPPOE back, and ran the same tests.
- download 3gig from pfsense using speedtest-cli
- and normal 1.7-1.9 on my desktop
Is there any chance that some old limiters, rules, traffic shaping, or other configurations are still active?
If this is a clean install, I would suspect a bug in pfSense -
I make a script to monitor the queues of the parent interface IX2 and all data goes through the queue 0 on the RX side.
BTW my ISP use a VLAN (IX2 --> VLAN40 --> PPPoE) maybe it doesn't help to manage the queues correctly...
In 24 Hours
-
@mr_nets
There may be only one queue on the network card, but packets can be distributed across CPU cores. Previously, due to netgraph, everything was handled by a single core. The deferred option slightly improved this, but it was a limited improvement—overall, performance was bottlenecked by a single CPU core. At least that limitation no longer exists now.In theory, if the network queue never fills up, there shouldn't be any issues. I don't know what the theoretical maximum throughput of a PPPoE session is—maybe Netgate specialists have already tested this? I would expect something like 2.5–3.5 Gbps with multithreaded download.
-
It still depends on the hardware but we were able to pass >8Gbps through a 6100 in raw testing: https://www.netgate.com/blog/optimizing-pppoe-performance-in-pfsense-software
-
@stephenw10 said in New PPPoE backend, some feedback:
It still depends on the hardware but we were able to pass >8Gbps through a 6100 in raw testing: https://www.netgate.com/blog/optimizing-pppoe-performance-in-pfsense-software
Thanks, great read. Too bad I missed it, but I'm glad you brought it to me today.
-
I am testing pfSense 2.8 on my pppoe WAN setup that has been working for many years.
I noticed that the weekly periodic reset I set on the pppoe WAN is not automatically performed on the weekend.
If I try to change the setting manually the pppoe connection resets and reconnects but then it will no longer do so automatically -
You're setting that in the PPP settings on the interface?
Interesting. That's an advanced setting that only applies to mpd5. It's hidden on the ppp interfaces page but looks like it's not hidden from the assigned interface and probably should be.
-
I use this setting on a pppoe connection with variable ip.
I need it to avoid the connection being reset by the ISP since I do not have a fixed IP.
On pfSense from 2.5 to 2.7.2 it worked on all versions.
-
I assume you have enabled if_pppoe?
-
Yes, I have enabled system/advanced /networking marked Use if_pppoe kernel module for PPPoE client.It seems like a bug in the new version 2.8
-
Yes, there is a bug there. However currently the bug is that the mpd5 specific options are not hidden correctly. And that the cron-job is still set even thought the script it calls is not present when if_pppoe is enabled.
If you need this for now you should add your own cronjob. This should work:
/etc/rc.interfaces_wan_configure opt2
Where opt2 is the pppoe interface internal name.This should probably also be a feature request. Let me see here...
-
The WAN interface is pppoe1 which is on igb1.
There is already a cron job with the script /var/etc/pppoe_restart_pppoe1.
If you can tell me if this is ok or if this script needs to be modified?
