speed loss through pfSense (PPPoE)

ck42

First, not sure how long this has been happening, so don't know if this is due to any changes I've made over the past year.
Basic issue is that I'm seeing a fairly large discrepancy between the speeds I'm getting directly from ISP w/o pfsense involved vs what workstations get when going through pfSense. Also, I've used multiple laptops and see the problem on each. It's not just an issue with one test device.

Been scouring the forums looking for things to test/try/change. Reached a point that I could use some pointers.

Here's what I've done so far:

Hooked up the ISP's modem by itself and connected laptop to it. Ran speedtest.
- Getting approx 800Mbps/up-down
Ran speedtest-cli on pfSense - consistently getting ~800Mbps DOWN and ~200Mbps UP
Connected LAN interface (em1) on pfSense directly to the laptop (switch removed) and ran speedtest.
- Getting approx 220Mbps (UP/DOWN) consistently

pfSense (2.4.4-p3)
Intel(R) Celeron(R) CPU G1610 @ 2.60GHz
2 CPUs: 1 package(s) x 2 core(s)
AES-NI CPU Crypto: No
Both interfaces are 'em'

Running these packages:
bandwidthd
freeradius3
LADVD
ntopng
nut
pfBlockerNG
RRD_Summary
Status_Traffic_Totals

Internet Service:
Fiber 1Gbps (symmetrical) PPPoE
Not using provider modem (running straight to pfSense WAN)

pfSense
Using Traffic Shaping (Wizard)
Installed speedtest-cli on pfSense

Some screenshots while the speedtest was running:

ck42

On a whim, I removed the traffic shaper and re-ran the speed test (dslreports/speedtest)
BAM!
Fixed...well sort of. (Getting ~550Mbps DOWN and ~900Mbps UP from laptop->switch->pfsense->WAN)

speedtest-cli directly on pfSense is still seeing about 750-800Mbps DOWN and only 200Mbps UP (UPLOAD speed is weird being so low - still don't understand this)
Workstation->switch-pfSense-WAN is giving me almost 400Mbps now UPLOAD. How can this be higher than the speedtest run directly on pfSense speedtest-cli?

Re-added traffic shaper and went back to lower speeds.
Deleted traffic shaper and back up again

Which then creates the question - Why did the traffic shaper cause this?

Put the shaper back in and then watched the queues as I ran a speedtest.

Below, the "DROP_B" column value for qOthersHigh for 'root_em1 (LAN interface) incremented while the speedtest was downloading.
The qOthersHigh value for the 'root_pppoe0' incremented during while uploading.

Not sure what this means though....but thinking that the speedtest traffic is somehow getting limited by the queue. Not sure why this particular traffic is getting put into this queue though and not the 'qInternet' queues. Is the filter recognizing something about the speedtest traffic that causes it to place it into a non-qInternet queue?

stephenw10

What CPU load do you see now and how is it spread across the cores?
PPPoE is limited to a single thread in pfSense.

Try removing the shaper entirely and retesting. You can always roll back that change.

Steve

ck42

CPU is Celeron(R) CPU G1610 @ 2.60GHz
2 CPUs: 1 package(s) x 2 core(s)

Top normally shows both cores in upper 90's% when NOT running any speed tests.

Top is shown WHILE the speedtest was running the download.

In this one, the Shaper is enabled except I removed any queues where I saw incrementing Drops when running speed tests.

This is with the Shaper completely removed

stephenw10

What does the output of vmstat -i look like?

I would expect both cores to be mostly idle when you're not testing. That's what the idle process there signifies.
Neither core is anywhere close to 100% even with no shaper so that doesn't look like what's limiting it.

What throughput do you see without any shaper?

Steve

ck42

Here's the vmstat:

Both cores are nearly 100% idle when not speed testing.
...and worse case I've seen is that a core goes down to maybe around 50% when speed testing.
But it's 1000% repeatable that if I remove the Shaper that speeds go up significantly.

stephenw10

Hmm, em0 only has one queue/interrupt. It may not make any difference if that's the PPPoE interface. Odd though.

Are those NICs indentical? On-board? Can you swap them out or add others?

Steve

ck42

@stephenw10
NICs are both onboard.
I don't think I have any NIC cards right now even if I wanted to do this.

stephenw10

Are they actually the same? Check pciconf -lv for the chip.

Maybe try swapping the assignments, see if that changes anything.

Steve

ck42

Not EXACTLY the same:

em0@pci0:0:25:0: class=0x020000 card=0x20368086 chip=0x15028086 rev=0x04 hdr=0x00
vendor = 'Intel Corporation'
device = '82579LM Gigabit Network Connection (Lewisville)'
class = network
subclass = ethernet

em1@pci0:2:0:0: class=0x020000 card=0x20368086 chip=0x10d38086 rev=0x00 hdr=0x00
vendor = 'Intel Corporation'
device = '82574L Gigabit Network Connection'
class = network
subclass = ethernet

stephenw10

Hmm, not seeing any easy details to compare there. The 825747 though is very common, I wouldn't expect to see any issues with.
I would try swapping them to see if it changes anything.

Steve