speed loss through pfSense (PPPoE)



  • First, not sure how long this has been happening, so don't know if this is due to any changes I've made over the past year.
    Basic issue is that I'm seeing a fairly large discrepancy between the speeds I'm getting directly from ISP w/o pfsense involved vs what workstations get when going through pfSense. Also, I've used multiple laptops and see the problem on each. It's not just an issue with one test device.

    Been scouring the forums looking for things to test/try/change. Reached a point that I could use some pointers.

    Here's what I've done so far:

    1. Hooked up the ISP's modem by itself and connected laptop to it. Ran speedtest.
      • Getting approx 800Mbps/up-down
    2. Ran speedtest-cli on pfSense - consistently getting ~800Mbps DOWN and ~200Mbps UP
    3. Connected LAN interface (em1) on pfSense directly to the laptop (switch removed) and ran speedtest.
      • Getting approx 220Mbps (UP/DOWN) consistently

    pfSense (2.4.4-p3)
    Intel(R) Celeron(R) CPU G1610 @ 2.60GHz
    2 CPUs: 1 package(s) x 2 core(s)
    AES-NI CPU Crypto: No
    Both interfaces are 'em'

    Running these packages:
    bandwidthd
    freeradius3
    LADVD
    ntopng
    nut
    pfBlockerNG
    RRD_Summary
    Status_Traffic_Totals

    Internet Service:
    Fiber 1Gbps (symmetrical) PPPoE
    Not using provider modem (running straight to pfSense WAN)

    pfSense
    Using Traffic Shaping (Wizard)
    Installed speedtest-cli on pfSense

    Some screenshots while the speedtest was running:
    sysctl.png
    top.png



  • On a whim, I removed the traffic shaper and re-ran the speed test (dslreports/speedtest)
    BAM!
    Fixed...well sort of. (Getting ~550Mbps DOWN and ~900Mbps UP from laptop->switch->pfsense->WAN)

    • speedtest-cli directly on pfSense is still seeing about 750-800Mbps DOWN and only 200Mbps UP (UPLOAD speed is weird being so low - still don't understand this)
    • Workstation->switch-pfSense-WAN is giving me almost 400Mbps now UPLOAD. How can this be higher than the speedtest run directly on pfSense speedtest-cli?

    Re-added traffic shaper and went back to lower speeds.
    Deleted traffic shaper and back up again

    Which then creates the question - Why did the traffic shaper cause this?

    Put the shaper back in and then watched the queues as I ran a speedtest.

    Below, the "DROP_B" column value for qOthersHigh for 'root_em1 (LAN interface) incremented while the speedtest was downloading.
    The qOthersHigh value for the 'root_pppoe0' incremented during while uploading.

    Not sure what this means though....but thinking that the speedtest traffic is somehow getting limited by the queue. Not sure why this particular traffic is getting put into this queue though and not the 'qInternet' queues. Is the filter recognizing something about the speedtest traffic that causes it to place it into a non-qInternet queue?

    pfTOP.png


  • Netgate Administrator

    What CPU load do you see now and how is it spread across the cores?
    PPPoE is limited to a single thread in pfSense.

    Try removing the shaper entirely and retesting. You can always roll back that change.

    Steve



  • CPU is Celeron(R) CPU G1610 @ 2.60GHz
    2 CPUs: 1 package(s) x 2 core(s)

    Top normally shows both cores in upper 90's% when NOT running any speed tests.

    Top is shown WHILE the speedtest was running the download.

    In this one, the Shaper is enabled except I removed any queues where I saw incrementing Drops when running speed tests.

    top2.png

    This is with the Shaper completely removed

    top3.png


  • Netgate Administrator

    What does the output of vmstat -i look like?

    I would expect both cores to be mostly idle when you're not testing. That's what the idle process there signifies.
    Neither core is anywhere close to 100% even with no shaper so that doesn't look like what's limiting it.

    What throughput do you see without any shaper?

    Steve



  • Here's the vmstat:

    vmstat.png

    Both cores are nearly 100% idle when not speed testing.
    ...and worse case I've seen is that a core goes down to maybe around 50% when speed testing.
    But it's 1000% repeatable that if I remove the Shaper that speeds go up significantly.


  • Netgate Administrator

    Hmm, em0 only has one queue/interrupt. It may not make any difference if that's the PPPoE interface. Odd though.

    Are those NICs indentical? On-board? Can you swap them out or add others?

    Steve



  • @stephenw10
    NICs are both onboard.
    I don't think I have any NIC cards right now even if I wanted to do this.


  • Netgate Administrator

    Are they actually the same? Check pciconf -lv for the chip.

    Maybe try swapping the assignments, see if that changes anything.

    Steve



  • Not EXACTLY the same:

    em0@pci0:0:25:0: class=0x020000 card=0x20368086 chip=0x15028086 rev=0x04 hdr=0x00
    vendor = 'Intel Corporation'
    device = '82579LM Gigabit Network Connection (Lewisville)'
    class = network
    subclass = ethernet

    em1@pci0:2:0:0: class=0x020000 card=0x20368086 chip=0x10d38086 rev=0x00 hdr=0x00
    vendor = 'Intel Corporation'
    device = '82574L Gigabit Network Connection'
    class = network
    subclass = ethernet


  • Netgate Administrator

    Hmm, not seeing any easy details to compare there. The 825747 though is very common, I wouldn't expect to see any issues with.
    I would try swapping them to see if it changes anything.

    Steve


Log in to reply