Gigabit troubleshooting



  • Update: iperf3 so far reporting <200/sec.

    Not to beat a dead horse because it has been discussed several times before, but I'm going to bring it up again anyway.

    I am in/around Des Moines, IA to start. I just got Centurylink 1gig hooked up today and they sent a tech out to do an 'install.' The first thing the tech says to me when he walks in the door is "So, they have you set up for the Gig, huh? Well, hate to break it to you, but we don't have any hardware that supports that speed." I inquire about the Technicolor, etc, at least to get me up and running at a higher speed - nope, they don't have it. Ok, so why even offer 1gig if you don't support it??

    Anyway, long story short he says you are on your own, he only knows of 1 guy IN THE ENTIRETY OF IOWA that is hitting anywhere near gig speeds and there's nothing he can do and leaves. I'm sorry, I don't believe that at all.

    SOOOO, here I am, with my brand spanking new Ferrari of an Intertube and I need some help squeaking out as much speed as possible.

    First off, the basics-

    Hardware
    Jetway JC320U93W-2930-B Intel Celeron N2930 Dual Intel LAN Fanless NUC (Quad core) http://www.amazon.com/Jetway-JC320U93W-2930-B-Intel-Celeron-Fanless/dp/B00SHYW6US?ie=UTF8&psc=1&redirect=true&ref_=oh_aui_detailpage_o08_s00
    Crucial 8Gb 1333 ram
    250Gb SSD

    A little system information:
    hw.machine: amd64
    hw.model: Intel(R) Celeron(R) CPU  N2930  @ 1.83GHz
    hw.ncpu: 4
    dev.em.0.%desc: Intel(R) PRO/1000 Network Connection 7.4.2
    dev.em.0.%driver: em
    dev.em.1.%desc: Intel(R) PRO/1000 Network Connection 7.4.2
    dev.em.1.%driver: em

    This is connected directly to the ONT by cat6. I am authenticating using PPPoE (most likely the issue) tagged VLAN 201.

    My computers/AppleTv's, etc are connected to the router through a Netgear gigabit switch. At first I had them running through an Apple Airport Extreme, but that definitely is not stout enough for 1g.

    I set the mbuf tunable to 1,000,000 per the pfsense documentation.

    So far, the max I have been able to get is 587/572. Any other advice?
    Thanks!

    –--A little more info. Tested that gigabit LAN is working. Pathetic speeds from the router to the iperf server.

    Router to iMac (hardwired, testing gigabit LAN connection)

    Accepted connection from 192.168.1.1, port 64195
    [  5] local 192.168.1.107 port 5201 connected to 192.168.1.1 port 61209
    [ ID] Interval          Transfer    Bandwidth
    [  5]  0.00-1.00  sec  69.3 MBytes  582 Mbits/sec                 
    [  5]  1.00-2.00  sec  72.6 MBytes  609 Mbits/sec                 
    [  5]  2.00-3.00  sec  92.3 MBytes  774 Mbits/sec                 
    [  5]  3.00-4.00  sec  104 MBytes  872 Mbits/sec                 
    [  5]  4.00-5.00  sec  105 MBytes  880 Mbits/sec                 
    [  5]  5.00-6.00  sec  105 MBytes  880 Mbits/sec                 
    [  5]  6.00-7.00  sec  104 MBytes  876 Mbits/sec                 
    [  5]  7.00-8.00  sec  104 MBytes  876 Mbits/sec                 
    [  5]  8.00-9.00  sec  104 MBytes  874 Mbits/sec                 
    [  5]  9.00-10.00  sec  104 MBytes  875 Mbits/sec                 
    [  5]  10.00-10.00  sec  313 KBytes  843 Mbits/sec


    [ ID] Interval          Transfer    Bandwidth      Retr
    [  5]  0.00-10.00  sec  966 MBytes  810 Mbits/sec    0            sender
    [  5]  0.00-10.00  sec  966 MBytes  810 Mbits/sec                  receiver

    iperf.scottlinux.com to Router

    Reverse mode, remote host iperf.scottlinux.com is sending
    [  4] local 63.224.182.9 port 26135 connected to 173.230.156.66 port 5201
    [ ID] Interval          Transfer    Bandwidth
    [  4]  0.00-1.00  sec  2.39 MBytes  20.1 Mbits/sec                 
    [  4]  1.00-2.00  sec  6.56 MBytes  55.1 Mbits/sec                 
    [  4]  2.00-3.00  sec  11.2 MBytes  94.2 Mbits/sec                 
    [  4]  3.00-4.00  sec  15.6 MBytes  131 Mbits/sec                 
    [  4]  4.00-5.00  sec  20.1 MBytes  168 Mbits/sec                 
    [  4]  5.00-6.00  sec  25.0 MBytes  210 Mbits/sec                 
    [  4]  6.00-7.00  sec  29.2 MBytes  245 Mbits/sec                 
    [  4]  7.00-8.00  sec  27.6 MBytes  232 Mbits/sec                 
    [  4]  8.00-9.00  sec  18.3 MBytes  153 Mbits/sec                 
    [  4]  9.00-10.00  sec  19.7 MBytes  165 Mbits/sec                 
    [  4]  10.00-11.00  sec  20.3 MBytes  170 Mbits/sec                 
    [  4]  11.00-12.00  sec  20.9 MBytes  176 Mbits/sec                 
    [  4]  12.00-13.00  sec  21.3 MBytes  178 Mbits/sec                 
    [  4]  13.00-14.00  sec  21.3 MBytes  179 Mbits/sec                 
    [  4]  14.00-15.00  sec  21.7 MBytes  182 Mbits/sec                 
    [  4]  15.00-16.00  sec  21.6 MBytes  181 Mbits/sec                 
    [  4]  16.00-17.00  sec  21.3 MBytes  179 Mbits/sec                 
    [  4]  17.00-18.00  sec  21.5 MBytes  181 Mbits/sec                 
    [  4]  18.00-19.00  sec  18.2 MBytes  153 Mbits/sec                 
    [  4]  19.00-20.00  sec  17.3 MBytes  145 Mbits/sec                 
    [  4]  20.00-21.00  sec  18.8 MBytes  158 Mbits/sec                 
    [  4]  21.00-22.00  sec  19.6 MBytes  165 Mbits/sec                 
    [  4]  22.00-23.00  sec  21.0 MBytes  176 Mbits/sec                 
    [  4]  23.00-24.00  sec  21.2 MBytes  178 Mbits/sec                 
    [  4]  24.00-25.00  sec  21.4 MBytes  180 Mbits/sec                 
    [  4]  25.00-26.00  sec  22.2 MBytes  186 Mbits/sec                 
    [  4]  26.00-27.00  sec  21.7 MBytes  182 Mbits/sec                 
    [  4]  27.00-28.00  sec  21.8 MBytes  183 Mbits/sec                 
    [  4]  28.00-29.00  sec  22.1 MBytes  185 Mbits/sec                 
    [  4]  29.00-30.00  sec  16.6 MBytes  139 Mbits/sec


    [ ID] Interval          Transfer    Bandwidth      Retr
    [  4]  0.00-30.00  sec  591 MBytes  165 Mbits/sec  404            sender
    [  4]  0.00-30.00  sec  588 MBytes  164 Mbits/sec                  receiver



  • Wow, nothing? Ok. Can anyone give me any suggestions on how to test my router's performance (CPU load, throughput saturation, etc)? Maybe any tweaks? In the meantime I will continue searching the forums.

    For what its worth, I am running 2.2.6-RELEASE (amd64).

    Thanks!



  • Check out the hardware requirements page -> https://www.pfsense.org/hardware

    In order to attain speeds approaching Gigabit, you need server class hardware with PCIe NICs:

    501+ Mbit = "Multiple cores at > 2.0GHz are required. Server class hardware with PCI-e network adapters."

    Unfortunately, you're looking at buying new hardware 'cause that tiny Celeron box is not going to cut it.  Also, don't forget about your switch.  Verify you can attain 900+ Mbit locally on your switch or it's not going to matter what you do with PFsense.



  • @marvosa:

    Unfortunately, you're looking at buying new hardware 'cause that tiny Celeron box is not going to cut it.  Also, don't forget about your switch.  Verify you can attain 900+ Mbit locally on your switch or it's not going to matter what you do with PFsense.

    Thanks for the advice. I'll look at the hardware, but from what I have come across during my searches is that the n2930 chip should be able to handle it. It scores a 1665 on the Passmark where an i7-610 scored a 1900, so it's not a slouch of a processor. I have yet to see the processor go above 40-50%, and RAM usage has been minimal given I typically have 6Gb free at all times.

    I've checked the wiring - from the ONT to the router is all cat6, maybe 50 feet. All tests have been ran directly connected to the router. I verified I was getting >800 from my iMac to my router through the netgear switch, so that isn't the issue.

    Two things I'm curious about - how accurate is speedtest.net and Centurylink's speed test? For both of those I am getting close to 600. But when I test downloading a torrent, I've never gone about 10M/s.

    Who knows, it may come down to hardware. I just need to ponder whether or not it is worth spending $$ on a new system when this one is only a month or so old, or do I want to continue with $120/month gigabit line that I am not utilizing. If I can prove that I'm running stable in the 600 range, I'll be happy with that until I decide to upgrade hardware.



  • Since you're in the US, I would recommend the speed test at www.dslreports.com/speedtest. It's not flash-based like speedtest.net (close other browser tabs to make sure the results aren't tainted by poor browser performance), and uses simultaneous connections to multiple geographically diverse servers (similar to how a torrent would likely be received). Speedtest.net uses multiple connections, but I believe they're all to the same server at the location selected.



  • Thanks for the advice. I checked dslreports and it came back with approximately the same results. However, this time I took some screenshots of what the system was doing.

    top -P showed that yes, the processors are working, but I wouldn't call them incapable. CPU 3 was at 91%, the other 3 were hovering in the 30-40% range. Memory usage was negligible.

    PFSense showed about 60% total utilization.

    I'm 'OK' with 410 for now, but would still like to see more. I may look at bumping up the specs, but for now, are there any tweaks I should know about?

    Thanks!



    ![CPU Usage.png](/public/imported_attachments/1/CPU Usage.png)
    ![CPU Usage.png_thumb](/public/imported_attachments/1/CPU Usage.png_thumb)



  • you can start by disabling/removing squid.

    if that doesn't help, you can try the new 2.3-rc snapshots. they are fairly stable & generally boost performance by some margin.



  • ROFL. yeah Squid is hosing your CPU. Hard to keep up with 1Gb. You may find you no longer need to cache data locally when you no longer have a sub-8Mb connection.

    I laugh because that's a good problem to have.



  • Yeah, its a good problem to have! I may work on the caching settings, because I mainly use squid/sarg for reporting on my children's online activities/websites visited, etc. I don't feel it's necessary, but the wife instructed me to do so. So yeah, kinda stuck there lol. Unless anyone knows of another pkg that will report on actual page visits, not just the top level domain?

    For example, sarg gives me this detail - www.dslreports.com/speedtest/3507765 as opposed to just www.dslreports.com



  • There are many different things that could be wrong in this case to archive 1 GBit/s at the WAN Port.
    Not all CPUs or their cores could or should be comparable each against the others. And I am really sure
    that the Intel Core i7 is blasting the Intel N2930 away, whatever was shown on a CPU comparison list,
    because we are talking here about Layer3 routing and forwarding and not other things.

    So to be on the safe side you should using iPerf on two machines, one as the server and one as the client
    and one in front of the WAN Port and the other on the LAN behind the WAN area. Speed tests over the
    Internet are also measuring the Internet connection speed with all its bads and goods, and not purely
    your pfSense hardware except the WAN routing performance.

    pfSense is a software firewall that is able to route network traffic also not a plain router likes DD-WRT or
    OpenWRT or the most consumer home routers, they realize and work it out mostly done in silicon or by
    the help of an ASIC/FPGA that would not being the same as a x86_64 based software firewall.

    If you really want to know what the Internet account and your pfSense hardware will be able to realize
    it should be better to do a fresh install with a 64Bit version of pfSense 2.2.6 or 2.3RC, configuring the
    WAN and LAN part and then do the measuring. No packets, no other services, no extra features or options
    enabled, no VLANs, massively VPN, DPI or QoS tasks running beside of this set up.
    Fresh install and plain configuration.

    Otherwise this would be not really matching the real world facts as I see it right, others may see this
    different for sure, but together with Squid or SquidGuard or Snort or pfBlockerNG and other packets
    each of them will eat some CPU power and narrows down the entire speed and throughput of your
    pfSense box, for sure there are many CPUs that are really strong and powerful and they will route
    1 GBit/s beside of any other installed packet likes the Intel 4 Core i3, i5, i7 or Xeon E3 or E5 CPUs
    running @3,xGHz. But this is then not really electric power saving at all.

    This board here is running from 1,86GHz till 2,16GHz with the same CPU!
    Link

    So I would suggest at first;

    • enable PowerD (hi adaptive or adaptive)
      lets scale the cpu frequency from its minimum to its maximum likes needed
    • enable TRIM support if a SSD or mSATA is in usage
      also not a must be but I personally fell better with it
    • the mbuf size was set to 1000000 (not 1,000,000)
      if it will be needed and not as a standard procedure
    • perhaps high up the RAM size for Squid
      by default Squid is using only 256 MB of RAM

    And if all of this will be not gain the entire throughput or WAN speed you really should have a look for
    stronger hardware. Also Squid can be fine tuned, what to cache, how great the objects should be that
    must be cached and the mode Squid is running on.



  • Thats a very good reply, thank you. I have some time this weekend that I can take the network down and check using iperf on the wan and lan ports of the router. I'm not running any packages other than squid, no VPN, no QoS, etc. Just a PPPoE VLAN to connect to CenturyLink.

    This board here is running from 1,86GHz till 2,16GHz with the same CPU!
    Link

    I can't tell if you are saying thats a good or bad thing? Thats the same board I am running.

    I have powerd set to hi-adaptive currently, TRIM support is enabled on my SSD, I upped the squid memory to 512 and verified mbuf is 1000000.

    I'll respond more sometime this weekend after I conduct a little more testing. Thanks!