Intel igb 125Mbps speeds?



  • OK, so I just built a new appliance and I'm trying to tune it. I ran a file transfer test using WinSCP to transfer a 1GB file to the tmp folder and then back again and I'm seeing 125Mbps. I don't know if this is a limitation of SSH, the processors or what so I thought I'd ask for some input. I'd like to see more like 500 to 800Mbps.
    Granted this is only a test on the LAN side, being all the interfaces are using the same driver the benchmarks will likely be the same.

    So, here is my setup…

    SuperMicro Atom 2758
    16GB ECC RAM
    WD 160GB 7200RPM Black
    Intel Pro 1000 Quad port NIC

    /boot/loader.conf
    autoboot_delay="3"
    vm.kmem_size="435544320"
    vm.kmem_size_max="535544320"
    if_igb_load="YES"
    kern.maxfilesperproc=32768
    kern.maxfiles=65536
    comconsole_speed="115200"
    hw.usb.no_pf="1"

    /boot/loader.conf.local
    kern.cam.boot_delay=10000
    kern.ipc.nmbclusters="1000000"
    net.inet.tcp.tso=0



  • Oh, and everything is running 1000baseT full duplex


  • LAYER 8 Netgate

    Where are you testing from and to?



  • Transfering to the firewall into the tmp folder from my desktop. I know my desktop can do 800Mbps, I do it all the time to my server


  • LAYER 8 Netgate

    All I can say is pfSense is not a file server. How does it do transferring data THROUGH it instead of TO it like it's designed to do.

    iperf is a better test to the device since it doesn't write to disk.



  • @Visseroth
    Could this be, perhaps?
    125 MB/s x 8Bit = 1 GBit/s

    In normal and pending on the used protocols you could only get something around 120 MBit/s
    from one GBit/s. If you want to know how many the LAN ports are able to deliver, you should use
    iPerf or NetIO through the pfSense box! 1 PC is the sender and the other one is a receiver, likes
    a client and server situation.



  • Tried iPerf but couldn't seem to get it to run very long nor pass traffic very fast. I'm getting better speed with internet traffic

    The LAN.jpg attachment is what I got with WinSCP, transferring a file. The idea was to see how fast the NICs would transfer files. This would let me know the NICs are capable. Then next would be to move traffic through more layers





    ![Server Abort.JPG](/public/imported_attachments/1/Server Abort.JPG)
    ![Server Abort.JPG_thumb](/public/imported_attachments/1/Server Abort.JPG_thumb)



  • The LAN.jpg attachment is what I got with WinSCP, transferring a file.

    WINSCP is not protocol independent likes NetIO or iPerf and the storage will also not be in the game!

    Tried iPerf but couldn't seem to get it to run very long nor pass traffic very fast. I'm getting better speed with internet traffic

    This would be not the trail I want to go to test the throughput of a device.

    e. The idea was to see how fast the NICs would transfer files.

    Then this might be then a bad idea as I see it right. The transport of the TCP/IP packets is interesting
    and the entire throughput pending on this.

    Then next would be to move traffic through more layers

    Good luck.



  • Your reply wasn't helpful in any way. Why bother replying?

    I'm obviously new to speed testing through the firewall.

    Does anyone have any suggestions?


  • Rebel Alliance

    Performance Testing "through" the Firewall:

    As you have 4 Interface maybe you have one spare for testing.
    Set that to an IP-Range on another subnet.
    Test if you can ping that ip from your Client.
    Install the iperf-Package.
    Start the iperf-Server on pfsense.

    iperf -c the-ip-you-set-the-pfsense-spare-interface-to

    Client <-> LAN [pfsense] OPT2

    My Results:
    Some Supermicro A1SAi-Board.
    Intel(R) Atom(TM) CPU C2550 @ 2.40GHz
    Quad Intel igb

    [root@burn ~]# iperf -c 172.XX.99.1 -t 60
    –----------------------------------------------------------
    Client connecting to 172.XX.99.1, TCP port 5001
    TCP window size: 85.0 KByte (default)

    [  3] local 172.XX.0.231 port 24708 connected with 172.XX.99.1 port 5001
    [ ID] Interval      Transfer    Bandwidth
    [  3]  0.0-60.0 sec  2.93 GBytes  420 Mbits/sec

    pfsense:
    LAN 172.XX.0.1
    OPT 172.XX.99.1

    burn:
    LAN 172.XX.0.231

    As for the WinSCP and 125MBit. I guess the Write-Cache on the WD-Black is off and you simply tested the maximum write speed to the Filesystem.



  • I setup a static on another interface, 172.16.0.1. I'm able to ping it but I can't get ipref to do anything. The client starts but there's no output

    I started the iperf server on PfSense and then started the client on my machine with iperf3 -p 5201 -c 172.16.0.1


  • Rebel Alliance

    Port 5201? Default is 5001 for me.

    Did you try to do a filter Rule to allow Traffic

    [your internal network] <-> TCP 172.16.0.1:5201



  • I don't know, I've tried 5001, 5201, added the rule in both the lan and the opt and still nothing. Even tried the local LAN GW address, still nothing.


  • Rebel Alliance

    Did you leave the iperf window (the one that doesn't seem to report anything) open after starting iperf in server-mode?



  • I indeed left it open then went to the command prompt and executed the client. Even checked the firewall to see if it was being blocked but didn't see anything.


  • Rebel Alliance

    Maybe TCP/UDP mixed up?

    For me the iperf-Server on pfsense works out-of-the-box.



  • or the version I downloaded.
    So I have a linux laptop and I tried it from the laptop and it worked fine but the weird thing is it was telling me that I was getting 5.75MB of transfer at only 4.67Mb/s.
    Seriously, I have a 20Mb connection, I've already fully saturated it with this thing and iperf is telling me I'm actually slower than that? Seems a bit off to me.

    I then setup my laptop as the server and pfsense as the client and here's the output…

    Client connecting to 10.1.1.111, TCP port 5001
    TCP window size: 65.0 KByte (default)
    ------------------------------------------------------------
    [  9] local 10.1.1.1 port 27528 connected with 10.1.1.111 port 5001
    [ ID] Interval       Transfer     Bandwidth
    [  9]  0.0- 2.0 sec  1.02 MBytes  4.26 Mbits/sec
    [  9]  2.0- 4.0 sec  1.01 MBytes  4.25 Mbits/sec
    [  9]  4.0- 6.0 sec  1.02 MBytes  4.28 Mbits/sec
    [  9]  6.0- 8.0 sec  1.02 MBytes  4.26 Mbits/sec
    [  9]  0.0-10.0 sec  5.09 MBytes  4.27 Mbits/sec
    

    UDP is saying….

    Client connecting to 10.1.1.111, UDP port 5001
    Sending 1470 byte datagrams
    UDP buffer size: 56.0 KByte (default)
    ------------------------------------------------------------
    [  9] local 10.1.1.1 port 50550 connected with 10.1.1.111 port 5001
    [ ID] Interval       Transfer     Bandwidth
    [  9]  0.0- 2.0 sec   247 KBytes  1.01 Mbits/sec
    [  9]  2.0- 4.0 sec   253 KBytes  1.03 Mbits/sec
    [  9]  4.0- 6.0 sec   253 KBytes  1.03 Mbits/sec
    [  9]  6.0- 8.0 sec   254 KBytes  1.04 Mbits/sec
    [  9]  8.0-10.0 sec   253 KBytes  1.03 Mbits/sec
    [  9]  0.0-10.0 sec  1.23 MBytes  1.03 Mbits/sec
    [  9] Sent 893 datagrams
    


  • OK, so I'm basically testing this thing before I implement it. I only have a 20Mbit connection and here is it fully saturated….








  • It's very important that you mention that you have snort. Snort has to inspect every packet and slows things down a lot.



  • This I understand and disabled snort during the test. I also have squid, this I didn't disable but didn't think it would hit on my speed to much.

    Seriously bud, sure I'm a bit ignorant but not completely stupid and you have been of no help at all, why even post at all?

    Anyhow…

    The other thing I thought of doing was disabling "Block private networks" and "Block bogon networks" and then plugging directly into my local network and then trying to connect to my storage server through the firewall so I could pull a file but unfortunately it seems I was unable to get to my server. I didn't see any blocks in the firewall logs so I'm not exactly sure why I was unable to reach the server. I figured it had something to do with routing.



  • This I understand and disabled snort during the test. I also have squid, this I didn't disable but didn't think it would hit on my speed to much.

    And perhaps Squid is acting as a caching proxy too?
    Then you might be sure cheating your self! Because if you then do the first time a test, you will get slow
    numbers from this test, but if you do it then again and again you will get more fine numbers.

    So in normal if you want to do a speed test you should do the following:

    • Do a fresh and full install
      – activate PowerD (hi adaptive)
      -- high up the mbuf size to 1.000.000
      -- enable TRIM support for the SSD or mSATA

    And then take two PCs or Laptops as a server and a client that are doing a test trough the pfSense machine
    and then on top you should activate Snort and do the test again. And then you should activate Squid and do
    the test again. So you get three independent and different numbers for the throughput of your pfSense machine.

    • One number is the raw throughput
    • One is the Snort throughput
    • One is the Snort & Squid throughput


  • Agreed. Unfortunately I need to get this one in place and have run out of time but I'll be building another one soon for CARP, that one will get further testing as this one will for now meet the needs of the two internet connections. One is 24Mbps the other 40 totally about 60Mbps.

    The reason I was trying to get more speed was for future reliability. So that if by chance the internet connections up here get faster and more stable this thing would be ready for the task and for general bench mark purposes.

    With this next one I'm going to do just as you said. Two machine, one on each side and then hammer traffic through it. First stock then with one package at a time.

    For those that were helpful, thank you. Some of you  ??? :-X


Log in to reply