gigabit WAN and capable pfSense hardware



  • I'm getting CenturyLink gigabit Internet next week. Google searching and forum browsing doesn't give a clear indication whether my pfSense hardware will be powerful enough to get me the full real world 930+ Mbps up & down speed. Based on the hardware listed below will I achieve full bandwidth or do I need to swap some things out??

    • Supermicro X9SCM mobo (LGA 1155, H2, XEON E3-1200 & E3-1200 v2, 2nd & 3rd Gen. i3, Pentium & Celeron CPUs)
    • XEON E3-1220 v2 CPU (3.1 GHz 4 cores, 4 threads)
    • Dual onboard gigabit NICs (Intel 82579LM and 82574L)
    • 4 GB ECC DDR3 unbuffered memory
    • 80 GB 2.5" SATA-I/1.5 Gbps HDD

  • LAYER 8 Global Moderator

    How is that pfsense hardware? Looks like some hardware your running pfsense on to me ;)

    Maybe you should contact who you bought it from on what sort of performance you can expect from it..

    Just saying. ;)

    Just saw a thread where actual hardware a sg-2440 was doing gig.. So Yeah have to assume that power sucking beast could do it as well.



  • Yes, I meant hardware I'm running pfSense on.

    I keep seeing posts about Intel NICs under performing (>500 Mbps). Quad core CPUs possibly contributing to the bad performance.


  • Netgate Administrator

    It will.

    Assuming that's not a PPPoE connection. If it is then it still might.

    Steve



  • @stephenw10 said in gigabit WAN and capable pfSense hardware:

    It will.

    Assuming that's not a PPPoE connection. If it is then it still might.

    Steve

    Yes, it's PPPoE. Can you elaborate why it might not?


  • Netgate Administrator

    Because PPPoE connections in FreeBSD (and hence pfSense) can only use a single thread/core.

    Single thread performance on that CPU is pretty good though.

    See: https://forum.netgate.com/topic/96129/gigabit-pppoe-and-intel-drivers

    Steve



  • ya, that's one of the threads that confused me and lead me to post my own thread. My NICs are showing up as em0 and em1 so I guess that means they're not using igb drivers which is a start. Guess I'll have to just test and see when CenturyLink installs it. I'll post my results here


  • Netgate Administrator

    I think there is some confusion here. The igb drivers are where this is most often seen because they support multiple queues per NIC so can use CPU cores far better. That means the PPPoE throughput can be far lower than the direct throughput.
    Other drivers such as em or re do not so the effect is less pronounced but the underlying cause is still there.

    At least that's how I understand it. 😉

    Steve


  • Netgate Administrator

    I setup a quick test here using another pfSense box as a PPPoE server. So all Gigabit but also all local.
    It's not a great test because I'm running iperf3 on the box under test. But since it uses only one core for PPPoE it should have plenty left for that.

    pppoe0: flags=88d1<UP,POINTOPOINT,RUNNING,NOARP,SIMPLEX,MULTICAST> metric 0 mtu 1492
    	inet 10.220.30.0 --> 10.220.30.129 netmask 0xffffffff 
    	inet6 fe80::290:7fff:fe9d:b635%pppoe0 prefixlen 64 scopeid 0x14 
    	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
    

    Without any tuning:

    [2.4.5-DEVELOPMENT][admin@xtm800.stevew.lan]/root: iperf3 -c 10.220.30.129
    Connecting to host 10.220.30.129, port 5201
    [  5] local 10.220.30.0 port 18962 connected to 10.220.30.129 port 5201
    [ ID] Interval           Transfer     Bitrate         Retr  Cwnd
    [  5]   0.00-1.00   sec   109 MBytes   918 Mbits/sec    0    288 KBytes       
    [  5]   1.00-2.00   sec   106 MBytes   891 Mbits/sec    0    288 KBytes       
    [  5]   2.00-3.00   sec   109 MBytes   910 Mbits/sec    0    288 KBytes       
    [  5]   3.00-4.00   sec   111 MBytes   930 Mbits/sec    0    288 KBytes       
    [  5]   4.00-5.00   sec   108 MBytes   903 Mbits/sec    0    288 KBytes       
    [  5]   5.00-6.00   sec   108 MBytes   904 Mbits/sec    0    288 KBytes       
    [  5]   6.00-7.00   sec   108 MBytes   904 Mbits/sec    0    288 KBytes       
    [  5]   7.00-8.00   sec   111 MBytes   930 Mbits/sec    0    288 KBytes       
    [  5]   8.00-9.00   sec   105 MBytes   885 Mbits/sec    0    288 KBytes       
    [  5]   9.00-10.00  sec   110 MBytes   919 Mbits/sec    0    288 KBytes       
    - - - - - - - - - - - - - - - - - - - - - - - - -
    [ ID] Interval           Transfer     Bitrate         Retr
    [  5]   0.00-10.00  sec  1.06 GBytes   909 Mbits/sec    0             sender
    [  5]   0.00-10.00  sec  1.06 GBytes   909 Mbits/sec                  receiver
    
    iperf Done.
    [2.4.5-DEVELOPMENT][admin@xtm800.stevew.lan]/root: iperf3 -c 10.220.30.129 -R
    Connecting to host 10.220.30.129, port 5201
    Reverse mode, remote host 10.220.30.129 is sending
    [  5] local 10.220.30.0 port 31734 connected to 10.220.30.129 port 5201
    [ ID] Interval           Transfer     Bitrate
    [  5]   0.00-1.00   sec  87.9 MBytes   737 Mbits/sec                  
    [  5]   1.00-2.00   sec  87.3 MBytes   732 Mbits/sec                  
    [  5]   2.00-3.00   sec  89.7 MBytes   753 Mbits/sec                  
    [  5]   3.00-4.00   sec  87.2 MBytes   732 Mbits/sec                  
    [  5]   4.00-5.00   sec  88.1 MBytes   739 Mbits/sec                  
    [  5]   5.00-6.00   sec  86.8 MBytes   728 Mbits/sec                  
    [  5]   6.00-7.00   sec  87.7 MBytes   736 Mbits/sec                  
    [  5]   7.00-8.00   sec  87.8 MBytes   736 Mbits/sec                  
    [  5]   8.00-9.00   sec  86.6 MBytes   726 Mbits/sec                  
    [  5]   9.00-10.00  sec  88.9 MBytes   746 Mbits/sec                  
    - - - - - - - - - - - - - - - - - - - - - - - - -
    [ ID] Interval           Transfer     Bitrate         Retr
    [  5]   0.00-10.01  sec   878 MBytes   736 Mbits/sec  3774             sender
    [  5]   0.00-10.00  sec   878 MBytes   737 Mbits/sec                  receiver
    
    iperf Done.
    

    Setting net.isr.dispatch to deferred:

    [2.4.5-DEVELOPMENT][admin@xtm800.stevew.lan]/root: sysctl net.isr.dispatch=deferred
    net.isr.dispatch: direct -> deferred
    [2.4.5-DEVELOPMENT][admin@xtm800.stevew.lan]/root: iperf3 -c 10.220.30.129 
    Connecting to host 10.220.30.129, port 5201
    [  5] local 10.220.30.0 port 59188 connected to 10.220.30.129 port 5201
    [ ID] Interval           Transfer     Bitrate         Retr  Cwnd
    [  5]   0.00-1.00   sec   111 MBytes   934 Mbits/sec    0    160 KBytes       
    [  5]   1.00-2.00   sec   111 MBytes   935 Mbits/sec    0    160 KBytes       
    [  5]   2.00-3.00   sec   111 MBytes   935 Mbits/sec    0    160 KBytes       
    [  5]   3.00-4.00   sec   111 MBytes   935 Mbits/sec    0    160 KBytes       
    [  5]   4.00-5.00   sec   111 MBytes   935 Mbits/sec    0    160 KBytes       
    [  5]   5.00-6.00   sec   111 MBytes   935 Mbits/sec    0    160 KBytes       
    [  5]   6.00-7.00   sec   111 MBytes   935 Mbits/sec    0    160 KBytes       
    [  5]   7.00-8.00   sec   111 MBytes   935 Mbits/sec    0    160 KBytes       
    [  5]   8.00-9.00   sec   111 MBytes   935 Mbits/sec    0    160 KBytes       
    [  5]   9.00-10.00  sec   111 MBytes   935 Mbits/sec    0    160 KBytes       
    - - - - - - - - - - - - - - - - - - - - - - - - -
    [ ID] Interval           Transfer     Bitrate         Retr
    [  5]   0.00-10.00  sec  1.09 GBytes   935 Mbits/sec    0             sender
    [  5]   0.00-10.04  sec  1.09 GBytes   932 Mbits/sec                  receiver
    
    iperf Done.
    [2.4.5-DEVELOPMENT][admin@xtm800.stevew.lan]/root: iperf3 -c 10.220.30.129 -R
    Connecting to host 10.220.30.129, port 5201
    Reverse mode, remote host 10.220.30.129 is sending
    [  5] local 10.220.30.0 port 5846 connected to 10.220.30.129 port 5201
    [ ID] Interval           Transfer     Bitrate
    [  5]   0.00-1.00   sec   112 MBytes   937 Mbits/sec                  
    [  5]   1.00-2.00   sec   112 MBytes   937 Mbits/sec                  
    [  5]   2.00-3.00   sec   112 MBytes   937 Mbits/sec                  
    [  5]   3.00-4.00   sec   112 MBytes   937 Mbits/sec                  
    [  5]   4.00-5.00   sec   112 MBytes   937 Mbits/sec                  
    [  5]   5.00-6.00   sec   112 MBytes   937 Mbits/sec                  
    [  5]   6.00-7.00   sec   112 MBytes   937 Mbits/sec                  
    [  5]   7.00-8.00   sec   112 MBytes   937 Mbits/sec                  
    [  5]   8.00-9.00   sec   112 MBytes   937 Mbits/sec                  
    [  5]   9.00-10.00  sec   112 MBytes   937 Mbits/sec                  
    - - - - - - - - - - - - - - - - - - - - - - - - -
    [ ID] Interval           Transfer     Bitrate         Retr
    [  5]   0.00-10.12  sec  1.09 GBytes   926 Mbits/sec    0             sender
    [  5]   0.00-10.00  sec  1.09 GBytes   937 Mbits/sec                  receiver
    
    iperf Done.
    

    That's using an E3-1225 v2 with em NICs.

    Interestingly the server end barely breaks a sweat when running that test.



  • What's the make and model of the sender and receiver NICs?


  • Netgate Administrator

    They are Intel NICs using the em driver as yours are.

    em9@pci0:14:0:0:	class=0x020000 card=0x00008086 chip=0x10d38086 rev=0x00 hdr=0x00
        vendor     = 'Intel Corporation'
        device     = '82574L Gigabit Network Connection'
        class      = network
        subclass   = ethernet
    

    Steve



  • Well, CenturyLink's gigabit service was installed and although I'm having massive latency issues when connecting CenturyLink's ONT to my pfSense build, I did get one good complete speed test in at 970 Mbps down (I forgot the up speed). This was without any tuning (I didn't change net.isr.dispatch to deferred). The CenturyLink tech's speed tests and my speed tests with their router/gateway device in place (Zyxel C3000Z) was/is roughly up to 930 Mbps up and down, so I'm not sure how much I can trust that 970 Mbps test. Anyways, I'll troubleshoot the pfSense box latency issues again later (I'll just use the Zyxel C3000Z for now).

    Thanks everyone for your help on this!


  • Netgate Administrator

    Yes, 970Mbps seems more than is possible!
    Still it's encouraging. ☺

    Steve


  • LAYER 8 Global Moderator

    970 is a bit overly optimistic ;) I think hehehehe

    They need to do some works on their maths maybe ;)


Log in to reply