Netgate SG-4860 Performance



  • Hi all

    I already posted this topic in " General pfSense Questions", but I didn't get an answer.

    Please have look at the picture to get an overview of my setup.
    pfSense is the routing instance of both VLAN's 103 & 104.

    drawing_pfsense.png

    Problem description
    If I copy large files from my workstation to the synology nas, cpu usage is very high and throughput is only ~45MByte.
    Ping latency to the internet is increasing dramatically and cpu usage of the pfSense is very high.

    What I did

    1. Separate LAGG for VLAN 103
      I moved VLAN 103 to a separate LAGG (1) that incoming and outgoing traffic is separated physically.

    2. System Tunables
      I have disabled the EEE settings on all interfaces.
      See this post https://community.spiceworks.com/topic/1221309-periodic-packet-loss-constant-carp-switchovers-with-intel-nics-i350-igb for details.

    [2.4.5-RELEASE][root@pfsense]/root: sysctl dev.igb | grep eee
    dev.igb.5.eee_disabled: 1
    dev.igb.4.eee_disabled: 1
    dev.igb.3.eee_disabled: 1
    dev.igb.2.eee_disabled: 1
    dev.igb.1.eee_disabled: 1
    dev.igb.0.eee_disabled: 1
    

    Question
    Is this behavior because of hardware limitations, or do I have a misconfiguration of my setup?
    I thought that a Netgate SG-4860 Appliance can handle 1GByte throughput without any limitations.

    Thanks in advance for your support!

    Some pfSense top -aSH outputs:

      PID USERNAME     PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
       11 root         155 ki31     0K    64K RUN     1 936:47  59.19% [idle{idle: cpu1}]
       11 root         155 ki31     0K    64K CPU3    3 935:25  52.52% [idle{idle: cpu3}]
       11 root         155 ki31     0K    64K RUN     0 937:59  45.01% [idle{idle: cpu0}]
       12 root         -92    -     0K   704K WAIT    0   0:45  27.15% [intr{irq272: igb4:que 0}]
    
    
      PID USERNAME     PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
        0 root         -92    -     0K   864K CPU2    2   2:06 100.00% [kernel{igb3 que (qid 0)}]
       11 root         155 ki31     0K    64K RUN     3 935:26  45.91% [idle{idle: cpu3}]
       11 root         155 ki31     0K    64K RUN     1 936:48  44.13% [idle{idle: cpu1}]
       12 root         -92    -     0K   704K WAIT    0   0:45  42.56% [intr{irq272: igb4:que 0}]
       11 root         155 ki31     0K    64K RUN     0 937:59  35.26% [idle{idle: cpu0}]
    
    
      PID USERNAME     PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
       11 root         155 ki31     0K    64K CPU3    3 935:33  94.23% [idle{idle: cpu3}]
       11 root         155 ki31     0K    64K CPU1    1 936:55  88.77% [idle{idle: cpu1}]
       11 root         155 ki31     0K    64K RUN     0 938:04  70.96% [idle{idle: cpu0}]
       12 root         -92    -     0K   704K WAIT    0   0:49  25.32% [intr{irq272: igb4:que 0}]
       12 root         -92    -     0K   704K WAIT    0   0:15   0.34% [intr{irq267: igb3:que 0}]
    

    Ping from workstation to 8.8.8.8 while copying a large file:

    Reply from 8.8.8.8: bytes=32 time=3ms TTL=52
    Reply from 8.8.8.8: bytes=32 time=3ms TTL=52
    Reply from 8.8.8.8: bytes=32 time=3ms TTL=52
    Reply from 8.8.8.8: bytes=32 time=3ms TTL=52
    Reply from 8.8.8.8: bytes=32 time=3ms TTL=52
    Reply from 8.8.8.8: bytes=32 time=3ms TTL=52
    Reply from 8.8.8.8: bytes=32 time=3057ms TTL=52
    Reply from 8.8.8.8: bytes=32 time=5ms TTL=52
    Request timed out.
    Request timed out.
    Reply from 8.8.8.8: bytes=32 time=2495ms TTL=52
    Reply from 8.8.8.8: bytes=32 time=3ms TTL=52
    Reply from 8.8.8.8: bytes=32 time=1007ms TTL=52
    Reply from 8.8.8.8: bytes=32 time=3ms TTL=52
    Reply from 8.8.8.8: bytes=32 time=3007ms TTL=52
    Reply from 8.8.8.8: bytes=32 time=3ms TTL=52
    Reply from 8.8.8.8: bytes=32 time=9ms TTL=52
    Request timed out.
    Reply from 8.8.8.8: bytes=32 time=2630ms TTL=52
    Reply from 8.8.8.8: bytes=32 time=3ms TTL=52
    Reply from 8.8.8.8: bytes=32 time=3ms TTL=52
    Reply from 8.8.8.8: bytes=32 time=3ms TTL=52
    Reply from 8.8.8.8: bytes=32 time=3ms TTL=52
    Reply from 8.8.8.8: bytes=32 time=3ms TTL=52
    Reply from 8.8.8.8: bytes=32 time=3ms TTL=52
    Reply from 8.8.8.8: bytes=32 time=3ms TTL=52
    




  • Hi Gertjan

    Thank you very much for that! You solved my issue. Speed is now ~90MByte
    I still have "Ping Timeouts" and "Connection Issues" to the internet, if I start a copy job. But I think this is because of 100% link load if I run a copy job from VLAN 104 to VLAN 103.



  • @asan said in Netgate SG-4860 Performance:

    You solved my issue.

    Thanks, but I did close to nothing.
    What did you do to make things work for you ?


  • LAYER 8 Global Moderator

    Really curious to what you did as well.. I don't see any such issues on my 4860

    So to duplicate your testing.. I don't have any laggs setup... But 2 different networks..

    PC on 192.168.9.100/24 iperf to Laptop on 192.168.2.225/24

    Ran a 60 second iperf to fill the pipe from pc to laptop.. Seeing what I would expect 900's mbps

    Now on pc pinging something outside, 8.8.8.8 - don't see any issues at all..

    test.jpg

    Done nothing that I recall to do any sort of tweaking of settings on the 4860.. Its currently running 2.4.5.. While it has quite a few packages installed.. Nothing that might be considered heavy like IPS or proxy, ntop, etc..



  • @Gertjan Sry I forgot to mention it. I Solved it with changing the following parameters:
    System -> Advanced -> Miscellaneous, enable PowerD and set all to maximum.
    2274374d-fc9d-4c4d-bc5b-cf6fac886c7e-image.png

    @johnpoz Really strange.
    If I start a copy job, it looks like this:

    pfSense

    Reply from 10.0.10.129: bytes=32 time<1ms TTL=64
    Reply from 10.0.10.129: bytes=32 time<1ms TTL=64
    Request timed out.
    Request timed out.
    Reply from 10.0.10.129: bytes=32 time=1116ms TTL=64
    Reply from 10.0.10.129: bytes=32 time<1ms TTL=64
    Request timed out.
    Request timed out.
    Reply from 10.0.10.129: bytes=32 time=600ms TTL=64
    Reply from 10.0.10.129: bytes=32 time=3565ms TTL=64
    Reply from 10.0.10.129: bytes=32 time<1ms TTL=64
    Reply from 10.0.10.129: bytes=32 time<1ms TTL=64
    Reply from 10.0.10.129: bytes=32 time<1ms TTL=64
    

    Google DNS

    Reply from 8.8.8.8: bytes=32 time=3ms TTL=52
    Reply from 8.8.8.8: bytes=32 time=3ms TTL=52
    Reply from 8.8.8.8: bytes=32 time=3ms TTL=52
    Reply from 8.8.8.8: bytes=32 time=3ms TTL=52
    Reply from 8.8.8.8: bytes=32 time=3ms TTL=52
    Reply from 8.8.8.8: bytes=32 time=3ms TTL=52
    Reply from 8.8.8.8: bytes=32 time=3ms TTL=52
    Reply from 8.8.8.8: bytes=32 time=3ms TTL=52
    Reply from 8.8.8.8: bytes=32 time=3ms TTL=52
    Reply from 8.8.8.8: bytes=32 time=3ms TTL=52
    Reply from 8.8.8.8: bytes=32 time=3ms TTL=52
    Request timed out.
    Request timed out.
    Reply from 8.8.8.8: bytes=32 time=1616ms TTL=52
    Reply from 8.8.8.8: bytes=32 time=3ms TTL=52
    Request timed out.
    Request timed out.
    Reply from 8.8.8.8: bytes=32 time=600ms TTL=52
    Reply from 8.8.8.8: bytes=32 time=3565ms TTL=52
    Reply from 8.8.8.8: bytes=32 time=2ms TTL=52
    Reply from 8.8.8.8: bytes=32 time=2ms TTL=52
    Reply from 8.8.8.8: bytes=32 time=3ms TTL=52
    

    I don't know, maybe it has something to do with:
    https://forum.netgate.com/topic/151690/increased-memory-and-cpu-spikes-causing-latency-outage-with-2-4-5/64
    or
    https://forum.netgate.com/topic/151819/2-4-5-high-latency-and-packet-loss-not-in-a-vm/80


  • LAYER 8 Global Moderator

    What were your powerd setting before... I do not recall ever touching those, maybe I did? But currently set like this

    powerd.jpg



  • @johnpoz PowerD was disabled.
    While it was disabled, my throughput was only ~45MByte.


  • LAYER 8 Global Moderator

    Odd..
    https://docs.netgate.com/pfsense/en/latest/book/config/advanced-misc.html

    From this, I would take it that should be hiadaptive

    Hiadaptive

    Similar to adaptive but tuned to keep performance high at the cost of increased power consumption. It raises the CPU frequency faster and drops it slower. This is the default mode.

    Are you running say the CE version of pfsense, vs the factory version?



  • I don't think that I am running the factory version.
    I bought the device second hand.

    How can I check, if the device has the correct default configuration?

    I also tried the Hiadaptive. No change.


  • LAYER 8 Global Moderator

    @asan said in Netgate SG-4860 Performance:

    I also tried the Hiadaptive. No change.

    You mean when changed it to that you still see your full speed, or it was no change and you still saw lower performance?

    As to easy way to tell of your factory or CE... Off the top pretty sure that if factory you will see the AWS and ipsec export stuff, if you were running a CE version those would not be there..

    factory.jpg

    There is prob some other way to tell, but that is what comes to mind right off the top.


  • LAYER 8 Rebel Alliance

    Factory:
    pfSense_factory.png

    CE:
    pfSense_CE.png

    -Rico


  • LAYER 8 Global Moderator

    Well that would be easier ;) heehehe



  • It looks like I have CE:
    cbea996e-0651-4059-a54b-470da394e389-image.png


  • LAYER 8 Global Moderator

    Well in the big picture shouldn't be any sort of real issue - but pretty sure if you want you could put in a ticket with netgate to get a copy of the factory image.

    I do not think you need to be the original purchaser of the hardware to be able to get the factory image.



  • Aren't those images online?
    What are the Netgate ADI Images for?

    Please have a look at the video which shows my issue.
    pfsense2.zip

    Do you think that there is a change to solve the issue with installing the factory image?


  • LAYER 8 Global Moderator

    I do not believe so - those ADI images are still just the CE versions from my understanding.. They are just serial vs vga

    https://www.pfsense.org/download/
    The Netgate ADI image only supports a serial installation from memstick and does not come with VGA option. If you purchased a Netgate product, refer to the product manual for your appliance to see which reinstall image you need.

    From the product page for the 4860
    https://docs.netgate.com/pfsense/en/latest/solutions/sg-4860/reinstall-pfsense.html

    Reinstalling pfSense Software

    Please open a support ticket to request access to the factory firmware by selecting Firmware Access as the General Problem and then select Netgate SG-4860 Desktop for the platform. Make sure to include the serial number in the ticket to expedite access.

    Once the ticket is processed, the latest stable version of the firmware will be attached to the ticket, with a name such as:

    pfSense-netgate-memstick-ADI-2.4.5-RELEASE-amd64.img.gz

    If you go to download the ADI versions on the download site you get
    pfSense-CE-memstick-ADI-2.4.5-RELEASE-amd64.img.gz

    You do not need a support contract for such questions, from my experience.. The support from netgate has always been just over the top great.. You are free to open a ticket, worse case I would think is they would tell you to help you with X you would need a support contract - but even with that they prob point you in the right direction either way..

    You prob get an answer to your ticket in a couple of minutes to be honest ;) I had opened a ticket to get a reinstall image for my 4860 on the off chance that something went horrible wrong, and I had a link to download the file in less than 2 minutes from the time I submitted to the time email with link showed up in my inbox.



  • @johnpoz I see. Thank you very much for your help and support! I'll try a reimage with the factory image following by manual reconfiguration. I don't want to make a config restore. In my point of view there is a change that I would restore wrong settings if I do so.
    It will take a few days to do that, but I'll give you feedback as soon as possible.

    BTW: Did you see the video? What do you think about that?


  • LAYER 8 Global Moderator

    From my understanding you can just reload your config.. But you might want to do a native configuration - just to see what is default and what is not..

    But I would for sure have a backup of your config, just for reference if need be.. Depending on how complex your config is - you could prob just take some screenshots so you don't forget any rules ;)

    Haven't take a look at the video as of yet - I will. Not normally a fan of videos, other than movies and such.. I prefer documents and screenshots vs having to wade through some video looking for the important pieces of the puzzle.

    edit:
    Yeah that is odd.. And your saying that goes away when you set powerd to max.

    Let me see if I can get a copy going to my laptop... I would do it on my nas, But its the same vlan as my PC... That test will prob have to wait til later, currently laptop is connected to my work network via vpn you know for "work" hehehe



  • No this issue doesn't go away. With the change to max or hiadaptive I have 90MByte througput, otherwise only 45MByte.


  • LAYER 8 Global Moderator

    But you still have loss of pings even when your seeing 90.. I have never noticed such a thing.. But happy to do some testing once I can disconnect my laptop from work (after working hours).. I could fire up one of my play laptops - but lazy ;) hehehe

    Maybe its an issue with your PC... do you see the same sort of issue when doing copies of files to something local on the same network, so your not routing through pfsense?



  • Yes I still have loss of pings and very slow - no connection.

    BTW: I already got the factory image from the support. What a service!!!
    Because of a complex configuration, setup will take a few hours. I don't want to risk another "misconfiguration" with a restore.


  • LAYER 8 Global Moderator

    Told you - they are fast and just over the top great!!

    edit: I have been around here for long time, and talk to some of the guys on back channels.. And what I can tell you is they "care" about the product, and what they do.. And they know their suff as well! Top notch group of guys for sure.



  • Yeah I guess so too. They are doing a great job and I really like pfSense as a firewall solution.


Log in to reply