Netgate SG-4860 Performance

asan

Hi all

I already posted this topic in " General pfSense Questions", but I didn't get an answer.

Please have look at the picture to get an overview of my setup.
pfSense is the routing instance of both VLAN's 103 & 104.

Problem description
If I copy large files from my workstation to the synology nas, cpu usage is very high and throughput is only ~45MByte.
Ping latency to the internet is increasing dramatically and cpu usage of the pfSense is very high.

What I did

Separate LAGG for VLAN 103
I moved VLAN 103 to a separate LAGG (1) that incoming and outgoing traffic is separated physically.
System Tunables
I have disabled the EEE settings on all interfaces.
See this post https://community.spiceworks.com/topic/1221309-periodic-packet-loss-constant-carp-switchovers-with-intel-nics-i350-igb for details.

[2.4.5-RELEASE][root@pfsense]/root: sysctl dev.igb | grep eee
dev.igb.5.eee_disabled: 1
dev.igb.4.eee_disabled: 1
dev.igb.3.eee_disabled: 1
dev.igb.2.eee_disabled: 1
dev.igb.1.eee_disabled: 1
dev.igb.0.eee_disabled: 1

Question
Is this behavior because of hardware limitations, or do I have a misconfiguration of my setup?
I thought that a Netgate SG-4860 Appliance can handle 1GByte throughput without any limitations.

Thanks in advance for your support!

Some pfSense top -aSH outputs:

  PID USERNAME     PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
   11 root         155 ki31     0K    64K RUN     1 936:47  59.19% [idle{idle: cpu1}]
   11 root         155 ki31     0K    64K CPU3    3 935:25  52.52% [idle{idle: cpu3}]
   11 root         155 ki31     0K    64K RUN     0 937:59  45.01% [idle{idle: cpu0}]
   12 root         -92    -     0K   704K WAIT    0   0:45  27.15% [intr{irq272: igb4:que 0}]


  PID USERNAME     PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
    0 root         -92    -     0K   864K CPU2    2   2:06 100.00% [kernel{igb3 que (qid 0)}]
   11 root         155 ki31     0K    64K RUN     3 935:26  45.91% [idle{idle: cpu3}]
   11 root         155 ki31     0K    64K RUN     1 936:48  44.13% [idle{idle: cpu1}]
   12 root         -92    -     0K   704K WAIT    0   0:45  42.56% [intr{irq272: igb4:que 0}]
   11 root         155 ki31     0K    64K RUN     0 937:59  35.26% [idle{idle: cpu0}]


  PID USERNAME     PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
   11 root         155 ki31     0K    64K CPU3    3 935:33  94.23% [idle{idle: cpu3}]
   11 root         155 ki31     0K    64K CPU1    1 936:55  88.77% [idle{idle: cpu1}]
   11 root         155 ki31     0K    64K RUN     0 938:04  70.96% [idle{idle: cpu0}]
   12 root         -92    -     0K   704K WAIT    0   0:49  25.32% [intr{irq272: igb4:que 0}]
   12 root         -92    -     0K   704K WAIT    0   0:15   0.34% [intr{irq267: igb3:que 0}]

Ping from workstation to 8.8.8.8 while copying a large file:

Reply from 8.8.8.8: bytes=32 time=3ms TTL=52
Reply from 8.8.8.8: bytes=32 time=3ms TTL=52
Reply from 8.8.8.8: bytes=32 time=3ms TTL=52
Reply from 8.8.8.8: bytes=32 time=3ms TTL=52
Reply from 8.8.8.8: bytes=32 time=3ms TTL=52
Reply from 8.8.8.8: bytes=32 time=3ms TTL=52
Reply from 8.8.8.8: bytes=32 time=3057ms TTL=52
Reply from 8.8.8.8: bytes=32 time=5ms TTL=52
Request timed out.
Request timed out.
Reply from 8.8.8.8: bytes=32 time=2495ms TTL=52
Reply from 8.8.8.8: bytes=32 time=3ms TTL=52
Reply from 8.8.8.8: bytes=32 time=1007ms TTL=52
Reply from 8.8.8.8: bytes=32 time=3ms TTL=52
Reply from 8.8.8.8: bytes=32 time=3007ms TTL=52
Reply from 8.8.8.8: bytes=32 time=3ms TTL=52
Reply from 8.8.8.8: bytes=32 time=9ms TTL=52
Request timed out.
Reply from 8.8.8.8: bytes=32 time=2630ms TTL=52
Reply from 8.8.8.8: bytes=32 time=3ms TTL=52
Reply from 8.8.8.8: bytes=32 time=3ms TTL=52
Reply from 8.8.8.8: bytes=32 time=3ms TTL=52
Reply from 8.8.8.8: bytes=32 time=3ms TTL=52
Reply from 8.8.8.8: bytes=32 time=3ms TTL=52
Reply from 8.8.8.8: bytes=32 time=3ms TTL=52
Reply from 8.8.8.8: bytes=32 time=3ms TTL=52

Gertjan

@asan said in Netgate SG-4860 Performance:

Netgate SG-4860

Hi, you saw https://forum.netgate.com/topic/119623/slow-throughput-on-sg-4860-600mbs-on-1gbs-line ?

asan

Hi Gertjan

Thank you very much for that! You solved my issue. Speed is now ~90MByte
I still have "Ping Timeouts" and "Connection Issues" to the internet, if I start a copy job. But I think this is because of 100% link load if I run a copy job from VLAN 104 to VLAN 103.

Gertjan

@asan said in Netgate SG-4860 Performance:

You solved my issue.

Thanks, but I did close to nothing.
What did you do to make things work for you ?

johnpoz

Really curious to what you did as well.. I don't see any such issues on my 4860

So to duplicate your testing.. I don't have any laggs setup... But 2 different networks..

PC on 192.168.9.100/24 iperf to Laptop on 192.168.2.225/24

Ran a 60 second iperf to fill the pipe from pc to laptop.. Seeing what I would expect 900's mbps

Now on pc pinging something outside, 8.8.8.8 - don't see any issues at all..

Done nothing that I recall to do any sort of tweaking of settings on the 4860.. Its currently running 2.4.5.. While it has quite a few packages installed.. Nothing that might be considered heavy like IPS or proxy, ntop, etc..

asan

@Gertjan Sry I forgot to mention it. I Solved it with changing the following parameters:
System -> Advanced -> Miscellaneous, enable PowerD and set all to maximum.

@johnpoz Really strange.
If I start a copy job, it looks like this:

pfSense

Reply from 10.0.10.129: bytes=32 time<1ms TTL=64
Reply from 10.0.10.129: bytes=32 time<1ms TTL=64
Request timed out.
Request timed out.
Reply from 10.0.10.129: bytes=32 time=1116ms TTL=64
Reply from 10.0.10.129: bytes=32 time<1ms TTL=64
Request timed out.
Request timed out.
Reply from 10.0.10.129: bytes=32 time=600ms TTL=64
Reply from 10.0.10.129: bytes=32 time=3565ms TTL=64
Reply from 10.0.10.129: bytes=32 time<1ms TTL=64
Reply from 10.0.10.129: bytes=32 time<1ms TTL=64
Reply from 10.0.10.129: bytes=32 time<1ms TTL=64

Google DNS

Reply from 8.8.8.8: bytes=32 time=3ms TTL=52
Reply from 8.8.8.8: bytes=32 time=3ms TTL=52
Reply from 8.8.8.8: bytes=32 time=3ms TTL=52
Reply from 8.8.8.8: bytes=32 time=3ms TTL=52
Reply from 8.8.8.8: bytes=32 time=3ms TTL=52
Reply from 8.8.8.8: bytes=32 time=3ms TTL=52
Reply from 8.8.8.8: bytes=32 time=3ms TTL=52
Reply from 8.8.8.8: bytes=32 time=3ms TTL=52
Reply from 8.8.8.8: bytes=32 time=3ms TTL=52
Reply from 8.8.8.8: bytes=32 time=3ms TTL=52
Reply from 8.8.8.8: bytes=32 time=3ms TTL=52
Request timed out.
Request timed out.
Reply from 8.8.8.8: bytes=32 time=1616ms TTL=52
Reply from 8.8.8.8: bytes=32 time=3ms TTL=52
Request timed out.
Request timed out.
Reply from 8.8.8.8: bytes=32 time=600ms TTL=52
Reply from 8.8.8.8: bytes=32 time=3565ms TTL=52
Reply from 8.8.8.8: bytes=32 time=2ms TTL=52
Reply from 8.8.8.8: bytes=32 time=2ms TTL=52
Reply from 8.8.8.8: bytes=32 time=3ms TTL=52

I don't know, maybe it has something to do with:
https://forum.netgate.com/topic/151690/increased-memory-and-cpu-spikes-causing-latency-outage-with-2-4-5/64
or
https://forum.netgate.com/topic/151819/2-4-5-high-latency-and-packet-loss-not-in-a-vm/80

johnpoz

What were your powerd setting before... I do not recall ever touching those, maybe I did? But currently set like this

asan

@johnpoz PowerD was disabled.
While it was disabled, my throughput was only ~45MByte.

johnpoz

Odd..
https://docs.netgate.com/pfsense/en/latest/book/config/advanced-misc.html

From this, I would take it that should be hiadaptive

Hiadaptive

Similar to adaptive but tuned to keep performance high at the cost of increased power consumption. It raises the CPU frequency faster and drops it slower. This is the default mode.

Are you running say the CE version of pfsense, vs the factory version?

asan

I don't think that I am running the factory version.
I bought the device second hand.

How can I check, if the device has the correct default configuration?

I also tried the Hiadaptive. No change.

johnpoz

@asan said in Netgate SG-4860 Performance:

I also tried the Hiadaptive. No change.

You mean when changed it to that you still see your full speed, or it was no change and you still saw lower performance?

As to easy way to tell of your factory or CE... Off the top pretty sure that if factory you will see the AWS and ipsec export stuff, if you were running a CE version those would not be there..

There is prob some other way to tell, but that is what comes to mind right off the top.

Rico

Factory:

CE:

-Rico

johnpoz

Well that would be easier ;) heehehe

asan

It looks like I have CE:

johnpoz

Well in the big picture shouldn't be any sort of real issue - but pretty sure if you want you could put in a ticket with netgate to get a copy of the factory image.

I do not think you need to be the original purchaser of the hardware to be able to get the factory image.

asan

Aren't those images online?
What are the Netgate ADI Images for?

Please have a look at the video which shows my issue.
pfsense2.zip

Do you think that there is a change to solve the issue with installing the factory image?

johnpoz

I do not believe so - those ADI images are still just the CE versions from my understanding.. They are just serial vs vga

https://www.pfsense.org/download/
The Netgate ADI image only supports a serial installation from memstick and does not come with VGA option. If you purchased a Netgate product, refer to the product manual for your appliance to see which reinstall image you need.

From the product page for the 4860
https://docs.netgate.com/pfsense/en/latest/solutions/sg-4860/reinstall-pfsense.html

Reinstalling pfSense Software

Please open a support ticket to request access to the factory firmware by selecting Firmware Access as the General Problem and then select Netgate SG-4860 Desktop for the platform. Make sure to include the serial number in the ticket to expedite access.

Once the ticket is processed, the latest stable version of the firmware will be attached to the ticket, with a name such as:

pfSense-netgate-memstick-ADI-2.4.5-RELEASE-amd64.img.gz

If you go to download the ADI versions on the download site you get
pfSense-CE-memstick-ADI-2.4.5-RELEASE-amd64.img.gz

You do not need a support contract for such questions, from my experience.. The support from netgate has always been just over the top great.. You are free to open a ticket, worse case I would think is they would tell you to help you with X you would need a support contract - but even with that they prob point you in the right direction either way..

You prob get an answer to your ticket in a couple of minutes to be honest ;) I had opened a ticket to get a reinstall image for my 4860 on the off chance that something went horrible wrong, and I had a link to download the file in less than 2 minutes from the time I submitted to the time email with link showed up in my inbox.

asan

@johnpoz I see. Thank you very much for your help and support! I'll try a reimage with the factory image following by manual reconfiguration. I don't want to make a config restore. In my point of view there is a change that I would restore wrong settings if I do so.
It will take a few days to do that, but I'll give you feedback as soon as possible.

BTW: Did you see the video? What do you think about that?

johnpoz

From my understanding you can just reload your config.. But you might want to do a native configuration - just to see what is default and what is not..

But I would for sure have a backup of your config, just for reference if need be.. Depending on how complex your config is - you could prob just take some screenshots so you don't forget any rules ;)

Haven't take a look at the video as of yet - I will. Not normally a fan of videos, other than movies and such.. I prefer documents and screenshots vs having to wade through some video looking for the important pieces of the puzzle.

edit:
Yeah that is odd.. And your saying that goes away when you set powerd to max.

Let me see if I can get a copy going to my laptop... I would do it on my nas, But its the same vlan as my PC... That test will prob have to wait til later, currently laptop is connected to my work network via vpn you know for "work" hehehe

asan

No this issue doesn't go away. With the change to max or hiadaptive I have 90MByte througput, otherwise only 45MByte.