Netgate SG-4860: Hardware Limitation or Misconfiguration

asan

Hi all

Please have look at the picture to get an overview of my setup.
pfSense is the routing instance of both VLAN's 103 & 104.

Problem description
If I copy large files from my workstation to the synology nas, cpu usage is very high and throughput is only ~45MByte.
Ping latency to the internet is increasing dramatically and cpu usage of the pfSense is very high.

What I did

Separate LAGG for VLAN 103
I moved VLAN 103 to a separate LAGG (1) that incoming and outgoing traffic is separated physically.
System Tunables
I have disabled the EEE settings on all interfaces.
See this post https://community.spiceworks.com/topic/1221309-periodic-packet-loss-constant-carp-switchovers-with-intel-nics-i350-igb for details.

[2.4.5-RELEASE][root@pfsense]/root: sysctl dev.igb | grep eee
dev.igb.5.eee_disabled: 1
dev.igb.4.eee_disabled: 1
dev.igb.3.eee_disabled: 1
dev.igb.2.eee_disabled: 1
dev.igb.1.eee_disabled: 1
dev.igb.0.eee_disabled: 1

Question
Is this behavior because of hardware limitations, or do I have a misconfiguration of my setup?
I thought that a Netgate SG-4860 Appliance can handle 1GByte throughput without any limitations.

Thanks in advance for your support!

Some pfSense top -aSH outputs:

  PID USERNAME     PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
   11 root         155 ki31     0K    64K RUN     1 936:47  59.19% [idle{idle: cpu1}]
   11 root         155 ki31     0K    64K CPU3    3 935:25  52.52% [idle{idle: cpu3}]
   11 root         155 ki31     0K    64K RUN     0 937:59  45.01% [idle{idle: cpu0}]
   12 root         -92    -     0K   704K WAIT    0   0:45  27.15% [intr{irq272: igb4:que 0}]


  PID USERNAME     PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
    0 root         -92    -     0K   864K CPU2    2   2:06 100.00% [kernel{igb3 que (qid 0)}]
   11 root         155 ki31     0K    64K RUN     3 935:26  45.91% [idle{idle: cpu3}]
   11 root         155 ki31     0K    64K RUN     1 936:48  44.13% [idle{idle: cpu1}]
   12 root         -92    -     0K   704K WAIT    0   0:45  42.56% [intr{irq272: igb4:que 0}]
   11 root         155 ki31     0K    64K RUN     0 937:59  35.26% [idle{idle: cpu0}]


  PID USERNAME     PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
   11 root         155 ki31     0K    64K CPU3    3 935:33  94.23% [idle{idle: cpu3}]
   11 root         155 ki31     0K    64K CPU1    1 936:55  88.77% [idle{idle: cpu1}]
   11 root         155 ki31     0K    64K RUN     0 938:04  70.96% [idle{idle: cpu0}]
   12 root         -92    -     0K   704K WAIT    0   0:49  25.32% [intr{irq272: igb4:que 0}]
   12 root         -92    -     0K   704K WAIT    0   0:15   0.34% [intr{irq267: igb3:que 0}]

Ping from workstation to 8.8.8.8 while copying a large file:

Reply from 8.8.8.8: bytes=32 time=3ms TTL=52
Reply from 8.8.8.8: bytes=32 time=3ms TTL=52
Reply from 8.8.8.8: bytes=32 time=3ms TTL=52
Reply from 8.8.8.8: bytes=32 time=3ms TTL=52
Reply from 8.8.8.8: bytes=32 time=3ms TTL=52
Reply from 8.8.8.8: bytes=32 time=3ms TTL=52
Reply from 8.8.8.8: bytes=32 time=3057ms TTL=52
Reply from 8.8.8.8: bytes=32 time=5ms TTL=52
Request timed out.
Request timed out.
Reply from 8.8.8.8: bytes=32 time=2495ms TTL=52
Reply from 8.8.8.8: bytes=32 time=3ms TTL=52
Reply from 8.8.8.8: bytes=32 time=1007ms TTL=52
Reply from 8.8.8.8: bytes=32 time=3ms TTL=52
Reply from 8.8.8.8: bytes=32 time=3007ms TTL=52
Reply from 8.8.8.8: bytes=32 time=3ms TTL=52
Reply from 8.8.8.8: bytes=32 time=9ms TTL=52
Request timed out.
Reply from 8.8.8.8: bytes=32 time=2630ms TTL=52
Reply from 8.8.8.8: bytes=32 time=3ms TTL=52
Reply from 8.8.8.8: bytes=32 time=3ms TTL=52
Reply from 8.8.8.8: bytes=32 time=3ms TTL=52
Reply from 8.8.8.8: bytes=32 time=3ms TTL=52
Reply from 8.8.8.8: bytes=32 time=3ms TTL=52
Reply from 8.8.8.8: bytes=32 time=3ms TTL=52
Reply from 8.8.8.8: bytes=32 time=3ms TTL=52