Abysmal Performance after pfSense hardware upgrade

8ayM

Let me know if there is anything you can think of me trying or something else you'd like to to check.

stephenw10

The throughput you're seeing now is as expected though?

8ayM

It is. You just now have me curious what is causing the interrupts.

I'm considering getting the 1u version of my new router. If I do, I'll preform a clean install and look to rebuild my system one brick at a time to see if I can figure out what is causing the GUI slowdown I've had since I moved to my last hardware. I can try to keep an eye on the interrupts as well.

Here is the stats from Status -> Interfaces

I seem to be a little beyond the interrupt range you said shouldn't be "unusual".

https://forum.netgate.com/topic/179674/netgate-6100-significant-interface-interrupt-rates/9

stephenw10

Mmm, but no where near 10K! I agree though I find it odd that you see the interrupt loading in the top output and I do not on a similar C3K system. Like whilst passing 1Gbps iperf traffic on a 5100:

last pid: 57718;  load averages:  0.55,  0.36,  0.34                                              up 0+06:27:17  22:57:28
339 threads:   7 running, 288 sleeping, 44 waiting
CPU 0:  0.0% user,  0.0% nice, 28.6% system,  0.0% interrupt, 71.4% idle
CPU 1:  0.0% user,  0.0% nice, 23.1% system,  0.0% interrupt, 76.9% idle
CPU 2:  0.4% user,  0.0% nice, 24.7% system,  0.0% interrupt, 74.9% idle
CPU 3:  0.0% user,  0.0% nice, 34.1% system,  0.0% interrupt, 65.9% idle
Mem: 45M Active, 258M Inact, 505M Wired, 3028M Free
ARC: 127M Total, 28M MFU, 93M MRU, 416K Anon, 962K Header, 4535K Other
     92M Compressed, 229M Uncompressed, 2.48:1 Ratio
Swap: 1024M Total, 1024M Free

  PID USERNAME    PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
   11 root        187 ki31     0B    64K RUN      0 377:22  73.68% [idle{idle: cpu0}]
   11 root        187 ki31     0B    64K RUN      2 376:55  73.23% [idle{idle: cpu2}]
   11 root        187 ki31     0B    64K RUN      1 377:47  72.55% [idle{idle: cpu1}]
   11 root        187 ki31     0B    64K CPU3     3 376:45  72.51% [idle{idle: cpu3}]
    0 root        -60    -     0B  1648K -        0   0:06  19.51% [kernel{if_io_tqg_0}]
    0 root        -60    -     0B  1648K -        3   0:05  18.78% [kernel{if_io_tqg_3}]
    0 root        -60    -     0B  1648K CPU1     1   0:04  18.60% [kernel{if_io_tqg_1}]
57718 root         34    0    19M  8644K CPU0     0   0:03  17.05% iperf3 -c 172.21.16.8 -P 3 -t 30{iperf3}
57718 root         36    0    19M  8644K sbwait   3   0:04  16.93% iperf3 -c 172.21.16.8 -P 3 -t 30{iperf3}
57718 root         40    0    19M  8644K sbwait   1   0:03  16.74% iperf3 -c 172.21.16.8 -P 3 -t 30{iperf3}
    0 root        -60    -     0B  1648K -        1   0:36   0.14% [kernel{if_config_tqg_0}]
78943 root         20    0    14M  4716K CPU2     2   0:00   0.12% top -HaSP
    7 root        -16    -     0B    16K pftm     0   0:09   0.03% [pf purge]

8ayM

@stephenw10

What settings did you utilize for the iperf test?

I turned on the server on my pfSense box

Then connected from my Windows box with
iperf3.exe -c 192.168.1.1 -P 2

This way I get over 1Gbps

stephenw10

I ran the server on a Linux box on my network and then ran a client on pfSense:

[24.03-BETA][admin@5100-2.stevew.lan]/root: iperf3 -c 172.21.16.8 -P 3 -t 30
Connecting to host 172.21.16.8, port 5201
[  5] local 172.21.16.75 port 22757 connected to 172.21.16.8 port 5201
[  7] local 172.21.16.75 port 1280 connected to 172.21.16.8 port 5201
[  9] local 172.21.16.75 port 22575 connected to 172.21.16.8 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  39.2 MBytes   329 Mbits/sec    0    938 KBytes       
[  7]   0.00-1.00   sec  39.2 MBytes   329 Mbits/sec    0    939 KBytes       
[  9]   0.00-1.00   sec  39.2 MBytes   329 Mbits/sec    0    937 KBytes       
[SUM]   0.00-1.00   sec   118 MBytes   986 Mbits/sec    0

The 5100 ix NICs are 1G but the SoC is the same.

8ayM

@stephenw10

I averaged about 1.9Gbps with the interrupts @ 24%-31%

with your settings

I'm guessingthere are gremlins in the config, from being carried forward over a decade, multiple hardware, and os upgrades

Gblenn

Perhaps a long shot, but could it be that you have have some power saving settings for the CPU? Pegging it at a low frequency? I remember reading about that in relation to high interrupt under load.

stephenw10

Hmm, possible.

8ayM

@Gblenn

I didn't see anything explicitly listed for power savings that was enabled.

8ayM

Ordering the Rackmount version shortly and I'll test restoring one component at a time to see if the interrupts persist, or at what point they may increase.