Load updated Intel IX module to get 10Gbps

stephenw10

Ok, what CPU is that exactly? You can see the driver is assigning the full 8 queues to the NICs which is good. Check the per-core CPU usage when you are testing the throughput using: top -HaSP

ogghi

@stephenw10 said in Load updated Intel IX module to get 10Gbps:

top -HaSP

Running:

iperf3 -c speedtest.shinternet.ch -p 5200-5209 -P 10 -t 100

last pid: 38688;  load averages:  1.34,  0.59,  0.47                                                                                                                  up 6+00:05:49  14:40:29
500 threads:   10 running, 464 sleeping, 26 waiting
CPU 0:  0.8% user,  0.0% nice, 64.7% system,  0.0% interrupt, 34.5% idle
CPU 1:  0.8% user,  0.0% nice, 20.0% system,  0.0% interrupt, 79.2% idle
CPU 2:  0.8% user,  0.0% nice, 33.3% system,  0.0% interrupt, 65.9% idle
CPU 3:  1.6% user,  0.0% nice, 27.1% system,  0.0% interrupt, 71.4% idle
CPU 4:  0.8% user,  0.0% nice, 32.2% system,  0.0% interrupt, 67.1% idle
CPU 5:  0.8% user,  0.0% nice, 27.8% system,  0.0% interrupt, 71.4% idle
CPU 6:  3.5% user,  0.0% nice,  0.8% system,  0.4% interrupt, 95.3% idle
CPU 7:  0.4% user,  0.0% nice, 36.9% system,  0.4% interrupt, 62.4% idle
Mem: 253M Active, 2162M Inact, 1160M Wired, 4179M Free
ARC: 489M Total, 210M MFU, 268M MRU, 776K Anon, 2855K Header, 7730K Other
     425M Compressed, 1529M Uncompressed, 3.59:1 Ratio
Swap: 1024M Total, 1024M Free

  PID USERNAME    PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
   11 root        187 ki31     0B   128K CPU6     6 137.8H  95.91% [idle{idle: cpu6}]
   11 root        187 ki31     0B   128K CPU1     1 138.5H  79.08% [idle{idle: cpu1}]
   11 root        187 ki31     0B   128K CPU5     5 137.6H  73.62% [idle{idle: cpu5}]
   11 root        187 ki31     0B   128K CPU4     4 137.5H  72.60% [idle{idle: cpu4}]
   11 root        187 ki31     0B   128K RUN      3 137.5H  71.97% [idle{idle: cpu3}]
   11 root        187 ki31     0B   128K RUN      2 138.2H  69.60% [idle{idle: cpu2}]
   11 root        187 ki31     0B   128K RUN      7 138.0H  64.44% [idle{idle: cpu7}]
    0 root        -60    -     0B  2368K CPU0     0 126:29  62.11% [kernel{if_io_tqg_0}]
   11 root        187 ki31     0B   128K CPU0     0 137.0H  36.71% [idle{idle: cpu0}]
    0 root        -60    -     0B  2368K -        7  85:13  34.52% [kernel{if_io_tqg_7}]
    0 root        -60    -     0B  2368K -        2 102:04  28.68% [kernel{if_io_tqg_2}]
    0 root        -60    -     0B  2368K -        3  94:49  25.96% [kernel{if_io_tqg_3}]
    0 root        -60    -     0B  2368K -        4  89:03  25.91% [kernel{if_io_tqg_4}]
    0 root        -60    -     0B  2368K -        5  98:08  24.47% [kernel{if_io_tqg_5}]
    0 root        -60    -     0B  2368K -        1 151:48  19.72% [kernel{if_io_tqg_1}]
88144 root         24    0    19M  8712K select   5  17.6H   6.36% /usr/local/sbin/openvpn --config /var/etc/openvpn/server1/config.ovpn
78123 root         20    0    22M    12M kqread   6   3:32   1.05% /usr/local/sbin/lighttpd_pfb -f /var/unbound/pfb_dnsbl_lighty.conf
29667 unbound      20    0  1771M  1655M kqread   2   0:49   0.90% /usr/local/sbin/unbound -c /var/unbound/unbound.conf{unbound}
 4206 root         20    0    29M    16M select   5  32:22   0.83% /usr/local/sbin/openvpn --config /var/etc/openvpn/server3/config.ovpn{openvpn}
29667 unbound      20    0  1771M  1655M kqread   4   0:47   0.60% /usr/local/sbin/unbound -c /var/unbound/unbound.conf{unbound}
29667 unbound      20    0  1771M  1655M kqread   3   2:03   0.57% /usr/local/sbin/unbound -c /var/unbound/unbound.conf{unbound}
    0 root        -60    -     0B  2368K -        6  98:06   0.54% [kernel{if_io_tqg_6}]
29667 unbound      20    0  1771M  1655M kqread   5   1:24   0.54% /usr/local/sbin/unbound -c /var/unbound/unbound.conf{unbound}
   12 root        -60    -     0B   288K WAIT     0   1:44   0.36% [intr{swi1: netisr 7}]
29667 unbound      20    0  1771M  1655M kqread   5   1:59   0.29% /usr/local/sbin/unbound -c /var/unbound/unbound.conf{unbound}
    2 root        -60    -     0B   128K WAIT     0  54:04   0.22% [clock{clock (0)}]
69491 root         20    0    26M    11M select   3  12:09   0.21% /usr/local/sbin/openvpn --config /var/etc/openvpn/server12/config.ovpn{openvpn}
29667 unbound      20    0  1771M  1655M kqread   6   2:02   0.21% /usr/local/sbin/unbound -c /var/unbound/unbound.conf{unbound}
29667 unbound      20    0  1771M  1655M kqread   2   2:13   0.17% /usr/local/sbin/unbound -c /var/unbound/unbound.conf{unbound}
   12 root        -60    -     0B   288K WAIT     2   3:11   0.16% [intr{swi1: netisr 2}]
    7 root        -16    -     0B    16K pftm     2   9:05   0.13% [pf purge]
17299 root         20    0    17M  4820K CPU3     3   0:00   0.10% top -HaSP
   12 root        -60    -     0B   288K WAIT     7   4:23   0.09% [intr{swi1: netisr 4}]
64908 root         20    0  5108M   189M uwait    7  29:25   0.08% /usr/local/bin/telegraf -config=/usr/local/etc/telegraf.conf{telegraf}

Edit: How do I get the CPU model from terminal or so?

stephenw10

It shows on the dashboard. It's shown in /var/log/dmesg.boot. sysctl hw.model should show it.

Nothing shown as limited there. Are you sure you can 10Gbps to that host?

ogghi

@stephenw10
LOL, easy, but sometimes still hidden from men's eyes...

Intel(R) Xeon(R) CPU D-1518 @ 2.20GHz
8 CPUs: 1 package(s) x 4 core(s) x 2 hardware threads
AES-NI CPU Crypto: Yes (active)
QAT Crypto: No

When our provider once ran a check (some raw ping flood I guess?) they filled up the 10Gbps.

Also I tried some other iperf3 servers, when they don't report busy, always the same limitation in speed.
Also this is something visible when downloading / uploading from/to a fast server in the net.

stephenw10

Hmm if you're able to test between two local machines with iperf I would try that first. And if you can confirm they will reach 10Gbps when on the same subnet that eliminates any doubt.

ogghi

@stephenw10 Thanks for getting back to me that quickly!
From a VM on an ESX host to a VM on another ESX host I get ~9Gbps, so looks fine.

From or to one of those VMs to the pFsense (I installed iperf3 to test) I get even less, around 2,7Gbps.

This seems odd? To exclude the network behind the pFsense I also once tried to have a PC with a 10Gb fiber card directly attached to the switch that comes right after the pFsense.
Same results though.

stephenw10

Running iperf to or from pfSense dircetly will always give a lower result. Both because iperf itself uses significant CPU cycles and because pfSense is optimised as a router and not a server.

Test through pfSense between different interfaces if you can.

ogghi

@stephenw10 our setup here is more simple than others perhaps.
Just ix0 for WAN, ix1 for LAN.
Some VLAN interfaces on LAN (ix1) and a slower backup link on igb5

stephenw10

Well if you're able to test against a local server on the WAN side that would be a good test.

ogghi

@stephenw10 I should probably enable post notifications...
The only way here would be for me to disconnect the WAN interface from the modem and attach a server to it?

Only other way could be if the provider modem has another 10Gb port and I connect a server there, then give it another public IP we still have. But then traffic would already passing through the modem (which should certainly support 10Gb as the provider uses it for lots of customers)

stephenw10

Hmm, you could use a switch on the WAN side but it would obviously have to be 10G capable.

That CPU is probably somewhere between the 1537 and 1541 for routing that traffic. I'd expect to see somewhere around 5-6Gbps with pf and NAT enabled.

ogghi

@stephenw10
What do you mean with between 1537 and 1541?
The Intel(R) Xeon(R) CPU D-1518 ?

So you think the limiting factor here could be the CPU?

Would we need to get another Netgate certified appliance here to get the full 10Gb?

stephenw10

It doesn't need to be a Netgate appliance but you might need some other device to pass 10Gbps.

However your top output above does not show any CPU core at 100%. It appears there is some other limit in play here before it exhausts the CPU.

ogghi

@stephenw10
That's a tough one then?
How could I figure out what is limiting here?

stephenw10

Well we did see one user report nearly 25Gbps using that CPU with Mellanox NICs: https://forum.netgate.com/post/1119611

I still find that hard to believe though.

ogghi

@stephenw10
Interesting!
I am wondering, should I try getting such a Mellanox card?
Or should we be fine achieving 10Gbps with a 6100 for example?
That would be fine to buy I guess...

stephenw10

The C3558 in the 6100 will not pass 10Gbps either. Again there are a lot of variables but I'd expect to see something in the 3-4Gbps range for an equivalent config.

ogghi

@stephenw10
So to get 10Gbps we would need much bigger hardware then?
I thought the 6100 could be an option, looking at this:

stephenw10

That is a total throughput value though. You won't see that in an iperf3 test across an single link.

Yes you need significant CPU power if you want to pass 10Gbps using a single TCP stream like downloading a file.

ogghi

@stephenw10 what appliance would you suggest here?
Not sure if budget allows me to get another PFsense, but I could ask
Otherwise, building something could be an option?
What specs would you recommend to have at least?