Load updated Intel IX module to get 10Gbps
-
@stephenw10 said in Load updated Intel IX module to get 10Gbps:
top -HaSP
Running:
iperf3 -c speedtest.shinternet.ch -p 5200-5209 -P 10 -t 100last pid: 38688; load averages: 1.34, 0.59, 0.47 up 6+00:05:49 14:40:29 500 threads: 10 running, 464 sleeping, 26 waiting CPU 0: 0.8% user, 0.0% nice, 64.7% system, 0.0% interrupt, 34.5% idle CPU 1: 0.8% user, 0.0% nice, 20.0% system, 0.0% interrupt, 79.2% idle CPU 2: 0.8% user, 0.0% nice, 33.3% system, 0.0% interrupt, 65.9% idle CPU 3: 1.6% user, 0.0% nice, 27.1% system, 0.0% interrupt, 71.4% idle CPU 4: 0.8% user, 0.0% nice, 32.2% system, 0.0% interrupt, 67.1% idle CPU 5: 0.8% user, 0.0% nice, 27.8% system, 0.0% interrupt, 71.4% idle CPU 6: 3.5% user, 0.0% nice, 0.8% system, 0.4% interrupt, 95.3% idle CPU 7: 0.4% user, 0.0% nice, 36.9% system, 0.4% interrupt, 62.4% idle Mem: 253M Active, 2162M Inact, 1160M Wired, 4179M Free ARC: 489M Total, 210M MFU, 268M MRU, 776K Anon, 2855K Header, 7730K Other 425M Compressed, 1529M Uncompressed, 3.59:1 Ratio Swap: 1024M Total, 1024M Free PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 11 root 187 ki31 0B 128K CPU6 6 137.8H 95.91% [idle{idle: cpu6}] 11 root 187 ki31 0B 128K CPU1 1 138.5H 79.08% [idle{idle: cpu1}] 11 root 187 ki31 0B 128K CPU5 5 137.6H 73.62% [idle{idle: cpu5}] 11 root 187 ki31 0B 128K CPU4 4 137.5H 72.60% [idle{idle: cpu4}] 11 root 187 ki31 0B 128K RUN 3 137.5H 71.97% [idle{idle: cpu3}] 11 root 187 ki31 0B 128K RUN 2 138.2H 69.60% [idle{idle: cpu2}] 11 root 187 ki31 0B 128K RUN 7 138.0H 64.44% [idle{idle: cpu7}] 0 root -60 - 0B 2368K CPU0 0 126:29 62.11% [kernel{if_io_tqg_0}] 11 root 187 ki31 0B 128K CPU0 0 137.0H 36.71% [idle{idle: cpu0}] 0 root -60 - 0B 2368K - 7 85:13 34.52% [kernel{if_io_tqg_7}] 0 root -60 - 0B 2368K - 2 102:04 28.68% [kernel{if_io_tqg_2}] 0 root -60 - 0B 2368K - 3 94:49 25.96% [kernel{if_io_tqg_3}] 0 root -60 - 0B 2368K - 4 89:03 25.91% [kernel{if_io_tqg_4}] 0 root -60 - 0B 2368K - 5 98:08 24.47% [kernel{if_io_tqg_5}] 0 root -60 - 0B 2368K - 1 151:48 19.72% [kernel{if_io_tqg_1}] 88144 root 24 0 19M 8712K select 5 17.6H 6.36% /usr/local/sbin/openvpn --config /var/etc/openvpn/server1/config.ovpn 78123 root 20 0 22M 12M kqread 6 3:32 1.05% /usr/local/sbin/lighttpd_pfb -f /var/unbound/pfb_dnsbl_lighty.conf 29667 unbound 20 0 1771M 1655M kqread 2 0:49 0.90% /usr/local/sbin/unbound -c /var/unbound/unbound.conf{unbound} 4206 root 20 0 29M 16M select 5 32:22 0.83% /usr/local/sbin/openvpn --config /var/etc/openvpn/server3/config.ovpn{openvpn} 29667 unbound 20 0 1771M 1655M kqread 4 0:47 0.60% /usr/local/sbin/unbound -c /var/unbound/unbound.conf{unbound} 29667 unbound 20 0 1771M 1655M kqread 3 2:03 0.57% /usr/local/sbin/unbound -c /var/unbound/unbound.conf{unbound} 0 root -60 - 0B 2368K - 6 98:06 0.54% [kernel{if_io_tqg_6}] 29667 unbound 20 0 1771M 1655M kqread 5 1:24 0.54% /usr/local/sbin/unbound -c /var/unbound/unbound.conf{unbound} 12 root -60 - 0B 288K WAIT 0 1:44 0.36% [intr{swi1: netisr 7}] 29667 unbound 20 0 1771M 1655M kqread 5 1:59 0.29% /usr/local/sbin/unbound -c /var/unbound/unbound.conf{unbound} 2 root -60 - 0B 128K WAIT 0 54:04 0.22% [clock{clock (0)}] 69491 root 20 0 26M 11M select 3 12:09 0.21% /usr/local/sbin/openvpn --config /var/etc/openvpn/server12/config.ovpn{openvpn} 29667 unbound 20 0 1771M 1655M kqread 6 2:02 0.21% /usr/local/sbin/unbound -c /var/unbound/unbound.conf{unbound} 29667 unbound 20 0 1771M 1655M kqread 2 2:13 0.17% /usr/local/sbin/unbound -c /var/unbound/unbound.conf{unbound} 12 root -60 - 0B 288K WAIT 2 3:11 0.16% [intr{swi1: netisr 2}] 7 root -16 - 0B 16K pftm 2 9:05 0.13% [pf purge] 17299 root 20 0 17M 4820K CPU3 3 0:00 0.10% top -HaSP 12 root -60 - 0B 288K WAIT 7 4:23 0.09% [intr{swi1: netisr 4}] 64908 root 20 0 5108M 189M uwait 7 29:25 0.08% /usr/local/bin/telegraf -config=/usr/local/etc/telegraf.conf{telegraf}Edit: How do I get the CPU model from terminal or so?
-
It shows on the dashboard. It's shown in /var/log/dmesg.boot.
sysctl hw.modelshould show it.Nothing shown as limited there. Are you sure you can 10Gbps to that host?
-
@stephenw10
LOL, easy, but sometimes still hidden from men's eyes...Intel(R) Xeon(R) CPU D-1518 @ 2.20GHz 8 CPUs: 1 package(s) x 4 core(s) x 2 hardware threads AES-NI CPU Crypto: Yes (active) QAT Crypto: NoWhen our provider once ran a check (some raw ping flood I guess?) they filled up the 10Gbps.
Also I tried some other iperf3 servers, when they don't report busy, always the same limitation in speed.
Also this is something visible when downloading / uploading from/to a fast server in the net. -
Hmm if you're able to test between two local machines with iperf I would try that first. And if you can confirm they will reach 10Gbps when on the same subnet that eliminates any doubt.
-
@stephenw10 Thanks for getting back to me that quickly!
From a VM on an ESX host to a VM on another ESX host I get ~9Gbps, so looks fine.From or to one of those VMs to the pFsense (I installed iperf3 to test) I get even less, around 2,7Gbps.
This seems odd? To exclude the network behind the pFsense I also once tried to have a PC with a 10Gb fiber card directly attached to the switch that comes right after the pFsense.
Same results though. -
Running iperf to or from pfSense dircetly will always give a lower result. Both because iperf itself uses significant CPU cycles and because pfSense is optimised as a router and not a server.
Test through pfSense between different interfaces if you can.
-
@stephenw10 our setup here is more simple than others perhaps.
Just ix0 for WAN, ix1 for LAN.
Some VLAN interfaces on LAN (ix1) and a slower backup link on igb5 -
Well if you're able to test against a local server on the WAN side that would be a good test.
-
@stephenw10 I should probably enable post notifications...
The only way here would be for me to disconnect the WAN interface from the modem and attach a server to it?Only other way could be if the provider modem has another 10Gb port and I connect a server there, then give it another public IP we still have. But then traffic would already passing through the modem (which should certainly support 10Gb as the provider uses it for lots of customers)
-
Hmm, you could use a switch on the WAN side but it would obviously have to be 10G capable.
That CPU is probably somewhere between the 1537 and 1541 for routing that traffic. I'd expect to see somewhere around 5-6Gbps with pf and NAT enabled.
-
@stephenw10
What do you mean with between 1537 and 1541?
The Intel(R) Xeon(R) CPU D-1518 ?So you think the limiting factor here could be the CPU?
Would we need to get another Netgate certified appliance here to get the full 10Gb?
-
It doesn't need to be a Netgate appliance but you might need some other device to pass 10Gbps.
However your top output above does not show any CPU core at 100%. It appears there is some other limit in play here before it exhausts the CPU.
-
@stephenw10
That's a tough one then?
How could I figure out what is limiting here? -
Well we did see one user report nearly 25Gbps using that CPU with Mellanox NICs: https://forum.netgate.com/post/1119611
I still find that hard to believe though.
-
@stephenw10
Interesting!
I am wondering, should I try getting such a Mellanox card?
Or should we be fine achieving 10Gbps with a 6100 for example?
That would be fine to buy I guess... -
The C3558 in the 6100 will not pass 10Gbps either. Again there are a lot of variables but I'd expect to see something in the 3-4Gbps range for an equivalent config.
-
@stephenw10
So to get 10Gbps we would need much bigger hardware then?
I thought the 6100 could be an option, looking at this:
-
That is a total throughput value though. You won't see that in an iperf3 test across an single link.
Yes you need significant CPU power if you want to pass 10Gbps using a single TCP stream like downloading a file.
-
@stephenw10 what appliance would you suggest here?
Not sure if budget allows me to get another PFsense, but I could ask
Otherwise, building something could be an option?
What specs would you recommend to have at least? -
For a single TCP stream like that I would look for fewer cores with a higher frequency. And something newer is obviously more efficient so some relatively recent i3 or i5 for example.
There a few threads discussing specific options.
