Netgate 1537 cannot achieve 10G throughput
-
Hello Netgate Community,
Our organization has purchased an Netgate 1537 and during recent testing in preparation for a 10G Internet installation we noticed that it would not achieve greater than 4G Firewall/VLAN throughput with iPerf3 either into/out of same LAN/VLAN 10G/SFP+ interface (router on a stick) or across WAN & LAN 10G/SFP+ interfaces. I have even tested added a third SPF+ interface with exactly the same results.
Test Details:
- iPerf3 Client on VLAN 10.40.1.200
- iPerf3 Server on VLAN 10.5.0.6
- iPerf3 Client command: iperf3 -t 300 -c 10.5.0.6
- Client & Server can achieve 10G when on same VLAN/subnet
- top command output: see below
- iperf output: see below
pfSense Details:
- Netgate 1537 32Gb pfSense Plus 23.09.1
- Kernel PTI: Disabled
- MDS Mitigation: Inactive
- net.inet.ip.intr_queue_maxlen: 4096 (System Tunables)
- kern.ipc.nmbclusters="1000000" (loader.conf.local)
- kern.ipc.nmbjumbop="524288" (loader.conf.local)
- No ALTQ
- Limiters enabled, but not on Test VLANs
- Captive portals enable, but not on Test VLANs
- pfBlockerNG; DNSBL only
It is possible to achieve 10G, however I have it use 5 threads (-P 5) in the iperf client which is highly, especially when the 1537 is rated for 18.62 Gbps.
Could someone please advise what further investigations I could perform? Thank you!
Connecting to host pbs01, port 5201 [ 5] local 10.40.1.200 port 40438 connected to 10.5.0.6 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 483 MBytes 4.05 Gbits/sec 1342 3.23 MBytes [ 5] 1.00-2.00 sec 455 MBytes 3.82 Gbits/sec 0 3.38 MBytes [ 5] 2.00-3.00 sec 452 MBytes 3.80 Gbits/sec 0 3.50 MBytes [ 5] 3.00-4.00 sec 454 MBytes 3.81 Gbits/sec 0 3.60 MBytes [ 5] 4.00-5.00 sec 456 MBytes 3.83 Gbits/sec 0 3.66 MBytes [ 5] 5.00-6.00 sec 455 MBytes 3.82 Gbits/sec 0 3.71 MBytes [ 5] 6.00-7.00 sec 455 MBytes 3.82 Gbits/sec 0 3.75 MBytes [ 5] 7.00-8.00 sec 456 MBytes 3.83 Gbits/sec 0 3.84 MBytes [ 5] 8.00-9.00 sec 455 MBytes 3.82 Gbits/sec 0 3.93 MBytes [ 5] 9.00-10.00 sec 455 MBytes 3.82 Gbits/sec 0 4.01 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 4.47 GBytes 3.84 Gbits/sec 1342 sender [ 5] 0.00-10.00 sec 4.44 GBytes 3.82 Gbits/sec receiver iperf Done.last pid: 42821; load averages: 0.76, 0.48, 0.42 754 threads: 18 running, 695 sleeping, 41 waiting CPU: 0.1% user, 0.0% nice, 2.4% system, 6.3% interrupt, 91.2% idle Mem: 142M Active, 2045M Inact, 1308M Wired, 56K Buf, 28G Free ARC: 302M Total, 35M MFU, 255M MRU, 315K Anon, 1714K Header, 9817K Other 224M Compressed, 764M Uncompressed, 3.42:1 Ratio Swap: 1024M Total, 1024M Free PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 11 root 187 ki31 0B 256K CPU11 11 213.0H 100.00% [idle{idle: cpu11}] 11 root 187 ki31 0B 256K CPU7 7 212.3H 100.00% [idle{idle: cpu7}] 11 root 187 ki31 0B 256K CPU13 13 212.3H 100.00% [idle{idle: cpu13}] 11 root 187 ki31 0B 256K RUN 8 208.6H 100.00% [idle{idle: cpu8}] 12 root -56 - 0B 416K CPU12 12 55:13 100.00% [intr{swi1: netisr 3}] 11 root 187 ki31 0B 256K CPU9 9 214.5H 99.42% [idle{idle: cpu9}] 11 root 187 ki31 0B 256K CPU3 3 212.6H 99.33% [idle{idle: cpu3}] 11 root 187 ki31 0B 256K CPU10 10 212.9H 99.31% [idle{idle: cpu10}] 11 root 187 ki31 0B 256K CPU4 4 213.4H 99.08% [idle{idle: cpu4}] 11 root 187 ki31 0B 256K CPU14 14 213.7H 99.01% [idle{idle: cpu14}] 11 root 187 ki31 0B 256K CPU2 2 212.8H 98.93% [idle{idle: cpu2}] 11 root 187 ki31 0B 256K CPU5 5 213.4H 98.92% [idle{idle: cpu5}] 11 root 187 ki31 0B 256K CPU15 15 212.6H 98.76% [idle{idle: cpu15}] 11 root 187 ki31 0B 256K CPU1 1 214.7H 96.97% [idle{idle: cpu1}] 11 root 187 ki31 0B 256K CPU0 0 208.6H 89.68% [idle{idle: cpu0}] 11 root 187 ki31 0B 256K RUN 6 212.7H 64.26% [idle{idle: cpu6}] 0 root -60 - 0B 3424K - 6 43:23 35.60% [kernel{if_io_tqg_6}] 0 root -60 - 0B 3424K - 0 125:04 9.77% [kernel{if_io_tqg_0}] 12 root -60 - 0B 416K WAIT 14 205:32 0.88% [intr{swi1: netisr 2}] 18880 unbound 20 0 1985M 1785M kqread 8 65:38 0.54% /usr/local/sbin/unbound -c /var/unbound/unbound.conf{unbound} 12 root -60 - 0B 416K WAIT 9 142:10 0.53% [intr{swi1: netisr 13}] 0 root -64 - 0B 3424K - 3 358:44 0.51% [kernel{dummynet}] 46954 root 21 0 112M 49M accept 9 0:11 0.50% php-fpm: pool nginx (php-fpm) 0 root -60 - 0B 3424K - 4 115:56 0.42% [kernel{if_io_tqg_4}] 12 root -60 - 0B 416K WAIT 8 113:39 0.37% [intr{swi1: netisr 8}] 18880 unbound 20 0 1985M 1785M kqread 13 56:38 0.34% /usr/local/sbin/unbound -c /var/unbound/unbound.conf{unbound} 18880 unbound 20 0 1985M 1785M kqread 14 66:22 0.34% /usr/local/sbin/unbound -c /var/unbound/unbound.conf{unbound} 0 root -60 - 0B 3424K - 10 100:59 0.30% [kernel{if_io_tqg_10}] 18880 unbound 20 0 1985M 1785M kqread 15 40:43 0.30% /usr/local/sbin/unbound -c /var/unbound/unbound.conf{unbound} 2830 root 21 0 112M 49M accept 10 0:18 0.25% php-fpm: pool nginx (php-fpm) 0 root -60 - 0B 3424K - 0 0:01 0.22% [kernel{softirq_0}] 7 root -16 - 0B 16K pftm 2 24:31 0.15% [pf purge] 18880 unbound 20 0 1985M 1785M kqread 1 30:44 0.12% /usr/local/sbin/unbound -c /var/unbound/unbound.conf{unbound} 65112 root 20 0 17M 5304K CPU8 8 0:00 0.12% top -aSH 18880 unbound 20 0 1985M 1785M kqread 3 58:03 0.10% /usr/local/sbin/unbound -c /var/unbound/unbound.conf{unbound} 18880 unbound 20 0 1985M 1785M kqread 7 45:37 0.10% /usr/local/sbin/unbound -c /var/unbound/unbound.conf{unbound} -
@darnoldvcs The 1537 will not do single stream/session 10Gbe throughput. I don’t actually think there is any netgate device that will do that.
It would need about 4 streams to achieve that kind of throughput, and any inspection packages like snort/suricata will make it impossible to reach sustained 10Gbe. -
In normal usage you may be able to see often 2 GBit/s - 4 Gbit/s
ordinary throughput, you may be able to get 4 GBit/s - 7 GBit/s
under heavy load and testing with iperf you will be able to see
also something reaching 9.2/9.4/9.6 GBit/s.- It is based on the art of the traffic
- It is based on the receiving device (CPU&RAM&SSD)
- It is base on the cables and the length to the end point
- It is often based on the switches there are in game!!!
The used chips I mean here
-
@darnoldvcs what iperf version were you using the latest 3.16 added multithreading support.. Before tests were limited to single thread.
Well yes the specs list like 18Ge, where on those specs does it say for 1 session from 1 client?
The thing has only 10ge ports.. So clearly if lists more than 10 ge in routing is via multiple interfaces at the same time, and more then 1 client using 1 connection, etc.
Also while testing, I would make when testing to vlans your not hairpinning the connection..
If you can get 10ge with parallel connections then clearly the box can route at 10ge, etc.
-
@keyser, @Dobby_ & @johnpoz Thank you for your input.
I was hoping it could achieve 10G single threaded and perhaps there was a configuration issue. Our team built out a test pfSense from PC spare parts; an old Intel Gen 9 i7 and was able to saturate 10G, cross vlan, single threaded, configured with a single LAN NIC (router on a stick). I guess that processor has more grunt that the Xeon D 1537.
I can achieve 10G with 5 threads on the 1537, and I appreciate I would need more interfaces to achieve the marketing numbers of 18+GB.
Again thank you for your input and guidance.
-
@darnoldvcs said in Netgate 1537 cannot achieve 10G throughput:
I guess that processor has more grunt that the Xeon D 1537.
They should be hard enough for the 10 GBit/s also Supermicro is
selling devices with such SOC´s and soldering 4 * 10 GBit/s port
on it.I can achieve 10G with 5 threads on the 1537, and I appreciate I would need more interfaces to achieve the marketing numbers of 18+GB.
The 18+GB would in my eyes the plain routing power! Please don´t mismatch
with firewall rules on top or vpn traffic.
