6100 10g port and vlans maxing at 1g speed
-
@stephenw10 said in 6100 10g port and vlans maxing at 1g speed:
top -HaSP
Looks like at least two core get pegged. Ran a test on two hosts between 10.15.1.0/24 and 10.15.100.0/24
/root: top -HaSP last pid: 53556; load averages: 1.27, 0.75, 0.55 up 1+21:18:12 22:35:13 656 threads: 8 running, 631 sleeping, 17 waiting CPU 0: 0.0% user, 0.0% nice, 100% system, 0.0% interrupt, 0.0% idle CPU 1: 0.0% user, 0.0% nice, 100% system, 0.0% interrupt, 0.0% idle CPU 2: 1.2% user, 0.0% nice, 51.0% system, 0.0% interrupt, 47.8% idle CPU 3: 1.2% user, 0.0% nice, 2.7% system, 0.0% interrupt, 96.1% idle Mem: 297M Active, 570M Inact, 715M Wired, 6225M Free ARC: 357M Total, 88M MFU, 265M MRU, 32K Anon, 1149K Header, 3206K Other 132M Compressed, 291M Uncompressed, 2.20:1 Ratio -
Hmm, you see that load with only ~500Mbps passing?
Does it show what process is using that in the full top output? ntop-ng perhaps?
-
@stephenw10 said in 6100 10g port and vlans maxing at 1g speed:
Hmm, you see that load with only ~500Mbps passing?
Does it show what process is using that in the full top output? ntop-ng perhaps?
It looks like it is if_io_tqg ... not sure what that is?
I ensured all SNMP is disabled, I uninstalled ntop-ng, still getting less than 1Gbit/secPID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 0 root -76 - 0B 736K CPU2 2 191:10 99.79% [kernel{if_io_tqg_2}] 0 root -76 - 0B 736K CPU3 3 190:40 99.79% [kernel{if_io_tqg_3}] 0 root -76 - 0B 736K - 0 223:36 95.08% [kernel{if_io_tqg_0}] 0 root -76 - 0B 736K - 1 182:50 72.32% [kernel{if_io_tqg_1}] 11 root 155 ki31 0B 64K RUN 1 57.7H 19.83% [idle{idle: cpu1}] -
Those are the NIC driver queues which is where the load for routing and filtering should appear.
But you should easily be able to pass 1Gbps there.Do you see the same restriction between other interfaces?
-
@stephenw10 I'm only using one 10g interface. I'll try the 2nd one and report back.
-
Learning a lot watching this thread.
That said, I'm not expecting more than 620ish kpps. Would be happy to be proven wrong!
https://ipng.ch/s/articles/2021/11/26/netgate-6100.html -
@dnavas
interesting! A lot of that post is beyond my level of expertise. But is if fair to infer from your findings that you never got full 10gb speed when routing across networks? -
The 6100 will not pass 10Gbps. There are many variables but I expect to see in the 3-4Gbps between the two 10G NICs.
But you are seeing a restriction at a far lower level. Even given the single TCP stream and tha5t it's between VLANs on the same NIC I expect to see more.
I'm setting up my own test now... -
@stephenw10 said in 6100 10g port and vlans maxing at 1g speed:
The 6100 will not pass 10Gbps.
Interesting
Does that mean, in the context of this marketing language: IPERF3 Traffic: 18.50 Gbps, 18.50 Gpbs only refers to LAN <-> WAN? I mean, that'd be bottlenecked by the 10g ports, right? So is that some sort of WAN LAGG setup?What about this: IPERF3 Traffic: 9.93 Gbps does that just mean LAN <-> WAN with firewalls rules? Not routing across subnets?
-
The 18.5Gbps figure is a total throughput (all interfaces) value for large packets (iperf, 1500B) of forwarding traffic. Without filtering.
For a single TCP stream when hairpinned on the same interface using VLANs you hit the additional complication of loading the queues/cores. You will probably find you see different results if you repeat the test with a single iperf stream. The throughput is better when the send and receive queues fall to different CPU cores. Testing with multiple streams avoid that, I usually use
-P 4since it's a 4 core CPU.This is the loading I see when testing between VLANs on the ix0 port:
last pid: 81510; load averages: 1.36, 0.81, 0.56 up 16+18:48:27 18:33:43 670 threads: 10 running, 623 sleeping, 4 zombie, 33 waiting CPU 0: 0.4% user, 0.0% nice, 20.4% system, 15.7% interrupt, 63.5% idle CPU 1: 2.7% user, 0.0% nice, 1.2% system, 67.8% interrupt, 28.2% idle CPU 2: 7.5% user, 0.0% nice, 5.1% system, 58.4% interrupt, 29.0% idle CPU 3: 0.4% user, 0.0% nice, 10.2% system, 19.2% interrupt, 70.2% idle Mem: 1233M Active, 211M Inact, 1230M Laundry, 761M Wired, 4272M Free ARC: 357M Total, 242M MFU, 106M MRU, 296K Anon, 1610K Header, 6869K Other 117M Compressed, 291M Uncompressed, 2.48:1 Ratio Swap: 1024M Total, 364M Used, 660M Free, 35% Inuse PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 12 root -72 - 0B 560K CPU3 3 3:41 81.58% [intr{swi1: netisr 0}] 12 root -72 - 0B 560K CPU0 0 2:47 72.43% [intr{swi1: netisr 3}] 11 root 155 ki31 0B 64K RUN 3 385.9H 70.77% [idle{idle: cpu3}] 11 root 155 ki31 0B 64K RUN 0 387.1H 66.14% [idle{idle: cpu0}] 11 root 155 ki31 0B 64K RUN 1 385.7H 32.37% [idle{idle: cpu1}] 11 root 155 ki31 0B 64K RUN 2 386.0H 30.00% [idle{idle: cpu2}] 0 root -76 - 0B 960K CPU0 0 1:40 17.55% [kernel{if_io_tqg_0}] 0 root -76 - 0B 960K - 3 1:06 11.07% [kernel{if_io_tqg_3}]That's between two 1G clients but with the port linked to a switch at 10G.
It passes 1G as expected:[ ID] Interval Transfer Bitrate [ 5] 0.00-60.00 sec 1.60 GBytes 230 Mbits/sec receiver [ 8] 0.00-60.00 sec 1.65 GBytes 237 Mbits/sec receiver [ 10] 0.00-60.00 sec 1.62 GBytes 232 Mbits/sec receiver [ 12] 0.00-60.00 sec 1.62 GBytes 232 Mbits/sec receiver [SUM] 0.00-60.00 sec 6.50 GBytes 930 Mbits/sec receiverntop-ng is also running on that box but not on either interface in the test.
Steve
-
@stephenw10
I do see a difference with -P 4 vs -P 14 streams I get about 1.5Gbits/sec and with 1 stream I get 650Mbits/sec.
Does that lead us to conclude that, when using a single 10g interface for VLANs, I will never get more than about 1.5 Gbits/sec because of processor constraints?
Would I see better performance if I put some of the vLANS on ix1 (the other 10g interface)?
-
Yes, I would expect to see better performance routing between different NICs. Especially for single TCP connections.
One interesting thing though is that your top output does not appear to show the interrupt load like mine does. It could just be missing from your screenshot but that would also imply is using less CPU time than the NIC queues unlike in my test. I wonder if you have something else running that appears as load there. Traffic shaping maybe?
Steve
-
@stephenw10 here's the full output:
from 10.15.1.111/24
command:
iperf3 -c 10.15.100.18 -P 4last pid: 30212; load averages: 1.42, 0.65, 0.48 up 3+21:37:16 22:54:17 649 threads: 8 running, 624 sleeping, 17 waiting CPU 0: 0.0% user, 0.0% nice, 100% system, 0.0% interrupt, 0.0% idle CPU 1: 0.0% user, 0.0% nice, 100% system, 0.0% interrupt, 0.0% idle CPU 2: 0.0% user, 0.0% nice, 57.3% system, 0.0% interrupt, 42.7% idle CPU 3: 0.0% user, 0.0% nice, 100% system, 0.0% interrupt, 0.0% idle Mem: 209M Active, 515M Inact, 751M Wired, 6332M Free ARC: 368M Total, 90M MFU, 272M MRU, 32K Anon, 1218K Header, 4383K Other 136M Compressed, 305M Uncompressed, 2.24:1 Ratio PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 0 root -76 - 0B 736K CPU1 1 251:45 99.85% [kernel{if_io_tqg_1}] 0 root -76 - 0B 736K CPU3 3 282:09 99.85% [kernel{if_io_tqg_3}] 0 root -76 - 0B 736K CPU0 0 316:09 99.75% [kernel{if_io_tqg_0}] 0 root -76 - 0B 736K - 2 259:37 58.90% [kernel{if_io_tqg_2}] 11 root 155 ki31 0B 64K RUN 2 83.9H 39.69% [idle{idle: cpu2}] 0 root -92 - 0B 736K - 0 12:48 0.39% [kernel{dummynet}] 16464 root 20 0 15M 5916K CPU2 2 0:00 0.16% top -HaSP 3134 root 20 0 19M 8156K select 2 0:08 0.12% /usr/local/sbin/openvpn --config /var/etc/openvpn/server21/co 387 root 20 0 12M 3120K bpf 2 11:26 0.10% /usr/local/sbin/filterlog -i pflog0 -p /var/run/filterlog.pid 4703 avahi 20 0 13M 4152K select 2 6:30 0.09% avahi-daemon: running [washington.local] (avahi-daemon) 1183 root 20 0 16M 7516K select 2 4:54 0.08% /usr/local/sbin/openvpn --config /var/etc/openvpn/client15/co 97937 root 20 0 11M 2816K select 2 18:30 0.08% /usr/sbin/syslogd -s -c -c -l /var/dhcpd/var/run/log -P /var/ 0 root -76 - 0B 736K - 2 3:28 0.06% [kernel{if_config_tqg_0}] 24 root -16 - 0B 16K - 2 3:12 0.04% [rand_harvestq] 72462 root 20 0 17M 7564K select 2 1:40 0.04% /usr/local/sbin/openvpn --config /var/etc/openvpn/client14/co 12 root -60 - 0B 272K WAIT 2 2:02 0.04% [intr{swi4: clock (0)}] -
Hmm, interesting. The 6100 I tested with is a test device I use for many things, it has a lot of config on it. I'll have to default it tomorrow and retest.
I'll try to get some results from 10G clients too.Steve
-
@stephenw10 thanks for all the help and testing!
-
@spacebass I don't recall, tbh. The article would seem to indicate not, but in reality most of my early connectivity issues were due to the bridge. I have just moved my data plane (servers) from my management plane (switches/gateways/etc), so I can retest across vlans easier.
On my 6100 w/ MTU of 1500, I'm getting a peak of about 850Mbps with one thread and 2.5Gbps with four. With an MTU of 9000 I'm getting a peak of 4.4Gbps with one thread and 9.91Gbps with four (I think it's pretty close to saturated with just two).
-
@stephenw10 just curious if you had done any more testing?
I'm still only getting about 1.5-2gbs across vLANS.I get that I'm hairpinned since all my vLANs go through a single 10g port - is that going to be the ultimate constraint?
Short of moving to an L3 switch, what else might I be able to do or try? I need the other 10g port for my 10g WAN connection.
I guess that also makes me wonder why the device is marketed as doing L3 forwarding at 18.50Gbps - does that assume an instance where there are only two subnets, each on one of the two 10g ports?
-
That's the maximum forwarding performance across all interfaces combined. That's also without filtering which is the biggest overhead you will hit there. 2Gbps is low though. I'd expect to see 3-4Gbps at least. There are a lot of variables there however.
I didn't have a chance to test the 10G port directly yet. I do have a new workstation that can test at those speeds far more easily so I should be able to get some numbers soon.
Steve
