SG-3100 slow not getting gigabit.

stephenw10

It's interrupt load from the NIC.

Are you running iperf3 on the 3100 directly? Or that's just when running through it?

A number of things will appear as interrupr load like that, notably pf. So if you have a very large number of rules or traffic shaping or maybe something complex in the ruleset somehow that's where it will show when loaded.

Steve

msf2000

@stephenw10

I'm running iperf between 2 nodes on different vlans... i.e., using the SG-3100 as a router/firewall only. With that I'm still maxed out at 480Mbps.... If I turn Suricata back on, it drops down to ~450. :(

stephenw10

VLANs on the same interface? Try between different NICs if you can.

msf2000

@stephenw10
That's brilliant... OK, with using separate interfaces for VLANs, I was able to get 760Mbps with iperf. Still significantly shy of advertised performance, but probably as good as the current network design can sustain (i.e., using a single trunk port).

Also, it's the same thing (different PID/NIC) that maxes out the CPU on the SG-3100....
[intr{mpic0: nmvneta0}]
[intr{mpic0: nmvneta1}]

david_moo

@msf2000
I think we need more of an explanation.....
If I am understanding correctly we have:
vlan #1 -> port 1 -> SG-3100 -> port 2 -> vlan #2.

If that is the case, the the SG-3100 is routing in a very standard way and should be pushing in/out 940Mbps (max for a 1Gbit port) . It's not doing that, why? Can the SG-3100 not handle it?

stephenw10

If both VLANs are using the switch ports they are sharing a single parent NIC.
The mvneta NIC/driver is single queue so only one CPU core can service it in any direction.
If you test between a VLAN on LAN and a VLAN on OPT, for example, you are using two NICs and hence two queues that both CPU cores can service.
I would not expect anything to have changed there between 2.4.5 and 21.0X.

Steve

msf2000

@stephenw10

The 760Mbps figure was routing between OPT1 and a LAN port. CPU was maxed with the nmvneta0 & 1 taking all of a core each.
I.e., this was my test setup:
Linux node 1 --> vlan #1 --> port 1 --> sg-3100 --> opt1 --> vlan 2 --> Linux node 2

david_moo

@msf2000
Does it make sense (if possible) to try the same setup with no vlans? I really feel you should be in the ~940Mbps region (full speed 1Gbps) with a simple setup.

some guy on the internet states:
it already known that the SG-3100 can’t do full gig speed over VLANs

stephenw10

He's actually posing that as a question there. I wouldn't expect it to make that much difference I agree. Unless you were doing something like VLAN0 where everything has to go through netgraph. Or maybe the additional 8 bytes on the packet is somehow causing fragmentation.

A sanity check test here shows some reduction in throughput when the LAN is configuered with a VLAN to one of the switch ports.
Testing in 21.09 I see 936/919Mbps LAN to WAN without any VLAN tagging. And 938/831 with the client on a LAN side VLAN.

That's local iperf3 testing with a single process.

Steve

msf2000

@stephenw10

Re-tested same setup. Disabled Suricata this time, and my best time was 815 Mbps with Iperf. The CPU limit was only on mvneta0 this time (as mvneta0 hit 100% but mvneta1 hit 70% range).

Not quite gigabit range, but that is real-world.

ashlm

I also get only ~650Mb/s down from both fast.com and speedtest.net on my SG-3100. Ended up buying a second to redo the config in a sterile, factory default environment and came up with the same result. This is traffic in through WAN, out LAN1, with no shaping or filtering packages installed. Haven't hooked it up to an iperf server yet, but I'm not really interested in the synthetic load throughput. I get 1.2Gb down from my ISP, again from fast.com and speedtest.net, when directly connected to the CPE onboard LAN switchports. I don't see why this router should be halving that.

Edit: Changed my config slightly so I've three VLANs going through LAN1 as a trunk, set LAN4 as a single VLAN port for my "priority" clients (through which testing for 1Gb is performed), DOCSIS 3.1 service on WAN, 80Gb PPPoE failover on member down on OPT1. No improvement.

ashlm

@ashlm Following on...
top -aSH shows mvnet02 (WAN) hitting over 98% utilisation of CPU core, bufferbloat kicks in after around 30 seconds of load and packets start being dropped. 665Mb peak downloading a Steam game (interface dropped completely and failed over once during 5 minutes download).