Throughput problems on 4100

ahxcjay

@steveits said in Why is the 4100 performance such trash?:

@ahxcjay Are you testing through the VPN?

Nope. I have Wireguard and IPsec enabled (with just as many tunnels as ER-4 has [5])) but only very specific traffic goes through those links.

Is hardware acceleration enabled? What settings?

I haven't touched the defaults:

Per Netgate's store page:
IPsec VPN
(AES-GCM-128 w/QAT)

IPERF3 Traffic: 960 Mbps
IMIX Traffic: 312 Mbps

..those 'tests' are junk, IMO. Did you see how they performed them? They literally bond 4 interfaces together. That is not even remotely 'real world'.

What gets me most about my issue is that I don't even see the system (single CPU or otherwise) go under any load. It looks like it has plenty of CPU cycles to spare, but here we are. I'm going to disable ALTQ support (probably enabled when I tried out the traffic shaper) and see what that does.

ahxcjay

@ahxcjay ..disabling ALTQ has zero effect. I can still only upload to google drive / youtube at ~200Mbit/sec. The moment I switch my cables back to the ER-4, I'm back up to ~500Mbit/sec. Infuriating.

I want to like this product, I really like some of the features, but it's letting me down on the core fundamentals.

Here's my software package list:

I have no interest in DPI etc..so that's not chugging my perf.

If I open 2x upload streams in the same browser to GDrive & YouTube my upload speed essentially doubles:

.. I just can not get a single upload to sustain past 200-250Mb/sec. A simple cable swap to ER4 and an individual upload is at 500+.

SteveITS

@ahxcjay said in Why is the 4100 performance such trash?:

Is hardware acceleration enabled? What settings?

I haven't touched the defaults

I meant the crypto. But that's irrelevant if you're not going through the VPN. (I was looking for, on the dashboard in the CPU Type section what are the "Crypto" entries? The config settings are under System/Advanced/Miscellaneous.)

Speed/duplex correct? Change patch cables? Small switch between the 4100 and ISP router? It may seem weird but those things come up on the forum reasonably often.

Edit: hmm, that doesn't make a lot of sense either if multiple streams are way faster. Does it change if you disable the traffic monitoring?

We have a client with a 4100 but they have a ~350 Mbps download.

ahxcjay

@steveits I meant the crypto. But that's irrelevant if you're not going through the VPN. (I > was looking for, on the dashboard in the CPU Type section what are the "Crypto" entries?

QAT Crypto: Yes (active)
Hardware crypto AES-CBC,AES-CCM,AES-GCM,AES-ICM,AES-XTS,SHA1,SHA256,SHA384,SHA512

The config settings are under System/Advanced/Miscellaneous.)

What settings should I post from here..?

Speed/duplex correct?

Yes. 1000, no issues.

Change patch cables?

Tried that, and remember, all I need to do is literally swap out the cables to the ER-4 (wan and to my Switch) and the upload speeds to GDrive / YT is in line with expectations.

Small switch between the 4100 and ISP router?

Nope. Direct into patch panel.

It may seem weird but those things come up on the forum reasonably often.

I understand, but isn't this issue beyond ridiculous? Why is pfSense failing on this? My mate told me to avoid pfSense at all costs and I ignored him (he recommend an SRX).

Does it change if you disable the traffic monitoring?

vnstat? Tried disabling that also; makes zero difference.

SteveITS

@ahxcjay said in Why is the 4100 performance such trash?:

Nope. Direct into patch panel.

I'm writing too fast. :) I meant add a switch. But that doesn't make sense if multiple streams are 2x fast because it's not being throttled at that network level.

re: System/Advanced/Miscellaneous, the crypto setting, but never mind, it's not relevant.

One thing you could try is to back up your config, set to factory defaults, and reproduce that way. If it's fast, it's a config issue. Worst case restore your config to get back to where you are now. And can restore parts of the config to test.

ahxcjay

@steveits ..that's one for the weekend.. ;)

All other (single threaded) web upload activities suffer this issue. Dropbox, iCloud..etc. Once I do something multi-threaded, performance is completely fine.

To @keyser - where is your gigabit connection with a 4100 against a single threaded upload via a web browser? I'm all eyes...

stephenw10

What interfaces are you using on the 4100?

Try connecting to command line and running top -HaSP whilst you're uploading.

Is either core hitting 100% load?

What loading from processes is there?

Steve

ahxcjay

@stephenw10

What interfaces are you using on the 4100?

WAN1 & LAN4.

Try connecting to command line and running top -HaSP whilst you're
uploading.

..did that earlier and noticed nothing. Let me try again.

Is either core hitting 100% load?

Barely under any load..

What loading from processes is there?

Hardly anything. See screenshot.

stephenw10

Hmm, that really is almost nothing.

I wonder if there's a flow-control issue there.

I assume WAN and LAN are linked at 1G?

As a test try reassigning the interfaces so both WAN and LAN are using the ix NICs or both using the igc NICs.

When you're testing that is the throughput constant or very 'peaky'?

Steve

ahxcjay

@stephenw10 said in Why is the 4100 performance such trash?:

Hmm, that really is almost nothing.

I wonder if there's a flow-control issue there.

..I would love to know what it is.. :)

I assume WAN and LAN are linked at 1G?

Yep. All good there.

As a test try reassigning the interfaces so both WAN and LAN are using the ix NICs or both using the igc NICs.

Let me go and do that now. I'll use the 'igc' NICs.

When you're testing that is the throughput constant or very 'peaky'?

Pretty constant I would say...

ahxcjay

O M G

IT FIXED IT !

..this is a bug, right?

keyser

@ahxcjay said in Why is the 4100 performance such trash?:

To @keyser - where is your gigabit connection with a 4100 against a single threaded upload via a web browser? I'm all eyes...

Sorry I cannot complete the test for you right now, as I only have SG-4100s at customer sites. My own boxes are SG-2100/SG-6100.
But i distinctly remember doing a single stream test and still seeing a 9xxMbps number on a QinQ line that will do about 940Mbps @ 100%.

I’m happy Steve found the issue (VERY likely flow control), and that’s exactly my point with my first post. People in here are very very helpfull and know A LOT about these boxes, potential issues and what not.
We alle want to help, but it’s not very motivating when posts like yours just flame the product by stating that the specs and performance of these boxes are invented ficticious numbers from netgate.

The problem is that google searches from people also finds this post, and some people only read headlines….

bingo600

@ahxcjay said in Why is the 4100 performance such trash?:

O M G

IT FIXED IT !

..this is a bug, right?

I have read this two times now ....
What fixed it ????

Factory reset or ????

Gertjan

@bingo600 said in Why is the 4100 performance such trash?:

I have read this two times now ....
What fixed it ????

Me 3 times.
And I have a 4100 in front of my.
I've 4 igc0-1-2-3 interfaces, an ix2 and an ix3. The latter two are combo ports :
See here.

I presume the fix was : use WAN on igc0 and all the LANs on igc1, 2 and 3 - not using ix2 and ix3.

stephenw10

Yeah, I read that as using igc as WAN fixed it. In which case it's probably linking differently than ix to whatever is upstream. Potentially with or without flow-control.
You'd have to check the ifconfig output from each to know more.

Steve

ahxcjay

@stephenw10 igc as WAN fixed it. What would cause this behaviour..?

ahxcjay

@gertjan correct. Once I removed ix from the ports used my line is at full speed on all uploads again! Amazing.

Thank you everyone!

stephenw10

@ahxcjay said in Throughput problems on 4100:

What would cause this behaviour..?

I would think it must be something in the link negotiation there. I would check the ifconfig -vv output for each NIC to start with. It 'feels' like a flow control issue. You would see 'rxpause, txpause' in that output. It can be disabled (or enabled) on ix if it is that.

Steve

ahxcjay

@stephenw10 great info. Thank you! I am so happy this is fixed as I love the product.

Gertjan

@stephenw10 said in Throughput problems on 4100:

I would think it must be something in the link negotiation there. I would check the ifconfig -vv output for each NIC to start with. It 'feels' like a flow control issue. You would see 'rxpause, txpause' in that output. It can be disabled (or enabled) on ix if it is that.

For the one and only 'ix' I use as a WAN to my 1Gbits/sec ISP router, I see :

ix3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: WAN
        options=e138bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 90:ec:77:xx:39:2a
        inet6 fe80::92ec:77ff:fe29:392a%ix3 prefixlen 64 scopeid 0x8
        inet 192.168.10.5 netmask 0xffffff00 broadcast 192.168.10.255
        media: Ethernet autoselect (1000baseT <full-duplex,rxpause,txpause>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

In short :

media: Ethernet autoselect (1000baseT <full-duplex,rxpause,txpause>)

for the port that is in use.

This is normal / not normal ?
Btw : I don't mind for now, as I have 23 Mbits/sec down and 2 Mbits/sec up for now.
Gbit fibre is coming at the end of the month.