Load updated Intel IX module to get 10Gbps

Patch

@ogghi said in Load updated Intel IX module to get 10Gbps:

@stephenw10
So to get 10Gbps we would need much bigger hardware then?
I thought the 6100 could be an option

What traffic mix is the operational requirement? More specifically

WAN type
VPN connections and load
typical number of concurrent connections, especially high bandwidth connections.

The reason I ask is I wonder how well the test results correlate with the operational performance under likely load conditions.

ogghi

@Patch
Hi there.

WAN Type is Init7 10/10Gbps fiber.
VPN connections (speed not relevant here) maximum of 20 ppl on OpenVPN, 3 WireGuard tunnels.

pfTop: Up State 1-100/14500
Are those concurrent connections?

High bandwidth usually would be the backup server sending backups to S3 storage outside...otherwise not too much high bandwidth things happening.

ogghi

@Patch any idea?

stephenw10

The fastest one you can get! If you need to pass a single stream TCP connection through it at close to 10Gbps at least. Like sending backups to S3.

ogghi

@stephenw10 What do you mean with the fastest?
One with the fastest possible CPU?

Also I am wondering if the problem is really due to CPU clock / single stream? If I run multiple tests / transfers at the same time to different hosts, those tests will share those ~5Gbit...
?

stephenw10

Well in a test like that I'd expect to see 7-8Gbps through the D1541 so seeing 5Gbps with a D1518 is not wildly low.

But as I said we have seen reports of dramatically higher throughput using other NICs. I've not tested that myself to confirm though.

ogghi

@stephenw10
I guess I figured out what HW we have here:
Must be one of those guys, except we don't have the 4 port Ethernet card in:
https://www.newegg.com/supermicro-sys-5018d-fn8t-intel-xeon-processor-d-1518-2-2-ghz-cpu-tdp-support-35w-fcbga-1667/p/370-0003-000G9

Also now I am thinking: If the Mellanox cards might give us better throughput, I could get a MCX4121A-ACAT which as of this list: https://www.freebsd.org/releases/12.1R/hardware/#support
is supported officially by FreeBSD?
I found that card for ~230 bucks :)

You think it's worth a try?

Best regards!

stephenw10

@ogghi said in Load updated Intel IX module to get 10Gbps:

MCX4121A-ACAT

We have seen a number of reports of issues with that card specifically so, no, I wouldn't get that one. The user who reported getting close to 25Gbps was using a ConnectX-5 NIC.

ogghi

@stephenw10
All right, I'll try and get a Mellanox MCX512A-ACAT aka ConnectX-5 EN and report back :)

Thanks!

stephenw10

I will say that I've never tested that myself. It would be good to get a second test with it though as that first result was very surprising.

tman222

I ran a Xeon D-1518 based pfSense system up until recently and the best performance I saw, if I recall correctly, was around ~6-7Gbit/s routing traffic between two different internal network segments (no NAT, no IDS/IPS) via an iperf3 test (single stream). I imagine with NAT in the picture, performance through WAN would have been a bit lower than that.

stephenw10

With the included ix NICs I assume? That's about what I'd expect. Which is why the reports of 25Gbps with Mellanox NICs are so surprising.

ogghi

@stephenw10
hi there, I got the card.
How would I go to install it actually?
Install the card, change LAN setting to one of the new ports, apply, switch over cables?
Then do the same for WAN?

stephenw10

Yes, pretty much exactly that. Just switch the interface assignments to the new NICs.

ogghi

@stephenw10
Hi there!
I was not successful here this morning, had to revert and restore backup. Seems the routes were not updating, same for firewall rules.
All back running. What I saw as warning output above in the webUI made me think the issue with the limitation might be something else:

Filter Reload
There were error(s) loading the rules: pfctl: interface ix0 bandwidth limited to 4294967295 bps because selected scheduler is 32-bit limited - The line in question reads [0]: @ 2024-03-07 07:38:45

So I tried disabling traffic shaper completely.
No change though.

But most important, what did go wrong with interface change? I tried in UI to re-assign, also from cmd locally...

stephenw10

Ah, looks like you have some limiters set on that and the link speed is above what it can handle. Which itself is interesting.

What traffic shaping do you have enabled there?

Do you have an interface with the bandwidth set as a percentage?

ogghi

@stephenw10
I disabled all traffic shaper queues this morning to test after I saw that output.
No change so far, but I haven't rebooted since, is it required?

stephenw10

No you should not need to reboot.

If you reload the ruleset in Status > Filter Reload does it regenerate the error?

If so that value must still be in the config somewhere.

tman222

@stephenw10 said in Load updated Intel IX module to get 10Gbps:

With the included ix NICs I assume? That's about what I'd expect. Which is why the reports of 25Gbps with Mellanox NICs are so surprising.

Actually, this was mostly tested between two interfaces on a Chelsio T540-SO-CR expansion card. I do recall testing between the Chelsio interfaces and onboard ix interfaces at a time or two as well and seeing similar speeds (i.e. no major increases or decreases).

ogghi

@stephenw10
I tried to do the reload while tail -f on /var/log/system.log
I only saw:
Mar 11 14:40:29 vm12 check_reload_status[331]: Reloading filter

That's where it had logged the error before I bet?