Load updated Intel IX module to get 10Gbps
-
@Patch any idea?
-
The fastest one you can get! If you need to pass a single stream TCP connection through it at close to 10Gbps at least. Like sending backups to S3.
-
@stephenw10 What do you mean with the fastest?
One with the fastest possible CPU?Also I am wondering if the problem is really due to CPU clock / single stream? If I run multiple tests / transfers at the same time to different hosts, those tests will share those ~5Gbit...
? -
Well in a test like that I'd expect to see 7-8Gbps through the D1541 so seeing 5Gbps with a D1518 is not wildly low.
But as I said we have seen reports of dramatically higher throughput using other NICs. I've not tested that myself to confirm though.
-
@stephenw10
I guess I figured out what HW we have here:
Must be one of those guys, except we don't have the 4 port Ethernet card in:
https://www.newegg.com/supermicro-sys-5018d-fn8t-intel-xeon-processor-d-1518-2-2-ghz-cpu-tdp-support-35w-fcbga-1667/p/370-0003-000G9Also now I am thinking: If the Mellanox cards might give us better throughput, I could get a MCX4121A-ACAT which as of this list: https://www.freebsd.org/releases/12.1R/hardware/#support
is supported officially by FreeBSD?
I found that card for ~230 bucks :)You think it's worth a try?
Best regards!
-
@ogghi said in Load updated Intel IX module to get 10Gbps:
MCX4121A-ACAT
We have seen a number of reports of issues with that card specifically so, no, I wouldn't get that one. The user who reported getting close to 25Gbps was using a ConnectX-5 NIC.
-
@stephenw10
All right, I'll try and get a Mellanox MCX512A-ACAT aka ConnectX-5 EN and report back :)Thanks!
-
I will say that I've never tested that myself. It would be good to get a second test with it though as that first result was very surprising.
-
I ran a Xeon D-1518 based pfSense system up until recently and the best performance I saw, if I recall correctly, was around ~6-7Gbit/s routing traffic between two different internal network segments (no NAT, no IDS/IPS) via an iperf3 test (single stream). I imagine with NAT in the picture, performance through WAN would have been a bit lower than that.
-
With the included ix NICs I assume? That's about what I'd expect. Which is why the reports of 25Gbps with Mellanox NICs are so surprising.
-
@stephenw10
hi there, I got the card.
How would I go to install it actually?
Install the card, change LAN setting to one of the new ports, apply, switch over cables?
Then do the same for WAN? -
Yes, pretty much exactly that. Just switch the interface assignments to the new NICs.
-
@stephenw10
Hi there!
I was not successful here this morning, had to revert and restore backup. Seems the routes were not updating, same for firewall rules.
All back running. What I saw as warning output above in the webUI made me think the issue with the limitation might be something else:Filter Reload There were error(s) loading the rules: pfctl: interface ix0 bandwidth limited to 4294967295 bps because selected scheduler is 32-bit limited - The line in question reads [0]: @ 2024-03-07 07:38:45So I tried disabling traffic shaper completely.
No change though.But most important, what did go wrong with interface change? I tried in UI to re-assign, also from cmd locally...
-
Ah, looks like you have some limiters set on that and the link speed is above what it can handle. Which itself is interesting.
What traffic shaping do you have enabled there?
Do you have an interface with the bandwidth set as a percentage?
-
@stephenw10
I disabled all traffic shaper queues this morning to test after I saw that output.
No change so far, but I haven't rebooted since, is it required? -
No you should not need to reboot.
If you reload the ruleset in Status > Filter Reload does it regenerate the error?
If so that value must still be in the config somewhere.
-
@stephenw10 said in Load updated Intel IX module to get 10Gbps:
With the included ix NICs I assume? That's about what I'd expect. Which is why the reports of 25Gbps with Mellanox NICs are so surprising.
Actually, this was mostly tested between two interfaces on a Chelsio T540-SO-CR expansion card. I do recall testing between the Chelsio interfaces and onboard ix interfaces at a time or two as well and seeing similar speeds (i.e. no major increases or decreases).
-
@stephenw10
I tried to do the reload while tail -f on /var/log/system.log
I only saw:
Mar 11 14:40:29 vm12 check_reload_status[331]: Reloading filterThat's where it had logged the error before I bet?
-
Yes it would show in the system log. It would also show on the filter reload page if the error was regenerated.
-
@stephenw10 So no errors here :)
But problem not solved either.https://hastebin.com/share/visalixawa.bash
I am confused why it still generates ALTQ queues?
Anyway, because of a power outage the firewall might reboot tonight (unless UPS holds up long enough)
