Load updated Intel IX module to get 10Gbps

[ogghi]

@stephenw10
All right, I'll try and get a Mellanox MCX512A-ACAT aka ConnectX-5 EN and report back :)

Thanks!

[stephenw10]

I will say that I've never tested that myself. It would be good to get a second test with it though as that first result was very surprising.

[tman222]

I ran a Xeon D-1518 based pfSense system up until recently and the best performance I saw, if I recall correctly, was around ~6-7Gbit/s routing traffic between two different internal network segments (no NAT, no IDS/IPS) via an iperf3 test (single stream). I imagine with NAT in the picture, performance through WAN would have been a bit lower than that.

[stephenw10]

With the included ix NICs I assume? That's about what I'd expect. Which is why the reports of 25Gbps with Mellanox NICs are so surprising.

[ogghi]

@stephenw10
hi there, I got the card.
How would I go to install it actually?
Install the card, change LAN setting to one of the new ports, apply, switch over cables?
Then do the same for WAN?

[stephenw10]

Yes, pretty much exactly that. Just switch the interface assignments to the new NICs.

[ogghi]

@stephenw10
Hi there!
I was not successful here this morning, had to revert and restore backup. Seems the routes were not updating, same for firewall rules.
All back running. What I saw as warning output above in the webUI made me think the issue with the limitation might be something else:

Filter Reload
There were error(s) loading the rules: pfctl: interface ix0 bandwidth limited to 4294967295 bps because selected scheduler is 32-bit limited - The line in question reads [0]: @ 2024-03-07 07:38:45

So I tried disabling traffic shaper completely.
No change though.

But most important, what did go wrong with interface change? I tried in UI to re-assign, also from cmd locally...

[stephenw10]

Ah, looks like you have some limiters set on that and the link speed is above what it can handle. Which itself is interesting.

What traffic shaping do you have enabled there?

Do you have an interface with the bandwidth set as a percentage?

[ogghi]

@stephenw10
I disabled all traffic shaper queues this morning to test after I saw that output.
No change so far, but I haven't rebooted since, is it required?

[stephenw10]

No you should not need to reboot.

If you reload the ruleset in Status > Filter Reload does it regenerate the error?

If so that value must still be in the config somewhere.

[tman222]

@stephenw10 said in Load updated Intel IX module to get 10Gbps:

With the included ix NICs I assume? That's about what I'd expect. Which is why the reports of 25Gbps with Mellanox NICs are so surprising.

Actually, this was mostly tested between two interfaces on a Chelsio T540-SO-CR expansion card. I do recall testing between the Chelsio interfaces and onboard ix interfaces at a time or two as well and seeing similar speeds (i.e. no major increases or decreases).

[ogghi]

@stephenw10
I tried to do the reload while tail -f on /var/log/system.log
I only saw:
Mar 11 14:40:29 vm12 check_reload_status[331]: Reloading filter

That's where it had logged the error before I bet?

[stephenw10]

Yes it would show in the system log. It would also show on the filter reload page if the error was regenerated.

[ogghi]

@stephenw10 So no errors here :)
But problem not solved either.

https://hastebin.com/share/visalixawa.bash

I am confused why it still generates ALTQ queues?
login-to-view

Anyway, because of a power outage the firewall might reboot tonight (unless UPS holds up long enough)

[stephenw10]

It doesn't. The script is simply logging that it has reached that section where it would be creating queues if they were configured.

[ogghi]

@stephenw10
Good!

The question remains on how to change NICs while maintaining the NAT and firewall rules etc?

Did I not use the right procedure?

[stephenw10]

I don't know. It should be trivial since all the interface config is abstracted from the NIC. You just assign WAN to the new ix NIC and all the settings follow it.

The traffic shaping is potentially an issue because it tries to detest the NIC link speed and obviously that can/will change.

[ogghi]

@stephenw10 As I disabled traffic shaping that should not be an issue anymore.
The new interface has a different name, mce0 and mce1 but that shouldn't matter?

I'll try again at an appropriate time where down-time is possible!

Thanks for the quick replies!

[stephenw10]

Nope the name shouldn't matter, that's the point of abstracting it. You can import a config into completely different hardware and just reassign the interfaces to the existing NICs.

[ogghi]

@stephenw10 That's what I thought.
The question is (as I asked above) the proper order:
Right now ix0 is WAN, ix1 is LAN
I would probably want to assign LAN to mce1 and apply. Then physically connect the fiber to mce1.
Then ix0 to mc0 can be done once webUI can be accessed again...