Help with 25G Speeds on HA pfSense Routers (LACP) Using Mellanox ConnectX-5 NIC
-
pfSense is a bad server. It's optimised as a router. You should test through it if you possibly can rather than to or from it directly.
Seeing 25Gbps in a single stream when pfSense is sending is surprising. Impressive. Do you see more if you run multiple streams? Or multiple simultaneous iperf instances?
-
What config do you have on pfSense for that test? A lot of rules? Basic install?
-
@stephenw10 Pretty much just a basic install with HA and some vlans created. We are in the testing phase so we wanted to have as much of a baseline as possible. On your previous post you asked some good questions that I will try to test later today. Right now I have to do "work" ie email and other admin nonsense.
-
Update: I was able to do some more testing and I rechecked the MTU settings for everything and found some things I missed. I then set up to ubuntu vms on each proxmox node. Each proxmox node had the necessary vlans created (I'm using openvswitch) and I was able to get 25gb across the vlans from one ubuntu box to another.
-
Interesting, I may want to do some additional testing in my lab on this, I've never managed to push PF much beyond about 10 gig, even with iperf and ideal scenarios, so this is super interesting.
-
@planedrop I spent all day at it and just started looking at everything again because it didn't add up. Heres a screen shot of one of the results. 
I also tried with parallel streams and it worked as expected the retrys went way down. -
@kilo40 Interesting, I'll see if I can duplicate this in my lab, that's crazy fast but awesome to see nonetheless.
-
That is crazy fast! Are you seeing that both ways?
-
@stephenw10 Yep, tested both ways and everything seems to be working great.
-
Nice. Surprising from that CPU. The NICs must really be helping.
-
Simply stunning performance. I think you will be helping the rest of us from now on!
Is this VT-d stretching its legs with the ConnectX?
️ -
S stephenw10 referenced this topic on
-
S stephenw10 referenced this topic on
