Load updated Intel IX module to get 10Gbps

[stephenw10]

Yes that should work. You should also be able to do both at the same time.

I would want to be sure I had some out of band access whilst making that change. That could be via the console or assigning an interface for management access.

[ogghi]

@stephenw10 That's kind of what I tried, but it wouldn't work, as I mentioned above it would not apply fw rules, NAT etc.
So I restored from backup...
When doing such things I always have a monitor + keyboard ready in the rack for local console access ;)

[ogghi]

@stephenw10
I managed to change interfaces.
Still no improvement, I even tried to remove some advanced settings:

I guess time to test some tuning, but given that we will need to restart each time, I need to do it early morning or during next maintenance window...

Any suggestions?
I still have some things in /boot/loader.conf.local

net.inet.tcp.tso="0"
if_ix_updated_load="YES"
hw.ix.flow_control="0"
hw.ix.num_queues=40
hw.ix.enable_aim=1
hw.ix.max_interrupt_rate=30000
kern.ipc.nmbclusters="1000000"
kern.ipc.nmbjumbop="524288"
machdep.hyperthreading_intr_allowed=1

Might make sense to remove, reboot and try?

[stephenw10]

Ok so you're using the Mellanox (mce) NICs now? And still seeing ~4Gbps? Per core CPU usage is still low when testing?

[ogghi]

@stephenw10

[  5]  37.00-38.00  sec   274 MBytes  2.30 Gbits/sec  812    462 KBytes       
[  5]  38.00-39.00  sec   297 MBytes  2.49 Gbits/sec  479    624 KBytes       
[  5]  39.00-40.00  sec   269 MBytes  2.26 Gbits/sec  724    632 KBytes       
[  5]  40.00-41.00  sec   265 MBytes  2.22 Gbits/sec  685    441 KBytes       
[  5]  41.00-42.00  sec   262 MBytes  2.20 Gbits/sec   65    510 KBytes       
[  5]  42.00-43.00  sec   248 MBytes  2.08 Gbits/sec  770    553 KBytes       
[  5]  43.00-44.00  sec   245 MBytes  2.06 Gbits/sec  971   96.2 KBytes       
[  5]  44.00-45.00  sec   285 MBytes  2.39 Gbits/sec  296    607 KBytes       
[  5]  45.00-46.00  sec   262 MBytes  2.20 Gbits/sec  1108    454 KBytes       
[  5]  46.00-47.00  sec   260 MBytes  2.18 Gbits/sec  690    481 KBytes       
[  5]  47.00-48.00  sec   257 MBytes  2.15 Gbits/sec  582    545 KBytes       
[  5]  48.00-49.00  sec   275 MBytes  2.31 Gbits/sec  253    568 KBytes       
[  5]  49.00-50.00  sec   248 MBytes  2.08 Gbits/sec  1163    399 KBytes       
[  5]  50.00-51.00  sec   264 MBytes  2.21 Gbits/sec  358    453 KBytes       
[  5]  51.00-52.00  sec   274 MBytes  2.30 Gbits/sec  175    446 KBytes       
[  5]  52.00-53.00  sec   267 MBytes  2.24 Gbits/sec  864    457 KBytes       
[  5]  53.00-54.00  sec   283 MBytes  2.37 Gbits/sec  250    496 KBytes       
[  5]  54.00-55.00  sec   276 MBytes  2.32 Gbits/sec  326    542 KBytes       
[  5]  55.00-56.02  sec   272 MBytes  2.25 Gbits/sec  866    552 KBytes       
[  5]  56.02-57.00  sec   268 MBytes  2.29 Gbits/sec  629    554 KBytes       
[  5]  57.00-58.01  sec   271 MBytes  2.24 Gbits/sec  122    594 KBytes       
[  5]  58.01-59.00  sec   247 MBytes  2.10 Gbits/sec  565    464 KBytes       
[  5]  59.00-60.00  sec   279 MBytes  2.34 Gbits/sec  864    457 KBytes       
[  5]  60.00-61.00  sec   260 MBytes  2.18 Gbits/sec  864    464 KBytes       
[  5]  61.00-62.00  sec   257 MBytes  2.16 Gbits/sec  776    506 KBytes       
[  5]  62.00-63.02  sec   268 MBytes  2.21 Gbits/sec  174    562 KBytes       
[  5]  63.02-64.01  sec   256 MBytes  2.15 Gbits/sec  873    158 KBytes       
[  5]  64.01-65.00  sec   249 MBytes  2.12 Gbits/sec  770    428 KBytes       
[  5]  65.00-66.00  sec   253 MBytes  2.12 Gbits/sec  619    481 KBytes       
[  5]  66.00-67.02  sec   280 MBytes  2.31 Gbits/sec  603    520 KBytes       
[  5]  67.02-68.02  sec   250 MBytes  2.10 Gbits/sec  779    428 KBytes       
[  5]  68.02-69.00  sec   261 MBytes  2.22 Gbits/sec  253    454 KBytes       
[  5]  69.00-70.00  sec   264 MBytes  2.22 Gbits/sec  632    484 KBytes       
[  5]  70.00-71.00  sec   256 MBytes  2.15 Gbits/sec  864    514 KBytes       
[  5]  71.00-72.00  sec   256 MBytes  2.14 Gbits/sec  772    564 KBytes       
[  5]  72.00-73.00  sec   255 MBytes  2.15 Gbits/sec  732    425 KBytes       
[  5]  73.00-74.00  sec   268 MBytes  2.25 Gbits/sec  652    474 KBytes  

This seems even worse right now.
CPU usage is not noticibly higher than no huge transfer.

Maybe it's about changing some settings and try now?

[stephenw10]

So.... not using Mellanox NICs?

[ogghi]

@stephenw10 I am using the Mellanox cards.

I will most likely test to change settings here (again)

Suggestions?

[stephenw10]

I would certainly disable the hardware offloading there in case something isn't correctly supported under mlx.

Those iperf figures are through the firewall rather than to/from it?

[ogghi]

@stephenw10
YAY!
This finally is all good and fast now!

Iperf (with 10 parallel streams) shows (a bit bouncing but hey) 9.2Gbps as peak

I enabled HW TSO and HW LRO

Thanks for the support!

Also: CPU load is not much either. So ++ for the Mellanox cards!

[stephenw10]

Nice. Just to be clear you 'set' those options in the GUI, actually disabled them?

[ogghi]

@stephenw10 yes, this is how it looks now:

[stephenw10]

Excellent, thanks!