Load updated Intel IX module to get 10Gbps
-
@stephenw10
I managed to change interfaces.
Still no improvement, I even tried to remove some advanced settings:
I guess time to test some tuning, but given that we will need to restart each time, I need to do it early morning or during next maintenance window...Any suggestions?
I still have some things in /boot/loader.conf.localnet.inet.tcp.tso="0" if_ix_updated_load="YES" hw.ix.flow_control="0" hw.ix.num_queues=40 hw.ix.enable_aim=1 hw.ix.max_interrupt_rate=30000 kern.ipc.nmbclusters="1000000" kern.ipc.nmbjumbop="524288" machdep.hyperthreading_intr_allowed=1Might make sense to remove, reboot and try?
-
Ok so you're using the Mellanox (mce) NICs now? And still seeing ~4Gbps? Per core CPU usage is still low when testing?
-
[ 5] 37.00-38.00 sec 274 MBytes 2.30 Gbits/sec 812 462 KBytes [ 5] 38.00-39.00 sec 297 MBytes 2.49 Gbits/sec 479 624 KBytes [ 5] 39.00-40.00 sec 269 MBytes 2.26 Gbits/sec 724 632 KBytes [ 5] 40.00-41.00 sec 265 MBytes 2.22 Gbits/sec 685 441 KBytes [ 5] 41.00-42.00 sec 262 MBytes 2.20 Gbits/sec 65 510 KBytes [ 5] 42.00-43.00 sec 248 MBytes 2.08 Gbits/sec 770 553 KBytes [ 5] 43.00-44.00 sec 245 MBytes 2.06 Gbits/sec 971 96.2 KBytes [ 5] 44.00-45.00 sec 285 MBytes 2.39 Gbits/sec 296 607 KBytes [ 5] 45.00-46.00 sec 262 MBytes 2.20 Gbits/sec 1108 454 KBytes [ 5] 46.00-47.00 sec 260 MBytes 2.18 Gbits/sec 690 481 KBytes [ 5] 47.00-48.00 sec 257 MBytes 2.15 Gbits/sec 582 545 KBytes [ 5] 48.00-49.00 sec 275 MBytes 2.31 Gbits/sec 253 568 KBytes [ 5] 49.00-50.00 sec 248 MBytes 2.08 Gbits/sec 1163 399 KBytes [ 5] 50.00-51.00 sec 264 MBytes 2.21 Gbits/sec 358 453 KBytes [ 5] 51.00-52.00 sec 274 MBytes 2.30 Gbits/sec 175 446 KBytes [ 5] 52.00-53.00 sec 267 MBytes 2.24 Gbits/sec 864 457 KBytes [ 5] 53.00-54.00 sec 283 MBytes 2.37 Gbits/sec 250 496 KBytes [ 5] 54.00-55.00 sec 276 MBytes 2.32 Gbits/sec 326 542 KBytes [ 5] 55.00-56.02 sec 272 MBytes 2.25 Gbits/sec 866 552 KBytes [ 5] 56.02-57.00 sec 268 MBytes 2.29 Gbits/sec 629 554 KBytes [ 5] 57.00-58.01 sec 271 MBytes 2.24 Gbits/sec 122 594 KBytes [ 5] 58.01-59.00 sec 247 MBytes 2.10 Gbits/sec 565 464 KBytes [ 5] 59.00-60.00 sec 279 MBytes 2.34 Gbits/sec 864 457 KBytes [ 5] 60.00-61.00 sec 260 MBytes 2.18 Gbits/sec 864 464 KBytes [ 5] 61.00-62.00 sec 257 MBytes 2.16 Gbits/sec 776 506 KBytes [ 5] 62.00-63.02 sec 268 MBytes 2.21 Gbits/sec 174 562 KBytes [ 5] 63.02-64.01 sec 256 MBytes 2.15 Gbits/sec 873 158 KBytes [ 5] 64.01-65.00 sec 249 MBytes 2.12 Gbits/sec 770 428 KBytes [ 5] 65.00-66.00 sec 253 MBytes 2.12 Gbits/sec 619 481 KBytes [ 5] 66.00-67.02 sec 280 MBytes 2.31 Gbits/sec 603 520 KBytes [ 5] 67.02-68.02 sec 250 MBytes 2.10 Gbits/sec 779 428 KBytes [ 5] 68.02-69.00 sec 261 MBytes 2.22 Gbits/sec 253 454 KBytes [ 5] 69.00-70.00 sec 264 MBytes 2.22 Gbits/sec 632 484 KBytes [ 5] 70.00-71.00 sec 256 MBytes 2.15 Gbits/sec 864 514 KBytes [ 5] 71.00-72.00 sec 256 MBytes 2.14 Gbits/sec 772 564 KBytes [ 5] 72.00-73.00 sec 255 MBytes 2.15 Gbits/sec 732 425 KBytes [ 5] 73.00-74.00 sec 268 MBytes 2.25 Gbits/sec 652 474 KBytesThis seems even worse right now.
CPU usage is not noticibly higher than no huge transfer.Maybe it's about changing some settings and try now?
-
So.... not using Mellanox NICs?
-
@stephenw10 I am using the Mellanox cards.
I will most likely test to change settings here (again)
Suggestions?
-
I would certainly disable the hardware offloading there in case something isn't correctly supported under mlx.
Those iperf figures are through the firewall rather than to/from it?
-
@stephenw10
YAY!
This finally is all good and fast now!
Iperf (with 10 parallel streams) shows (a bit bouncing but hey) 9.2Gbps as peak
I enabled HW TSO and HW LRO
Thanks for the support!
Also: CPU load is not much either. So ++ for the Mellanox cards!
-
Nice. Just to be clear you 'set' those options in the GUI, actually disabled them?
-
@stephenw10 yes, this is how it looks now:
-
Excellent, thanks!
