In which PCIe slot better to install Intel QAT in 2xCPU packages systems ?
-
Dear pfSense Gurus!
In which PCIe slot better to install Intel QAT in 2xCPU packages server systems in terms of performance?
Let’s take for example this mainboard with 2xCPU:
From my point of view there are several key moments that impacts on system performance:
- PCIe 1 to 4 belongs to CPU1, but PCIe 5-7 belongs to CPU2. NOTE, that controversial fact that in table with description below the PCIe 5-6 AVAILABLE ONLY WHEN CPU2 installed, but PCIe 7 (RAID extra card installed) working even with only CPU1 installed (? so, it’s belong to CPU1, breaking the numbering order?);
- If FreeBSD’s crypto drivers are independent on which CPU in multi-CPU system they working. Because a lot of services in FreeBSD still prefer to loading CPU1 and no CPU2 (am I wrong here and stick in a past years?);
Now I thinking that would be better to install Intel QAT on a PCIe 7, that belongs to CPU1:
- discard the extra RAID card and connect to on-board-RAID (anyway it used as mirror RAID with SSD, no cache enabled);
- point crypto drivers in FreeBSD preferable using CPU1 to work with;
- point all others processes and drivers preferable using CPU2 at first and CPU1 at last;
- anyway, that would be some BSD drivers and services that only able to work with CPU1 (default);
So, doing this, we receive balanced on CPUs loading system or this idea are totally idealistic and wrong?
(I think no matter which brand are motherboard: IBM, Dell, HP, MSI,… The principle would be the same. Nope?)
Thank You so much for detailed explanation of Your thoughts about this!
-
@Sergei_Shablovsky Depends on which CPU and how many other devices will be on which bus but ideally place them on the bus with your NIC's.
Since the whole purpose of them in this application is to offload from NIC to QAT to free up your cpu cycles for encryption/decryption and compression. You would want them as close to the nic as possible.
-
@Epimpin said in In which PCIe slot better to install Intel QAT in 2xCPU packages systems ?:
@Sergei_Shablovsky Depends on which CPU and how many other deviced will be on which bus but ideally place them on the bus with your NIC's.
I’m damn tied, forgot to wrote: ALL slots except PCIe 7 are filled by NICs.
Since the whole purpose of them in this application is to offload from NIC to QAT to free up your cpu cycles for encryption/decryption and compression. You would want then as close to the nic as possible.
Thank You for explanation. I know that, but I have lack of experience how this would work in case all other PCIe slots are filled by NICs…
-
@Sergei_Shablovsky When my systems get full of nics I consolidate with something like this with more ports.
-
@Sergei_Shablovsky When my ports become that filled with NIC's, I consolidated them by buying nics with more ports per nic. For instance, back in the day when my pfsense box was full I got 1gbe nics with 6 ports from hotlava.
-
@Epimpin said in In which PCIe slot better to install Intel QAT in 2xCPU packages systems ?:
@Sergei_Shablovsky When my systems get full of nics I consolidate with something like this with more ports.
Thank You for suggestion!
But I have some restrictions that not make this option possible:
-
(more valuable)
From the infrastructure perspective always are MORE CRITICAL and less critical nets.
So, REASONABLE TO COMBINE in each NIC “1 MORE CRITICAL + 1 less critical” (for example for 2-head NIC) TO ELIMINATE DOWNTIME AND SERVICE DEGRADATION IF CERTAIN NIC COME OUT OF ORDER (and You have no another one absolutely identical on a spare shelf). -
Most of all equipment that we work are RACK 2U servers, so only half-height PCIe NICs possible;
Reasonable?
-
-
@Epimpin So, back to initial questions, which PCIe slot would be better in this certain case ?
Your opinion…
