Does upgrade to a modern 10G NIC make sense !!??

[louis2]

@louis2

I realized that I could compare the transfer between NAS and PC in two situations:

• PC and NAS in the samen vlan and
• PC to NAS via the pfSense

I had to make some changes in my network, however it would show the impact of pfSense

The test setup I used was as follows

• The TrueNAS-scale system connected to my 10G-main-switch
• The PC via ConnectX4-port-1 connected to the 10G-main-switch routed to the NAS via the FW
• An extra 10G switch connected to the 10G-main-switch NAS-vlan and to the PC via ConnectX4 port-2
• NAS and PC both equipped with a better quality NVME SSD
• testing using iSCSI drive on the NAS
• Using one big file which is not really a representative test

This setup allowed me to test both situations, via the FW and using the same vlan by switching the used PC NIC-port

The result was better that I expected! But of course there is some impact due to the FW. It is also clear that the PC (Windows11 pro 64bit), does not manage to send data to the NAS at full speed.

Note that I am using jumbo frames (9014), to minimize the number of frames the FW has to handle and a bit more efficient ethernet frames.

See for the results the picture below

login-to-view

[Gblenn]

@louis2 That was a good test to do and it is showing you where to focus, and it doesn't look like it's the NIC's...

What HW are you running pfsense on?

[stephenw10]

Mmm, that's a pretty good result for the routed/filtered throughput.

[louis2]

@Gblenn

CPU
Core(TM) i5-6600K CPU @ 3.50GHz
Current: 3602 MHz, Max: 3500 MHz
4 CPUs : 1 package(s) x 4 core(s)
AES-NI CPU Crypto: Yes (active)
IPsec-MB Crypto: Yes (inactive)

DISC
200GB NVME SSD

RAM
16 GB

NIC's
Intel I219 (On MB)
Intel X520 DA2 2x SFP+
Intel x550 2x 10G UTP

The test was done via de SFP+ card

[louis2]

@stephenw10

Note that I did choose to test with one big file about 17 TB despite that that is not exactly representative.

I did that to avoid major impact from windows and the nas. I know that .e.g. windows small file performance is ..... terrrible

I do not know what is the effect of one big file versus many small one's on pfSense trough put, but I assume that the num,ber of packages has more impact on pfsense, than te number of files.

[Gblenn]

@louis2 said in Does upgrade to a modern 10G NIC make sense !!??:

CPU
Core(TM) i5-6600K CPU @ 3.50GHz

I'm guessing the results actually scale quite well based on CPU performance in such a test. And I'm seeing iperf results around 9 Gbit on my i5-11400 (VLAN to VLAN). Comparing the CPU's on cpubenchmark I see roughly 30% better single thread performance on my 11400 vs your 6600. Which seems to be about right based on the VLAN results you have...
https://www.cpubenchmark.net/compare/2570vs4233/Intel-i5-6600K-vs-Intel-i5-11400

BTW, I'm also running X520's, and have 4 cores assigned from that CPU.

[louis2]

@Gblenn

I just do not understand what the limiting factor is. The complexity & number of the rules when sending data towards the NAS is higher than the from the NAS to the PC. That might be a reason. However that can not be the main reason since the transfers PC to NAS are also far from 10G when pfSense is not involved.

So I do not know if the NAS or the PC is the main reason for the 'slow' transfer. Given the fact that writing is always slower than reading the change that the NAS is the main factor seems a bit higher.

Where to add that the NAS interface stack is a lot more complex than the PC-interface stack and the PC is equipped with a ConnectX4 and the NAS with a connectX3. If that matters, I do not know.

[stephenw10]

Yeah I would definitely run an iperf test to eliminate disk reads and smb issues.

That also means you can try parallel streams to see if it's a core/queue issue in the firewall. Though that's unlikely IMO with that CPU.

[Gblenn]

@louis2 said in Does upgrade to a modern 10G NIC make sense !!??:

However that can not be the main reason since the transfers PC to NAS are also far from 10G when pfSense is not involved.

Sorry, you are right, I clearly didn't read the graphs (or rather the labels)... You clearly have more or less the same performance in both scenarios. Testing with iperf will, probably give you more realistic data on the actual throughput.

One difference between the NAS to PC and PC to NAS is the cache you may have in the NAS. Whilst the PC is likely reading the file from the SSD before sending, the NAS may perhaps be reading from cache, completely bypassing SSD...

[louis2]

@Gblenn

Yep TrueNas is using ZFS and a big ram cache, however the NVME-SSD should be ... fast enough to write 10G ... I think & hope. However I must admit that SSD's are not by far as fast as advertised if you are writing larger amounts of data ..

It is a 4TB WD_BLACK SN850X not the worst ssd ....