Zotac CI 321 Dual NIC Nano

highwire

I have a Zotac ZBOX with the same RealTek PHYs. I was also getting WAN disconnects. I solved the discconnects by setting System -> Advanced -> Networking -> Disable Hardware Checksum Offload.

perth

@highwire:

I have a Zotac ZBOX with the same RealTek PHYs. I was also getting WAN disconnects. I solved the discconnects by setting System -> Advanced -> Networking -> Disable Hardware Checksum Offload.

I've applied this setting and rebooted the pfSense box.
We'll see how it goes. crosses fingers

perth

So the Disable Hardware Checksum Offload setting may have improved the reliability of my connection.
However I'm still getting excessive disconnects running this setup.

I think I'm going to try setting System > Advanced > Firewall and NAT > Disable Firewall Scrub "Disables the PF scrubbing option which can sometimes interfere with NFS and PPTP traffic." Since PPPOE is PPTP OE, if I understand correctly. Lets also disable IPv6, thought I had already turned that off…

Need to replace the modem and see what that does.

perth

Checked the "Configure a Null service name" under advanced WAN config, based off of other posts here for similar issues.
Think I've tried that already though…

unknownUser

Very interesting topic, I'm just about to order CI323, unfortunately my "trusted" shops don't have it on stock atm, so I had some time finding this topic ;)
Perth maybe you can help me with a few questions, as I'm not sure if this will work, so the following is my setup:
I've a NAS running at 1600MhZ (NSA 325 Zyxel). I've sabnzbd & Twonky only installed on it.
My actual router is the ASUS RT-AC87U and its running OpenVPN at 50 Mbit/s up and 10Mbit/s down.
Wifi is running on the wireless router which is an Apple TimeCapsule used for Backup and Wifi.

My aim is to reduce this setup, I'll keep the TimeCapsule for Wifi & Backup.
The ASUS and the NAS should be replaced with the ZBOX and I need some additional functions.

What I need running at the same time on the CI323 would be a VPN connection at 50 MBit/s,Sabnzbd using SSL (downloading & decompressing),Twonky (streaming) and an Adblocker or atleast a script using hosts files.
Will this CPU will be enough? Have you tried getting 100% CPU usage, how did you succeed?

perth

Disclaimer: Lots of the numbers below come from my memory. I'm about to go to sleep, and my memory isn't ever any good anyway. Double check any numbers I didn't copy/paste. I also suck at math.

Blindly following pfSense Hardware Crypto Doc (https://doc.pfsense.org/index.php/Are_cryptographic_accelerators_supported)

[2.2.6-RELEASE][admin@redacted]/root: openssl engine -t -c
(cryptodev) BSD cryptodev engine
 [RSA, DSA, DH]
     [ available ]
(rsax) RSAX engine support
 [RSA]
     [ available ]
(rdrand) Intel RDRAND engine
 [RAND]
     [ available ]
(dynamic) Dynamic engine loading support
     [ unavailable ]

[2.2.6-RELEASE][admin@redacted]/root: openssl speed -evp RSA
RSA is an unknown cipher or digest

[2.2.6-RELEASE][admin@redacted]/root: openssl speed -evp DSA
Doing dsaEncryption for 3s on 16 size blocks: 3413578 dsaEncryption's in 2.99s
Doing dsaEncryption for 3s on 64 size blocks: 2729635 dsaEncryption's in 2.99s
Doing dsaEncryption for 3s on 256 size blocks: 1553738 dsaEncryption's in 3.01s
Doing dsaEncryption for 3s on 1024 size blocks: 577673 dsaEncryption's in 3.00s
Doing dsaEncryption for 3s on 8192 size blocks: 88519 dsaEncryption's in 3.00s
OpenSSL 1.0.1l-freebsd 15 Jan 2015
built on: date not available
options:bn(64,64) rc4(16x,int) des(idx,cisc,16,int) aes(partial) idea(int) blowfish(idx) 
compiler: clang
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
dsaEncryption    18253.28k    58384.26k   132241.26k   197179.05k   241715.88k

[2.2.6-RELEASE][admin@redacted]/root: openssl speed -evp DH
DH is an unknown cipher or digest

Doing the above crypto performance test didn't cause my CPU to hit even 10%. I find the results quite confusing, the output states "The 'numbers' are in 1000s of bytes per second processed." and then throws a 'k' on the end of the numbers. Does that mean thousand thousand, or is the output redundant, but not multiplicative? No freaking clue, using the raw data :). Lets look at the worst: 3413578 16B blocks in 2.99 seconds, ((3413578 * 16) * 8 / 2.99)/(1024^2) = 139Mb/s of dsa Encryption. So I think my box is more than capable of the VPN workload you mention; if that's true the CI323 won't even be tickled. But then, I have almost no clue what I'm looking at; sorry.

I don't think you news reader program (SABnzbd) is of any significance, work load wise, in this setup.
I don't think Twonky is doing transcoding, but if it is, I wouldn't put it on the pfSense box. I'd estimate 1 maxed out thread per stream, possibly more?, I see that eating up enough CPU/RAM to impact your network throughput, plus cause lots of jitter. Since Twonky does embedded & Android systems though, it's probably not overly resource intensive, and probably fine.

All I can find (quickly) on your NAS' CPU is 1.6 GHz, so I'm assuming 1 core, and probably atom architecture. That's not a lot. If I'm wrong in that assumption, then maybe it's an x86 dual core; maybe. That would be the worst case scenario, and would be less than 50% of the CI323's compute capability. So looking at things that way, my box is 50% of the CI323, your NAS (worst case) is <50% of the CI323. I'm peaking at 20% CPU. So worst case is 70% (maxing out your NAS that I've made more awesome than I think it is) + VPN work load. Looking at it this way cuts things close, but I really doubt your NAS is that awesome; it's a NAS.

So can the CI323 handle your work load? I would say, "yeah". That said, I haven't asked my box to handle encryption/decryption of network traffic. I'm only on a 3Mb DSL connection. My connection is not stable (though I'm not blaming the CI321 for that). But the pfSense Hardware guide (https://www.pfsense.org/hardware/#requirements) recommends a 1GHz processor for your connection speed; CI323 has 8x that (4x 2GHz)?

So… yeah, it seems likely the CI323 can do what you are asking. I think the question you should consider is: do you like the architecture, including all the inherent benefits and draw backs, that you have outlined? Your pfSense box is the first line of defence, which is another way of saying that it's what gets shot full of holes first when bad stuff happens; also lightning. You are wanting to put your data there? Lots of people on these forums recommend against that design, it puts your data on the front line. (I was going to do the same thing, though pfSense and the NAS were going to be separate VMs. I understand the protection offered by VMs has been busted out of before. I did not implement that idea.) Also you have the problem/added complexity of dealing w/ the storage that won't fit inside of the CI323, but is electrically directly connected (lightning). Just food for thought.

If you do buy the CI323, I'd love to hear how it works out for you!

unknownUser

Thanks for the info.

I will take a closer look to understand the results you posted, as of now I don't really get it what they mean.
The newsreader sabnzbd is used for downloading big files, so it runs with full speed of 50 mbit/s, that causes my NAS to be used by 100%, depending on the resources avaiable the download speed get's slower (if I run twonky while it's downloading or unpacking). Also the unpacking takes forever. Yes It's 1.6Ghz single core Marvell Kirkwood, I don't know what that mean for it's architecture compared to celeron or atom.
All in all it sound good, what you are saying, it looks like running smooth.
The last thing I have to think about it is really the Harddrive, which has no sensitive data, I have to decide what and how and where to put it, I wanted it to replace the NAS.

If/When I get a CI323 I will report my finding!
Cheers!

interfasys

That's probably the cipher you need for OpenVPN

# openssl speed -elapsed -evp aes-256-cbc
You have chosen to measure elapsed time instead of user CPU time.
Doing aes-256-cbc for 3s on 16 size blocks: 944551 aes-256-cbc's in 3.01s
Doing aes-256-cbc for 3s on 64 size blocks: 874272 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 256 size blocks: 707852 aes-256-cbc's in 3.01s
Doing aes-256-cbc for 3s on 1024 size blocks: 410113 aes-256-cbc's in 3.03s
Doing aes-256-cbc for 3s on 8192 size blocks: 80373 aes-256-cbc's in 3.01s
OpenSSL 1.0.2f  28 Jan 2016
built on: reproducible build, date unspecified
options:bn(64,64) rc4(16x,int) des(idx,cisc,16,int) aes(partial) idea(int) blowfish(idx)
compiler: clang37 -I. -I.. -I../include  -fPIC -DOPENSSL_PIC -DOPENSSL_THREADS -pthread -D_THREAD_SAFE -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -DL_ENDIAN -O3 -Wall -O2 -pipe -D_FORTIFY_SOURCE=2 -flto -march=native  -fstack-protector-strong --param ssp-buffer-size=4 -fno-strict-aliasing -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DMD5_ASM -DAES_ASM -DVPAES_ASM -DBSAES_ASM -DWHIRLPOOL_ASM -DGHASH_ASM -DECP_NISTZ256_ASM
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-256-cbc       5024.52k    18651.14k    60246.48k   138542.09k   218901.82k

You'll easily reach 100Mbit, using one core, which leaves you with 3 for other things to do.

perth

@interfasys:

That's probably the cipher you need for OpenVPN
…
You'll easily reach 100Mbit, using one core, which leaves you with 3 for other things to do.

Thank you interfasys! Here's the result of the aes-256-cbc scheme speed test on the CI321:

[2.2.6-RELEASE][admin@redacted]/root: openssl speed -elapsed -evp aes-256-cbc
You have chosen to measure elapsed time instead of user CPU time.
Doing aes-256-cbc for 3s on 16 size blocks: 14575858 aes-256-cbc's in 3.03s
Doing aes-256-cbc for 3s on 64 size blocks: 3854920 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 256 size blocks: 973141 aes-256-cbc's in 3.01s
Doing aes-256-cbc for 3s on 1024 size blocks: 245621 aes-256-cbc's in 3.01s
Doing aes-256-cbc for 3s on 8192 size blocks: 30538 aes-256-cbc's in 3.00s
OpenSSL 1.0.1l-freebsd 15 Jan 2015
built on: date not available
options:bn(64,64) rc4(16x,int) des(idx,cisc,16,int) aes(partial) idea(int) blowfish(idx) 
compiler: clang
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-256-cbc      76936.49k    82238.29k    82825.67k    83620.87k    83389.10k

Applying the same formula I made up earlier:
((blocksEncrypted * bytesPerBlock) * 8 / seconds)/(1024^2) = Mb/s
16B = 587 Mb/s
64B = 627 Mb/s
256B = 631 Mb/s
1KB = 637 Mb/s
8KB = 636 Mb/s
(decimals truncated)

Did a little more looking around on the internet regarding throughput on custom firewalls, and the internet says we are paying too much attention to the CPU.
We should be looking at the bus the NIC is on. https://calomel.org/network_performance.html

interfasys

Interesting to see how much slower the C321 (1.1Ghz) is in this openssl test compared to the C323 (1.6Ghz) with AESNI.

But regardless, the speed is going to depend a lot on how you're able to tune your connection. If you're connecting to a public provider, you'll be limited to what they offer and ~100Mbit is what you're going to reach.

Regarding the bus, there is nothing to worry about today. PCI Express 1x is enough to drive even a dual-port Intel NIC.

And beware of tips such as

Ideally you want to use a server based add on card with a TCP offload engine or TCP accelerator.

Netmap is fast and requires all hardware acceleration to be turned off.
It's still a good idea to get Intel NICs for Gigabit WAN connections (or peace of mind), but it's tricky to add to a Zotac Nano…

perth

@interfasys:

Interesting to see how much slower the C321 (1.1Ghz) is in this openssl test compared to the C323 (1.6Ghz) with AESNI.
…

Interfasys, are you running the CI323? All my tests are on the CI321. If one of my posts was misleading please let me know which one & I'll edit it for clarity.
Secondly, I find the differences between our encryption speed tests very interesting, the CI321 is stomping your platform on the smaller block sizes. Yet your platform is annihilating mine on the large block sizes.

Comparing results of 'openssl speed -elapsed -evp aes-256-cbc' on our platforms:


CI321 ------------------------------------------------------------------------------
type                  16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-256-cbc           76936.49k    82238.29k   82825.67k   83620.87k    83389.10k
Recalculated as Mb/s  587          627         631         637          636
Interfasys -------------------------------------------------------------------------
type                  16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-256-cbc           5024.52k    18651.14k    60246.48k   138542.09k   218901.82k
Recalculated as Mb/s  38           142         459         1057         1668

Seeing the compiler string in your output, I wonder:
Did you custom compile openssl?
Is that compiler string caused by installing openssl as a port instead of a pkg?
Looking forward to a little more detail on the platform you ran that test on. I think I have more questions for you. :-)

interfasys

All my tests are on the CI321. If one of my posts was misleading please let me know which one & I'll edit it for clarity.

It was clear that you were on the CI321. I provided my numbers for @unknownUser since he was considering the CI323 to fill his requirements.

Secondly, I find the differences between our encryption speed tests very interesting, the CI321 is stomping your platform on the smaller block sizes. Yet your platform is annihilating mine on the large block sizes.

Indeed, here are the results without using evp

type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-256 cbc      24665.50k    26819.67k    27680.79k    75580.20k    76382.63k

I find it unusual to get lower numbers for small block sizes using AES-NI vs not using it.

Your formula is wrong by the way, the numbers given are already in MB/s
(80373 * 8192) / 3.01 ~= 218.9 MB/s

Did you custom compile openssl?
Is that compiler string caused by installing openssl as a port instead of a pkg?

Yes

interfasys

Look at the APU2 numbers, same thing happens when enabling AES-NI.
https://forum.pfsense.org/index.php?topic=106444.msg593101#msg593101

interfasys

A test you can run

# openvpn --genkey --secret /tmp/secret
# time openvpn --test-crypto --secret /tmp/secret --verb 0 --tun-mtu 20000 --cipher aes-256-cbc
Sat Feb  6 18:39:04 2016 WARNING: file '/tmp/secret' is group or others accessible

real    0m26.146s
user    0m24.424s
sys     0m1.709s

interfasys

And another one

[code]# dd if=/dev/zero bs=1M count=100 >> /dev/null
100+0 records in
100+0 records out
104857600 bytes transferred in 0.015134 secs (6928676210 bytes/sec)

# dd if=/dev/zero bs=1M count=100 | openssl aes-256-cbc -e -pass pass:secretpwd | openssl aes-256-cbc -d -pass pass:secretpwd >> /dev/null
100+0 records in
100+0 records out
104857600 bytes transferred in 0.872586 secs (120168784 bytes/sec)[/code]

Rango

@perth:

So the Disable Hardware Checksum Offload setting may have improved the reliability of my connection.
However I'm still getting excessive disconnects running this setup.

I think I'm going to try setting System > Advanced > Firewall and NAT > Disable Firewall Scrub "Disables the PF scrubbing option which can sometimes interfere with NFS and PPTP traffic." Since PPPOE is PPTP OE, if I understand correctly. Lets also disable IPv6, thought I had already turned that off…

Need to replace the modem and see what that does.

Can you guys confirm the wan disconnections have been fixed by doing this optioned mentioned above?
Thanks you in advance. BTW could it be possible it's cause you're on PPPOE?

How long after you guys placed the order did you guys get it?

Rango

Does this box ci321 mobo support dual channel memory and does it have 2 slots for memory?
I'm looking at cpu and seems to support it but that's based on mobo not cpu.
If so would it be better for performance to install 2x4GB ram as it would utilize dual channel ?

Also i'm assuming you guys are running sata drive not msata correct?

Memory Specifications
Max Memory Size (dependent on memory type) 8 GB
Memory Types DDR3L-1600
Max # of Memory Channels 2
ECC Memory Supported ‡ No

http://ark.intel.com/products/87258/Intel-Celeron-Processor-N3150-2M-Cache-up-to-2_08-GHz

hardsense

@Perth

Would you do us a favour by testing below ?

It's a simple and effective test .

Test 1:

Connect 1 host to WAN interface of your pfsense box, connect another host to LAN interface of your pfsense box and perform a iperf test and provide us with screenshots as usual.

Test 2:

Install the DHCP service on the host connected on the WAN side and configured as WAN DHCP server and let it run for 24 hour or so and see if the connection drop or not.

P.S The ideal test -> The host are equipped with Intel LAN card and running current Desktop processor and hardware . If you don't have it , just run the test with whatever you have in your inventory.

Thanks.

Rango

@G.D.:

Another deficiency (though maybe not very important in case of pfSense) is that despite having two memory slots this box is configured for single channel operation.

Anyway, has anyone tried Zotac CI321 with pfSence yet?

Where did you get that this box is single channel operation?

In fact cpu cpecs on i323 say opposite that it has 2 channels for memory.

Rango

Regarding the bus, there is nothing to worry about today. PCI Express 1x is enough to drive even a dual-port Intel NIC.

Hmmm…..looking at the box it seems impossible to add any pci-e card unless one changes the box and it would have to be sideways not vertical?
hmmm seems impossible to add any external nic to this???

I hope the realtek nics are working good with pfsense?