Zotac CI 321 Dual NIC Nano

perth

From the previously linked sysctl -a output:

re0: <realtek 8111="" 8168="" b="" c="" cp="" d="" dp="" e="" f="" g="" pcie="" gigabit="" ethernet=""> port 0xe000-0xe0ff mem 0xf0104000-0xf0104fff,0xf0100000-0xf0103fff irq 19 at device 0.0 on pci3
re0: Using 1 MSI-X message
re0: Chip rev. 0x2c800000
re0: MAC rev. 0x00100000
miibus0: <mii bus=""> on re0
rgephy0: <rtl8169s 8211="" 8110s="" 1000base-t="" media="" interface=""> PHY 1 on miibus0
rgephy0:  none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow
re0: Ethernet address: 00:01:2e:64:ee:d3
pcib4: <acpi pci-pci="" bridge=""> irq 16 at device 28.4 on pci0
pci4: <acpi pci="" bus=""> on pcib4
re1: <realtek 8111="" 8168="" b="" c="" cp="" d="" dp="" e="" f="" g="" pcie="" gigabit="" ethernet=""> port 0xd000-0xd0ff mem 0xf0004000-0xf0004fff,0xf0000000-0xf0003fff irq 16 at device 0.0 on pci4
re1: Using 1 MSI-X message
re1: Chip rev. 0x2c800000
re1: MAC rev. 0x00100000
miibus1: <mii bus=""> on re1
rgephy1: <rtl8169s 8211="" 8110s="" 1000base-t="" media="" interface=""> PHY 1 on miibus1
rgephy1:  none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow</rtl8169s></mii></realtek></acpi></acpi></rtl8169s></mii></realtek>

This last line seems to match what the menu for forcing the interface speeds offered. So I'd bet that gigabit works just fine.

perth

Conclusion
I believe I can say without hesitation, that the Zotac CI321 is entirely capable of being a basic pfSense appliance. The below graphs, taken directly from pfSense, should make that fairly obvious. The load average seems to (almost) always be below .1 for 1, 5, and 15 minute averages. The temperature seems to stay between 50 & 55 degrees C. (As reported by pfSense's System Information.)
I cannot say what the limitations of this fanless system will be, or how many additional features/applications/packages can be implemented on it w/out causing issue.
While I have had some issues with WAN disconnections since implementing my new network solution; I do not believe that the Zotac CI321 is responsible. I believe the issue is either that my ISP is irritated with me removing their solution (to which they seemed to have root access), or something in how I've configured the modem or pfSense box. (Example: disconnections have decreased since disabling gateway monitoring, which had been pinging the gateway once every second.)

With that said, I would not buy the CI321 again. As previously mentioned, Zotac has since released the CI323. That system, spec wise, is significantly better for an extremely minor increase in cost (at the time). Considering the success of this system, I would feel reasonably comfortable assuming the CI323 will also be compatible.
Caveats:

I have not attempted to use the included WiFi or Bluetooth. In pfSense, under Interfaces > (assign) > Wireless the Parent Interface drop-down is not populated. I have made absolutely no attempt to get the wireless NIC to work; I have no interest in it. See the sysctl or pciconf output in one of my earlier posts to determine if this interface is compatible with pfSense/FreeBSD.
You have to change the Boot Mode option in the UEFI BIOS before the CI321 will boot something other than Windows. See random YouTube video of the BIOS I found: https://www.youtube.com/watch?v=Cznx10PqoR0

Hope this helps! Feel free to ask me questions about this device, or for some specific output (provide instructions, just in case).

RRD Graphs, 8 hour period, 1 minute average
Throughput States Processor Memory Mbuf Clusters

RRD Graphs, 1 week period, 1 hour average
Throughput States Processor Memory Mbuf Clusters

This post is deleted!

perth

I installed, and am more or less constantly viewing, ntopnp. Small increase in processor usage and memory.

RRD Graphs, 1 day period, 5 minute average
Processor Memory

I'm loving this thing. Just need to figure out where the problem is that is causing all these disconnects…

yashiharu

can you try with the ac wifi?

just want to know if it works & the performance of 1T1R

thanks

ReFleX

Well this is a very intresting topic to read. This is also the main reason i bought a CI323 so i can install pfSense on it.
When i recieve my order i will start a topic with my findings, it will be a setup with a OpenWRT router and several VLAN's.
Somebody that has this kind of setup?

duren

I have this kind of setup. While waiting for a suitable box, which now is looking like the ci323, I've been running pfsense very comfortably in a VM. I have an Asus n66u running Tomato/shibby that acts as an access point and backup router.

The cable modem is vlan'd onto a switch and serves one public IP to the AP and one public IP to pfsense on the VM, soon to be a box like the 323.

The router runs a heartbeat script against pfsense such that if the VM goes down, within one minute the router will create a virtual interface matching the IP of pfsense so that devices continue to function transparently. When the VM comes back up, the router will tear down the interface and all is well again. Dhcp is not a problem because the ap is responsible for that, not pfsense. The heartbeat is on an aliased IP of pfsense.

I needed vlans to achieve this. I also experimented with vlans in case I had to settle on a box with one nic. This works perfectly also.

To tell you the truth, the VM setup is working so well, I'm questioning getting a box, especially with this redundancy in place.

ReFleX

@duren:

I have this kind of setup. While waiting for a suitable box, which now is looking like the ci323, I've been running pfsense very comfortably in a VM. I have an Asus n66u running Tomato/shibby that acts as an access point and backup router.

The cable modem is vlan'd onto a switch and serves one public IP to the AP and one public IP to pfsense on the VM, soon to be a box like the 323.

The router runs a heartbeat script against pfsense such that if the VM goes down, within one minute the router will create a virtual interface matching the IP of pfsense so that devices continue to function transparently. When the VM comes back up, the router will tear down the interface and all is well again. Dhcp is not a problem because the ap is responsible for that, not pfsense. The heartbeat is on an aliased IP of pfsense.

I needed vlans to achieve this. I also experimented with vlans in case I had to settle on a box with one nic. This works perfectly also.

To tell you the truth, the VM setup is working so well, I'm questioning getting a box, especially with this redundancy in place.

Well that sounds interesting, did you make the heartbeat script?
Sounds like a homebrew way to provide a FHRP.

duren

@ReFleX:

Well that sounds interesting, did you make the heartbeat script?
Sounds like a homebrew way to provide a FHRP.

That's right, it's a bash script that executes as a cron job on the router. It simply setups or or tears down that virtual interface depending on the result of a ping every minute.

TWiST

How are you guys doing with the CI323, I was about to order one on Amazon for 149 barebones, but I am just doing one last round of research. The processor should be great, just wondering about the NICs. I have tried to find an N3150 with Intel NICs, but it was not very fruitful.

messerchmidt

http://www.amazon.com/gp/product/B0179S50UU/ref=pd_lpo_sbs_dp_ss_2?pf_rd_p=1944687622&pf_rd_s=lpo-top-stripe-1&pf_rd_t=201&pf_rd_i=B00M4OEPLA&pf_rd_m=ATVPDKIKX0DER&pf_rd_r=0B6FPWEY6V86EKM17Y0X

seems to be available

8gb ram and a cheap 128gb ssd and your set

highwire

I have a Zotac ZBOX with the same RealTek PHYs. I was also getting WAN disconnects. I solved the discconnects by setting System -> Advanced -> Networking -> Disable Hardware Checksum Offload.

perth

@highwire:

I have a Zotac ZBOX with the same RealTek PHYs. I was also getting WAN disconnects. I solved the discconnects by setting System -> Advanced -> Networking -> Disable Hardware Checksum Offload.

I've applied this setting and rebooted the pfSense box.
We'll see how it goes. crosses fingers

perth

So the Disable Hardware Checksum Offload setting may have improved the reliability of my connection.
However I'm still getting excessive disconnects running this setup.

I think I'm going to try setting System > Advanced > Firewall and NAT > Disable Firewall Scrub "Disables the PF scrubbing option which can sometimes interfere with NFS and PPTP traffic." Since PPPOE is PPTP OE, if I understand correctly. Lets also disable IPv6, thought I had already turned that off…

Need to replace the modem and see what that does.

perth

Checked the "Configure a Null service name" under advanced WAN config, based off of other posts here for similar issues.
Think I've tried that already though…

unknownUser

Very interesting topic, I'm just about to order CI323, unfortunately my "trusted" shops don't have it on stock atm, so I had some time finding this topic ;)
Perth maybe you can help me with a few questions, as I'm not sure if this will work, so the following is my setup:
I've a NAS running at 1600MhZ (NSA 325 Zyxel). I've sabnzbd & Twonky only installed on it.
My actual router is the ASUS RT-AC87U and its running OpenVPN at 50 Mbit/s up and 10Mbit/s down.
Wifi is running on the wireless router which is an Apple TimeCapsule used for Backup and Wifi.

My aim is to reduce this setup, I'll keep the TimeCapsule for Wifi & Backup.
The ASUS and the NAS should be replaced with the ZBOX and I need some additional functions.

What I need running at the same time on the CI323 would be a VPN connection at 50 MBit/s,Sabnzbd using SSL (downloading & decompressing),Twonky (streaming) and an Adblocker or atleast a script using hosts files.
Will this CPU will be enough? Have you tried getting 100% CPU usage, how did you succeed?

perth

Disclaimer: Lots of the numbers below come from my memory. I'm about to go to sleep, and my memory isn't ever any good anyway. Double check any numbers I didn't copy/paste. I also suck at math.

Blindly following pfSense Hardware Crypto Doc (https://doc.pfsense.org/index.php/Are_cryptographic_accelerators_supported)

[2.2.6-RELEASE][admin@redacted]/root: openssl engine -t -c
(cryptodev) BSD cryptodev engine
 [RSA, DSA, DH]
     [ available ]
(rsax) RSAX engine support
 [RSA]
     [ available ]
(rdrand) Intel RDRAND engine
 [RAND]
     [ available ]
(dynamic) Dynamic engine loading support
     [ unavailable ]

[2.2.6-RELEASE][admin@redacted]/root: openssl speed -evp RSA
RSA is an unknown cipher or digest

[2.2.6-RELEASE][admin@redacted]/root: openssl speed -evp DSA
Doing dsaEncryption for 3s on 16 size blocks: 3413578 dsaEncryption's in 2.99s
Doing dsaEncryption for 3s on 64 size blocks: 2729635 dsaEncryption's in 2.99s
Doing dsaEncryption for 3s on 256 size blocks: 1553738 dsaEncryption's in 3.01s
Doing dsaEncryption for 3s on 1024 size blocks: 577673 dsaEncryption's in 3.00s
Doing dsaEncryption for 3s on 8192 size blocks: 88519 dsaEncryption's in 3.00s
OpenSSL 1.0.1l-freebsd 15 Jan 2015
built on: date not available
options:bn(64,64) rc4(16x,int) des(idx,cisc,16,int) aes(partial) idea(int) blowfish(idx) 
compiler: clang
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
dsaEncryption    18253.28k    58384.26k   132241.26k   197179.05k   241715.88k

[2.2.6-RELEASE][admin@redacted]/root: openssl speed -evp DH
DH is an unknown cipher or digest

Doing the above crypto performance test didn't cause my CPU to hit even 10%. I find the results quite confusing, the output states "The 'numbers' are in 1000s of bytes per second processed." and then throws a 'k' on the end of the numbers. Does that mean thousand thousand, or is the output redundant, but not multiplicative? No freaking clue, using the raw data :). Lets look at the worst: 3413578 16B blocks in 2.99 seconds, ((3413578 * 16) * 8 / 2.99)/(1024^2) = 139Mb/s of dsa Encryption. So I think my box is more than capable of the VPN workload you mention; if that's true the CI323 won't even be tickled. But then, I have almost no clue what I'm looking at; sorry.

I don't think you news reader program (SABnzbd) is of any significance, work load wise, in this setup.
I don't think Twonky is doing transcoding, but if it is, I wouldn't put it on the pfSense box. I'd estimate 1 maxed out thread per stream, possibly more?, I see that eating up enough CPU/RAM to impact your network throughput, plus cause lots of jitter. Since Twonky does embedded & Android systems though, it's probably not overly resource intensive, and probably fine.

All I can find (quickly) on your NAS' CPU is 1.6 GHz, so I'm assuming 1 core, and probably atom architecture. That's not a lot. If I'm wrong in that assumption, then maybe it's an x86 dual core; maybe. That would be the worst case scenario, and would be less than 50% of the CI323's compute capability. So looking at things that way, my box is 50% of the CI323, your NAS (worst case) is <50% of the CI323. I'm peaking at 20% CPU. So worst case is 70% (maxing out your NAS that I've made more awesome than I think it is) + VPN work load. Looking at it this way cuts things close, but I really doubt your NAS is that awesome; it's a NAS.

So can the CI323 handle your work load? I would say, "yeah". That said, I haven't asked my box to handle encryption/decryption of network traffic. I'm only on a 3Mb DSL connection. My connection is not stable (though I'm not blaming the CI321 for that). But the pfSense Hardware guide (https://www.pfsense.org/hardware/#requirements) recommends a 1GHz processor for your connection speed; CI323 has 8x that (4x 2GHz)?

So… yeah, it seems likely the CI323 can do what you are asking. I think the question you should consider is: do you like the architecture, including all the inherent benefits and draw backs, that you have outlined? Your pfSense box is the first line of defence, which is another way of saying that it's what gets shot full of holes first when bad stuff happens; also lightning. You are wanting to put your data there? Lots of people on these forums recommend against that design, it puts your data on the front line. (I was going to do the same thing, though pfSense and the NAS were going to be separate VMs. I understand the protection offered by VMs has been busted out of before. I did not implement that idea.) Also you have the problem/added complexity of dealing w/ the storage that won't fit inside of the CI323, but is electrically directly connected (lightning). Just food for thought.

If you do buy the CI323, I'd love to hear how it works out for you!

unknownUser

Thanks for the info.

I will take a closer look to understand the results you posted, as of now I don't really get it what they mean.
The newsreader sabnzbd is used for downloading big files, so it runs with full speed of 50 mbit/s, that causes my NAS to be used by 100%, depending on the resources avaiable the download speed get's slower (if I run twonky while it's downloading or unpacking). Also the unpacking takes forever. Yes It's 1.6Ghz single core Marvell Kirkwood, I don't know what that mean for it's architecture compared to celeron or atom.
All in all it sound good, what you are saying, it looks like running smooth.
The last thing I have to think about it is really the Harddrive, which has no sensitive data, I have to decide what and how and where to put it, I wanted it to replace the NAS.

If/When I get a CI323 I will report my finding!
Cheers!

interfasys

That's probably the cipher you need for OpenVPN

# openssl speed -elapsed -evp aes-256-cbc
You have chosen to measure elapsed time instead of user CPU time.
Doing aes-256-cbc for 3s on 16 size blocks: 944551 aes-256-cbc's in 3.01s
Doing aes-256-cbc for 3s on 64 size blocks: 874272 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 256 size blocks: 707852 aes-256-cbc's in 3.01s
Doing aes-256-cbc for 3s on 1024 size blocks: 410113 aes-256-cbc's in 3.03s
Doing aes-256-cbc for 3s on 8192 size blocks: 80373 aes-256-cbc's in 3.01s
OpenSSL 1.0.2f  28 Jan 2016
built on: reproducible build, date unspecified
options:bn(64,64) rc4(16x,int) des(idx,cisc,16,int) aes(partial) idea(int) blowfish(idx)
compiler: clang37 -I. -I.. -I../include  -fPIC -DOPENSSL_PIC -DOPENSSL_THREADS -pthread -D_THREAD_SAFE -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -DL_ENDIAN -O3 -Wall -O2 -pipe -D_FORTIFY_SOURCE=2 -flto -march=native  -fstack-protector-strong --param ssp-buffer-size=4 -fno-strict-aliasing -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DMD5_ASM -DAES_ASM -DVPAES_ASM -DBSAES_ASM -DWHIRLPOOL_ASM -DGHASH_ASM -DECP_NISTZ256_ASM
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-256-cbc       5024.52k    18651.14k    60246.48k   138542.09k   218901.82k

You'll easily reach 100Mbit, using one core, which leaves you with 3 for other things to do.

perth

@interfasys:

That's probably the cipher you need for OpenVPN
…
You'll easily reach 100Mbit, using one core, which leaves you with 3 for other things to do.

Thank you interfasys! Here's the result of the aes-256-cbc scheme speed test on the CI321:

[2.2.6-RELEASE][admin@redacted]/root: openssl speed -elapsed -evp aes-256-cbc
You have chosen to measure elapsed time instead of user CPU time.
Doing aes-256-cbc for 3s on 16 size blocks: 14575858 aes-256-cbc's in 3.03s
Doing aes-256-cbc for 3s on 64 size blocks: 3854920 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 256 size blocks: 973141 aes-256-cbc's in 3.01s
Doing aes-256-cbc for 3s on 1024 size blocks: 245621 aes-256-cbc's in 3.01s
Doing aes-256-cbc for 3s on 8192 size blocks: 30538 aes-256-cbc's in 3.00s
OpenSSL 1.0.1l-freebsd 15 Jan 2015
built on: date not available
options:bn(64,64) rc4(16x,int) des(idx,cisc,16,int) aes(partial) idea(int) blowfish(idx) 
compiler: clang
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-256-cbc      76936.49k    82238.29k    82825.67k    83620.87k    83389.10k

Applying the same formula I made up earlier:
((blocksEncrypted * bytesPerBlock) * 8 / seconds)/(1024^2) = Mb/s
16B = 587 Mb/s
64B = 627 Mb/s
256B = 631 Mb/s
1KB = 637 Mb/s
8KB = 636 Mb/s
(decimals truncated)

Did a little more looking around on the internet regarding throughput on custom firewalls, and the internet says we are paying too much attention to the CPU.
We should be looking at the bus the NIC is on. https://calomel.org/network_performance.html