Zotac CI 321 Dual NIC Nano



  • Hi,

    I'm looking to set up a dedicated pfsense box, here are my requirements:

    1. I want to encrypt all my traffic with OpenVPN and ensure it can handle ~100mbps for a bit of future proofing of my internet speed.
    2. I want as small a form factor as possible.
    3. I want it as cheap as possible.
    4. I want it to have dual NIC, not to mess with somehow getting another NIC into it.

    I was eye-ing the OEM Production 2550L2D - Dual Broadcomm NIC, Dual Core Atom 1.86Ghz. It fits the bill everything except 1 from what I've read.
    http://www.newegg.com/Product/Product.aspx?Item=N82E16856205007

    Now I see Zotac announced the ZBOX CI 321 which will be dual nic (unfortunately likely Realtek) and a 1.1 GHz Intel Celeron 2961Y dual-core “Haswell” processor.
    http://www.anandtech.com/Gallery/Album/4187

    What are your thoughts about this ZBox handling 100mbps? I can't find any cheap boxes with recent hardware, it's either that old Atom or this new ZBOX from what I can find.



  • I actually found your post just because I was wondering myself about CI321 as a pfSense box. I can't find it for sale at the moment, but I think this would be a great all in one box to get pfSense running. Hopefully it's not too expensive. I see ci320 (1x NIC) is $135 on amazon.



  • Does seem to be available in the US: http://www.newegg.com/Product/Product.aspx?Item=N82E16883218044

    interesting, I wonder if it's fanless -


  • Netgate Administrator

    Bit late to the party here but it won't do 100Mbps of OpenVPN. I would think 60-70 max.

    Steve



  • The Newegg product is the CI320 which does not have Dual Ethernet, while the still interesting but yet unavailable for purchase CI321 does have Dual Ethernet … I think that it would be a great box, will have a lot of power compared to the 2550 series, etc. I have been using the J1900 processors with a lot of success. Zotac has finally posted an official page on their website now for the CI321 ...

    http://www.zotac.com/products/mini-pcs/zbox-c-series/product/zbox-c-series/detail/zbox-ci321-nano-zbox-ci321nano.html



  • I found the ci321 on a german webpage (all though with an unknown delivery date) at €180 which is $193. Compared to that I payed the equivalent of $160 for the ci320 half a year ago, today the price of the ci320 seems to have risen to $183 where I'm at. So roughly the same price for the ci320 and ci321. Both the barebone configuration where you need to add ram and a system disk of some sort.

    The ci320 is a brilliant little machine, I've fitted an ssd and it runs completely silent, no heat issues at all or anything. Adding a second nic seems to make it even more perfect, but for some odd reason they downgraded the n2930 1.8 GHz quad-core cpu in the ci320 to a 1.1 GHz Celeron 2961Y dual-core cpu in the ci321. Fewer cores, slower clock rate and higher energy consumption, I wonder what that's all about?

    A bit unrelated, I'm running pfsense on my ci320 using vlan's on the single nic and paired with a Netgear GS105E switch. The E-version of the GS105 is a little gem, costs very little and is configurable so you can do port cloning for wireshark use, split the 5 ports into different vlan's and other stuff. Only issue is that the configuration software is windows only. If you go for the 8-port GS108E instead you get a web interface for configuring stuff, but it's also in a different price range.  Once configured, the GS105E remembers it's settings so no need for windows on a daily basis.

    I've rigged my GS105E to act like two separate switches in one. I've got port 1 connected to my ISP and port 2 to my pfsense. The switch tags all traffic to and from port 1 with a vlan tag and only allows it to reach port 2. When pfsense gets traffic with that tag it treats it as WAN traffic.

    Local LAN traffic gets a different vlan tag from pfsense and the switch forwards that traffic to port 3, which is connected to my regular LAN switch.

    I don't currently use port 4 and 5 for anything, but I could set them up to clone the traffic on port 3 if I wan't to sniff packets with wireshark or something.

    So in essence my GS105E currently runs as two switches in one - one that connects port 1 and 2 and another that connects port 2 and 3 and keeps everything nice and separate. The nic in my ci320 functions as two separate nics depending on which vlan tag the packets have. A bit tricky to set up, but works as a charm.



  • That is an interesting idea, I totally forgot about the application of VLAN. I actually have the bigger brother GS108T switch that adds PoE and LACP. I guess I could do something similar if I end up with one port router box, however I am still lurking around for either CI321 or a Chinese dual Intel nic box with Celeron. I am just leery to buy direcly from aliexpress and wait for some local distributor to pick those up here in US.



  • I ordered a sample unit to test, it finally shipped last week and should be in this week. I understand that a May 2015 release date has been promised on the CI321 with a price around $140, still don't see it available anywhere else other than the German site mentioned above.



  • I contacted Zotac twice about the CI321 and they were worthless.  They refused to answer when the item would reach distribution channels and kept directing me to contact a distributor.  The distributors have no interest in talking to consumers who are buying single quantity products.  So I'm getting ready to give up on Zotac and look for something else.



  • Another deficiency (though maybe not very important in case of pfSense) is that despite having two memory slots this box is configured for single channel operation.

    Anyway, has anyone tried Zotac CI321 with pfSence yet?



  • Wondering if anyone else has used this box for pfSense yet as well.



  • There is a new ZBOX-CI323NANO from Zotac http://liliputing.com/2015/10/zotac-launches-mini-pcs-with-intel-braswell-chips.html

    with Dual LAN and a Quad-Core Intel N3150 http://ark.intel.com/products/87258/Intel-Celeron-Processor-N3150-2M-Cache-up-to-2_08-GHz with AES-NI !! it should have plenty of Power for a Fast OpenVPN connection.

    Greetings Auric



  • The ZBOX-CI321NANO-U is now for sale!
    Amazon: https://www.amazon.com/gp/product/B00W8XXAJU
    Newegg: http://www.newegg.com/Product/Product.aspx?Item=N82E16856173122

    People should take note of the 1 review currently on Newegg:
    … Cons: NICs are realtek but I knew that buying it but one of the NICs will not auto-negotiate with a unmanaged switch defeating the purpose of the second NIC (was using PFSense on it). ...

    I'm currently buying one w/ 2x2GB memory, and an SSD (nothing lying around that'll work;) for my first pfSense venture anyway. If necessary I'll just manually set the speed on the port and life will be just fine (pretty sure I can do that, it's *nix afterall).

    Here's hoping!



  • @perth:

    The ZBOX-CI321NANO-U is now for sale!
    Amazon: https://www.amazon.com/gp/product/B00W8XXAJU
    Newegg: http://www.newegg.com/Product/Product.aspx?Item=N82E16856173122

    People should take note of the 1 review currently on Newegg:
    … Cons: NICs are realtek but I knew that buying it but one of the NICs will not auto-negotiate with a unmanaged switch defeating the purpose of the second NIC (was using PFSense on it). ...

    I'm currently buying one w/ 2x2GB memory, and an SSD (nothing lying around that'll work;) for my first pfSense venture anyway. If necessary I'll just manually set the speed on the port and life will be just fine (pretty sure I can do that, it's *nix afterall).

    Here's hoping!

    Hello, did you finally buy it ? Let me know if Pfsense works please, i want to buy this one



  • @milocheri:

    Hello, did you finally buy it ? Let me know if Pfsense works please, i want to buy this one

    Had some nonsense w/ Amazon, the previously linked system came bare-bones. Had to return it and ordered the bare-bones version (~$70 less) +RAM/SSD (~$70); the Zotac system is taking forever to ship… Not past the estimated delivery date yet though, and Amazon warned me. System should be here Tuesday, but I probably won't get to touch it until the week after that.

    I'll definitely update this thread when I know something. :)



  • The new Zotac CI323 previously mentioned is up for sale on Newegg: http://www.newegg.com/Product/Product.aspx?Item=N82E16856173128
    Currently at ~$10 over what I paid for the 321 for more than 2x the compute power of the 321 w/ a ~40% reduction in TDP.
    CI321 processor Intel spec. sheet: http://ark.intel.com/products/78943/Intel-Celeron-Processor-2961Y-2M-Cache-1_10-GHz
    CI323 processor Intel spec. sheet: http://ark.intel.com/products/87258/Intel-Celeron-Processor-N3150-2M-Cache-up-to-2_08-GHz

    Rather frustrating since the 321 just shipped today. -_-; Ah technology, you cruel cruel mistress.
    Return it ("again"), and buy the 323 delaying the project for another week? That'll give me more headroom for doing interesting or different things with the box.
    Buuuut it's going on a measly 3Mb DSL connection for traffic shaping & bandwidth monitoring. Guess I'll keep it, any input?
    My goals:

    • Have a learning experience.

    • Fairly and dynamically split the 3Mb connection into 4 logical groups (I expect only partial success).

    • Be a firewall.

    • Bandwidth usage monitoring (which group, what %). No clue if pfSense has this built in.

    • It'll be freaking cool



  • Well this second one seems to be better for the Intel Processor N3150 (Quad Core) keep us posted please, i'm waiting for your review to order mine Thanks !!

    http://cpuboss.com/cpus/Intel-Celeron-N3150-vs-Intel-Celeron-2961Y
    


  • Going w/ the Zotac CI321; decided I didn't want to wait any more. :)
    Negative: No serial port. All configuration, management, and/or recovery will have to be performed via HDMI/DP connected display & USB connected keyboard, or SSH.
    In other words, there is no low-level fall back recovery/configuration option (well, you could pull the drive…?).

    Booting pfSense on Zotac CI321:
    Following pfSense's guide to creating a bootable USB drive: https://doc.pfsense.org/index.php/Writing_Disk_Images

    • Used pfSense-memstick-2.2.5-RELEASE-amd64.img.gz
      • sha256 checksum verified
      • used bs=512 instead of bs=1M due to fdisk reporting that my dive was using 512 chunks
        Could not boot from USB. After playing w/ creating the bootable USB drive in different ways, finally found a PS2 to USB adaptor and got to look at the BIOS settings.
        You will need to modify the BIOS Boot settings
    • 'Del' gets you into BIOS Settings
      • Boot > Boot OS Selection: Set to Legacy Only (was set to uEFI Win8 by default IIRC).
        • I made some other changes in there, so it's possible you'll have to poke around some more.
          Now I could boot from the USB stick prepared according to pfSense directions linked above.
          Notes: Quick boot was disabled by default.
          Notes: "Intelligent" keyboards that take a long time to initialize (gaming keyboards) will most likely take too long to become available, and you won't be able to gain access to the BIOS. Have a basic USB keyboard available. There's a setting in the BIOS to increase the wait time for USB devices to initialize, I set mine to an insane 20 secs, could probably get away with 8. I'll worry about that later, the additional delay is worth increased reliability w/ my primary keyboard (assuming it works;).

    Installing pfSense on Zotac CI321:
    See: https://doc.pfsense.org/index.php/Installing_pfSense
    USB 2.0 boot drive was in a 3.0 front port.
    Chose '1'/'Enter'. Boot Multi User.
    Chose 'i' install pfSense when prompted.
    (Was unable to change Video Font, Screenmap, nor Keymap.)
    Chose Quick/Easy Install.
    MUST: Choose Standard Kernal; lack of serial on the box makes the Embedded kernal (no VGA) a bad choice, my opinion.
    Removed USB drive and restarted when prompted.

    1st Boot:
    (My Zotac box is not connected to any network.
    These are my answers, not a guide. Usefull for seeing what options pfSense makes available to you.
    Disclamer: This is the first time I'm touching pfSense; I'm probably going to break something;)
    Setup VLANs now [y|n]: N
    WAN interface name a=auto-detect (re0 re1 or a): re0
    LAN interface name (re1 a or nothing if finished): re1
    Optional 1 interface name ( a or nothing if finished): [Return]
    Confirm above config.

    CLI Config:
    pfSense finished booting (LOL it plays happy music!) and then gives you some options. I did the following:
    3) Reset webConfigurator password

    • Reset password to default. admin/pfsense
    1. Enable sshd
      8 ) Shell
    • Changed root account's password
    • Shell's available: sh, csh, tcsh, others?
      • No bash
      • passwd lists /bin/sh as default for root acount
      • passwd lists /etc/rc.initial as default for admin account
        • runs /bin/tcsh if in recovery console mode
        • is what creates that initial menu used above
    • exit takes you back to numeric menu created by /etc/rc.initial
      • Deduction: After boot you start as 'admin' account and choosing '8 ) Shell' is similar to typing su on a nomal *nix CLI.
      • Choosing 8 ) Shell bypasses root password even after being set?
    1. Set interface(s) IP address
      • Note: Configuring pfSense for shoving on my existing network for inital configuration.
    • Set my LAN interface to a safe IP (not in use, outside of DHCP range) valid for my LAN.
    • Subnet mask is set by CIDR notation, CIDR exaples for standard classful ranges are provided.
    • Disabled DHCP for LAN
    • Did not revert webConfigurator to HTTP (left as HTTPS)
    1. Reboot system
    • Confirm (Plays shutdown music;)
    • Config changes seem to be retained. Was never asked to save the above changes; all changes seem to be written to disk instantly. There isn't a 'backup' option in this menu, though there is a '15) Restore recent configuration' option; unsure of how this works.

    webConfigurator initial setup:
    Plugged pfSense box into (one of) my routers, and pulled up the webConfigurator.

    • Guessed that Ethernet port closest to antenna was re1; it was.
      Logged in w/ default admin/pfsense, was greated by an initial configuration wizard.
    • hostname: bridgekeeper ;)
    • Set DNS Servers (8.8.8.8, 8.8.4.4 for now)
    • Set timeserver & timezone
    • WAN Config
      • DHCP
      • All other fields left blank/default
    • LAN Config
      • Pre-filled w/ settings from earlier CLI config.
    • Set Admin WebGUI Password (also for SSH)
    • pfSense will 'reload' at this point.

    System resource usage at this point:
        MBUF Usage: 5% (1270/26584)
      Temperature: 27.8°C
      Load average: 0.01, 0.01, 0.00
        CPU usage: 0%
      Memory usage: 4% of 3984 MB
        SWAP usage: 0% of 8191 MB
        Disk usage: / (ufs): 1% of 50G
        Disk usage: /var/run (ufs in RAM): 3% of 3.4M

    System Specs:
    System: Zotac CI321: Intel 2961Y: 2 Thread, 2 Core, 1.1GHz: https://www.amazon.com/gp/product/B00W8XXAJU (http://ark.intel.com/products/78943/Intel-Celeron-Processor-2961Y-2M-Cache-1_10-GHz)
    RAM: 2x Kingston KVR16LS11S6/2: 2GB, 204-SODIMM, DDR3L-1600, CL11: https://www.amazon.com/gp/product/B00HVTHQ4Q
    SSD: ADATA SP600 ASP600S3-64GM-C: 64GB, SATA III, Synchronous NAND: https://www.amazon.com/gp/product/B009SX8WEQ
    Total cost to me: $196.42

    Output from 'sysctl -a': https://bpaste.net/show/978ef8d843d6
    Output from 'pciconf -lv': https://bpaste.net/show/11dd1f703c04

    More to follow…



  • Of course, while I was writing this post my connection to the internet flaked out again; and I clicked preview and lost everything. :/ Looks like an IP address change is the culprit:

      Nov 14 06:18:41  php-fpm[71380]: /rc.newwanip: IP has changed, killing states on former IP 172.78.111.78.
      Nov 14 06:18:41  php-fpm[71380]: /rc.newwanip: ROUTING: setting default route to 74.42.148.214
      Nov 14 06:18:46  php-fpm[71380]: /rc.newwanip: phpDynDNS (Redacted): PAYLOAD: ERROR: Invalid update URL (2)
      Nov 14 06:18:46  php-fpm[71380]: /rc.newwanip: phpDynDNS (Redacted): (Unknown Response)
      Nov 14 06:18:48  php-fpm[71380]: /rc.newwanip: phpDynDNS (Redacted): PAYLOAD: ERROR: Invalid update URL (2)
      Nov 14 06:18:48  php-fpm[71380]: /rc.newwanip: phpDynDNS (Redacted): (Unknown Response)
      Nov 14 06:18:49  php-fpm[71380]: /rc.newwanip: Resyncing OpenVPN instances for interface WAN.
      Nov 14 06:18:49  php-fpm[71380]: /rc.newwanip: Creating rrd update script
      Nov 14 06:18:51  php-fpm[71380]: /rc.newwanip: pfSense package system has detected an IP change or dynamic WAN reconnection - 172.78.1xx.xx -> 172.78.1yy.yy - Restarting packages.
      Nov 14 06:18:51  check_reload_status: Starting packages
      Nov 14 06:18:52  php-fpm[98867]: /rc.start_packages: Restarting/Starting all packages
    

    I'll have to look into how to make pfSense handle this better, if that's possible.

    ANYWAY

    The experience so far: (fun)
    I'm using the Zotac CI321 running pfSense. It seems to be working just fine, sans above flakyness. I had that earlier today after the new setup had replaced the ISP's provided modem/router/AP solution. It was bad enough I switched back to the ISP's device for a few hours (someone needed the internet:). I don't know if that was IP changes, or me poking around in pfSense's settings. I actually managed to lock myself out of the web GUI; even though I left the safety rules enabled. Still had SSH access though so I got it fixed. When I've had this new setup in place for some more time I'll give you a more definitive go ahead; if applicable.

    As to the review I quoted earlier, that said the Zotac CI321 running pfSense would only do 100Mbps. I can't say if this is true or not. Both of my interfaces have auto negotiated 100Mbps links, but the switch on the router/AP is only a 100Mbps link, and the other device is an ADSL 2+ modem where a 100Mbps link seems likely to be correct (why would it be higher?). Maybe tomorrow when I'm not thinking about climbing into bed I'll plug the Zotac box into something capable of gigabit speeds and see what happens. I guess I should have paid attention when I was preconfiguring it. : ) I can tell you that the pfSense web GUI will allow me to force 1000Mbps speeds on the interfaces. I don't know if that menu is adaptive to the hardware/drivers or not though. See: https://doc.pfsense.org/index.php/Forcing_Interface_Speed_or_Duplex_Settings

    My Setup:
    Frontier ADSL 2+ 3Mbit/~800bps D/U -> TP-LINK TD-8616 -> (re0 PPPoE) Zotac CI321, pfSense 2.2.5-RELEASE (re1) -> Linksys E2500, Bridged Mode, everything disabled



  • From the previously linked sysctl -a output:

    re0: <realtek 8111="" 8168="" b="" c="" cp="" d="" dp="" e="" f="" g="" pcie="" gigabit="" ethernet=""> port 0xe000-0xe0ff mem 0xf0104000-0xf0104fff,0xf0100000-0xf0103fff irq 19 at device 0.0 on pci3
    re0: Using 1 MSI-X message
    re0: Chip rev. 0x2c800000
    re0: MAC rev. 0x00100000
    miibus0: <mii bus=""> on re0
    rgephy0: <rtl8169s 8211="" 8110s="" 1000base-t="" media="" interface=""> PHY 1 on miibus0
    rgephy0:  none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow
    re0: Ethernet address: 00:01:2e:64:ee:d3
    pcib4: <acpi pci-pci="" bridge=""> irq 16 at device 28.4 on pci0
    pci4: <acpi pci="" bus=""> on pcib4
    re1: <realtek 8111="" 8168="" b="" c="" cp="" d="" dp="" e="" f="" g="" pcie="" gigabit="" ethernet=""> port 0xd000-0xd0ff mem 0xf0004000-0xf0004fff,0xf0000000-0xf0003fff irq 16 at device 0.0 on pci4
    re1: Using 1 MSI-X message
    re1: Chip rev. 0x2c800000
    re1: MAC rev. 0x00100000
    miibus1: <mii bus=""> on re1
    rgephy1: <rtl8169s 8211="" 8110s="" 1000base-t="" media="" interface=""> PHY 1 on miibus1
    rgephy1:  none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow</rtl8169s></mii></realtek></acpi></acpi></rtl8169s></mii></realtek>
    

    This last line seems to match what the menu for forcing the interface speeds offered. So I'd bet that gigabit works just fine.



  • Conclusion
    I believe I can say without hesitation, that the Zotac CI321 is entirely capable of being a basic pfSense appliance. The below graphs, taken directly from pfSense, should make that fairly obvious. The load average seems to (almost) always be below .1 for 1, 5, and 15 minute averages. The temperature seems to stay between 50 & 55 degrees C. (As reported by pfSense's System Information.)
    I cannot say what the limitations of this fanless system will be, or how many additional features/applications/packages can be implemented on it w/out causing issue.
    While I have had some issues with WAN disconnections since implementing my new network solution; I do not believe that the Zotac CI321 is responsible. I believe the issue is either that my ISP is irritated with me removing their solution (to which they seemed to have root access), or something in how I've configured the modem or pfSense box. (Example: disconnections have decreased since disabling gateway monitoring, which had been pinging the gateway once every second.)

    With that said, I would not buy the CI321 again. As previously mentioned, Zotac has since released the CI323. That system, spec wise, is significantly better for an extremely minor increase in cost (at the time). Considering the success of this system, I would feel reasonably comfortable assuming the CI323 will also be compatible.
    Caveats:

    • I have not attempted to use the included WiFi or Bluetooth. In pfSense, under Interfaces > (assign) > Wireless the Parent Interface drop-down is not populated. I have made absolutely no attempt to get the wireless NIC to work; I have no interest in it. See the sysctl or pciconf output in one of my earlier posts to determine if this interface is compatible with pfSense/FreeBSD.
    • You have to change the Boot Mode option in the UEFI BIOS before the CI321 will boot something other than Windows. See random YouTube video of the BIOS I found: https://www.youtube.com/watch?v=Cznx10PqoR0

    Hope this helps! Feel free to ask me questions about this device, or for some specific output (provide instructions, just in case).

    RRD Graphs, 8 hour period, 1 minute average
    Throughput States Processor Memory Mbuf Clusters

    RRD Graphs, 1 week period, 1 hour average
    Throughput States Processor Memory Mbuf Clusters



  • This post is deleted!


  • I installed, and am more or less constantly viewing, ntopnp. Small increase in processor usage and memory.

    RRD Graphs, 1 day period, 5 minute average
          Processor                  Memory

    I'm loving this thing. Just need to figure out where the problem is that is causing all these disconnects…



  • can you try with the ac wifi?

    just want to know if it works & the performance of 1T1R

    thanks



  • Well this is a very intresting topic to read. This is also the main reason i bought a CI323 so i can install pfSense on it.
    When i recieve my order i will start a topic with my findings, it will be a setup with a OpenWRT router and several VLAN's.
    Somebody that has this kind of setup?



  • I have this kind of setup. While waiting for a suitable box, which now is looking like the ci323, I've been running pfsense very comfortably in a VM. I have an Asus n66u running Tomato/shibby that acts as an access point and backup router.

    The cable modem is vlan'd onto a switch and serves one public IP to the AP and one public IP to pfsense on the VM, soon to be a box like the 323.

    The router runs a heartbeat script against pfsense such that if the VM goes down, within one minute the router will create a virtual interface matching the IP of pfsense so that devices continue to function transparently. When the VM comes back up, the router will tear down the interface and all is well again. Dhcp is not a problem because the ap is responsible for that, not pfsense. The heartbeat is on an aliased IP of pfsense.

    I needed vlans to achieve this. I also experimented with vlans in case I had to settle on a box with one nic. This works perfectly also.

    To tell you the truth, the VM setup is working so well, I'm questioning getting a box, especially with this redundancy in place.



  • @duren:

    I have this kind of setup. While waiting for a suitable box, which now is looking like the ci323, I've been running pfsense very comfortably in a VM. I have an Asus n66u running Tomato/shibby that acts as an access point and backup router.

    The cable modem is vlan'd onto a switch and serves one public IP to the AP and one public IP to pfsense on the VM, soon to be a box like the 323.

    The router runs a heartbeat script against pfsense such that if the VM goes down, within one minute the router will create a virtual interface matching the IP of pfsense so that devices continue to function transparently. When the VM comes back up, the router will tear down the interface and all is well again. Dhcp is not a problem because the ap is responsible for that, not pfsense. The heartbeat is on an aliased IP of pfsense.

    I needed vlans to achieve this. I also experimented with vlans in case I had to settle on a box with one nic. This works perfectly also.

    To tell you the truth, the VM setup is working so well, I'm questioning getting a box, especially with this redundancy in place.

    Well that sounds interesting, did you make the heartbeat script?
    Sounds like a homebrew way to provide a FHRP.



  • @ReFleX:

    Well that sounds interesting, did you make the heartbeat script?
    Sounds like a homebrew way to provide a FHRP.

    That's right, it's a bash script that executes as a cron job on the router. It simply setups or or tears down that virtual interface depending on the result of a ping every minute.



  • How are you guys doing with the CI323, I was about to order one on Amazon for 149 barebones, but I am just doing one last round of research. The processor should be great, just wondering about the NICs. I have tried to find an N3150 with Intel NICs, but it was not very fruitful.





  • I have a Zotac ZBOX with the same RealTek PHYs.  I was also getting WAN disconnects.  I solved the discconnects by setting System -> Advanced -> Networking -> Disable Hardware Checksum Offload.



  • @highwire:

    I have a Zotac ZBOX with the same RealTek PHYs.  I was also getting WAN disconnects.  I solved the discconnects by setting System -> Advanced -> Networking -> Disable Hardware Checksum Offload.

    I've applied this setting and rebooted the pfSense box.
    We'll see how it goes. crosses fingers



  • So the Disable Hardware Checksum Offload setting may have improved the reliability of my connection.
    However I'm still getting excessive disconnects running this setup.

    I think I'm going to try setting System > Advanced > Firewall and NAT > Disable Firewall Scrub "Disables the PF scrubbing option which can sometimes interfere with NFS and PPTP traffic." Since PPPOE is PPTP OE, if I understand correctly. Lets also disable IPv6, thought I had already turned that off…

    Need to replace the modem and see what that does.



  • Checked the "Configure a Null service name" under advanced WAN config, based off of other posts here for similar issues.
    Think I've tried that already though…



  • Very interesting topic, I'm just about to order CI323, unfortunately my "trusted" shops don't have it on stock atm, so I had some time finding this topic ;)
    Perth maybe you can help me with a few questions, as I'm not sure if this will work, so the following is my setup:
    I've a NAS running at 1600MhZ (NSA 325 Zyxel). I've sabnzbd & Twonky only installed on it.
    My actual router is the ASUS RT-AC87U and its running OpenVPN at 50 Mbit/s up and 10Mbit/s down.
    Wifi is running on the wireless router which is an Apple TimeCapsule used for Backup and Wifi.

    My aim is to reduce this setup, I'll keep the TimeCapsule for Wifi & Backup.
    The ASUS and the NAS should be replaced with the ZBOX and I need some additional functions.

    What I need running at the same time on the CI323 would be a VPN connection at 50 MBit/s,Sabnzbd using SSL (downloading & decompressing),Twonky (streaming) and an Adblocker or atleast a script using hosts files.
    Will this CPU will be enough? Have you tried getting 100% CPU usage, how did you succeed?



  • Disclaimer: Lots of the numbers below come from my memory. I'm about to go to sleep, and my memory isn't ever any good anyway. Double check any numbers I didn't copy/paste. I also suck at math.

    Blindly following pfSense Hardware Crypto Doc (https://doc.pfsense.org/index.php/Are_cryptographic_accelerators_supported)

    [2.2.6-RELEASE][admin@redacted]/root: openssl engine -t -c
    (cryptodev) BSD cryptodev engine
     [RSA, DSA, DH]
         [ available ]
    (rsax) RSAX engine support
     [RSA]
         [ available ]
    (rdrand) Intel RDRAND engine
     [RAND]
         [ available ]
    (dynamic) Dynamic engine loading support
         [ unavailable ]
    
    [2.2.6-RELEASE][admin@redacted]/root: openssl speed -evp RSA
    RSA is an unknown cipher or digest
    
    [2.2.6-RELEASE][admin@redacted]/root: openssl speed -evp DSA
    Doing dsaEncryption for 3s on 16 size blocks: 3413578 dsaEncryption's in 2.99s
    Doing dsaEncryption for 3s on 64 size blocks: 2729635 dsaEncryption's in 2.99s
    Doing dsaEncryption for 3s on 256 size blocks: 1553738 dsaEncryption's in 3.01s
    Doing dsaEncryption for 3s on 1024 size blocks: 577673 dsaEncryption's in 3.00s
    Doing dsaEncryption for 3s on 8192 size blocks: 88519 dsaEncryption's in 3.00s
    OpenSSL 1.0.1l-freebsd 15 Jan 2015
    built on: date not available
    options:bn(64,64) rc4(16x,int) des(idx,cisc,16,int) aes(partial) idea(int) blowfish(idx) 
    compiler: clang
    The 'numbers' are in 1000s of bytes per second processed.
    type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
    dsaEncryption    18253.28k    58384.26k   132241.26k   197179.05k   241715.88k
    
    [2.2.6-RELEASE][admin@redacted]/root: openssl speed -evp DH
    DH is an unknown cipher or digest
    

    Doing the above crypto performance test didn't cause my CPU to hit even 10%. I find the results quite confusing, the output states "The 'numbers' are in 1000s of bytes per second processed." and then throws a 'k' on the end of the numbers. Does that mean thousand thousand, or is the output redundant, but not multiplicative? No freaking clue, using the raw data :). Lets look at the worst: 3413578 16B blocks in 2.99 seconds, ((3413578 * 16) * 8 / 2.99)/(1024^2) = 139Mb/s of dsa Encryption. So I think my box is more than capable of the VPN workload you mention; if that's true the CI323 won't even be tickled. But then, I have almost no clue what I'm looking at; sorry.

    I don't think you news reader program (SABnzbd) is of any significance, work load wise, in this setup.
    I don't think Twonky is doing transcoding, but if it is, I wouldn't put it on the pfSense box. I'd estimate 1 maxed out thread per stream, possibly more?, I see that eating up enough CPU/RAM to impact your network throughput, plus cause lots of jitter. Since Twonky does embedded & Android systems though, it's probably not overly resource intensive, and probably fine.

    All I can find (quickly) on your NAS' CPU is 1.6 GHz, so I'm assuming 1 core, and probably atom architecture. That's not a lot. If I'm wrong in that assumption, then maybe it's an x86 dual core; maybe. That would be the worst case scenario, and would be less than 50% of the CI323's compute capability. So looking at things that way, my box is 50% of the CI323, your NAS (worst case) is <50% of the CI323. I'm peaking at 20% CPU. So worst case is 70% (maxing out your NAS that I've made more awesome than I think it is) + VPN work load. Looking at it this way cuts things close, but I really doubt your NAS is that awesome; it's a NAS.

    So can the CI323 handle your work load? I would say, "yeah". That said, I haven't asked my box to handle encryption/decryption of network traffic. I'm only on a 3Mb DSL connection. My connection is not stable (though I'm not blaming the CI321 for that). But the pfSense Hardware guide (https://www.pfsense.org/hardware/#requirements) recommends a 1GHz processor for your connection speed; CI323 has 8x that (4x 2GHz)?

    So… yeah, it seems likely the CI323 can do what you are asking. I think the question you should consider is: do you like the architecture, including all the inherent benefits and draw backs, that you have outlined? Your pfSense box is the first line of defence, which is another way of saying that it's what gets shot full of holes first when bad stuff happens; also lightning. You are wanting to put your data there? Lots of people on these forums recommend against that design, it puts your data on the front line. (I was going to do the same thing, though pfSense and the NAS were going to be separate VMs. I understand the protection offered by VMs has been busted out of before. I did not implement that idea.) Also you have the problem/added complexity of dealing w/ the storage that won't fit inside of the CI323, but is electrically directly connected (lightning). Just food for thought.

    If you do buy the CI323, I'd love to hear how it works out for you!



  • Thanks for the info.

    I will take a closer look to understand the results you posted, as of now I don't really get it what they mean.
    The newsreader sabnzbd is used for downloading big files, so it runs with full speed of 50 mbit/s, that causes my NAS to be used by 100%, depending on the resources avaiable the download speed get's slower (if I run twonky while it's downloading or unpacking). Also the unpacking takes forever. Yes It's 1.6Ghz single core Marvell Kirkwood, I don't know what that mean for it's architecture compared to celeron or atom.
    All in all it sound good, what you are saying, it looks like running smooth.
    The last thing I have to think about it is really the Harddrive, which has no sensitive data, I have to decide what and how and where to put it, I wanted it to replace the NAS.

    If/When I get a CI323 I will report my finding!
    Cheers!



  • That's probably the cipher you need for OpenVPN

    # openssl speed -elapsed -evp aes-256-cbc
    You have chosen to measure elapsed time instead of user CPU time.
    Doing aes-256-cbc for 3s on 16 size blocks: 944551 aes-256-cbc's in 3.01s
    Doing aes-256-cbc for 3s on 64 size blocks: 874272 aes-256-cbc's in 3.00s
    Doing aes-256-cbc for 3s on 256 size blocks: 707852 aes-256-cbc's in 3.01s
    Doing aes-256-cbc for 3s on 1024 size blocks: 410113 aes-256-cbc's in 3.03s
    Doing aes-256-cbc for 3s on 8192 size blocks: 80373 aes-256-cbc's in 3.01s
    OpenSSL 1.0.2f  28 Jan 2016
    built on: reproducible build, date unspecified
    options:bn(64,64) rc4(16x,int) des(idx,cisc,16,int) aes(partial) idea(int) blowfish(idx)
    compiler: clang37 -I. -I.. -I../include  -fPIC -DOPENSSL_PIC -DOPENSSL_THREADS -pthread -D_THREAD_SAFE -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -DL_ENDIAN -O3 -Wall -O2 -pipe -D_FORTIFY_SOURCE=2 -flto -march=native  -fstack-protector-strong --param ssp-buffer-size=4 -fno-strict-aliasing -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DMD5_ASM -DAES_ASM -DVPAES_ASM -DBSAES_ASM -DWHIRLPOOL_ASM -DGHASH_ASM -DECP_NISTZ256_ASM
    The 'numbers' are in 1000s of bytes per second processed.
    type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
    aes-256-cbc       5024.52k    18651.14k    60246.48k   138542.09k   218901.82k
    

    You'll easily reach 100Mbit, using one core, which leaves you with 3 for other things to do.



  • @interfasys:

    That's probably the cipher you need for OpenVPN

    You'll easily reach 100Mbit, using one core, which leaves you with 3 for other things to do.

    Thank you interfasys! Here's the result of the aes-256-cbc scheme speed test on the CI321:

    [2.2.6-RELEASE][admin@redacted]/root: openssl speed -elapsed -evp aes-256-cbc
    You have chosen to measure elapsed time instead of user CPU time.
    Doing aes-256-cbc for 3s on 16 size blocks: 14575858 aes-256-cbc's in 3.03s
    Doing aes-256-cbc for 3s on 64 size blocks: 3854920 aes-256-cbc's in 3.00s
    Doing aes-256-cbc for 3s on 256 size blocks: 973141 aes-256-cbc's in 3.01s
    Doing aes-256-cbc for 3s on 1024 size blocks: 245621 aes-256-cbc's in 3.01s
    Doing aes-256-cbc for 3s on 8192 size blocks: 30538 aes-256-cbc's in 3.00s
    OpenSSL 1.0.1l-freebsd 15 Jan 2015
    built on: date not available
    options:bn(64,64) rc4(16x,int) des(idx,cisc,16,int) aes(partial) idea(int) blowfish(idx) 
    compiler: clang
    The 'numbers' are in 1000s of bytes per second processed.
    type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
    aes-256-cbc      76936.49k    82238.29k    82825.67k    83620.87k    83389.10k
    

    Applying the same formula I made up earlier:
    ((blocksEncrypted * bytesPerBlock) * 8 / seconds)/(1024^2) = Mb/s
    16B  =  587 Mb/s
    64B  =  627 Mb/s
    256B =  631 Mb/s
    1KB  =  637 Mb/s
    8KB  =  636 Mb/s
    (decimals truncated)

    Did a little more looking around on the internet regarding throughput on custom firewalls, and the internet says we are paying too much attention to the CPU.
    We should be looking at the bus the NIC is on. https://calomel.org/network_performance.html



  • Interesting to see how much slower the C321 (1.1Ghz) is in this openssl test compared to the C323 (1.6Ghz) with AESNI.

    But regardless, the speed is going to depend a lot on how you're able to tune your connection. If you're connecting to a public provider, you'll be limited to what they offer and ~100Mbit is what you're going to reach.

    Regarding the bus, there is nothing to worry about today. PCI Express 1x is enough to drive even a dual-port Intel NIC.

    And beware of tips such as

    Ideally you want to use a server based add on card with a TCP offload engine or TCP accelerator.

    Netmap is fast and requires all hardware acceleration to be turned off.
    It's still a good idea to get Intel NICs for Gigabit WAN connections (or peace of mind), but it's tricky to add to a Zotac Nano…


Log in to reply