PC Engines APU4C4 Experience



  • @gtrdriver said in PC Engines APU4C4 Experience:

    Hello

    I just made some tests today - on APU Board with a mSata ZFS Installation works like a Charm - fine !
    I also Tryed to install ZFS VErsion on a APU Board with a SD Card and here it seem that ZFS dont work - is that possible ?
    The ZFS Installer report a Error - but no more ...

    Installation on SD witz Default UFS runs fine ...

    I would not recommend to use a SD card like this. My estimated guess is that the SD card will wear out very soon if you are able to format it ZFS. That said: I guess the SD card is too slow compared to an SSD. I would UFS format it and use RAM disk. On the other hand: a RAM disk isn't really feasible with 4GB RAM.

    I would stick with an SSD. I normally format these UFS if there's a UPS available. I will format ZFS if there isn't any.



  • I installed pfSense with ZFS on an APU4C4 on a 16 GB mSATA. Works without issues.



  • @viragomann said in PC Engines APU4C4 Experience:

    I installed pfSense with ZFS on an APU4C4 on a 16 GB mSATA. Works without issues.

    So have I too but on two APU2C0 one with a Zheino 16GB SSD and on the other on a Transcend 32GB - no issues so far ...



  • Hello

    Regarding ZFS on SD: There is no specific Error - the Installer Says:

    ZFS setup failed x
    x x
    x An installation step has been aborted. Would x
    x you like to restart the installation or exit x
    x the installer? x
    tqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq



  • Ideally, ZFS requires a system with ECC memory, otherwise you're still not 100% safeguarded against bit errors..

    I read that https://pcengines.ch/apu2.htm ECC is optional, can some comment on that?



  • From what I've read https://blog.3mdeb.com/2018/2018-10-16-enabling-ecc-on-pc-engines-apu2/ every APU2 equal and above 4GB memory it is enabled....



  • Recently I went over to from UFS to ZFS on a APU2C4 (RAM 4GB) with mSATA 16Gb, fresh install from USB then I restored the config.xml. I have a few Q's maybe someone can help, log file below...

    
    
    
    
     [1+ Disks] Striping provides maximum storage but no redundancy
     pfSense Installer
     qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq
    
                 lqqqqqqqqqqqqqqqqZFS Configurationqqqqqqqqqqqqqqqqqqqk
                 x Configure Options:                                 x
                 x lqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqk x
                 x x >>> Install          Proceed with Installation x x
                 x x T Pool Type/Disks:   stripe: 0 disks           x x
                 x x - Rescan Devices     *                         x x
                 x x - Disk Info          *                         x x
                 x x N Pool Name          zroot                     x x
                 x x 4 Force 4K Sectors?  YES                       x x
                 x x E Encrypt Disks?     NO                        x x
                 x x P Partition Scheme   GPT (BIOS)                x x
                 x x S Swap Size          2g                        x x
                 x x M Mirror Swap?       NO                        x x
                 x x W Encrypt Swap?      NO                        x x
                 x mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj x
                 tqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqu
                 x             <Select>       <Cancel>                x
                 mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj
    
    
     Create ZFS boot pool with displayed options
     pfSense Installer
     qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq
    
    
    
                    lqqqqqqqqqqqqqZFS Configurationqqqqqqqqqqqqqqqk
                    x Select Virtual Device type:                 x
                    x lqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqk x
                    x x stripe  Stripe - No Redundancy          x x
                    x x mirror  Mirror - n-Way Mirroring        x x
                    x x raid10  RAID 1+0 - n x 2-Way Mirrors    x x
                    x x raidz1  RAID-Z1 - Single Redundant RAID x x
                    x x raidz2  RAID-Z2 - Double Redundant RAID x x
                    x x raidz3  RAID-Z3 - Triple Redundant RAID x x
                    x mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj x
                    tqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqu
                    x           <  OK  >      <Cancel>            x
                    mqqqqqqqq[Press arrows, TAB or ENTER]qqqqqqqqqj
    
    
    
    
    
     [1+ Disks] Striping provides maximum storage but no redundancy
     pfSense Installer
     qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq
    
    
    
    
    
    
                         lqqqqqqqqZFS Configurationqqqqqqqqqqk
                         x lqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqk x
                         x x[*] ada0  SATA SSD             x x
                         x x[ ] da0   VendorCo ProductCode x x
                         x mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj x
                         tqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqu
                         x       <  OK  >    < Back >        x
                         mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj
     pfSense Installer
     qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq
    
    
    
    
    
                   lqqqqqqqqqqqqqqZFS Configurationqqqqqqqqqqqqqqqqqk
                   x Last Chance! Are you sure you want to destroy  x
                   x the current contents of the following disks:   x
                   x                                                x
                   x     ada0                                       x
                   x                                                x
                   tqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqu
                   x             < YES >       < NO  >              x
                   mqqqqqqqqq[Press arrows, TAB or ENTER]qqqqqqqqqqqj
    
    
    
    
    
    
    
    ZFS NOTICE: Prefetch is disabled by default if less than 4GB of RAM is present;
                to enable, add "vfs.zfs.prefetch_disable=0" to /boot/loader.conf.
    ZFS filesystem version: 5
    ZFS storage pool version: features support (5000)
    kern.geom.part.mbr.enforce_chs: 0 -> 0
     pfSense Installer
     qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq
    
    
    
    
                     lqqqqqqqqqqFetching Distributionqqqqqqqqqqqqk
                     x                                           x
                     x MANIFEST                   [   Done     ] x
                     x base.txz                   [   100%     ] x
                     x                                           x
                     x Fetching distribution files...            x
                     x                                           x
                     x  lqOverall Progressqqqqqqqqqqqqqqqqqqqqk  x
                     x  x                100%                 x  x
                     x  mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj  x
                     mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj
     pfSense Installer
     qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq
    
    
    
    
                     lqqqqqqqqqqChecksum Verificationqqqqqqqqqqqqk
                     x                                           x
                     x base.txz                   [In Progress ] x
                     x                                           x
                     x Verifying checksums of selected           x
                     x distributions.                            x
                     x                                           x
                     x  lqOverall Progressqqqqqqqqqqqqqqqqqqqqk  x
                     x  x                  0%                 x  x
                     x  mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj  x
                     mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj
     pfSense Installer
     qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq
    
    
    
    
    
    
    
    
                          lqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqk
                          x Checking distribution archives. x
                          x Please wait...                  x
                          mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj
     pfSense Installer
     qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq
    
    
    
    
    
                          lqqqqqqqArchive Extractionqqqqqqqqqk
                          x Extracting distribution files... x
                          x                                  x
                          x                                  x
                          x   Overall Progress:              x
                          x  lqqqqqqqqqqqqqqqqqqqqqqqqqqqqk  x
                          x  x            100%            x  x
                          x  mqqqqqqqqqqqqqqqqqqqqqqqqqqqqj  x
                          mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj
    
    
    
    
    
    
    
    
    Looking for config.xml on da0s3: [found msdos] [no config.xml on da0s3, unmounting] diskid/DISK-920702597E285218784s3: [Could not locate an existing config.xml file!] Done.
     pfSense Installer
     qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq
    
    
    
    
    
                        lqqqqqqqManual Configurationqqqqqqqqqqk
                        x The installation is now finished.   x
                        x Before exiting the installer, would x
                        x you like to open a shell in the new x
                        x system to make any final manual     x
                        x modifications?                      x
                        tqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqu
                        x         < Yes >     < No  >         x
                        mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj
     pfSense Installer
     qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq
    
    
    
    
    
    
                             lqqqqqqqqqCompleteqqqqqqqqqqqk
                             x Installation of pfSense    x
                             x complete! Would you like   x
                             x to reboot into the         x
                             x installed system now?      x
                             tqqqqqqqqqqqqqqqqqqqqqqqqqqqqu
                             x   <Reboot>  <Shell >       x
                             mqqqqqqqqqqqqqqqqqqqqqqqqqqqqj
    
    
    
    
    
    
    
    Nov  6 10:28:00  reboot: rebooted by root
    Nov  6 10:28:00 Waiting (max 60 seconds) for system process `vnlru' to stop... done
    Waiting (max 60 seconds) for system process `bufdaemon' to stop... done
    Waiting (max 60 seconds) for system process `syncer' to stop...
    Syncing disks, vnodes remaining... 0 0 done
    All buffers synced.
    Uptime: 26m58s
    umass0: detached
    uhub2: detached
    PC Engines apu2
    coreboot build 20190810
    BIOS version v4.10.0.2
    4080 MB ECC DRAM
    SeaBIOS (version rel-1.12.1.3-0-g300e8b7)
    
    Press F10 key now for boot menu
    
    Booting from Hard Disk...
    Boot failed: could not read the boot disk
    
    Trying next device: 2
    Booting from Hard Disk...
    gptzfsboot: error 12 lba 32
    gptzfsboot: error 12 lba 544
    gptzfsboot: error 12 lba 1965088
    gptzfsboot: error 12 lba 1965600
    gptzfsboot: error 12 lba 1966072
    gptzfsboot: error 12 lba 1
    /boot/config: -S115200 -h
    
    Consoles: serial port
    BIOS drive C: is disk0
    BIOS drive D: is disk1
    
    
    read 1 from 0 to 0xcdf89440, error: 0xcread 1 from 0 to 0xcdf89440, error: 0xcBy
    
    FreeBSD/x86 ZFS enabled bootstrap loader, Revision 1.1
    (Wed Nov 21 08:03:10 EST 2018 root@buildbot2.nyi.netgate.com)
    Loading /boot/defaults/loader.conf
    -
              __
       _ __  / _|___  ___ _ __  ___  ___
      | '_ \| |_/ __|/ _ \ '_ \/ __|/ _ \
      | |_) |  _\__ \  __/ | | \__ \  __/
      | .__/|_| |___/\___|_| |_|___/\___|
      |_|
    
    
     +============Welcome to pfSense===========+   __________________________
     |                                         |  /                       ___\
     |  1. Boot Multi User [Enter]             | |                      /`
     |  2. Boot [S]ingle User                  | |                     /    :-|
     |  3. [Esc]ape to loader prompt           | |      _________  ___/    /_ |
     |  4. Reboot                              | |    /` ____   / /__    ___/ |
     |                                         | |   /  /   /  /    /   /     |
     |  Options:                               | |  /  /___/  /    /   /      |
     |  5. [K]ernel: kernel (1 of 2)           | | /   ______/    /   /       |
     |  6. Configure Boot [O]ptions...         | |/   /          /   /        |
     |  7. Select Boot [E]nvironment...        |     /          /___/         |
     |                                         |    /                         |
     |                                         |   /_________________________/
     +=========================================+
    
    /boot/kernel/kernel text=0x17c1930 data=0xb93d38+0x557b28 syms=[0x8+0x197400+0x]
    /boot/entropy size=0x1000
    /boot/kernel/zfs.ko size 0x3821b0 at 0x2fde000
    loading required module 'opensolaris'
    /boot/kernel/opensolaris.ko size 0xa3b0 at 0x3361000
    

    First Q

    As you can see, there are a few error on first boot (did not see them at second boot)
    gptzfsboot: error 12 lba 32
    gptzfsboot: error 12 lba 544
    gptzfsboot: error 12 lba 1965088
    gptzfsboot: error 12 lba 1965600
    gptzfsboot: error 12 lba 1966072
    gptzfsboot: error 12 lba 1

    What are they?

    Second Q

    The log mentioned

    ZFS NOTICE: Prefetch is disabled by default if less than 4GB of RAM is present;
    to enable, add "vfs.zfs.prefetch_disable=0" to /boot/loader.conf.
    ZFS filesystem version: 5
    ZFS storage pool version: features support (5000)
    kern.geom.part.mbr.enforce_chs: 0 -> 0
    pfSense Installer

    As the APU2C4 has 4Gb memory, do I have to adjust it (see loader.conf below) and what does it bring?

    [2.4.4-RELEASE][root@pfSense.localdomain]/boot: cat loader.conf
    kern.cam.boot_delay=10000
    kern.ipc.nmbclusters="1000000"
    kern.ipc.nmbjumbop="524288"
    kern.ipc.nmbjumbo9="524288"
    kern.geom.label.disk_ident.enable="0"
    kern.geom.label.gptid.enable="0"
    zfs_load="YES"
    boot_serial="YES"
    console="comconsole"
    comconsole_speed="115200"
    autoboot_delay="3"
    hw.usb.no_pf="1"
    
    

    Third Q

    Why does fstyp still mention UFS?

    [2.4.4-RELEASE][root@pfSense.localdomain]/root: df -h
    Filesystem                Size    Used   Avail Capacity  Mounted on
    zroot/ROOT/default         12G    487M     11G     4%    /
    devfs                     1.0K    1.0K      0B   100%    /dev
    zroot/tmp                  11G    2.9M     11G     0%    /tmp
    zroot/var                  12G    150M     11G     1%    /var
    zroot                      11G     88K     11G     0%    /zroot
    /dev/md0                  3.4M    148K    3.0M     5%    /var/run
    devfs                     1.0K    1.0K      0B   100%    /var/dhcpd/dev
    /lib                       12G    487M     11G     4%    /var/unbound/lib
    devfs                     1.0K    1.0K      0B   100%    /var/unbound/dev
    /var/log/pfblockerng       12G    150M     11G     1%    /var/unbound/var/log/pfblockerng
    /usr/local/share/GeoIP     12G    487M     11G     4%    /var/unbound/usr/local/share/GeoIP
    /usr/local/bin             12G    487M     11G     4%    /var/unbound/usr/local/bin
    /usr/local/lib             12G    487M     11G     4%    /var/unbound/usr/local/lib
    [2.4.4-RELEASE][root@pfSense.localdomain]/root: fstyp -l /dev/md0
    ufs
    
    

    Fourth Q

    From what I read TRIM is enabled on ZFS, but how can I check it? ...

    [2.4.4-RELEASE][root@pfSense.localdomain]/root: tunefs -p /
    tunefs: /: could not open special device
    
    

    Thanks for any help

    Cheers Qinn


  • Netgate Administrator

    1. Looks like some problem with the GPT header that it was able to recover. Of you don't see it again then it's likely been fixed at the first boot.

    2. You don't have to set that. It would probably only be a marginal drive performance increase and disk performance is not hugely important for pfSense. If you do choose to set it put it in /boot/loader.conf.local to avoid it being overwritten.

    3. Ram drives are still formatted UFS.

    4. No idea. 😉

    Steve



  • Thanks Steve, would it be wise to set ( then maybe I have data redundancy, but not disk redundancy)

    zfs set copies=2 yourpoolname
    

    BTW I hope that anyone can shed a light on TRIM?


  • Netgate Administrator

    That's not something I've ever tried. If you have the space I guess it makes sense. It would double the writes to the drive though.



  • @Qinn said in PC Engines APU4C4 Experience:

    BTW I hope that anyone can shed a light on TRIM?

    What is the output of sysctl vfs.zfs.trim

    You should see something like:

    [2.4.4-RELEASE] [root@b..]/root: sysctl vfs.zfs.trim
    vfs.zfs.trim.max_interval: 1
    vfs.zfs.trim.timeout: 30
    vfs.zfs.trim.txg_delay: 32
    vfs.zfs.trim.enabled: 1
    [2.4.4-RELEASE] [root@b..]/root:
    

    vfs.zfs.trim.enabled: 1 means TRIM is enabled.



  • Thanks @bigsy 👍

    [2.4.4-RELEASE][root@pfSense.localdomain]/root: sysctl vfs.zfs.trim
    vfs.zfs.trim.max_interval: 1
    vfs.zfs.trim.timeout: 30
    vfs.zfs.trim.txg_delay: 32
    vfs.zfs.trim.enabled: 1
    
    

    Did not change anything after install, seems @doktornotor was right https://forum.netgate.com/topic/84867/2-2-x-zfs-full-install-howto/9

    Also here https://www.freebsd.org/doc/en_US.ISO8859-1/books/faq/all-about-zfs.html

    Nice thread here https://forum.netgate.com/topic/112490/how-to-2-4-0-zfs-install-ram-disk-hot-spare-snapshot-resilver-root-drive from @pfBasic every command uses "yourpoolname" although I think it's in most cases "zroot" is there anyway to check, it seems he did not mentioned it.

    For now I used :

    [2.4.4-RELEASE][root@pfSense.localdomain]/root: zpool status
      pool: zroot
     state: ONLINE
      scan: none requested
    config:
    
            NAME        STATE     READ WRITE CKSUM
            zroot       ONLINE       0     0     0
              ada0p3    ONLINE       0     0     0
    
    errors: No known data errors
    
    

    btw whats ada0p3 and why is the ram drive on UFS?



  • Hello

    Today i got msata Drives and i tryed to do a ZFS install - Install runs fine - no errors at all - but after Reboot System dont boot from msata

    Tryed 5-6 Times - allways same issue - then installed standard UFS Auto installer - and this Installation boot without any problems.

    Can anyone confirm this ?



  • Some Additional Information:

    I have now successfully installed pfsense with ZFS on internal SATA Drive - System boot from MSATA - everything seem to be fine.

    Do some Software Reboots everything is ok

    The i do a Power Off (hardware) and then Reboot - and then System dont boot from ZFS Installation anymore.

    Dont get me wrong - i choose ZFS because i read about that ZFS is more stable at Unplaned Power OFF Situations but thats does not seem to work here ?!?

    Does anyone have other Information or better Results here ?



  • I just grabbed an APU4C4 (BIOS 4.10.0.2) and did a zfs install to test. Boots fine. Power off, pull plug, replace plug- boots fine. Reboot from console-works. Yank power cord, then plug back in- boots normally.
    What are you seeing on the console when it doesn't boot?



  • Hi

    just make Bios Update to 4.10.0.2

    Then make new install using ZFS (AUTO)
    I Dont make any changes only "Stripe" and sda0 for the single msata Disk
    Nothing else changed - is this right ?

    Then Installation runs trough until end is reached.
    ba22c761-2d3b-41ba-a8a1-32d65750825a-image.png

    Then Reboot.
    19b0aee8-5ef1-4a38-a549-2157d205ee6d-image.png

    Looking good.

    I will try again in report ...



  • My test was a vanilla install using zfs. My msata (Samsung 64 GB) shows as ada0, not sda0.



  • Hi

    just did 25 Power Shutdowns and It boot well - dont know why other tests today wernt wrong ....
    Perhaps its a Firmware Issue - i dont used 4.10.0.2 ....



  • @gtrdriver said in PC Engines APU4C4 Experience:

    Some Additional Information:

    I have now successfully installed pfsense with ZFS on internal SATA Drive - System boot from MSATA - everything seem to be fine.

    Do some Software Reboots everything is ok

    The i do a Power Off (hardware) and then Reboot - and then System dont boot from ZFS Installation anymore.

    Dont get me wrong - i choose ZFS because i read about that ZFS is more stable at Unplaned Power OFF Situations but thats does not seem to work here ?!?

    Does anyone have other Information or better Results here ?

    What I normally do is this:

    • accept the fact that stuff can break (it will break sooner or later... it's a fact)
    • keep a spare USB thumb drive with a working pfSense install image
    • create a backup after changing the configuration - and store it safely
    • enable auto backup and document the details required for a restore
    • connect the APU2/3/4 to a UPS to overcome (short) power outages
    • never pull the power cord; except when there isn't another option

    I did install quite some APU2/3/4 devices with pfSense(R) open source firewall/router software distribution. Some mSATA SSDs are UFS formatted and others are ZFS formatted. Some are connected to an UPS and some aren't. Over the years we have had some rare cases where we had to reinstall pfSense and restore the configuration from backup.

    Also be aware of facts like these: [quote]msata16a on shipment hold. Some customer reported problems leading to data loss, e.g. with Sophos UTM. We suspect that it is related to the TRIM function included in modern file systems (e.g. Linux EXT4). We have finally received a firmware update from our supplier. Customers may return modules to us for firmware reprogramming (data is lost), or for full credit. Modules reprogrammed by us can be recognized by a + mark on the label.[/quote] Source: https://www.pcengines.ch/msata16a.htm



  • ZFS seems the most vigorous FS available in pfS, that's why I moved over from UFS to ZFS. The (re)install is a breeze, with a current config file at hand.
    My APU2C4 has ECC and to me this and the ZFS FS seems max what I can do against sudden power surges, that being said, I would always use the "halt" command before I pull the power cord ;)

    @bouke Kudos for pointing out the flaky mSATA issue link, didn't knew that.


Log in to reply