PC Engines APU4C4 Experience

Qinn

Is there any reason why Auto (UFS) Guided Disk Setup is on top in the partitioning setup and Auto(ZFS) Guided Root-on-ZFS is way below in the pfSense Installer, as most will enter through the setup and thus choose the first?

gtrdriver

That Question ist very Interesting.

stephenw10

UFS is the supported install type at this time. It's likely we will move to ZFS as the default at some point but currectly the vast majority of installs are UFS and that's what we test against.
I'm not aware of any issues with ZFS though, I have several boxes running that.

Steve

Qinn

Thanks Steve, from what I remember UFS is the default Unix file system and ZFS is open source, but that's al I know, as I have more experience with ext3/4. So as those filesystems are a bit out of my wheelhouse, is there anything to say pro or contra, apart from the "auto repairing" that ZFS seems to have, when using a 16GB SSD durable and fast Solid state drive?

stephenw10

Well not much beyond UFS is far more tested and uses less resouces. ZFS uses significantly more resources but is far better at recovering from data loss or not losing it in the first place!
I'm far from an expert there either.

Steve

gtrdriver

Hello

I just made some tests today - on APU Board with a mSata ZFS Installation works like a Charm - fine !
I also Tryed to install ZFS VErsion on a APU Board with a SD Card and here it seem that ZFS dont work - is that possible ?
The ZFS Installer report a Error - but no more ...

Installation on SD witz Default UFS runs fine ...

stephenw10

Not sure I've ever tried that. What size/type of card? What was the error?

Steve

bouke

@gtrdriver said in PC Engines APU4C4 Experience:

Hello

I just made some tests today - on APU Board with a mSata ZFS Installation works like a Charm - fine !
I also Tryed to install ZFS VErsion on a APU Board with a SD Card and here it seem that ZFS dont work - is that possible ?
The ZFS Installer report a Error - but no more ...

Installation on SD witz Default UFS runs fine ...

I would not recommend to use a SD card like this. My estimated guess is that the SD card will wear out very soon if you are able to format it ZFS. That said: I guess the SD card is too slow compared to an SSD. I would UFS format it and use RAM disk. On the other hand: a RAM disk isn't really feasible with 4GB RAM.

I would stick with an SSD. I normally format these UFS if there's a UPS available. I will format ZFS if there isn't any.

viragomann

I installed pfSense with ZFS on an APU4C4 on a 16 GB mSATA. Works without issues.

fireodo

@viragomann said in PC Engines APU4C4 Experience:

I installed pfSense with ZFS on an APU4C4 on a 16 GB mSATA. Works without issues.

So have I too but on two APU2C0 one with a Zheino 16GB SSD and on the other on a Transcend 32GB - no issues so far ...

gtrdriver

Hello

Regarding ZFS on SD: There is no specific Error - the Installer Says:

ZFS setup failed x
x x
x An installation step has been aborted. Would x
x you like to restart the installation or exit x
x the installer? x
tqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq

Qinn

Ideally, ZFS requires a system with ECC memory, otherwise you're still not 100% safeguarded against bit errors..

I read that https://pcengines.ch/apu2.htm ECC is optional, can some comment on that?

Qinn

From what I've read https://blog.3mdeb.com/2018/2018-10-16-enabling-ecc-on-pc-engines-apu2/ every APU2 equal and above 4GB memory it is enabled....

Qinn

Recently I went over to from UFS to ZFS on a APU2C4 (RAM 4GB) with mSATA 16Gb, fresh install from USB then I restored the config.xml. I have a few Q's maybe someone can help, log file below...





 [1+ Disks] Striping provides maximum storage but no redundancy
 pfSense Installer
 qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq

             lqqqqqqqqqqqqqqqqZFS Configurationqqqqqqqqqqqqqqqqqqqk
             x Configure Options:                                 x
             x lqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqk x
             x x >>> Install          Proceed with Installation x x
             x x T Pool Type/Disks:   stripe: 0 disks           x x
             x x - Rescan Devices     *                         x x
             x x - Disk Info          *                         x x
             x x N Pool Name          zroot                     x x
             x x 4 Force 4K Sectors?  YES                       x x
             x x E Encrypt Disks?     NO                        x x
             x x P Partition Scheme   GPT (BIOS)                x x
             x x S Swap Size          2g                        x x
             x x M Mirror Swap?       NO                        x x
             x x W Encrypt Swap?      NO                        x x
             x mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj x
             tqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqu
             x             <Select>       <Cancel>                x
             mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj


 Create ZFS boot pool with displayed options
 pfSense Installer
 qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq



                lqqqqqqqqqqqqqZFS Configurationqqqqqqqqqqqqqqqk
                x Select Virtual Device type:                 x
                x lqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqk x
                x x stripe  Stripe - No Redundancy          x x
                x x mirror  Mirror - n-Way Mirroring        x x
                x x raid10  RAID 1+0 - n x 2-Way Mirrors    x x
                x x raidz1  RAID-Z1 - Single Redundant RAID x x
                x x raidz2  RAID-Z2 - Double Redundant RAID x x
                x x raidz3  RAID-Z3 - Triple Redundant RAID x x
                x mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj x
                tqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqu
                x           <  OK  >      <Cancel>            x
                mqqqqqqqq[Press arrows, TAB or ENTER]qqqqqqqqqj





 [1+ Disks] Striping provides maximum storage but no redundancy
 pfSense Installer
 qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq






                     lqqqqqqqqZFS Configurationqqqqqqqqqqk
                     x lqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqk x
                     x x[*] ada0  SATA SSD             x x
                     x x[ ] da0   VendorCo ProductCode x x
                     x mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj x
                     tqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqu
                     x       <  OK  >    < Back >        x
                     mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj
 pfSense Installer
 qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq





               lqqqqqqqqqqqqqqZFS Configurationqqqqqqqqqqqqqqqqqk
               x Last Chance! Are you sure you want to destroy  x
               x the current contents of the following disks:   x
               x                                                x
               x     ada0                                       x
               x                                                x
               tqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqu
               x             < YES >       < NO  >              x
               mqqqqqqqqq[Press arrows, TAB or ENTER]qqqqqqqqqqqj







ZFS NOTICE: Prefetch is disabled by default if less than 4GB of RAM is present;
            to enable, add "vfs.zfs.prefetch_disable=0" to /boot/loader.conf.
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
kern.geom.part.mbr.enforce_chs: 0 -> 0
 pfSense Installer
 qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq




                 lqqqqqqqqqqFetching Distributionqqqqqqqqqqqqk
                 x                                           x
                 x MANIFEST                   [   Done     ] x
                 x base.txz                   [   100%     ] x
                 x                                           x
                 x Fetching distribution files...            x
                 x                                           x
                 x  lqOverall Progressqqqqqqqqqqqqqqqqqqqqk  x
                 x  x                100%                 x  x
                 x  mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj  x
                 mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj
 pfSense Installer
 qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq




                 lqqqqqqqqqqChecksum Verificationqqqqqqqqqqqqk
                 x                                           x
                 x base.txz                   [In Progress ] x
                 x                                           x
                 x Verifying checksums of selected           x
                 x distributions.                            x
                 x                                           x
                 x  lqOverall Progressqqqqqqqqqqqqqqqqqqqqk  x
                 x  x                  0%                 x  x
                 x  mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj  x
                 mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj
 pfSense Installer
 qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq








                      lqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqk
                      x Checking distribution archives. x
                      x Please wait...                  x
                      mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj
 pfSense Installer
 qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq





                      lqqqqqqqArchive Extractionqqqqqqqqqk
                      x Extracting distribution files... x
                      x                                  x
                      x                                  x
                      x   Overall Progress:              x
                      x  lqqqqqqqqqqqqqqqqqqqqqqqqqqqqk  x
                      x  x            100%            x  x
                      x  mqqqqqqqqqqqqqqqqqqqqqqqqqqqqj  x
                      mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj








Looking for config.xml on da0s3: [found msdos] [no config.xml on da0s3, unmounting] diskid/DISK-920702597E285218784s3: [Could not locate an existing config.xml file!] Done.
 pfSense Installer
 qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq





                    lqqqqqqqManual Configurationqqqqqqqqqqk
                    x The installation is now finished.   x
                    x Before exiting the installer, would x
                    x you like to open a shell in the new x
                    x system to make any final manual     x
                    x modifications?                      x
                    tqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqu
                    x         < Yes >     < No  >         x
                    mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj
 pfSense Installer
 qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq






                         lqqqqqqqqqCompleteqqqqqqqqqqqk
                         x Installation of pfSense    x
                         x complete! Would you like   x
                         x to reboot into the         x
                         x installed system now?      x
                         tqqqqqqqqqqqqqqqqqqqqqqqqqqqqu
                         x   <Reboot>  <Shell >       x
                         mqqqqqqqqqqqqqqqqqqqqqqqqqqqqj







Nov  6 10:28:00  reboot: rebooted by root
Nov  6 10:28:00 Waiting (max 60 seconds) for system process `vnlru' to stop... done
Waiting (max 60 seconds) for system process `bufdaemon' to stop... done
Waiting (max 60 seconds) for system process `syncer' to stop...
Syncing disks, vnodes remaining... 0 0 done
All buffers synced.
Uptime: 26m58s
umass0: detached
uhub2: detached
PC Engines apu2
coreboot build 20190810
BIOS version v4.10.0.2
4080 MB ECC DRAM
SeaBIOS (version rel-1.12.1.3-0-g300e8b7)

Press F10 key now for boot menu

Booting from Hard Disk...
Boot failed: could not read the boot disk

Trying next device: 2
Booting from Hard Disk...
gptzfsboot: error 12 lba 32
gptzfsboot: error 12 lba 544
gptzfsboot: error 12 lba 1965088
gptzfsboot: error 12 lba 1965600
gptzfsboot: error 12 lba 1966072
gptzfsboot: error 12 lba 1
/boot/config: -S115200 -h

Consoles: serial port
BIOS drive C: is disk0
BIOS drive D: is disk1


read 1 from 0 to 0xcdf89440, error: 0xcread 1 from 0 to 0xcdf89440, error: 0xcBy

FreeBSD/x86 ZFS enabled bootstrap loader, Revision 1.1
(Wed Nov 21 08:03:10 EST 2018 root@buildbot2.nyi.netgate.com)
Loading /boot/defaults/loader.conf
-
          __
   _ __  / _|___  ___ _ __  ___  ___
  | '_ \| |_/ __|/ _ \ '_ \/ __|/ _ \
  | |_) |  _\__ \  __/ | | \__ \  __/
  | .__/|_| |___/\___|_| |_|___/\___|
  |_|


 +============Welcome to pfSense===========+   __________________________
 |                                         |  /                       ___\
 |  1. Boot Multi User [Enter]             | |                      /`
 |  2. Boot [S]ingle User                  | |                     /    :-|
 |  3. [Esc]ape to loader prompt           | |      _________  ___/    /_ |
 |  4. Reboot                              | |    /` ____   / /__    ___/ |
 |                                         | |   /  /   /  /    /   /     |
 |  Options:                               | |  /  /___/  /    /   /      |
 |  5. [K]ernel: kernel (1 of 2)           | | /   ______/    /   /       |
 |  6. Configure Boot [O]ptions...         | |/   /          /   /        |
 |  7. Select Boot [E]nvironment...        |     /          /___/         |
 |                                         |    /                         |
 |                                         |   /_________________________/
 +=========================================+

/boot/kernel/kernel text=0x17c1930 data=0xb93d38+0x557b28 syms=[0x8+0x197400+0x]
/boot/entropy size=0x1000
/boot/kernel/zfs.ko size 0x3821b0 at 0x2fde000
loading required module 'opensolaris'
/boot/kernel/opensolaris.ko size 0xa3b0 at 0x3361000

First Q

As you can see, there are a few error on first boot (did not see them at second boot)
gptzfsboot: error 12 lba 32
gptzfsboot: error 12 lba 544
gptzfsboot: error 12 lba 1965088
gptzfsboot: error 12 lba 1965600
gptzfsboot: error 12 lba 1966072
gptzfsboot: error 12 lba 1

What are they?

Second Q

The log mentioned

ZFS NOTICE: Prefetch is disabled by default if less than 4GB of RAM is present;
to enable, add "vfs.zfs.prefetch_disable=0" to /boot/loader.conf.
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
kern.geom.part.mbr.enforce_chs: 0 -> 0
pfSense Installer

As the APU2C4 has 4Gb memory, do I have to adjust it (see loader.conf below) and what does it bring?

[2.4.4-RELEASE][root@pfSense.localdomain]/boot: cat loader.conf
kern.cam.boot_delay=10000
kern.ipc.nmbclusters="1000000"
kern.ipc.nmbjumbop="524288"
kern.ipc.nmbjumbo9="524288"
kern.geom.label.disk_ident.enable="0"
kern.geom.label.gptid.enable="0"
zfs_load="YES"
boot_serial="YES"
console="comconsole"
comconsole_speed="115200"
autoboot_delay="3"
hw.usb.no_pf="1"

Third Q

Why does fstyp still mention UFS?

[2.4.4-RELEASE][root@pfSense.localdomain]/root: df -h
Filesystem                Size    Used   Avail Capacity  Mounted on
zroot/ROOT/default         12G    487M     11G     4%    /
devfs                     1.0K    1.0K      0B   100%    /dev
zroot/tmp                  11G    2.9M     11G     0%    /tmp
zroot/var                  12G    150M     11G     1%    /var
zroot                      11G     88K     11G     0%    /zroot
/dev/md0                  3.4M    148K    3.0M     5%    /var/run
devfs                     1.0K    1.0K      0B   100%    /var/dhcpd/dev
/lib                       12G    487M     11G     4%    /var/unbound/lib
devfs                     1.0K    1.0K      0B   100%    /var/unbound/dev
/var/log/pfblockerng       12G    150M     11G     1%    /var/unbound/var/log/pfblockerng
/usr/local/share/GeoIP     12G    487M     11G     4%    /var/unbound/usr/local/share/GeoIP
/usr/local/bin             12G    487M     11G     4%    /var/unbound/usr/local/bin
/usr/local/lib             12G    487M     11G     4%    /var/unbound/usr/local/lib
[2.4.4-RELEASE][root@pfSense.localdomain]/root: fstyp -l /dev/md0
ufs

Fourth Q

From what I read TRIM is enabled on ZFS, but how can I check it? ...

[2.4.4-RELEASE][root@pfSense.localdomain]/root: tunefs -p /
tunefs: /: could not open special device

Thanks for any help

Cheers Qinn

stephenw10

Looks like some problem with the GPT header that it was able to recover. Of you don't see it again then it's likely been fixed at the first boot.
You don't have to set that. It would probably only be a marginal drive performance increase and disk performance is not hugely important for pfSense. If you do choose to set it put it in /boot/loader.conf.local to avoid it being overwritten.
Ram drives are still formatted UFS.
No idea.

Steve

Qinn

Thanks Steve, would it be wise to set ( then maybe I have data redundancy, but not disk redundancy)

zfs set copies=2 yourpoolname

BTW I hope that anyone can shed a light on TRIM?

stephenw10

That's not something I've ever tried. If you have the space I guess it makes sense. It would double the writes to the drive though.

bigsy

@Qinn said in PC Engines APU4C4 Experience:

BTW I hope that anyone can shed a light on TRIM?

What is the output of sysctl vfs.zfs.trim

You should see something like:

[2.4.4-RELEASE] [root@b..]/root: sysctl vfs.zfs.trim
vfs.zfs.trim.max_interval: 1
vfs.zfs.trim.timeout: 30
vfs.zfs.trim.txg_delay: 32
vfs.zfs.trim.enabled: 1
[2.4.4-RELEASE] [root@b..]/root:

vfs.zfs.trim.enabled: 1 means TRIM is enabled.

Qinn

Thanks @bigsy

[2.4.4-RELEASE][root@pfSense.localdomain]/root: sysctl vfs.zfs.trim
vfs.zfs.trim.max_interval: 1
vfs.zfs.trim.timeout: 30
vfs.zfs.trim.txg_delay: 32
vfs.zfs.trim.enabled: 1

Did not change anything after install, seems @doktornotor was right https://forum.netgate.com/topic/84867/2-2-x-zfs-full-install-howto/9

Also here https://www.freebsd.org/doc/en_US.ISO8859-1/books/faq/all-about-zfs.html

Nice thread here https://forum.netgate.com/topic/112490/how-to-2-4-0-zfs-install-ram-disk-hot-spare-snapshot-resilver-root-drive from @pfBasic every command uses "yourpoolname" although I think it's in most cases "zroot" is there anyway to check, it seems he did not mentioned it.

For now I used :

[2.4.4-RELEASE][root@pfSense.localdomain]/root: zpool status
  pool: zroot
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        zroot       ONLINE       0     0     0
          ada0p3    ONLINE       0     0     0

errors: No known data errors

btw whats ada0p3 and why is the ram drive on UFS?

gtrdriver

Hello

Today i got msata Drives and i tryed to do a ZFS install - Install runs fine - no errors at all - but after Reboot System dont boot from msata

Tryed 5-6 Times - allways same issue - then installed standard UFS Auto installer - and this Installation boot without any problems.

Can anyone confirm this ?