Soekris 5501, CF, write performance issue

Guest

Hello,

perhaps this depends on the different meaning from the word "industrial".
I am personally using this cards often
Transcend Industrial Grade CF200I 4GB Compact Flash Speicherkarte
and in some rarely cases I am using this ones, but not so often!
Transcend Industrial 100x 4GB Compact Flash Speicherkarte

For high grade industrial CF Cards I suggest you to try out Amtron, Apro or Pretec.

elector

Yea ok, I meant industrial, my bad.

It's a 1GB "Cactus" CF card and I have also tried a 2GB Cactus card and two non-industrial Kingston and a noname cards but they all deliver the same result. All my CF cards is writeable at at least 5MB/s on a box running windows 7 and a USB card reader the Cactus card topping out at about 10MB/s.

In the mean time I've tried another box similar to the Soekris. I had a old PC Engines Alix 2d3 laying around, and it has the exact same issue.

I also have a newer PC Engines APU with a nice msata (ssd) card in it and it's FAST…

This leads me to think there might be some kind of issue with pfSense and CF drivers?

cmb

Windows isn't using the small block size dd uses by default. On an ALIX with a Kingston 4 GB "Elite Pro" CF (nothing fancy):

dd if=/dev/zero of=testfile count=10000

5120000 bytes transferred in 104.233252 secs (49121 bytes/sec)

dd if=/dev/zero of=testfile count=10 bs=1M

10485760 bytes transferred in 23.767149 secs (441187 bytes/sec)

Just a fact of life with CF and that method of testing.

elector

I should have mentioned that I already tried that (only bs=4096 though) and yea it did improved quite a bit but it's still floated around in the KB/s range.

On the production system that started all of my testing/benchmarking/writingInHere I got:

dd if=/dev/zero of=testfile count=10000
5120000 bytes transferred in 417.913803 secs (12251 bytes/sec)

dd if=/dev/zero of=testfile2 count=10 bs=1M
10485760 bytes transferred in 58.843716 secs (178197 bytes/sec)

And just to make sure it's not just dd that's the issue I tried to use fetch to download a 10MB file on a 100Mbit/s link close to the pfSense box and it also ran at pitiful speeds, around the 90KB/s mark.

elector

@cmb:

Windows isn't using the small block size dd uses by default. On an ALIX with a Kingston 4 GB "Elite Pro" CF (nothing fancy):

dd if=/dev/zero of=testfile count=10 bs=1M

10485760 bytes transferred in 23.767149 secs (441187 bytes/sec)

Btw. that is also quite poor speeds. That CF card should be doing at least 10x that.

elector

I just tried a couple of older releases on my Alix board and it turns out there's a MAJOR difference…

2.0 still hovers in the 170KB/s range
1.2.3 jumps up to ~4MB/s

and the blocksize parameter for dd doesn't change much now only 5-10% or so difference between no option and bs=1M

I know a lot has changed since 1.2.3 but maybe some of you can use this info to narrow down the issue? I'm still guess it's got something to do with the kernel drivers for accessing the CF card.

cmb

It's possibly a difference with whether or not DMA is enabled with that specific card.
https://doc.pfsense.org/index.php/Upgrade_Guide#Disk_Driver_Changes

elector

@cmb:

It's possibly a difference with whether or not DMA is enabled with that specific card.
https://doc.pfsense.org/index.php/Upgrade_Guide#Disk_Driver_Changes

You might very well be right.

I have been trying out various ATA modes today, I've gone through: BIOSDMA, PIO0 (alias BIOSPIO), PIO1, PIO2, PIO3, PIO4, WDMA2, UDMA2 (alias UDMA33), UDMA4, (alias UDMA66). None of them changes much, a few KB up or down, nothing major.

I have also experimented a bit with write cache on and off, and doesn't change much either.

PS. My Cactus card is UDMA4 enabled and is detected as such if I don't force the PIO/DMA mode.

demco

Elector,

Try mounting / filesystem as asynchronous (nanobsd default to synchronous) and see if it improve the write speed.

This involve editing /etc/inc/config.lib.inc, so backup the file and only proceed if you're comfortable with hand editing the files.
1. Under functions conf_mount_rw & conf_mount_ro, look for the mount command for / filesystem.

2. Remove the 'sync' flag. Save the file.

3. Cycle through rc.conf_mount_ro, rc.conf_mount_rw. You should see / no longer mount as sync.

If it improve your write speed, you will need to decide if the increase risk of filesystem corruption is worth it.

/ is mount as synchronous to minimize filesystem inconsistency after improper shutdown.

elector

@demco:

Try mounting / filesystem as asynchronous (nanobsd default to synchronous) and see if it improve the write speed.

Excellent, that did the trick, now I can write at ~7MB/s.

I'm not that worried about a broken filesystem, since pfSense are in read only mode most of the time and I do make fairly regular backups of the config so when/if it blows up I can get a new image up and running fairly quickly.

Thank you very much for you help.

Grogdor

Below is my experience with this issue. Should I use a system patch instead of editing the _rw/_ro functions in /etc/inc/config.lib.inc by hand? Is there a config toggle or better way? Should I leave /cf mounted sync, as that's where the conf/rrd/leases live? With the system on a UPS and the NanoBSD "backup partition" to fall back on, I'll take the usable performance.

I have an embedded system (it's rebranded, not sure of exact model#) with a 2GB consumer CF card that ran 2.1.3 fine. Pulled the system out of storage recently and decided to do a clean 2.2.2 install; I booted an Ubuntu livecd and downloaded then wrote the x86 2GB embedded VGA pfSense 2.2.2 image to CF at ~7MB/sec.

After getting it running, I tried to install some packages but it was taking ages. With iostat/dd/fetch, traced it down to the slow CF card, writing at 17-20KB/sec. Followed some booting tips to disable DMA, write caching and ACPI with no write-speed improvement. Had a poke at BIOS settings but nothing really jumped out. Fired up a FreeBSD 10.1R livecd and got good performance there, so it was back to poking pfSense… continued onto the second page of forum search results for "cf card" this time, which brings us here.


[2.2.2-RELEASE][admin@pash]/boot: mount
/dev/ufs/pfsense0 on / (ufs, local, noatime, synchronous)
/dev/ufs/cf on /cf (ufs, local, noatime, synchronous)

[2.2.2-RELEASE][admin@pash]/root/tmp: dd if=/dev/zero of=1MB bs=1M count=1
1048576 bytes transferred in 49.369825 secs (21239 bytes/sec)

[2.2.2-RELEASE][admin@pash]/root: mount -o noatime,async  /dev/ufs/pfsense0 /
/dev/ufs/pfsense0 on / (ufs, asynchronous, local, noatime)
/dev/ufs/cf on /cf (ufs, local, noatime, synchronous)

[2.2.2-RELEASE][admin@pash]/root/tmp: dd if=/dev/zero of=10MB bs=1m count=10
10485760 bytes transferred in 1.950883 secs (5374879 bytes/sec)

Edit: below is the system patch that I've applied…


--- config.lib.inc      2015-04-13 19:16:38.000000000 -0600
+++ config.lib.inc.async        2015-05-31 04:26:02.000000000 -0600
@@ -328,12 +328,12 @@
        /*    if the platform is soekris or wrap or pfSense, lets mount the
         *    compact flash cards root.
          */
-       $status = mwexec("/sbin/mount -u -w -o sync,noatime /");
+       $status = mwexec("/sbin/mount -u -w -o async,noatime /");
        /* we could not mount this correctly.  kick off fsck */
        if($status <> 0) {
                log_error(gettext("File system is dirty.  Launching FSCK for /"));
                mwexec("/sbin/fsck -y /");
-               $status = mwexec("/sbin/mount -u -w -o sync,noatime /");
+               $status = mwexec("/sbin/mount -u -w -o async,noatime /");
        }

        mark_subsystem_dirty('mount');
@@ -367,7 +367,7 @@
        /* sync data, then force a remount of /cf */
        pfSense_sync();
        mwexec("/sbin/mount -u -r -f -o sync,noatime {$g['cf_path']}");
-       mwexec("/sbin/mount -u -r -f -o sync,noatime /");
+       mwexec("/sbin/mount -u -r -f -o async,noatime /");
 }

 /****f* config/convert_config