Ad0 errors with CompactFlash Sandisk cards and Soekris net5501

jimp · Oct 24, 2011, 1:45 PM

Can you find the real size of the available space on those cards?

We have been shrinking the image sizes more and more lately because the cards have been shrinking. The images for 2.0.1+ are smaller than ever, because Sandisk has been shipping cards with sizes smaller than those being shipped in current hardware from Netgate and such. We have given the folks at Netgate a new set of images that have slightly smaller sizes, which is likely what you need here.

I don't know if Netgate has posted theirs anywhere yet. We'll be releasing 2.0.1 in the very near future, so it'd be worth trying once we get a new set of images out.

fabix · Oct 24, 2011, 2:00 PM

I don't know how to get more details. Even if the card is smaller than 4GB, I used a 2GB image to be sure it can be contained on the card.

Wim Vandeputte confirmed me last SanDisk cards are smaller than 4GB. He adviced me to install a smaller image of pfSense. But that doesn't solve the problem.

Actually, I installed recently a new firewall (same hardware) and I don't have these errors. So, maybe it's a problem of card…

jimp · Oct 24, 2011, 2:24 PM

That may be, then… But it did look like the same sort of error (can't read repeatedly from the same high LBA) that I've seen when someone has written an image too large to the card.

nexus_VI · Oct 31, 2011, 1:10 PM

I'm seeing this same error here using a SanDisk card with the 4G image, namely:

Master: ad0 <sandisk sdcfh-004g="" hdx="" 6.03="">ATA/ATAPI revision 0

The main question is, will I face problems when operating the system like this, should I wait for the 2.0.1 images?</sandisk>

jimp · Oct 31, 2011, 1:15 PM

If you are running the 4G version and getting those errors, then yeah you'll want to wait if you're seeing those errors.

If you are running a 1G/2G version and getting errors, then you may have some other issue with the card not related to the size.

wm408 · Nov 1, 2011, 8:00 PM

In a net6501, I am having sort of similar errors on a 4G mSATA card that I purchased from Soekris, its a Transcend 4g MSA500 (I think?, something like this). I tested 2G and 4G nanobsd images on it with the same result.

Here is the link to my post: http://forum.pfsense.org/index.php/topic,42028.msg219929.html#msg219929

And here is some of the output without having to go to the link

_ad4: 3775MB <ts4gmsa500 20101212="">at ata2-master PIO4 SATA 3Gb/s
uhub6: 1 port with 1 removable, self powered
uhub3: 3 ports with 3 removable, self powered
uhub7: 3 ports with 3 removable, self powered
ad4: TIMEOUT - READ retrying (1 retry left) LBA=3861855
Trying to mount root from ufs:/dev/ufs/pfsense0
ad4: TIMEOUT - READ retrying (1 retry left) LBA=3452247
Configuring crash dumps…
Mounting filesystems...
Setting up embedded specific environment...ad4: TIMEOUT - READ retrying (1 retry left) LBA=3450831
ad4: TIMEOUT - READ retrying (0 retries left) LBA=3450831
ad4: FAILURE - READ timed out LBA=3450831
g_vfs_done():ufs/pfsense0[READ(offset=1766785024, length=14336)]error = 5
vnode_pager_getpages: I/O read error
ad4: TIMEOUT - READ retrying (1 retry left) LBA=1514407
ad4: TIMEOUT - READ retrying (0 retries left) LBA=1514407
ad4: FAILURE - READ timed out LBA=1514407
g_vfs_done():ufs/pfsense0[READ(offset=775335936, length=2048)]error = 5
/etc/rc.embedded: mdmfs: Input/output error
ad4: TIMEOUT - READ retrying (1 retry left) LBA=3450831
ad4: TIMEOUT - READ retrying (0 retries left) LBA=3450831
ad4: FAILURE - READ timed out LBA=3450831
g_vfs_done():ufs/pfsense0[READ(offset=1766785024, length=14336)]error = 5
vnode_pager_getpages: I/O read error
ad4: TIMEOUT - READ retrying (1 retry left) LBA=1514407
/etc/rc.embedded: mdmfs: Input/output error
ad4: TIMEOUT - READ retrying (1 retry left) LBA=1866375
done.
ad4: TIMEOUT - READ retrying (1 retry left) LBA=311183

___
/ f
/ p _/ Sense
_/
__/

Welcome to pfSense 2.0-RELEASE …

ad4: TIMEOUT - READ retrying (1 retry left) LBA=3447463
ad4: TIMEOUT - READ retrying (1 retry left) LBA=1814527
Creating symlinks...g_vfs_done():ufs/cf[READ(offset=13193216, length=4096)]error = 5
g_vfs_done():ufs/cf[READ(offset=13193216, length=4096)]error = 5
mkdir: /cf/conf: Input/output error
.g_vfs_done():ufs/cf[READ(offset=13193216, length=4096)]error = 5
cat: /cf/conf/config.xml: Input/output error
ad4: TIMEOUT - READ retrying (1 retry left) LBA=1764455
g_vfs_done():ufs/cf[READ(offset=13193216, length=4096)]error = 5
cat: /cf/conf/config.xml: Input/output error
ad4: TIMEOUT - READ retrying (1 retry left) LBA=1503551
..done.
ad4: TIMEOUT - READ retrying (1 retry left) LBA=327975
ad4: TIMEOUT - READ retrying (0 retries left) LBA=327975
ad4: FAILURE - READ timed out LBA=327975
g_vfs_done():ufs/pfsense0[READ(offset=167882752, length=6144)]error = 5
vnode_pager_getpages: I/O read error
/etc/rc: /usr/bin/cap_mkdb: Input/output error</ts4gmsa500>_

Honeybadger · Oct 31, 2011, 11:11 PM

I was getting these errors until I put my OS on a 2g (2g image) Sandisk Extreme IV card.

It is designed for heavy writing in warm environments (SLR cameras) and my problem went away.

Hope your problem does the same…

wm408 · Nov 1, 2011, 5:19 PM

the net6501's don't support CF cards. mSATA is an option which seems fine. I have tried 4G and 2G on the 4G card, no luck.

I am going to try a different mSATA card.

@Honeybadger:

I was getting these errors until I put my OS on a 2g (2g image) Sandisk Extreme IV card.

It is designed for heavy writing in warm environments (SLR cameras) and my problem went away.

Hope your problem does the same…

jimp · Nov 1, 2011, 5:31 PM

One thing I'd be wary of in the net6501 case is putting an mSATA card in the slot by the CPU. With no active cooling in the case, the CPU gets quite hot and the card in the slot next to the CPU will get a bit of heat from that as well. In the wrong environment I could see that killing an mSATA drive (or at least harming its performance in some way)

nexus_VI · Nov 1, 2011, 7:11 PM

I tried with the 2G image, sadly the results are the same:

ad0: FAILURE - READ status=51 <ready,dsc,error>error=10 <nid_not_found>LBA=7813119
ad0: FAILURE - READ status=51 <ready,dsc,error>error=10 <nid_not_found>LBA=7813119
ad0: FAILURE - READ status=51 <ready,dsc,error>error=10 <nid_not_found>LBA=7813119

Is buying a new CF card my best bet, or even my only choice?</nid_not_found></ready,dsc,error></nid_not_found></ready,dsc,error></nid_not_found></ready,dsc,error>

wm408 · Nov 1, 2011, 8:05 PM

Thx Jimp. Yeah slot 00 is the farthest from, 01 being closer. The run time on this mSATA card is so little that I don't think that's a factor yet for me. But I will follow that rule for sure.

Hey Jimp with this card, I am not sure if its relevant but I found a thread with sort of similar problems to mine here: http://freebsd.1045724.n5.nabble.com/HDD-READ-Timeout-messages-after-8-2-Upgrade-td3733475.html

Adding ahci_load="YES" to loader.conf solved the problem

Do you think its relevant to the problem I am having?

Right now I am trying to mount the *.img file to edit the file via:

mount -t auto -o loop,offset=$((63*512)) /media/B4E1-5C16/pfSense-2.0-RELEASE-2g-i386-20110913-1802-nanobsd.img /mnt/test

If you have thoughts let me know!

@jimp:

One thing I'd be wary of in the net6501 case is putting an mSATA card in the slot by the CPU. With no active cooling in the case, the CPU gets quite hot and the card in the slot next to the CPU will get a bit of heat from that as well. In the wrong environment I could see that killing an mSATA drive (or at least harming its performance in some way)

depiman · Nov 2, 2011, 11:04 AM

I have exactly the same problem on a Soekris net4801. A 4 GB image on a new 4 GB SanDisk Ultra card gives a write error at the end. The same image on an older SanDisk Ultra II card goes well. The newer card boots and runs with the 4 GB image, but is unstable. On the old card it's stable. The 2 GB image on the newer card runs, but still to early to say if it's stable. Fact is that it's not relyable for my production environment (…) Big question: where's the problem? On the image, the CF card or the Soekris?

Metu69salemi · Nov 2, 2011, 11:18 AM

@depiman:

Big question: where's the problem? On the image, the CF card or the Soekris?

Shaking from the sleeve(meaning this more like i feel than i know): CF card

depiman · Nov 2, 2011, 12:26 PM

@Metu69salemi:

Shaking from the sleeve(meaning this more like i feel than i know): CF card

I think so to. And that looks the same as Wim van der Putten his opinion: newer SanDisk has less bytes then before. Or something else is wrong. So we can say newer doesn't have to be better ;-) Sh*t, I bought ten of those cards two months ago! :-(

Metu69salemi · Nov 2, 2011, 12:32 PM

Use 2GB images, and your problem is solved

wm408 · Nov 2, 2011, 10:39 PM

Tried that, no luck.

@Metu69salemi:

Use 2GB images, and your problem is solved

Metu69salemi · Nov 2, 2011, 10:43 PM

Okay then it has to be something else on that card. Can you use it with your camera or windows machine without errors?

wm408 · Nov 2, 2011, 10:46 PM

I haven't tried moving files in and out of it on Windows, but it seems to format it just fine and then detect it as a drive. Also in Linux if I format, seems like I can access it thereafter.

@Metu69salemi:

Okay then it has to be something else on that card. Can you use it with your camera or windows machine without errors?

wallabybob · Nov 3, 2011, 12:59 AM

I think there are a few different types of errors reported in this thread:

@wm408:

I haven't tried moving files in and out of it on Windows, but it seems to format it just fine and then detect it as a drive. Also in Linux if I format, seems like I can access it thereafter.

Neither of these tests are good tests that the whole card can be read and written.

On linux something like dd if=/dev/sdxx of=/dev/null bs=65536 can be used to read the whole card and something like dd if=/dev/zero of=/dev/sdaxx bs=65536 can be used to write zeroes to the whole card (where sdxx is carefully chosen to match the device assignment for the card).

@depiman:

I have exactly the same problem on a Soekris net4801. A 4 GB image on a new 4 GB SanDisk Ultra card gives a write error at the end.

This is almost certainly because the card is not large enough. It seems that once a 4GB card could hold 4102410241024 bytes of data an then some "genius" decided that a 4GB card needed to be able to hold only 4100010001000 bytes of data and then some "genius" decided the 4100010001000 bytes could include error detection bytes and then some genius decided the 4100010001000 byes could also include "replacement" blocks and then some genius decided that 3.9100010001000 could be rounded up to 4GB etc etc with each act of "genius" reducing the data space available to the end user. On what I've seen reported I would guess there could be a good case made to an aggressive consumer affairs department that a card advertised as 4GB must have at least 4100010001000 bytes available to the end user for file storage. But that is a digression.

@nexus_VI:

I tried with the 2G image, sadly the results are the same:

ad0: FAILURE - READ status=51 <ready,dsc,error>error=10 <nid_not_found>LBA=7813119
ad0: FAILURE - READ status=51 <ready,dsc,error>error=10 <nid_not_found>LBA=7813119
ad0: FAILURE - READ status=51 <ready,dsc,error>error=10 <nid_not_found>LBA=7813119

Is buying a new CF card my best bet, or even my only choice?</nid_not_found></ready,dsc,error></nid_not_found></ready,dsc,error></nid_not_found></ready,dsc,error>

This doesn't look as if you are running the 2GB image. The logical blocks are 512 bytes long. LBA (Logical Block Address) 7813119 * 512 is about 4,000,316,928 which is well over 2GB. The 2GB image should not be referencing any block above something a little less than 210241024*1024/512.

@wm408:

In a net6501, I am having sort of similar errors on a 4G mSATA card that I purchased from Soekris, its a Transcend 4g MSA500 (I think?, something like this). I tested 2G and 4G nanobsd images on it with the same result.

_ad4: 3775MB <ts4gmsa500 20101212="">at ata2-master PIO4 SATA 3Gb/s
ad4: TIMEOUT - READ retrying (1 retry left) LBA=3861855
ad4: TIMEOUT - READ retrying (1 retry left) LBA=3452247
ad4: TIMEOUT - READ retrying (1 retry left) LBA=3450831
ad4: TIMEOUT - READ retrying (0 retries left) LBA=3450831
ad4: FAILURE - READ timed out LBA=3450831
g_vfs_done():ufs/pfsense0[READ(offset=1766785024, length=14336)]error = 5
vnode_pager_getpages: I/O read error
ad4: TIMEOUT - READ retrying (1 retry left) LBA=1514407
ad4: TIMEOUT - READ retrying (0 retries left) LBA=1514407
ad4: FAILURE - READ timed out LBA=1514407
g_vfs_done():ufs/pfsense0[READ(offset=775335936, length=2048)]error = 5
ad4: TIMEOUT - READ retrying (1 retry left) LBA=3450831
ad4: TIMEOUT - READ retrying (0 retries left) LBA=3450831
ad4: FAILURE - READ timed out LBA=3450831
g_vfs_done():ufs/pfsense0[READ(offset=1766785024, length=14336)]error = 5
vnode_pager_getpages: I/O read error
ad4: TIMEOUT - READ retrying (1 retry left) LBA=1514407
ad4: TIMEOUT - READ retrying (1 retry left) LBA=1866375
ad4: TIMEOUT - READ retrying (1 retry left) LBA=311183
ad4: TIMEOUT - READ retrying (1 retry left) LBA=3447463
ad4: TIMEOUT - READ retrying (1 retry left) LBA=1814527
ad4: TIMEOUT - READ retrying (1 retry left) LBA=1764455
ad4: TIMEOUT - READ retrying (1 retry left) LBA=1503551
ad4: TIMEOUT - READ retrying (1 retry left) LBA=327975
ad4: TIMEOUT - READ retrying (0 retries left) LBA=327975
ad4: FAILURE - READ timed out LBA=327975

These all look like the disk has bad spots at the referenced LBAs. However the problem could be that you are using quite new motherboard and FreeBSD needs some tweaks to work well with that chipset (or that "disk").

@fabix:

On a pfsense 2.0, I have a lot of I/O errors:
ad0: FAILURE - READ status=51 <ready,dsc,error>error=10 <nid_not_found>LBA=7813119
ad0: FAILURE - READ status=51 <ready,dsc,error>error=10 <nid_not_found>LBA=7813119</nid_not_found></ready,dsc,error></nid_not_found></ready,dsc,error>

As discussed earlier in this reply, the disk image you wrote to the card is too large for the card. In fact it appears that it is just too large. Your card claims to have user capacity of 4,000,317,440 bytes, the read starts at byte 4,000,316,928 which is the last block. Perhaps it was a multi-block read.

@BrianX:

i am getting the same errors. tried two different cards. they are transcend industrial CF cards in an ide carrier in a dual atom d525 system. (hacom mars) dmesg | grep ad0
ad0: 1943MB <transcend 20070418="">at ata0-master PIO4
ad0: FAILURE - READ status=ff <busy,ready,dma_ready,dsc,drq,correctable,index,error>error=0 LBA=532431
ad0: FAILURE - WRITE status=ff <busy,ready,dma_ready,dsc,drq,correctable,index,error>error=58 <uncorrectable,nid_not_found,media_change_reqest>LBA=88375
ad0: TIMEOUT - WRITE retrying (1 retry left) LBA=88631
ad0: FAILURE - WRITE status=ff <busy,ready,dma_ready,dsc,drq,correctable,index,error>error=58 <uncorrectable,nid_not_found,media_change_reqest>LBA=88887
ad0: TIMEOUT - WRITE retrying (0 retries left) LBA=88631
ad0: FAILURE - READ status=ff <busy,ready,dma_ready,dsc,drq,correctable,index,error>error=0 LBA=233943</busy,ready,dma_ready,dsc,drq,correctable,index,error></uncorrectable,nid_not_found,media_change_reqest></busy,ready,dma_ready,dsc,drq,correctable,index,error></uncorrectable,nid_not_found,media_change_reqest></busy,ready,dma_ready,dsc,drq,correctable,index,error></busy,ready,dma_ready,dsc,drq,correctable,index,error></transcend>

Notice the READ failures are flagged CORRECTABLE. I presume these are corrected and are reported as an indication of a developing bad spot on the "disk". (They might go uncorrectable one day soon.) The write failures (retries exhausted) indicate a bad spot. These are not related to the disk being too small to hold the file system that was written to it. Maybe you could use the Linux commands I gave earlier to test out the card with a different operating system. Did your two different cards report errors at the same LBAs?</ts4gmsa500>_

wm408 · Nov 4, 2011, 5:27 PM

Thx for your input. I am still scratching my head.

@wm408:

In a net6501, I am having sort of similar errors on a 4G mSATA card that I purchased from Soekris, its a Transcend 4g MSA500 (I think?, something like this). I tested 2G and 4G nanobsd images on it with the same result.

_ad4: 3775MB <ts4gmsa500 20101212="">at ata2-master PIO4 SATA 3Gb/s
ad4: TIMEOUT - READ retrying (1 retry left) LBA=3861855
ad4: TIMEOUT - READ retrying (1 retry left) LBA=3452247
ad4: TIMEOUT - READ retrying (1 retry left) LBA=3450831
ad4: TIMEOUT - READ retrying (0 retries left) LBA=3450831
ad4: FAILURE - READ timed out LBA=3450831
g_vfs_done():ufs/pfsense0[READ(offset=1766785024, length=14336)]error = 5
vnode_pager_getpages: I/O read error
ad4: TIMEOUT - READ retrying (1 retry left) LBA=1514407
ad4: TIMEOUT - READ retrying (0 retries left) LBA=1514407
ad4: FAILURE - READ timed out LBA=1514407
g_vfs_done():ufs/pfsense0[READ(offset=775335936, length=2048)]error = 5
ad4: TIMEOUT - READ retrying (1 retry left) LBA=3450831
ad4: TIMEOUT - READ retrying (0 retries left) LBA=3450831
ad4: FAILURE - READ timed out LBA=3450831
g_vfs_done():ufs/pfsense0[READ(offset=1766785024, length=14336)]error = 5
vnode_pager_getpages: I/O read error
ad4: TIMEOUT - READ retrying (1 retry left) LBA=1514407
ad4: TIMEOUT - READ retrying (1 retry left) LBA=1866375
ad4: TIMEOUT - READ retrying (1 retry left) LBA=311183
ad4: TIMEOUT - READ retrying (1 retry left) LBA=3447463
ad4: TIMEOUT - READ retrying (1 retry left) LBA=1814527
ad4: TIMEOUT - READ retrying (1 retry left) LBA=1764455
ad4: TIMEOUT - READ retrying (1 retry left) LBA=1503551
ad4: TIMEOUT - READ retrying (1 retry left) LBA=327975
ad4: TIMEOUT - READ retrying (0 retries left) LBA=327975
ad4: FAILURE - READ timed out LBA=327975

These all look like the disk has bad spots at the referenced LBAs. However the problem could be that you are using quite new motherboard and FreeBSD needs some tweaks to work well with that chipset (or that "disk").</ts4gmsa500>_