Ad0 errors with CompactFlash Sandisk cards and Soekris net5501

jimp · Nov 1, 2011, 5:31 PM

One thing I'd be wary of in the net6501 case is putting an mSATA card in the slot by the CPU. With no active cooling in the case, the CPU gets quite hot and the card in the slot next to the CPU will get a bit of heat from that as well. In the wrong environment I could see that killing an mSATA drive (or at least harming its performance in some way)

nexus_VI · Nov 1, 2011, 7:11 PM

I tried with the 2G image, sadly the results are the same:

ad0: FAILURE - READ status=51 <ready,dsc,error>error=10 <nid_not_found>LBA=7813119
ad0: FAILURE - READ status=51 <ready,dsc,error>error=10 <nid_not_found>LBA=7813119
ad0: FAILURE - READ status=51 <ready,dsc,error>error=10 <nid_not_found>LBA=7813119

Is buying a new CF card my best bet, or even my only choice?</nid_not_found></ready,dsc,error></nid_not_found></ready,dsc,error></nid_not_found></ready,dsc,error>

wm408 · Nov 1, 2011, 8:05 PM

Thx Jimp. Yeah slot 00 is the farthest from, 01 being closer. The run time on this mSATA card is so little that I don't think that's a factor yet for me. But I will follow that rule for sure.

Hey Jimp with this card, I am not sure if its relevant but I found a thread with sort of similar problems to mine here: http://freebsd.1045724.n5.nabble.com/HDD-READ-Timeout-messages-after-8-2-Upgrade-td3733475.html

Adding ahci_load="YES" to loader.conf solved the problem

Do you think its relevant to the problem I am having?

Right now I am trying to mount the *.img file to edit the file via:

mount -t auto -o loop,offset=$((63*512)) /media/B4E1-5C16/pfSense-2.0-RELEASE-2g-i386-20110913-1802-nanobsd.img /mnt/test

If you have thoughts let me know!

@jimp:

One thing I'd be wary of in the net6501 case is putting an mSATA card in the slot by the CPU. With no active cooling in the case, the CPU gets quite hot and the card in the slot next to the CPU will get a bit of heat from that as well. In the wrong environment I could see that killing an mSATA drive (or at least harming its performance in some way)

depiman · Nov 2, 2011, 11:04 AM

I have exactly the same problem on a Soekris net4801. A 4 GB image on a new 4 GB SanDisk Ultra card gives a write error at the end. The same image on an older SanDisk Ultra II card goes well. The newer card boots and runs with the 4 GB image, but is unstable. On the old card it's stable. The 2 GB image on the newer card runs, but still to early to say if it's stable. Fact is that it's not relyable for my production environment (…) Big question: where's the problem? On the image, the CF card or the Soekris?

Metu69salemi · Nov 2, 2011, 11:18 AM

@depiman:

Big question: where's the problem? On the image, the CF card or the Soekris?

Shaking from the sleeve(meaning this more like i feel than i know): CF card

depiman · Nov 2, 2011, 12:26 PM

@Metu69salemi:

Shaking from the sleeve(meaning this more like i feel than i know): CF card

I think so to. And that looks the same as Wim van der Putten his opinion: newer SanDisk has less bytes then before. Or something else is wrong. So we can say newer doesn't have to be better ;-) Sh*t, I bought ten of those cards two months ago! :-(

Metu69salemi · Nov 2, 2011, 12:32 PM

Use 2GB images, and your problem is solved

wm408 · Nov 2, 2011, 10:39 PM

Tried that, no luck.

@Metu69salemi:

Use 2GB images, and your problem is solved

Metu69salemi · Nov 2, 2011, 10:43 PM

Okay then it has to be something else on that card. Can you use it with your camera or windows machine without errors?

wm408 · Nov 2, 2011, 10:46 PM

I haven't tried moving files in and out of it on Windows, but it seems to format it just fine and then detect it as a drive. Also in Linux if I format, seems like I can access it thereafter.

@Metu69salemi:

Okay then it has to be something else on that card. Can you use it with your camera or windows machine without errors?

wallabybob · Nov 3, 2011, 12:59 AM

I think there are a few different types of errors reported in this thread:

@wm408:

I haven't tried moving files in and out of it on Windows, but it seems to format it just fine and then detect it as a drive. Also in Linux if I format, seems like I can access it thereafter.

Neither of these tests are good tests that the whole card can be read and written.

On linux something like dd if=/dev/sdxx of=/dev/null bs=65536 can be used to read the whole card and something like dd if=/dev/zero of=/dev/sdaxx bs=65536 can be used to write zeroes to the whole card (where sdxx is carefully chosen to match the device assignment for the card).

@depiman:

I have exactly the same problem on a Soekris net4801. A 4 GB image on a new 4 GB SanDisk Ultra card gives a write error at the end.

This is almost certainly because the card is not large enough. It seems that once a 4GB card could hold 4102410241024 bytes of data an then some "genius" decided that a 4GB card needed to be able to hold only 4100010001000 bytes of data and then some "genius" decided the 4100010001000 bytes could include error detection bytes and then some genius decided the 4100010001000 byes could also include "replacement" blocks and then some genius decided that 3.9100010001000 could be rounded up to 4GB etc etc with each act of "genius" reducing the data space available to the end user. On what I've seen reported I would guess there could be a good case made to an aggressive consumer affairs department that a card advertised as 4GB must have at least 4100010001000 bytes available to the end user for file storage. But that is a digression.

@nexus_VI:

I tried with the 2G image, sadly the results are the same:

ad0: FAILURE - READ status=51 <ready,dsc,error>error=10 <nid_not_found>LBA=7813119
ad0: FAILURE - READ status=51 <ready,dsc,error>error=10 <nid_not_found>LBA=7813119
ad0: FAILURE - READ status=51 <ready,dsc,error>error=10 <nid_not_found>LBA=7813119

Is buying a new CF card my best bet, or even my only choice?</nid_not_found></ready,dsc,error></nid_not_found></ready,dsc,error></nid_not_found></ready,dsc,error>

This doesn't look as if you are running the 2GB image. The logical blocks are 512 bytes long. LBA (Logical Block Address) 7813119 * 512 is about 4,000,316,928 which is well over 2GB. The 2GB image should not be referencing any block above something a little less than 210241024*1024/512.

@wm408:

In a net6501, I am having sort of similar errors on a 4G mSATA card that I purchased from Soekris, its a Transcend 4g MSA500 (I think?, something like this). I tested 2G and 4G nanobsd images on it with the same result.

_ad4: 3775MB <ts4gmsa500 20101212="">at ata2-master PIO4 SATA 3Gb/s
ad4: TIMEOUT - READ retrying (1 retry left) LBA=3861855
ad4: TIMEOUT - READ retrying (1 retry left) LBA=3452247
ad4: TIMEOUT - READ retrying (1 retry left) LBA=3450831
ad4: TIMEOUT - READ retrying (0 retries left) LBA=3450831
ad4: FAILURE - READ timed out LBA=3450831
g_vfs_done():ufs/pfsense0[READ(offset=1766785024, length=14336)]error = 5
vnode_pager_getpages: I/O read error
ad4: TIMEOUT - READ retrying (1 retry left) LBA=1514407
ad4: TIMEOUT - READ retrying (0 retries left) LBA=1514407
ad4: FAILURE - READ timed out LBA=1514407
g_vfs_done():ufs/pfsense0[READ(offset=775335936, length=2048)]error = 5
ad4: TIMEOUT - READ retrying (1 retry left) LBA=3450831
ad4: TIMEOUT - READ retrying (0 retries left) LBA=3450831
ad4: FAILURE - READ timed out LBA=3450831
g_vfs_done():ufs/pfsense0[READ(offset=1766785024, length=14336)]error = 5
vnode_pager_getpages: I/O read error
ad4: TIMEOUT - READ retrying (1 retry left) LBA=1514407
ad4: TIMEOUT - READ retrying (1 retry left) LBA=1866375
ad4: TIMEOUT - READ retrying (1 retry left) LBA=311183
ad4: TIMEOUT - READ retrying (1 retry left) LBA=3447463
ad4: TIMEOUT - READ retrying (1 retry left) LBA=1814527
ad4: TIMEOUT - READ retrying (1 retry left) LBA=1764455
ad4: TIMEOUT - READ retrying (1 retry left) LBA=1503551
ad4: TIMEOUT - READ retrying (1 retry left) LBA=327975
ad4: TIMEOUT - READ retrying (0 retries left) LBA=327975
ad4: FAILURE - READ timed out LBA=327975

These all look like the disk has bad spots at the referenced LBAs. However the problem could be that you are using quite new motherboard and FreeBSD needs some tweaks to work well with that chipset (or that "disk").

@fabix:

On a pfsense 2.0, I have a lot of I/O errors:
ad0: FAILURE - READ status=51 <ready,dsc,error>error=10 <nid_not_found>LBA=7813119
ad0: FAILURE - READ status=51 <ready,dsc,error>error=10 <nid_not_found>LBA=7813119</nid_not_found></ready,dsc,error></nid_not_found></ready,dsc,error>

As discussed earlier in this reply, the disk image you wrote to the card is too large for the card. In fact it appears that it is just too large. Your card claims to have user capacity of 4,000,317,440 bytes, the read starts at byte 4,000,316,928 which is the last block. Perhaps it was a multi-block read.

@BrianX:

i am getting the same errors. tried two different cards. they are transcend industrial CF cards in an ide carrier in a dual atom d525 system. (hacom mars) dmesg | grep ad0
ad0: 1943MB <transcend 20070418="">at ata0-master PIO4
ad0: FAILURE - READ status=ff <busy,ready,dma_ready,dsc,drq,correctable,index,error>error=0 LBA=532431
ad0: FAILURE - WRITE status=ff <busy,ready,dma_ready,dsc,drq,correctable,index,error>error=58 <uncorrectable,nid_not_found,media_change_reqest>LBA=88375
ad0: TIMEOUT - WRITE retrying (1 retry left) LBA=88631
ad0: FAILURE - WRITE status=ff <busy,ready,dma_ready,dsc,drq,correctable,index,error>error=58 <uncorrectable,nid_not_found,media_change_reqest>LBA=88887
ad0: TIMEOUT - WRITE retrying (0 retries left) LBA=88631
ad0: FAILURE - READ status=ff <busy,ready,dma_ready,dsc,drq,correctable,index,error>error=0 LBA=233943</busy,ready,dma_ready,dsc,drq,correctable,index,error></uncorrectable,nid_not_found,media_change_reqest></busy,ready,dma_ready,dsc,drq,correctable,index,error></uncorrectable,nid_not_found,media_change_reqest></busy,ready,dma_ready,dsc,drq,correctable,index,error></busy,ready,dma_ready,dsc,drq,correctable,index,error></transcend>

Notice the READ failures are flagged CORRECTABLE. I presume these are corrected and are reported as an indication of a developing bad spot on the "disk". (They might go uncorrectable one day soon.) The write failures (retries exhausted) indicate a bad spot. These are not related to the disk being too small to hold the file system that was written to it. Maybe you could use the Linux commands I gave earlier to test out the card with a different operating system. Did your two different cards report errors at the same LBAs?</ts4gmsa500>_

wm408 · Nov 4, 2011, 5:27 PM

Thx for your input. I am still scratching my head.

@wm408:

In a net6501, I am having sort of similar errors on a 4G mSATA card that I purchased from Soekris, its a Transcend 4g MSA500 (I think?, something like this). I tested 2G and 4G nanobsd images on it with the same result.

_ad4: 3775MB <ts4gmsa500 20101212="">at ata2-master PIO4 SATA 3Gb/s
ad4: TIMEOUT - READ retrying (1 retry left) LBA=3861855
ad4: TIMEOUT - READ retrying (1 retry left) LBA=3452247
ad4: TIMEOUT - READ retrying (1 retry left) LBA=3450831
ad4: TIMEOUT - READ retrying (0 retries left) LBA=3450831
ad4: FAILURE - READ timed out LBA=3450831
g_vfs_done():ufs/pfsense0[READ(offset=1766785024, length=14336)]error = 5
vnode_pager_getpages: I/O read error
ad4: TIMEOUT - READ retrying (1 retry left) LBA=1514407
ad4: TIMEOUT - READ retrying (0 retries left) LBA=1514407
ad4: FAILURE - READ timed out LBA=1514407
g_vfs_done():ufs/pfsense0[READ(offset=775335936, length=2048)]error = 5
ad4: TIMEOUT - READ retrying (1 retry left) LBA=3450831
ad4: TIMEOUT - READ retrying (0 retries left) LBA=3450831
ad4: FAILURE - READ timed out LBA=3450831
g_vfs_done():ufs/pfsense0[READ(offset=1766785024, length=14336)]error = 5
vnode_pager_getpages: I/O read error
ad4: TIMEOUT - READ retrying (1 retry left) LBA=1514407
ad4: TIMEOUT - READ retrying (1 retry left) LBA=1866375
ad4: TIMEOUT - READ retrying (1 retry left) LBA=311183
ad4: TIMEOUT - READ retrying (1 retry left) LBA=3447463
ad4: TIMEOUT - READ retrying (1 retry left) LBA=1814527
ad4: TIMEOUT - READ retrying (1 retry left) LBA=1764455
ad4: TIMEOUT - READ retrying (1 retry left) LBA=1503551
ad4: TIMEOUT - READ retrying (1 retry left) LBA=327975
ad4: TIMEOUT - READ retrying (0 retries left) LBA=327975
ad4: FAILURE - READ timed out LBA=327975

These all look like the disk has bad spots at the referenced LBAs. However the problem could be that you are using quite new motherboard and FreeBSD needs some tweaks to work well with that chipset (or that "disk").</ts4gmsa500>_

wm408 · Nov 8, 2011, 4:40 PM

On my net6501 I am trying a different mSATA here, its 20GB in size. Applied the 4g nanobsd image to it /w dd in Linux.

On boot and during use I get this (still scratching my head):

Trying to mount root from ufs:/dev/ufs/pfsense0
ad4: TIMEOUT - READ_MUL retrying (1 retry left) LBA=332239
ad4: TIMEOUT - READ_MUL retrying (1 retry left) LBA=1513199
ad4: TIMEOUT - READ_MUL retrying (0 retries left) LBA=1513199
ad4: FAILURE - READ_MUL timed out LBA=1513199
g_vfs_done():ufs/pfsense0[READ(offset=774717440, length=18944)]error = 5
vnode_pager_getpages: I/O read error
ad4: TIMEOUT - READ_MUL retrying (1 retry left) LBA=3450767
ad4: TIMEOUT - READ_MUL retrying (0 retries left) LBA=3450767
ad4: FAILURE - READ_MUL timed out LBA=3450767
g_vfs_done():ufs/pfsense0[READ(offset=1766752256, length=18944)]error = 5
vnode_pager_getpages: I/O read error
ad4: TIMEOUT - READ_MUL retrying (1 retry left) LBA=1135623
ad4: TIMEOUT - READ_MUL retrying (1 retry left) LBA=1765855
ad4: TIMEOUT - READ_MUL retrying (1 retry left) LBA=1768839
ad4: TIMEOUT - READ_MUL retrying (1 retry left) LBA=442207
ad4: TIMEOUT - READ_MUL retrying (0 retries left) LBA=442207
ad4: FAILURE - READ_MUL timed out LBA=442207
g_vfs_done():ufs/pfsense0[READ(offset=226369536, length=35328)]error = 5
vnode_pager_getpages: I/O read error
ad4: TIMEOUT - READ_MUL retrying (1 retry left) LBA=442207
ad4: TIMEOUT - READ_MUL retrying (1 retry left) LBA=1513199
ad4: TIMEOUT - READ_MUL retrying (1 retry left) LBA=1765455
ad4: TIMEOUT - READ_MUL retrying (1 retry left) LBA=1134127
ad4: TIMEOUT - READ_MUL retrying (1 retry left) LBA=332151

I can work in the CLI/console but I am unsure here on continuing.

Update:

I took a 2.5" 320GB Sata drive and wrote the 4G nanobsd image to it in Linux via: dd if=pfsense.img of=/dev/sdX

booted up the net6501 and have the same result.

So either my ability to write the nanobsd image sucks (which doesn't make sense)
or
Something with the nanobsd and the net6501?

hdokes · Nov 17, 2011, 8:34 PM

Getting same issues here with ANY compactflash card I use. All are 2gb and I am getting the identical messages whether using the 2gb or 1gb of pfsense 2.0 nano image for i386 vga.

The specific message is:

ad0: 1953MB <sandisk sdcfj-2048="" hdx4.03="">ad0: Failure Read_DMA timed out LBA=4001759
ad0: Failure Read_DMA timed out LBA=4001743
ad0: Failure Read_DMA timed out LBA=4001756

etc.

The hardware I am using is an asus Pundit P1-PH1. This very unit has been running flawlessly for the last 2 years with an embedded version of pfsense 1.2. The errors I am receiving now occur with the original compact flash card that was used for the last 2 years as well as a brand new one from Sandisk and new units from R Data. It doesn't matter who's cf card I use…. all are 2gb... all are having the exact same issue. This CAN NOT be an issue with compact flash cards. It's happening to far too many people with unrelated hardware. Has anyone actually had success flashing and using the nano versions of pfsense 2.0?

In all cases the write process to these flash drives is completed without errors. In all cases, pfsense boots and runs through it's motions until it gets to the ad0 device which is the compact flash card.

I have tested these cards (all of them) on a win7 machine. I am able to partition and format at will, copy files to and from the flash drive and have confirmed integrity of the files once copied from the flash drive back to the hard drive.

I'm happy to provide any additional information/testing that might be requested to help narrow down this issue as it is without question a big time pain in the behind right now.

It should be noted, while it is not the desired configuration, I have reluctantly installed a hard drive on the same asus Pundit and have successfully loaded pfsense 2.0 i386. Pretty certain it's not a hardware issue.</sandisk>

wm408 · Nov 29, 2011, 3:41 AM

I have been able to install embedded/nanobsd 2.0.x on a net5501 with no problems / no errors. I have run into problems with a net6501 and the nanobsd/embedded however. Still not sure what is going on there.

@hdokes:

Getting same issues here with ANY compactflash card I use. All are 2gb and I am getting the identical messages whether using the 2gb or 1gb of pfsense 2.0 nano image for i386 vga.

The specific message is:

ad0: 1953MB <sandisk sdcfj-2048="" hdx4.03="">ad0: Failure Read_DMA timed out LBA=4001759
ad0: Failure Read_DMA timed out LBA=4001743
ad0: Failure Read_DMA timed out LBA=4001756

etc.

The hardware I am using is an asus Pundit P1-PH1. This very unit has been running flawlessly for the last 2 years with an embedded version of pfsense 1.2. The errors I am receiving now occur with the original compact flash card that was used for the last 2 years as well as a brand new one from Sandisk and new units from R Data. It doesn't matter who's cf card I use…. all are 2gb... all are having the exact same issue. This CAN NOT be an issue with compact flash cards. It's happening to far too many people with unrelated hardware. Has anyone actually had success flashing and using the nano versions of pfsense 2.0?

In all cases the write process to these flash drives is completed without errors. In all cases, pfsense boots and runs through it's motions until it gets to the ad0 device which is the compact flash card.

I have tested these cards (all of them) on a win7 machine. I am able to partition and format at will, copy files to and from the flash drive and have confirmed integrity of the files once copied from the flash drive back to the hard drive.

I'm happy to provide any additional information/testing that might be requested to help narrow down this issue as it is without question a big time pain in the behind right now.

It should be noted, while it is not the desired configuration, I have reluctantly installed a hard drive on the same asus Pundit and have successfully loaded pfsense 2.0 i386. Pretty certain it's not a hardware issue.</sandisk>

cmb · Dec 6, 2011, 10:49 AM

There seems to be multiple different issues here, but the one that results in any kind of NID_NOT_FOUND READ errors like:


ad0: FAILURE - READ status=51 <ready,dsc,error>error=10 <nid_not_found>LBA=7813119
ad0: FAILURE - READ status=51 <ready,dsc,error>error=10 <nid_not_found>LBA=7813119</nid_not_found></ready,dsc,error></nid_not_found></ready,dsc,error>

is normal on newer SanDisk cards and safe to ignore. It has nothing to do with image size, you can put a 1 GB image on a 4 GB card and they still do that. From what I've been able to gather on that, the cards report they have a larger usable size than they do, and when the FreeBSD ATA code initializes it tries to read the end of the disk even if its partition is nowhere near that, which the CF is reporting doesn't actually exist, which is what that log message says.

"according to ATA-7 specification, Section 6.59.6 is: "IDNF shall be set to one if a user-accessible address could not be found. IDNF shall be set to one if an address outside of the range of user-accessible addresses is requested if command aborted is not returned." FreeBSD labels this bit as NID_NOT_FOUND"

jdamnation · Dec 7, 2011, 4:19 PM

Here are my results. Four brand new CF cards, two Sandisk Ultra 4GB and two Sandisk Extreme 2GB.

Test unit: Seokris Net 5501

Flash method: Unmounted drive from OSX, then did "gzcat pfSense-2.0-RELEASE-2g-i386-nanobsd.img.gz | dd of=/dev/disk4 bs=16k"

Sandisk Ultra CompactFlash 30MB/s 4GB - card 1 of 2

pfSense-2.0-RELEASE-4g-i386-nanobsd.img.gz = LBA errors
pfSense-2.0-RELEASE-2g-i386-nanobsd.img.gz = LBA errors

Sandisk Ultra CompactFlash 30MB/s 4GB - card 2 of 2

pfSense-2.0-RELEASE-4g-i386-nanobsd.img.gz = LBA errors
pfSense-2.0-RELEASE-2g-i386-nanobsd.img.gz = LBA errors

Sandisk Extreme III 2GB - card 1 of 2

pfSense-2.0-RELEASE-2g-i386-nanobsd.img.gz = No errors, all good!

Sandisk Extreme III 2GB - card 2 of 2

pfSense-2.0-RELEASE-2g-i386-nanobsd.img.gz = No errors, all good!

Other Observations

When flashing the 4GB image to the 4GB cards, dd returned the following when the flash complete: 'dd: /dev/disk4: end of device'. The card still booted, though with LBA errors.
The 2GB image did not return 'dd: /dev/disk4: end of device', however still gave LBA errors for some reason.

Summary Of Findings

If you want to use a CF card with a Seokris net5501, make sure you get yourself a Sandisk Extreme III - and avoid Sandisk Ultra 4GB.

Hope this is useful for someone - cost me an afternoons work!

jimp · Dec 7, 2011, 5:51 PM

@jdamnation:

The 2GB image did not return 'dd: /dev/disk4: end of device', however still gave LBA errors for some reason.

That is likely because of what cmb mentioned in the post before yours - it's trying to access the last bit of the disk which the card claims exists by size but then returns an error when something actually tries to read it.

jdamnation · Dec 7, 2011, 5:59 PM

Yep it was the NID_NOT_FOUND errors that I was seeing.

Will this smaller image in the next release solve this particular problem?

JD

jimp · Dec 7, 2011, 6:08 PM

Not likely because that is the fault of the hardware in the CF. If the image size would have done anything, the 2GB image on the 4GB card would not have had any problems.