NanoBSD Image Block Alignment [MisAlignment]
-
Ok, my bad, one added thing doesn't check out, which is the 2K flash erase block size said to be "common". So common in fact I've never ever seen it that small, unless we are getting into SSD's.
Cheap flash, that we should be focused on, has an erase block size of 4K or larger. Some USB flash drives have been reported to use minimum erase blocks as large as 1MB, also all noted in my first post. Yeah, generally people just throw these away because the performance is so horrible, but it's good to let people know they might be dealing with such a device.
All of this can be found using a simple dd raw write test script, I can post my version of it if anyone is interested. SD cards on a native (non USB) interface report this value to Linux where it can easily be read, not sure about FreeBSD. A native SD interface would be one such as found on most Android devices, which handily is already running Linux.
-
Honestly, who on the pfSense dev team doesn't know about 512 byte, 512e, and 4K magnetic discs.
No one, but if you get Jim on his soap box… :)
If you need references, I will be happy to post a set of links for fact checking every single word I've typed.
Please do. Better yet, post real world results of "wrong" vs. "right" as relevant to our embedded images.
And don't get all worked up with me because doktornotor is a dick.
-
@cmb:
Please do. Better yet, post real world results of "wrong" vs. "right" as relevant to our embedded images.
The results I have posted are real world, my references will be as well. There are some on the forum I've linked to already, see my second post, and comments under it. Anything that is misaligned, more specifically to sector 63, I would think is relevant, would you agree?
Someone would have to manually rebuild one of the NanoBSD images to test proper alignment on an actual NanoBSD install. I don't have a good FreeBSD VM to work with right now, or time unfortunately. Someone else could do this much easier / faster.
The fact that OpenBSD has made this change already, noting performance, is pretty solid as far as I'm concerned.
@cmb:
And don't get all worked up with me because doktornotor is a dick.
Fair enough, lol.
References in next post…
-
References
Relevant Alignment Info for FAT32 SD. All applies except for the bits about FAT, obviously. Pay attention to the MBR / partition layout using sectors to calculate alignment.
http://3gfp.com/wp/2014/07/formatting-sd-cards-for-speed-and-lifetime/Block Device Attributes I spoke about.
https://www.kernel.org/doc/Documentation/mmc/mmc-dev-attrs.txtMeasuring Flash Block Size, wrote my test script based on this. I just dd write from /dev/zero to smaller block sizes in descending order starting from 8MB, it works great. The first size that bombs out as being way slower / longer, is one below your smallest erase block. This is for ANY flash based storage device.
http://kim.oyhus.no/FlashBlockSize.htmlOp, I was off, here we have flash devices using erase block sizes of 8MB, I said much smaller, so the issue can be worse than I laid out.
https://www.raspberrypi.org/forums/viewtopic.php?t=11258&p=123670Speed of USB Flash Devices. So wish they recorded more filesystem details, but still a good reference. You can easily see how the exact same device reads / writes far slower in different circumstances, even with the same format. And which devices to look for when buying.
http://usbflashspeed.com/Edit: FreeBSD Specific References Added
Good Alignment Info for UFS / ZFS on FreeBSD
http://ivoras.net/blog/tree/2011-01-01.freebsd-on-4k-sector-drives.htmlDiscussion about forcing alignment with fdisk, and why it's not updated I suppose (gpart).
https://forums.freebsd.org/threads/gpart-trying-to-force-mbr-partitions-to-be-cylinder-aligned.36439/Awesome example on FreeBSD using an actual drive that reports 4K sectors. This specific example I had not seen before. Remember, our flash drives still report 512 byte sectors, the NanoBSD images will end up aligned to KB or MB boundaries, not specific sectors. And the existing UFS filesystem is already using a 4K fragment size, so the partition is all that's left to fix.
https://forums.freebsd.org/threads/ufs-sector-and-alignment-explanation.42208/ -
Dumb dick question: When you do /etc/rc.conf_mount_rw; /etc/rc.conf_mount_rw the it still takes damn ages without any config changes done and with nothing to write anywhere. How does that fit your write amplification theories? For some mysterious reason still unanswered. ::)
Regards,
Mr. Dumb Dick
-
Dumb dick question: When you do /etc/rc.conf_mount_rw; /etc/rc.conf_mount_rw the it still takes damn ages without any config changes done and with nothing to write anywhere. How does that fit your write amplification theories? For some mysterious reason still unanswered. ::)
Regards,
Mr. Dumb Dick
A. Because you keep referring to documented referenced facts as theories, like an ignorant dick.
B. Because no one likes teaching ignorant dicks how to use Google.
-
Right. The most amplified writes are those where there's nothign to write. Excellent theory.
-
Right. The most amplified writes are those where there's nothign to write. Excellent theory.
I can't count now, the number of times I've said, remounting UFS results in writes to disk.
Nor can I count, the number of times I've referenced the source of the old kernel patch as hard proof of this.
BTW, the answer was actually…
E: Your Mom
-
Ok, seriously now… the reason I'm taking a minute to post something useful...
If you question the alignment, simply flash a drive with a NanoBSD image, boot Gparted Live from a disc or USB [YUMI works great]. If the storage device with pfSense on it uses 512 byte sector emulation, the very first [only] MBR partition that it can read, will start at sector 63. You can see all of this from the GUI.
If your device with pfSense NanoBSD on it uses a different size for sector emulation, just multiply whatever that sector size is by the sector number the first MBR partition starts on, and you will get 31.5K.
Then, look at the very first [of three] FreeBSD slice inside that first MBR partition, and it starts at sector 0 of the MBR partition, the very same sector that the MBR partition starts on.
-
Oh, fuck me gently sideways. Yeah, it writes even when there's nothing to write.
Look: This shit worked until 8.3. Then someone broke it.
Now, go find the real bug and align your ass instead.
-
Oh, fuck me gently sideways. Yeah, it writes even when there's nothing to write.
Look: This shit worked until 8.3. Then someone broke it.
Now, go find the real bug and align your ass instead.
Sounds like I'm too busy aligning your ass to me…
Anyways, yeah, I've read that thread. They changed some kernel code in 8.3 that added filesystem stability. pfSense devs wrote a patch to unpatch, if you will, 8.3 so it behaved again like 8.2, which the devs currently call an "incorrect" solution, and will not reintroduce that patch. So, forward we go with said bug hunt… ah shit, now you have me giving history lessons.
-
Oh yeah, and, for the… (I can't count how many times now), you're forgetting the whole topic of the thread your posting in. It's not called "fix remount issue" for a reason. That's not what it's about.
It's about fixing the alignment issue, because it is such a dirt simple thing to fix in the build system. Hell, wouldn't fixing it in the installer also make a great upstream patch to get FreeBSD on the same page as every other OS that's still maintained today? I would be infinitely happy if it simply made packages install at a reasonable speed, or made upgrades install at a reasonable speed, or even made slice duplication happen at a reasonable speed.
When I have a CF device that I can write to at (lets round) around 4MB/sec, and after flashing it with pfSense NanoBSD, am now writing to it at 0.04MB/sec, there is clearly some slowdown to be made up there. Even doktornodick or whatever that spells, can't argue against that, not coherently anyways.
-
Bored waiting for Windows to update, added following FreeBSD specific references to articles / threads in the references post:
Edit: FreeBSD Specific References Added
Good Alignment Info for UFS / ZFS on FreeBSD
http://ivoras.net/blog/tree/2011-01-01.freebsd-on-4k-sector-drives.htmlDiscussion about forcing alignment with fdisk, and why it's not updated I suppose (gpart).
https://forums.freebsd.org/threads/gpart-trying-to-force-mbr-partitions-to-be-cylinder-aligned.36439/Awesome example on FreeBSD using an actual drive that reports 4K sectors. This specific example I had not seen before. Remember, our flash drives still report 512 byte sectors, the NanoBSD images will end up aligned to KB or MB boundaries, not specific sectors. And the existing UFS filesystem is already using a 4K fragment size, so the partition is all that's left to fix.
https://forums.freebsd.org/threads/ufs-sector-and-alignment-explanation.42208/This one's for you cmb, for the only reasonable discussion that's taken place here. Thank you for a little sanity.
-
They changed some kernel code in 8.3 that added filesystem stability
ROFL. Look, we added stability. No regression at all. It's not slower than molasses – it's just very stable now!
-
Then find the bug that you are clearly capable of finding, submit it to FreeBSD, and get it patched upstream. pfSense is the wrong place for messy kernel patches.
-
The bug was already filed. Considering people have been complaining just for 3+ years, this definitely can wait. The IPv6 fragmention bug in pf has been filed just ~7 years ago – and of course still well alive and kicking. Welcome to FreeBSD's kernel land, brace for impact!
-
The bug was already filed. Considering people have been complaining just for 3+ years, this definitely can wait. The IPv6 fragmention bug in pf has been filed just ~7 years ago – and of course still well alive and kicking. Welcome to FreeBSD's kernel land, brace for impact!
Hahahaha, epic.
Fragmentation of packets is against the entire IPv6 protocol, for good reason. And you cite the unstable kernel patch as not being applied upstream.
All in the same post.
You absolutely do not belong in this discussion. Any further posts by you will be pertinently ignored by me. If anyone else feels your worthy of a response… never mind, they won't.
-
-
Added info posted in another thread. Also simple testing method found, see end.
What "fixes" the issue? from disk to disk, regardless of type (flash / spinny) and write rate, isn't necessarily the disk controller, but how much cache the disk controller has to work with.
If it has enough cache to absorb all the random writes and spit them out to the physical layer on its own time, it tells the kernel it has all that data, and you don't see hanging issues.
If it doesn't have enough cache to absorb all the random writes, you wait until it has actually written most of it to the physical layer, then it tells the kernel it has all the data, and this is the hang you experience.
So, it's not a disk speed issue at all, it's that faster / larger / newer disks tend to have better controllers with more cache than slower / smaller / older disks.
It doesn't help that NanoBSD images are not 4k aligned, so writes take even longer than they should on flash & 4k drives. Hopefully this gets fixed for the 2.3 branch, or we get a NanoBSD installer so we can fix it manually.
Want to test? Run a pfSense NanoBSD VM on a system with a RAID controller that has a descent size cache in write-back mode. Now try it with the controller cache disabled. Now try it with the controller cache disabled and running on a true 4k sector drive. Spoiler: good, bad, worse.
-
ToDo: https://redmine.pfsense.org/issues/4847
