Hacking "SD Formatter" voodoo partitioning?

[romanaOne]

Why do special "SD Formatter" programs (currently for Mac OS X and Windows only) create strange, small gaps (~2-10MB) before or after the partition containing the filesystem?

I'm asking in this forum because I have noticed that my Supercard DSTWO seems to require SD cards formatted with this util, otherwise I get "NO FIND SYSTEM FILE" or some such Engrish ellol message.

If I just format with the Ubuntu's "Disks" application or "Disk Utility" in OS X, it uses the whole card w/o any freespace at the beginning or end.

 

[TBx]

I imagine it's similar to TRIM for SSDs. Need some extra unused space to elongate the life of the flash memory.

 

[romanaOne]

I doubt cheap (micro)SD cards (or the generic card readers in most PCs) can do any type of wear leveling. Besides, the details would vary from card to card. (Even name brand SSDs vary a lot, sometimes with disastrous results for the unsuspecting user who might think this a mature technology with standards. As a customer, you pay a big company so you can do their QC testing for them b/c they don't care.) Aaaanyway....

2MB (or the biggest gap I've seen) 4MB is not much overprovisioning for a 16GB card.

I've been looking into this and I now think the voodoo is that "SD Formatter" has a way of figuring out how to align the partition for better speed. The details of how to do this in Linux are described here under "finding page, erase block, and segment sizes."

Speed isn't just for recording video or junky flashcards: I've had FreeBSD fail to boot from a slow microSD card. (You'd want to boot FreeBSD from the SD in order to use all the SATA controllers for hard drives if you are setting up a system for network attached storage (NAS).)

For sneakernet though, I think even the dumb formatters like "Disks" and "Disk Utility" do well enough.

 

[urherenow]

usually, the space at the beginning is for a backup MFT, isn't it? But Windows also takes all of the space if you make it a PRIMARY partition...

 

[d0k3]

Why do special "SD Formatter" programs (currently for Mac OS X and Windows only) create strange, small gaps (~2-10MB) before or after the partition containing the filesystem?

I'm asking in this forum because I have noticed that my Supercard DSTWO seems to require SD cards formatted with this util, otherwise I get "NO FIND SYSTEM FILE" or some such Engrish ellol message.

If I just format with the Ubuntu's "Disks" application or "Disk Utility" in OS X, it uses the whole card w/o any freespace at the beginning or end.

If you're really interested in knowing about what these gaps are, here's a recommended read:
http://wiki.laptop.org/go/How_to_Damage_a_FLASH_Storage_Device

the Supercard issue is a different one, might have to do with the file system or the cluster sizes... whatever.

And, by the way, my EmuNAND9 implements all these techniques to optimze the performance and lifetime of your SD cards .

 

[romanaOne]

the Supercard issue is a different one, might have to do with the file system or the cluster sizes... whatever.

After reading the link above, I have to go home and recheck to be absolutely sure, but I think the only difference (using fdisk -l /dev/mmcblk0 to check) between partitioning+formatting with "Disks" and "SD Formatter" (Mac OS X) was the size of the partition+filesystem; Id, Type, sector size 512 were unchanged. I think....

 

[romanaOne]

Well, it appears that Ubuntu's "Disks" formats everything (SSDs, SD cards, USB sticks) with the first partition start at block 2048.

Typical example (a 16GB card I use in DSTWO+, not DSTWO):

sudo fdisk -l /dev/mmcblk0
Disk /dev/mmcblk0: 14.9 GiB, 15931539456 bytes, 31116288 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xb847d94e

Device  Boot Start  End  Sectors  Size Id Type
/dev/mmcblk0p1  2048 31116287 31114240 14.9G  c W95 FAT32 (LBA)

I had used a friend's windows 10 computer to use the windows version of the SD formatter to format my uSD for the Supercard DSTWO and it did not even have a partition table: it mounts as /dev/mmcblck0 and fdisk -l reports nothing at all except for "Disklabel type: dos"
Ubuntu Disks says the filesystem type is FAT32 (32-bit) type b.

The article linked below says an SD data parition should start on an erase block boundary and the the FTL(controller, voodoo, whatever you wanna call it) assumes there is a FAT and gives special priority to those first couple blocks. Crazy.
Are you doing something like this (summarized here for convenience):

0. Set heads and sectors to sane (I guess) value (sudo fdisk -H 255 -S 63 -u -c /dev/mmcblk0)
1. Use flashbench to determine erase_block_size (or just guess 4MB (8192 blocks) or 8MB (16384 blocks)
2. Determine FAT_size (using dosfsck)
3. data_start_block = erase_block_size - 2*FAT_size
4. sudo mkdosfs -n NAME -s 128 -R data_start_block -v /dev/mmcblk0p1

I tried this on a well-used Sandisk Cruzer Blade 8GB USB stick. Flashbench seemed to indicate that the erase block size was 8MB, so I used 16384 for erase block size. I threw a few large files at it and there was no huge speed difference. Then I tried it with probably wrong erase block size of 8192. It seemed a little slower, but nothing to go crazy over.
All of this is for MBR + FAT32 SD cards. I wonder what to do with ext4?


EDIT: It looks like emunand9 guesses 4MB here:

#define PARTITION_ALIGN ((4 * 1024 * 1024) / 0x200) // align at 4MB

Do all (u)SD cards likely to be used in (n)3DS (eg. 8-32GB) have 4MB erase block size?

I guess so. I have 2,4,8,16 GB Sandisk and Lexar cards and they all say this when I stick them in a linux pc:

cat /sys/block/mmcblk0/device/preferred_erase_size
4194304