Hacking "SD Formatter" voodoo partitioning?

romanaOne

Well-Known Member
Member
Joined
Apr 18, 2014
Messages
402
Trophies
0
Age
47
Location
where the potential goes to zero
XP
777
Country
United States
Why do special "SD Formatter" programs (currently for Mac OS X and Windows only) create strange, small gaps (~2-10MB) before or after the partition containing the filesystem?

I'm asking in this forum because I have noticed that my Supercard DSTWO seems to require SD cards formatted with this util, otherwise I get "NO FIND SYSTEM FILE" or some such Engrish ellol message.

If I just format with the Ubuntu's "Disks" application or "Disk Utility" in OS X, it uses the whole card w/o any freespace at the beginning or end.
 

TBx

New Member
Newbie
Joined
Jan 4, 2016
Messages
1
Trophies
0
XP
25
Country
United States
I imagine it's similar to TRIM for SSDs. Need some extra unused space to elongate the life of the flash memory.
 

romanaOne

Well-Known Member
Member
Joined
Apr 18, 2014
Messages
402
Trophies
0
Age
47
Location
where the potential goes to zero
XP
777
Country
United States
I doubt cheap (micro)SD cards (or the generic card readers in most PCs) can do any type of wear leveling. Besides, the details would vary from card to card. (Even name brand SSDs vary a lot, sometimes with disastrous results for the unsuspecting user who might think this a mature technology with standards. As a customer, you pay a big company so you can do their QC testing for them b/c they don't care.) Aaaanyway....

2MB (or the biggest gap I've seen) 4MB is not much overprovisioning for a 16GB card.

I've been looking into this and I now think the voodoo is that "SD Formatter" has a way of figuring out how to align the partition for better speed. The details of how to do this in Linux are described here under "finding page, erase block, and segment sizes."

Speed isn't just for recording video or junky flashcards: I've had FreeBSD fail to boot from a slow microSD card. (You'd want to boot FreeBSD from the SD in order to use all the SATA controllers for hard drives if you are setting up a system for network attached storage (NAS).)

For sneakernet though, I think even the dumb formatters like "Disks" and "Disk Utility" do well enough.
 

urherenow

Well-Known Member
Member
Joined
Mar 8, 2009
Messages
3,879
Trophies
0
Age
45
Location
Japan
XP
2,362
Country
United States
usually, the space at the beginning is for a backup MFT, isn't it? But Windows also takes all of the space if you make it a PRIMARY partition...
 

d0k3

3DS Homebrew Legend
Member
Joined
Dec 3, 2004
Messages
2,784
Trophies
0
XP
3,818
Country
Germany
Why do special "SD Formatter" programs (currently for Mac OS X and Windows only) create strange, small gaps (~2-10MB) before or after the partition containing the filesystem?

I'm asking in this forum because I have noticed that my Supercard DSTWO seems to require SD cards formatted with this util, otherwise I get "NO FIND SYSTEM FILE" or some such Engrish ellol message.

If I just format with the Ubuntu's "Disks" application or "Disk Utility" in OS X, it uses the whole card w/o any freespace at the beginning or end.
If you're really interested in knowing about what these gaps are, here's a recommended read:
http://wiki.laptop.org/go/How_to_Damage_a_FLASH_Storage_Device

the Supercard issue is a different one, might have to do with the file system or the cluster sizes... whatever.

And, by the way, my EmuNAND9 implements all these techniques to optimze the performance and lifetime of your SD cards :wink: :wink:.
 

romanaOne

Well-Known Member
Member
Joined
Apr 18, 2014
Messages
402
Trophies
0
Age
47
Location
where the potential goes to zero
XP
777
Country
United States
the Supercard issue is a different one, might have to do with the file system or the cluster sizes... whatever.

After reading the link above, I have to go home and recheck to be absolutely sure, but I think the only difference (using fdisk -l /dev/mmcblk0 to check) between partitioning+formatting with "Disks" and "SD Formatter" (Mac OS X) was the size of the partition+filesystem; Id, Type, sector size 512 were unchanged. I think.... :)
 

romanaOne

Well-Known Member
Member
Joined
Apr 18, 2014
Messages
402
Trophies
0
Age
47
Location
where the potential goes to zero
XP
777
Country
United States
Well, it appears that Ubuntu's "Disks" formats everything (SSDs, SD cards, USB sticks) with the first partition start at block 2048.

Typical example (a 16GB card I use in DSTWO+, not DSTWO):
Code:
sudo fdisk -l /dev/mmcblk0
Disk /dev/mmcblk0: 14.9 GiB, 15931539456 bytes, 31116288 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xb847d94e

Device  Boot Start  End  Sectors  Size Id Type
/dev/mmcblk0p1  2048 31116287 31114240 14.9G  c W95 FAT32 (LBA)

I had used a friend's windows 10 computer to use the windows version of the SD formatter to format my uSD for the Supercard DSTWO and it did not even have a partition table: it mounts as /dev/mmcblck0 and fdisk -l reports nothing at all except for "Disklabel type: dos"
Ubuntu Disks says the filesystem type is FAT32 (32-bit) type b.

The article linked below says an SD data parition should start on an erase block boundary and the the FTL(controller, voodoo, whatever you wanna call it) assumes there is a FAT and gives special priority to those first couple blocks. Crazy.
Are you doing something like this (summarized here for convenience):

0. Set heads and sectors to sane (I guess) value (sudo fdisk -H 255 -S 63 -u -c /dev/mmcblk0)
1. Use flashbench to determine erase_block_size (or just guess 4MB (8192 blocks) or 8MB (16384 blocks)
2. Determine FAT_size (using dosfsck)
3. data_start_block = erase_block_size - 2*FAT_size
4. sudo mkdosfs -n NAME -s 128 -R data_start_block -v /dev/mmcblk0p1

I tried this on a well-used Sandisk Cruzer Blade 8GB USB stick. Flashbench seemed to indicate that the erase block size was 8MB, so I used 16384 for erase block size. I threw a few large files at it and there was no huge speed difference. Then I tried it with probably wrong erase block size of 8192. It seemed a little slower, but nothing to go crazy over.
All of this is for MBR + FAT32 SD cards. I wonder what to do with ext4?


EDIT: It looks like emunand9 guesses 4MB here:
Code:
#define PARTITION_ALIGN ((4 * 1024 * 1024) / 0x200) // align at 4MB

Do all (u)SD cards likely to be used in (n)3DS (eg. 8-32GB) have 4MB erase block size?

I guess so. I have 2,4,8,16 GB Sandisk and Lexar cards and they all say this when I stick them in a linux pc:
Code:
cat /sys/block/mmcblk0/device/preferred_erase_size
4194304
 
Last edited by romanaOne,

Site & Scene News

General chit-chat
Help Users
  • No one is chatting at the moment.
  • Tom Bombadildo @ Tom Bombadildo:
    Maybe we can break it again Flame ;O;
  • Flame @ Flame:
    tom whats something cool to mod?
  • Flame @ Flame:
    i have a itch to mod something
  • Flame @ Flame:
    like broken or something

    I mod you <3 - Sweebs
  • Tom Bombadildo @ Tom Bombadildo:
    Urmum ;O;
  • Flame @ Flame:
    you cut
  • DinohScene @ DinohScene:
    You chop
  • XAIXER @ XAIXER:
    Computer repair is always fun
  • DinohScene @ DinohScene:
    It's bloody awful
  • XAIXER @ XAIXER:
    I find it quite vexing and soothing at the same time :P
  • DinohScene @ DinohScene:
    Shit always breaks at the wrong moment :')
  • XAIXER @ XAIXER:
    That too
  • DinohScene @ DinohScene:
    Doing it for a living and in me spare time.. Fuck it at times
  • XAIXER @ XAIXER:
    Yup
  • XAIXER @ XAIXER:
    If I get sick of computers, I just go back to linguistics.
  • DinohScene @ DinohScene:
    Perhaps yelling in latin might help speed up the process of troubleshooting haha
  • XAIXER @ XAIXER:
    It actually does :P
  • The Real Jdbye @ The Real Jdbye:
    there needs to be a compact mode
  • The Real Jdbye @ The Real Jdbye:
    i can only see 6 messages at a time
  • WiiMiiSwitch @ WiiMiiSwitch:
    How is everyone?
  • Plazorn @ Plazorn:
    I am good, also glad Temporary Mode is over
  • WiiMiiSwitch @ WiiMiiSwitch:
    Nice too know @Plazorn
  • ZeroT21 @ ZeroT21:
    Need lifemod, me bored
    ZeroT21 @ ZeroT21: Need lifemod, me bored