"SD Formatter" voodoo partitioning?

Discussion in 'NDS - Flashcarts and Accessories' started by romanaOne, Jan 11, 2016.

  1. romanaOne
    OP

    romanaOne GBAtemp Regular

    Member
    272
    33
    Apr 18, 2014
    United States
    where the potential goes to zero
    Why do special "SD Formatter" programs (currently for Mac OS X and Windows only) create strange, small gaps (~2-10MB) before or after the partition containing the filesystem?

    I'm asking in this forum because I have noticed that my Supercard DSTWO seems to require SD cards formatted with this util, otherwise I get "NO FIND SYSTEM FILE" or some such Engrish ellol message.

    If I just format with the Ubuntu's "Disks" application or "Disk Utility" in OS X, it uses the whole card w/o any freespace at the beginning or end.
     
  2. TBx

    TBx Newbie

    Newcomer
    1
    0
    Jan 4, 2016
    United States
    I imagine it's similar to TRIM for SSDs. Need some extra unused space to elongate the life of the flash memory.
     
  3. romanaOne
    OP

    romanaOne GBAtemp Regular

    Member
    272
    33
    Apr 18, 2014
    United States
    where the potential goes to zero
    I doubt cheap (micro)SD cards (or the generic card readers in most PCs) can do any type of wear leveling. Besides, the details would vary from card to card. (Even name brand SSDs vary a lot, sometimes with disastrous results for the unsuspecting user who might think this a mature technology with standards. As a customer, you pay a big company so you can do their QC testing for them b/c they don't care.) Aaaanyway....

    2MB (or the biggest gap I've seen) 4MB is not much overprovisioning for a 16GB card.

    I've been looking into this and I now think the voodoo is that "SD Formatter" has a way of figuring out how to align the partition for better speed. The details of how to do this in Linux are described here under "finding page, erase block, and segment sizes."

    Speed isn't just for recording video or junky flashcards: I've had FreeBSD fail to boot from a slow microSD card. (You'd want to boot FreeBSD from the SD in order to use all the SATA controllers for hard drives if you are setting up a system for network attached storage (NAS).)

    For sneakernet though, I think even the dumb formatters like "Disks" and "Disk Utility" do well enough.
     
  4. urherenow

    urherenow GBAtemp Addict

    Member
    2,917
    847
    Mar 8, 2009
    United States
    Japan
    usually, the space at the beginning is for a backup MFT, isn't it? But Windows also takes all of the space if you make it a PRIMARY partition...
     
  5. d0k3

    d0k3 3DS Homebrew Legend

    Member
    2,605
    2,627
    Dec 3, 2004
    Gambia, The
    If you're really interested in knowing about what these gaps are, here's a recommended read:
    http://wiki.laptop.org/go/How_to_Damage_a_FLASH_Storage_Device

    the Supercard issue is a different one, might have to do with the file system or the cluster sizes... whatever.

    And, by the way, my EmuNAND9 implements all these techniques to optimze the performance and lifetime of your SD cards :wink: :wink:.
     
  6. romanaOne
    OP

    romanaOne GBAtemp Regular

    Member
    272
    33
    Apr 18, 2014
    United States
    where the potential goes to zero
    After reading the link above, I have to go home and recheck to be absolutely sure, but I think the only difference (using fdisk -l /dev/mmcblk0 to check) between partitioning+formatting with "Disks" and "SD Formatter" (Mac OS X) was the size of the partition+filesystem; Id, Type, sector size 512 were unchanged. I think.... :)
     
  7. romanaOne
    OP

    romanaOne GBAtemp Regular

    Member
    272
    33
    Apr 18, 2014
    United States
    where the potential goes to zero
    Well, it appears that Ubuntu's "Disks" formats everything (SSDs, SD cards, USB sticks) with the first partition start at block 2048.

    Typical example (a 16GB card I use in DSTWO+, not DSTWO):
    Code:
    sudo fdisk -l /dev/mmcblk0
    Disk /dev/mmcblk0: 14.9 GiB, 15931539456 bytes, 31116288 sectors
    Units: sectors of 1 * 512 = 512 bytes
    Sector size (logical/physical): 512 bytes / 512 bytes
    I/O size (minimum/optimal): 512 bytes / 512 bytes
    Disklabel type: dos
    Disk identifier: 0xb847d94e
    
    Device  Boot Start  End  Sectors  Size Id Type
    /dev/mmcblk0p1  2048 31116287 31114240 14.9G  c W95 FAT32 (LBA)
    
    I had used a friend's windows 10 computer to use the windows version of the SD formatter to format my uSD for the Supercard DSTWO and it did not even have a partition table: it mounts as /dev/mmcblck0 and fdisk -l reports nothing at all except for "Disklabel type: dos"
    Ubuntu Disks says the filesystem type is FAT32 (32-bit) type b.

    The article linked below says an SD data parition should start on an erase block boundary and the the FTL(controller, voodoo, whatever you wanna call it) assumes there is a FAT and gives special priority to those first couple blocks. Crazy.
    Are you doing something like this (summarized here for convenience):

    0. Set heads and sectors to sane (I guess) value (sudo fdisk -H 255 -S 63 -u -c /dev/mmcblk0)
    1. Use flashbench to determine erase_block_size (or just guess 4MB (8192 blocks) or 8MB (16384 blocks)
    2. Determine FAT_size (using dosfsck)
    3. data_start_block = erase_block_size - 2*FAT_size
    4. sudo mkdosfs -n NAME -s 128 -R data_start_block -v /dev/mmcblk0p1

    I tried this on a well-used Sandisk Cruzer Blade 8GB USB stick. Flashbench seemed to indicate that the erase block size was 8MB, so I used 16384 for erase block size. I threw a few large files at it and there was no huge speed difference. Then I tried it with probably wrong erase block size of 8192. It seemed a little slower, but nothing to go crazy over.
    All of this is for MBR + FAT32 SD cards. I wonder what to do with ext4?


    EDIT: It looks like emunand9 guesses 4MB here:
    Code:
    #define PARTITION_ALIGN ((4 * 1024 * 1024) / 0x200) // align at 4MB 
    Do all (u)SD cards likely to be used in (n)3DS (eg. 8-32GB) have 4MB erase block size?

    I guess so. I have 2,4,8,16 GB Sandisk and Lexar cards and they all say this when I stick them in a linux pc:
    Code:
    cat /sys/block/mmcblk0/device/preferred_erase_size
    4194304
    
     
    Last edited by romanaOne, Jan 15, 2016