Fat32 cluster size

Discussion in 'Wii - Backup Loaders' started by PsyBlade, Jan 28, 2012.

Jan 28, 2012

Fat32 cluster size by PsyBlade at 10:39 PM (12,106 Views / 0 Likes) 22 replies

  1. PsyBlade
    OP

    Member PsyBlade Snake Charmer

    Joined:
    Jul 30, 2009
    Messages:
    2,204
    Location:
    Sol III
    Country:
    Germany
    At the moment using 32k clusters seems all the rage.
    I wnt to know if there really is something that works only or at least better with it.

    Because at the moment wbfs2fat tries to keep the clustersize small if possible.
    (Mainly because thats what I have done since the introduction of fat32)
    And I am pondering wether to change that.
     


  2. fig2k4

    Member fig2k4 GBAtemp Advanced Fan

    Joined:
    Oct 30, 2009
    Messages:
    758
    Location:
    Scotland
    Country:
    United Kingdom
    I'm sure that 32KB is just the optimal size. Given that Wii disc sectors are 32KB, it means that there's no wasted space (half full clusters) and it's faster to read too because reading 1 Wii disc sector means reading 1 full cluster.

    I honestly don't think there's much difference using 8KB or 16KB. Like I said, it's just the "optimal" size.

    IIRC, there used to be some compatibility issues with 64KB clusters, but I don't think that's the case now either.
     
  3. tueidj

    Member tueidj I R Expert

    Joined:
    Jan 8, 2009
    Messages:
    2,569
    Country:
    Since when are wii disc sectors 32KB?
     
  4. fig2k4

    Member fig2k4 GBAtemp Advanced Fan

    Joined:
    Oct 30, 2009
    Messages:
    758
    Location:
    Scotland
    Country:
    United Kingdom
    Really? :)

    Anyway, here's a little something from the CFG FAQ that backs up my theory that the size doesn't really matter.

     
  5. tueidj

    Member tueidj I R Expert

    Joined:
    Jan 8, 2009
    Messages:
    2,569
    Country:
    All DVDs use 2KB sectors. Wii discs are no exception.
    The 32KB reference comes from the cluster size used to group sectors together for hashing and encryption. But that imposes no limitations on random access; decryption can be started on any 16 byte boundary and raw disc access can be (and is frequently) started on any 4 byte boundary. Seems like someone saw "cluster size" in two different places and assumed they must mean the same thing when in reality they're based on different concepts.
     
  6. fig2k4
    This message by fig2k4 has been removed from public view by a moderator, Dec 9, 2016.
    Jan 29, 2012
  7. PsyBlade
    OP

    Member PsyBlade Snake Charmer

    Joined:
    Jul 30, 2009
    Messages:
    2,204
    Location:
    Sol III
    Country:
    Germany
    I care.

    But my question was more about sneek, DML and the like too, not only normal usb-loaders
     
  8. Taleweaver

    Member Taleweaver Storywriter

    Joined:
    Dec 23, 2009
    Messages:
    5,124
    Location:
    Belgium
    Country:
    Belgium
    I don't really follow your question, PsyBlade. UNEEK+DI doesn't work on a FAT32 64k clustersize formatted drive*. It just doesn't boot. The exact same setup & files on FAT32 32k clustersize works fine. I've been told it's the same with the lower clustersizes, , but I haven't checked this myself.




    *note: this was a while ago...somewhat around revision 50. Perhaps the "32k clustersize or lower" limitation is not needed anymore, but I doubt it.
     
  9. PsyBlade
    OP

    Member PsyBlade Snake Charmer

    Joined:
    Jul 30, 2009
    Messages:
    2,204
    Location:
    Sol III
    Country:
    Germany
    32k is already the maximum allowed by the fat32 standart.
    I am more interested in smaller clusters, not some ugly hacks.
     
  10. smf

    Member smf GBAtemp Advanced Fan

    Joined:
    Feb 23, 2009
    Messages:
    839
    Country:
    United Kingdom
    You have to read the entire 32k cluster and decrypt all the data from the start as AES-CBC doesn't allow mid stream decryption (and the 0x7c00 bytes of data are encrypted as one stream).

    http://wiibrew.org/wiki/Wii_Disc#Partition_Data

    the 32k clusters are grouped in larger numbers for hashing.
     
  11. tueidj

    Member tueidj I R Expert

    Joined:
    Jan 8, 2009
    Messages:
    2,569
    Country:
    Nope, that's exactly what CBC is designed for. The IV for a 16-byte encrypted block is the ciphertext from the preceeding 16-byte block so you can just jump into the middle and start reading.
     
  12. fig2k4

    Member fig2k4 GBAtemp Advanced Fan

    Joined:
    Oct 30, 2009
    Messages:
    758
    Location:
    Scotland
    Country:
    United Kingdom
    "Knowing" things is trolling when it's totally off topic. It's got literally nothing to do with what I was talking about either. In my world Wii disc sectors at 32KB, full stop.

    If there's going to be a lot of small files, it might benefit from the smaller cluster size. But if the majority of files are going to be WBFS files for example, there's no reason to make the cluster size smaller than 32KB, IMO.

    Unless the drive is too small to support 32KB clusters obviously. As you probably know, FAT32 needs at least 65535 clusters, so making the cluster size 32KB on smaller drives is invalid. In that case I'd use the next largest valid size.

    From reading that back I realise that I tend to use the largest possible size, rather than the smallest.
     
  13. PsyBlade
    OP

    Member PsyBlade Snake Charmer

    Joined:
    Jul 30, 2009
    Messages:
    2,204
    Location:
    Sol III
    Country:
    Germany
    My questions are:
    1. Are there Wii programs that only work with a specific cluster size?
    2. Are there Wii programs that work better with specific cluster sizes?

    Because what I heared as advise for e.g. sneek contradicts itself.
    like "32k only" "
     
  14. tueidj

    Member tueidj I R Expert

    Joined:
    Jan 8, 2009
    Messages:
    2,569
    Country:
    @[member='fig2k4']: I'm pretty sure it was exactly the same subject you were talking about, and I think pointing out that d2x considers disc sectors to be only 2048 bytes is especially relevant.

    The maximum size of transfers for ye olde USB2 module (used in almost every cios ever) is 32KB. I think the maximum DMA size of the SD controller is the same. So you don't want to (can't) go over that.
    The older cioses that used libfat didn't really care about the cluster size at all; libfat uses a configurable intermediate cache. It was probably hardcoded to something small in the cios, like 4KB or 8KB. All disc accesses would have used multiples of that.
    Now everything seems to be using ChaN's fat filesystem code instead which reads whole clusters at once. This is fine if you actually want the whole cluster, but otherwise it might be reading a lot of data that it doesn't actually need. Only one cluster is cached at a time so if two files from different parts of the disc are being alternately accessed, it has to keep reading in the same two clusters over and over which makes the wasted reading time much worse.

    A lot of files on the NAND are small (< 4KB), so for SNEEK I think it would be faster with smaller clusters. For discs it would vary, some game makers know how to pack their resources properly to reduce access time (sequential reading = good for big clusters) while others (like the idiots at TellTale) have thousands of small files scattered all over the place (random reading = bad for big clusters).
     
  15. smf

    Member smf GBAtemp Advanced Fan

    Joined:
    Feb 23, 2009
    Messages:
    839
    Country:
    United Kingdom
    Hmm, didn't know that.. I'll do some investigation. However the Wii will always read from the beginning of the cluster as it needs to read the hashes. Which it does even if you tell it to read from within the middle of a cluster.
     
  16. tueidj

    Member tueidj I R Expert

    Joined:
    Jan 8, 2009
    Messages:
    2,569
    Country:
    That's assuming the disc is actually encrypted/hashed (GC discs aren't, so it's irrelevant for DML), the hashes for the target cluster aren't already in memory from a previous partial read and the cios authors remain clueless enough to not bother disabling hash checking (even though the TMD signature check is already disabled so verification is pointless).
     
  17. Taleweaver

    Member Taleweaver Storywriter

    Joined:
    Dec 23, 2009
    Messages:
    5,124
    Location:
    Belgium
    Country:
    Belgium
    Hmm...I just noticed these posts, suggesting that 64k cluster size is better for DML regarding compatibility. I haven't checked myself yet, but it's worth mentioning here.
     
  18. JoostinOnline

    Member JoostinOnline Certified Crash Test Dummy

    Joined:
    Apr 2, 2011
    Messages:
    10,834
    Location:
    The Twilight Zone
    Country:
    United States
    32k is the maximum for NEEK, but DML works on 64k clusters. I don't know how it works on 32k from personal experience, but reports suggest it doesn't work well.

    I say stick with 32k for HDD's.
     
  19. smf

    Member smf GBAtemp Advanced Fan

    Joined:
    Feb 23, 2009
    Messages:
    839
    Country:
    United Kingdom
    verification is still useful, because it ensures integrity. It would be better if the TMD was checked as well of course.

    The larger the cluster size, the less fragmentation there is and the smaller the file allocation table.
     
  20. tueidj

    Member tueidj I R Expert

    Joined:
    Jan 8, 2009
    Messages:
    2,569
    Country:
    Without checking the entire integrity chain it's worthless. And if the hash check fails, the game forcibly locks up - what's the advantage of that? May as well skip it, let the game eat the bad data (it probably won't crash, which is better than always crashing) and gain a nice speed boost.
     
    1 person likes this.
  21. cwjakesteel

    Newcomer cwjakesteel Member

    Joined:
    Apr 7, 2012
    Messages:
    17
    Country:
    Jamaica
    It's in the guide. That the SD card works 'best' with 64kb cluster sizes and that the USB's can be no larger than 32kb. But then that news was changed to: The USB's cluster size should be 32kb, not less than or equal to.
     

Share This Page