Hacking Fat32 cluster size

[PsyBlade]

At the moment using 32k clusters seems all the rage.
I wnt to know if there really is something that works only or at least better with it.

Because at the moment wbfs2fat tries to keep the clustersize small if possible.
(Mainly because thats what I have done since the introduction of fat32)
And I am pondering wether to change that.

 

[fig2k4]

I'm sure that 32KB is just the optimal size. Given that Wii disc sectors are 32KB, it means that there's no wasted space (half full clusters) and it's faster to read too because reading 1 Wii disc sector means reading 1 full cluster.

I honestly don't think there's much difference using 8KB or 16KB. Like I said, it's just the "optimal" size.

IIRC, there used to be some compatibility issues with 64KB clusters, but I don't think that's the case now either.

 

[tueidj]

Since when are wii disc sectors 32KB?

 

[fig2k4]

Really?

Anyway, here's a little something from the CFG FAQ that backs up my theory that the size doesn't really matter.

Q: What is the recommended cluster size when formatting FAT?
A: It's not important - any size should work fine. But 32kb is a good choice as that size is aligned to wii disc sector size.

 

[tueidj]

All DVDs use 2KB sectors. Wii discs are no exception.
The 32KB reference comes from the cluster size used to group sectors together for hashing and encryption. But that imposes no limitations on random access; decryption can be started on any 16 byte boundary and raw disc access can be (and is frequently) started on any 4 byte boundary. Seems like someone saw "cluster size" in two different places and assumed they must mean the same thing when in reality they're based on different concepts.

 

[PsyBlade]

I care.

But my question was more about sneek, DML and the like too, not only normal usb-loaders

 

[Taleweaver]

I don't really follow your question, PsyBlade. UNEEK+DI doesn't work on a FAT32 64k clustersize formatted drive*. It just doesn't boot. The exact same setup & files on FAT32 32k clustersize works fine. I've been told it's the same with the lower clustersizes, , but I haven't checked this myself.




*note: this was a while ago...somewhat around revision 50. Perhaps the "32k clustersize or lower" limitation is not needed anymore, but I doubt it.

 

[PsyBlade]

32k is already the maximum allowed by the fat32 standart.
I am more interested in smaller clusters, not some ugly hacks.

 

[smf]

But that imposes no limitations on random access; decryption can be started on any 16 byte boundary and raw disc access can be (and is frequently) started on any 4 byte boundary.

You have to read the entire 32k cluster and decrypt all the data from the start as AES-CBC doesn't allow mid stream decryption (and the 0x7c00 bytes of data are encrypted as one stream).

http://wiibrew.org/wiki/Wii_Disc#Partition_Data

the 32k clusters are grouped in larger numbers for hashing.

 

[tueidj]

You have to read the entire 32k cluster and decrypt all the data from the start as AES-CBC doesn't allow mid stream decryption (and the 0x7c00 bytes of data are encrypted as one stream).

Nope, that's exactly what CBC is designed for. The IV for a 16-byte encrypted block is the ciphertext from the preceeding 16-byte block so you can just jump into the middle and start reading.

 

[fig2k4]

"Knowing" things is trolling when it's totally off topic. It's got literally nothing to do with what I was talking about either. In my world Wii disc sectors at 32KB, full stop.

If there's going to be a lot of small files, it might benefit from the smaller cluster size. But if the majority of files are going to be WBFS files for example, there's no reason to make the cluster size smaller than 32KB, IMO.

Unless the drive is too small to support 32KB clusters obviously. As you probably know, FAT32 needs at least 65535 clusters, so making the cluster size 32KB on smaller drives is invalid. In that case I'd use the next largest valid size.

From reading that back I realise that I tend to use the largest possible size, rather than the smallest.

 

[PsyBlade]

I don't really follow your question, PsyBlade. UNEEK+DI doesn't work on a FAT32 64k clustersize formatted drive*. It just doesn't boot. The exact same setup & files on FAT32 32k clustersize works fine. I've been told it's the same with the lower clustersizes, , but I haven't checked this myself.

My questions are:
1. Are there Wii programs that only work with a specific cluster size?
2. Are there Wii programs that work better with specific cluster sizes?

Because what I heared as advise for e.g. sneek contradicts itself.
like "32k only" "

 

[tueidj]

@[member='fig2k4']: I'm pretty sure it was exactly the same subject you were talking about, and I think pointing out that d2x considers disc sectors to be only 2048 bytes is especially relevant.

My questions are:
1. Are there Wii programs that only work with a specific cluster size?
2. Are there Wii programs that work better with specific cluster sizes?

The maximum size of transfers for ye olde USB2 module (used in almost every cios ever) is 32KB. I think the maximum DMA size of the SD controller is the same. So you don't want to (can't) go over that.
The older cioses that used libfat didn't really care about the cluster size at all; libfat uses a configurable intermediate cache. It was probably hardcoded to something small in the cios, like 4KB or 8KB. All disc accesses would have used multiples of that.
Now everything seems to be using ChaN's fat filesystem code instead which reads whole clusters at once. This is fine if you actually want the whole cluster, but otherwise it might be reading a lot of data that it doesn't actually need. Only one cluster is cached at a time so if two files from different parts of the disc are being alternately accessed, it has to keep reading in the same two clusters over and over which makes the wasted reading time much worse.

A lot of files on the NAND are small (< 4KB), so for SNEEK I think it would be faster with smaller clusters. For discs it would vary, some game makers know how to pack their resources properly to reduce access time (sequential reading = good for big clusters) while others (like the idiots at TellTale) have thousands of small files scattered all over the place (random reading = bad for big clusters).

 

[smf]

You have to read the entire 32k cluster and decrypt all the data from the start as AES-CBC doesn't allow mid stream decryption (and the 0x7c00 bytes of data are encrypted as one stream).

Nope, that's exactly what CBC is designed for. The IV for a 16-byte encrypted block is the ciphertext from the preceeding 16-byte block so you can just jump into the middle and start reading.

Hmm, didn't know that.. I'll do some investigation. However the Wii will always read from the beginning of the cluster as it needs to read the hashes. Which it does even if you tell it to read from within the middle of a cluster.

 

[tueidj]

That's assuming the disc is actually encrypted/hashed (GC discs aren't, so it's irrelevant for DML), the hashes for the target cluster aren't already in memory from a previous partial read and the cios authors remain clueless enough to not bother disabling hash checking (even though the TMD signature check is already disabled so verification is pointless).

 

[Taleweaver]

Hmm...I just noticed these posts, suggesting that 64k cluster size is better for DML regarding compatibility. I haven't checked myself yet, but it's worth mentioning here.

 

[JoostinOnline]

At the moment using 32k clusters seems all the rage.
I wnt to know if there really is something that works only or at least better with it.

Because at the moment wbfs2fat tries to keep the clustersize small if possible.
(Mainly because thats what I have done since the introduction of fat32)
And I am pondering wether to change that.

32k is the maximum for NEEK, but DML works on 64k clusters. I don't know how it works on 32k from personal experience, but reports suggest it doesn't work well.

I say stick with 32k for HDD's.

 

[smf]

(even though the TMD signature check is already disabled so verification is pointless).

verification is still useful, because it ensures integrity. It would be better if the TMD was checked as well of course.

The larger the cluster size, the less fragmentation there is and the smaller the file allocation table.

 

[tueidj]

Without checking the entire integrity chain it's worthless. And if the hash check fails, the game forcibly locks up - what's the advantage of that? May as well skip it, let the game eat the bad data (it probably won't crash, which is better than always crashing) and gain a nice speed boost.

 

[cwjakesteel]

Hmm...I just noticed these posts, suggesting that 64k cluster size is better for DML regarding compatibility. I haven't checked myself yet, but it's worth mentioning here.

It's in the guide. That the SD card works 'best' with 64kb cluster sizes and that the USB's can be no larger than 32kb. But then that news was changed to: The USB's cluster size should be 32kb, not less than or equal to.