Just on the off chance anyone was wondering, the reboot patches work just fine with Monster Hunter Generations. Test on my US version old3DS, running the leaked EU copy of the game. This also means that the reboot patches work fine with the region-free patch.
We may not be that far from agreement. I'd appreciate your help understanding what aspects I may have gotten wrong.
See quick TLDR and then full-blown technical details that I hope you agree avoids 'cargo cult debugging'.
As to C, we totally agree. Neither Windows nor the official SD formatter allow FAT32 on SD media larger than 32GB. We also agree that partition alignment is critical to performance on SD, due at least in part to the physical flash page sizes (erase and write) being larger than the logical sector size of 512 bytes.
As to B, I think you were disagreeing that creation of zero-byte files first would improve performance on FAT32. I still believe this to be true, and provided some technical details below as to how this helps. In return, can you please help me understand if there are parts which don't make sense, are wrong, etc.?
As to A, I think you were stating that having FS metadata and file data contigous on the SD would never be beneficial, and defrag is bad. I agree in part; Old style defragmenters would not improve speed and would shorten the SD lifespan. FYI, in the linked post, I was responding to someone with a heavily-used SD card, which likely had both file and file system metadata fragmentation. As listed below via the technical details, I think there are some benefits to be had.
I agree that most of these wouldn't have any appreciable effect on PC hardware. Given the terrible read and write perf of the 3DS's SD, small changes can add up.
I will reference the 1999 Microsoft whitepaper on FAT32, named FatGen102.pdf (google it).
The micro-optimizations I will address target two aspects: the FAT and file system metadata. I'll start with an overview of the FAT, then directory entry structure (file system metadata), and then the FAT.
Background reading can include page 12-18 of the FatGen102.pdf whitepaper.
The File Allocation Table (FAT) is a singly-linked list indicating the clusters (extents) of a file. See page 12. One can logically view the FAT as a single large array of 32-bit entries. However, each 32-bit FAT entry has only 28 valid bits... the top four bits of each are reserved (ignored). See page 15. The first two clusters are reserved. See page 14.
There are two special values in the FAT. The first is "end of clusterchain", indicating that no further clusters are allocated for the file. The second is a "bad cluster" mark, indicating that the cluster should not be considered as free (e.g., due to unreliability). See pages 16-17.
The FAT entry for cluster zero (reserved cluster) is always set to the special value indicating "end of clusterchain". A zero-length file has cluster zero listed as its first cluster number. See page 16.
Background reading can include pages 16, 20-22.
Any directory (including the root directory) is simply a file whose contents contain "directory entry" structures. See page 16.
The first two entries for all directories (other than the root directory) are the special directory entries "." and "..". See page 23.
FAT32 uses 32-byte "directory entry" structures. See page 22.
Thus, a single 512 byte sector may only contain 16 entries, and a 32k cluster size will allow a single cluster to contain at most 1024 entries. See page 22.
Each file (or directory) in FAT32 uses one or more of these 32-byte entries. See page 22.
Any filename that doesn't fit in the 8dot3 notation uses at least one additional directory entry for the long file name. (not explicitly listed in this document).
If all files have a long file name, the maximums are 8 entries per sector, and 512 entries per cluster.
The first byte of the directory entry structure may have special values. The value 0xE5 indicates the entry is free and can be re-used. The value 0x00 also indicates the entry is free and can be re-used, but also indicates that there are no other valid entries following it. See page 22.
Interestingly, it does not appear to be explicitly required that invalid entries following the first 0x00 entry have any specific contents, although they are commonly also set to 0x00.
There is no requirement that entries are sorted in any way. See Note 1 on page 25.
Each directory entry for a file includes 32-bits for the entry's first cluster (DIR_FstClusLO and DIR_FstClusHI). See page 22.
As file sizes are not always multiples of cluster size, each directory entry for a file has a 32-bit entry storing the size, in bytes, of the file. See page 22.
FAT32 stores a hint for the next free cluster. See page 20.
Zero-byte files are a special case. A zero-byte file's directory entry puts zero as the first cluster for the file. This avoids any allocation of "real" space in the FAT for the file. See last paragraph of page 16.
I suggest there is some benefit to preventing the need to read more than a single sector of data to find the most commonly used files.
For this section, please presume the volume has just been mounted. Thus, the only cached information is the location of the FAT, and the location of the first cluster of the root directory. All reads will read an entire cluster at a time (not 512 byte sector). Edge cases ignored (such as a directory entry's long file name being in different cluster than the directory entry with the 8dot3 name, or corrupt entries).
# Directory entries are not sorted
# Thus must sequentially search
unsigned long currentCluster <== dirEntry.FirstCluster
signed long remainingEntries <== dirEntry.FileSize / 32 # count of directory entries
bool keepParsing <== totalEntries > 0
short entryIndex <== -1;
while(keepParsing) {
# read the directory entries from the cluster
ReadCluster(currentCluster, buffer);
# determine number of valid entries read
short validEntries <== min(CLUSTER_SIZE/sizeof(DIR_ENTRY32), remainingEntries);
remainingEntries <== remainingEntries - validEntries;
# check these entries for a match
entryIndex <== FindMatchingEntry(nameToFind, buffer);
if(-1 <> entryIndex) {
# found entry, so stop parsing
keepParsing <== false;
}else if(0 < remainingEntries) {
# more entries, so figure out next cluster
currentCluster <== DetermineNextClusterFromFAT(currentCluster);
}else{
# no more entries, and not found...
keepParsing <== false;
}
}
if (-1 == entryIndex) return null;
else ... # allocate and return copy of directory entry;
Normally only a single cluster is read from the SD for each directory's metadata. As can be seen, if the number of entries exceeds what fits in that cluster, then TWO additional reads may be required: one to parse the FAT and determine the next cluster, and a second to read that cluster's actual data.
Let's say someone loaded a directory with 3000 files (numbered 0..2999), and then deleted all but the last 20 files. In this case, multiple clusters of directory_entry will all be filled with "free" entries, and it would take additional reads and processing time to find the directory from the last cluster. This is not an unusual pattern for consumer electronics, such as cameras. It may also occur when installing many applications, then deleting many of the older applications.
If the first 20 files were not deleted, they would be found with a single sector read.
Micro-Optimization: By copy files off, re-formatting, and copying the files back, the "free" directory entries are removed and the files end up in the first few directory entries in the sector. Reduces sectors read from the SD card, as well as processor time spent parsing the data.
As previously noted, zero-byte files don't have any clusters allocated to them.
Therefore, consider the allocated cluster layout, in cheesy ASCII, where "D" is used for directory entries (directory data) and "f" is used for file data.
Normal recursive copy of 3000 files:
DfffffDfffffDfffffDfffff...
After robocopy of 3000 zero-byte files:
DDDD.......................
After robocopy of 3000 zero-byte files, then robocopy of full file data:
DDDDffffffffffffffffffff...
In the first case, parsing that directory's data may require reading multiple sectors of the FAT. In the second / third, the first read of the FAT will include the clusters for all file system metadata.
This improvement in metadata layout can be further multiplied in any of the following situations:
1. Lack of any caching of file system metadata, such as during initial boot / A9LH
2. On-demand caching of file system metadata, as it only reads what is actually needed
3. Any read-ahead caching implemented at any layer (hardware, firmware, OS, app, ...)
5. Any other additional hardware/firmware optimizations for sequential reads of data
Micro-Optimization: By first creating zero-byte files, the file system metadata is located contiguously on the media. Contiguously allocated file system metadata reduces the amount of the FAT that is read and/or cached, and allows benefits from read-ahead caching. In some file systems, it can also allow multi-cluster read commands to the SD card (when detecting contiguous allocation from the FAT).
Similar to the above two micro-optimizations, this is simply putting the files needed during boot onto the media first. This ensures that the first sector of that directory has the information needed, and thus avoids additional reads from the FAT to determine second (or third, or...) clusters for the directory.
I only briefly referred to this in the post, because the information needed is not really something most people would have available. It's most useful when the reading of the data can occur in parallel with parsing previously-read data....
This might have some benefit for systems with many, many applications installed....
I suggest there is some benefit to having free space be contiguous.
When allocating space using FAT32, the file system must parse the FAT to find the first cluster that is marked as free. Especially with larger FATs, it can be costly to read the FAT from SD and/or parse the entire FAT as the SD card reaches capacity. The FAT32 file system even includes a hint to help optimize this by recording the last-allocated cluster as a good starting point for future searches. See FSI_Nxt_Free in FSInfo structure on page 20. Open source implementations of FAT32 also keep this hint in memory, to optimize finding the next free cluster.
Without this hint, allocation of a cluster on a half-full SD card would require reading and parsing half the FAT, for each cluster being allocated. With this hint, allocation becomes a single check that succeeds the first time, when the free space is contiguous. Even mostly-contiguous free space is beneficial here. In contrast, a heavily-fragmented SD card would negate the benefits of the hint.
Note: We likely agree that read speed, presuming a properly aligned partition, is not affected because of the cluster-based alignment and allocation unit.
Micro-Optimization: When the free space is contiguous, allocations become marginally faster, because the hint points directly to a free cluster.
Thanks for reading!
As you can see, I freely admit these are all micro-optimizations. Given the terrible performance of the 3DS SD card and the CPU, the reduced processor use, memory overhead, and IO through the SD card can still add up..
Yeah, it's the concept I am referring to, not the old-school style defrag utilities that copy the data around on the same piece of media.
There is some (admittedly small) benefit to having the file system metadata, as well as the data for each file, be contiguously allocated on the media. There is additional benefit
See a related post I wrote a few minutes ago. It's long, but I try to organize in SPOILER tags to help readability.
I think I see what you're getting at. However, robocopy isn't the solution.
FAT has no logical separation of the data region and metadata region, so this will help on spinning media. Not flash drives or random-access media. That said - one does want to avoid extra unneeded sector reads.
A full defragment is NOT a good idea. SD cards can store large amounts with abysmally short lifespans, and performing a full recopy will be severely detrimental, as well reording data clusters for contiguity. Actual file data is aligned to cluster sizes, so the performance of reads/writes should never be an issue on actual file data as long as it's properly aligned.
Aside from this, according to #Cakey at least, the 3DS does have an r/w cache in-kernel (but I would need to verify this.)
Now then, rather than spend time debating various points since I see what you're getting at: Partial defragmentation of just metadata is not infeasible, and won't shorten a card's lifespan. A full defragmentation should never, I repeat, never be performed due to lifespan concerns. Additionally, I'm not convinced that it would offer a huge measurable improvement. Performance will always decay from a fresh copy due to extended use (since FAT variants are poorly designed.)
Because I love reporting when things work, the US version of Monster Hunter Generations also works just fine with the Reboot patches on my old3DS. So you can all rest easy knowing this amazing CFW will continue to be great!
@chaoskagami
I've been trying to install a 4.5 on my New 3DS EmuNAND (I know 4.5 and N3DS are incompatibles but I wanted to try some stuff), i had some trouble to do it, but now I did it!
Well, when I boot it with Corbenik, when I try to turn off the console the screens turns off but the rest don't (I don't think it is a bug cause I'm doing something that it isn't natural, but I thought you maybe could help me find out why it is doing it)
I tried with Luma, it can turn off the 3DS (thought it reboot the console if I enter on the System Settings of any version)
As I said, I don't think it is a bug, but I'll send the logs in case you want to check it out
Sorry to make you waste your time with my shit (if you think I'm doing shit things)
And thanks in advance if you know why it is doing that
@chaoskagami
I've been trying to install a 4.5 on my New 3DS EmuNAND (I know 4.5 and N3DS are incompatibles but I wanted to try some stuff), i had some trouble to do it, but now I did it!
Well, when I boot it with Corbenik, when I try to turn off the console the screens turns off but the rest don't (I don't think it is a bug cause I'm doing something that it isn't natural, but I thought you maybe could help me find out why it is doing it)
I tried with Luma, it can turn off the 3DS (thought it reboot the console if I enter on the System Settings of any version)
As I said, I don't think it is a bug, but I'll send the logs in case you want to check it out
Sorry to make you waste your time with my shit (if you think I'm doing shit things)
And thanks in advance if you know why it is doing that
I have no clue. I have not even slightly tested 4.5, never mind on N3DS (like, what the heck?) But uh, neat I guess? You're in uncharted territory. I can't guarantee the patches are even correctly applied, tbh. Luma likely works correctly because it extracts the FIRM off NAND, and you seem at least to be booting with 11.0 NFIRM which is majorly mismatched (and I'm surprised it works)
@chaoskagami
I've been trying to install a 4.5 on my New 3DS EmuNAND (I know 4.5 and N3DS are incompatibles but I wanted to try some stuff), i had some trouble to do it, but now I did it!
Well, when I boot it with Corbenik, when I try to turn off the console the screens turns off but the rest don't (I don't think it is a bug cause I'm doing something that it isn't natural, but I thought you maybe could help me find out why it is doing it)
I tried with Luma, it can turn off the 3DS (thought it reboot the console if I enter on the System Settings of any version)
As I said, I don't think it is a bug, but I'll send the logs in case you want to check it out
Sorry to make you waste your time with my shit (if you think I'm doing shit things)
And thanks in advance if you know why it is doing that
I actually don't think it can work that low. I was only able to get my emuNAND down to 9.2, going to 9.0 ended up giving me a black screen. My results during the FIRM0/1 protection has shown that Corbenik has trouble working with anything below 10.2 for emuNAND and sysNANDS below 10.2 won't even boot.
I actually don't think it can work that low. I was only able to get my emuNAND down to 9.2, going to 9.0 ended up giving me a black screen. My results during the FIRM0/1 protection has shown that Corbenik has trouble working with anything below 10.2 for emuNAND and sysNANDS below 10.2 won't even boot.
I have no clue. I have not even slightly tested 4.5, never mind on N3DS (like, what the heck?) But uh, neat I guess? You're in uncharted territory. I can't guarantee the patches are even correctly applied, tbh. Luma likely works correctly because it extracts the FIRM off NAND, and you seem at least to be booting with 11.0 NFIRM which is majorly mismatched (and I'm surprised it works)
I actually don't think it can work that low. I was only able to get my emuNAND down to 9.2, going to 9.0 ended up giving me a black screen. My results during the FIRM0/1 protection has shown that Corbenik has trouble working with anything below 10.2 for emuNAND and sysNANDS below 10.2 won't even boot.
Maybe it is a corbenik problem dealing with old 00040130 - System Modules, since that I had to delete all cia of System Modules with the low titleID ended with 2 to make it work on corbenik (but i don't need the System Modules for I'm trying to do)
Edit: now that I used the FIRM off NAND with luma, I have trouble trying to turn off
I did test 10.2 and up with an emuNAND and can say they all work just fine.
I didn't however test them on sysNAND since it didn't relate to the tests I was doing at the time. Sadly I can't do much testing now since my 2DS was not only bricked (I bricked it on purpose for science,) but is now in a pile of pieces since I used the parts to fix my 3DS.
I did test 10.2 and up with an emuNAND and can say they all work just fine.
I didn't however test them on sysNAND since it didn't relate to the tests I was doing the time. Sadly I can't do much testing now since my 2DS was not only bricked (I bricked it on purpose for science,) but is now in a pile of pieces since I used the parts to fix my 3DS.
Well, at least corbenik didn't cause that brick, right?
Sorry that I've gone without a stable release for a while, by the way. It's been 23 days since last release, but the git repo is still moving. I should probably make one pretty soon here, but I'm not sure the new code is entirely tested enough yet.
Well, at least corbenik didn't cause that brick, right?
Sorry that I've gone without a stable release for a while, by the way. It's been 23 days since last release, but the git repo is still moving. I should probably make one pretty soon here, but I'm not sure the new code is entirely tested enough yet.
The brick was my own fault, I was dicking around with the ShadowNAND code to see what it would take to brick my system. I was intending to brick it, to see if I could brick it.
And I've been testing it the entire 23 days. The only "bug" I can think of, is when you turn "Dim background" on, it doesn't completely dim it at first, it only dims the background around the text, but leaving the screen fixes it. That's really not much of a big deal though.
I did test 10.2 and up with an emuNAND and can say they all work just fine.
I didn't however test them on sysNAND since it didn't relate to the tests I was doing at the time. Sadly I can't do much testing now since my 2DS was not only bricked (I bricked it on purpose for science,) but is now in a pile of pieces since I used the parts to fix my 3DS.
Probably it worked with 11.0 NATIVE_FIRM 'cause the SystemModules are compatible with 11.0 NATIVE_FIRM
I wonder if I do the entire downgrade and use the 4.5 NATIVE_FIRM will make it work like on Old 3DS...
The brick was my own fault, I was dicking around with the ShadowNAND code to see what it would take to brick my system. I was intending to brick it, to see if I could brick it.
And I've been testing it the entire 23 days. The only "bug" I can think of, is when you turn "Dim background" on, it doesn't completely dim it at first, it only dims the background around the text, but leaving the screen fixes it. That's really not much of a big deal though.
I wish there was an easy way to fix that, but it's a byproduct of my use of dirty renders for efficiency's sake. I don't redraw the entire screen normally, only individual characters because otherwise the flicker would be unbearable. So, known issue I guess.
Probably it worked with 11.0 NATIVE_FIRM 'cause the SystemModules are compatible with 11.0 NATIVE_FIRM
I wonder if I do the entire downgrade and use the 4.5 NATIVE_FIRM will make it work like on Old 3DS...
Probably it worked with 11.0 NATIVE_FIRM 'cause the SystemModules are compatible with 11.0 NATIVE_FIRM
I wonder if I do the entire downgrade and use the 4.5 NATIVE_FIRM will make it work like on Old 3DS...
I did it on purpose, I could have fixed it myself, but ended up needing the parts. I actually wrote a blog about my adventures of fixing my 3DS with 2DS parts.
Still, you can try, but remember, that's up to you to risk that.
I wish there was an easy way to fix that, but it's a byproduct of my use of dirty renders for efficiency's sake. I don't redraw the entire screen normally, only individual characters because otherwise the flicker would be unbearable. So, known issue I guess.
I don't see the issue in it. It barely bothered me and was only starling at first, but once i realized that I could use more themes with it, I was happier with it.
I'm doing this because I got bored with so much stability on the 3DS scene right now so I started trying things that no one (that I know) tried xD
And don't worry, I'm only using my RedNAND, so if I brick my RedNAND I always can recover it :3
I don't see the issue in it. It barely bothered me and was only starling at first, but once i realized that I could use more themes with it, I was happier with it.
I did it on purpose, I could have fixed it myself, but ended up needing the parts. I actually wrote a blog about my adventures of fixing my 3DS with 2DS parts.
Still, you can try, but remember, that's up to you to risk that.
Seemingly out of nowhere a PC port for Pokemon Platinum has surfaced online, bundled alongside the source code for those interested in building and developing it for...
With very little in the way of announcement, Valve has today increased the price of the Steam Deck but some fairly considerable margins. Both of the available models...
Earlier this year, Sony announced major price increases for the PS5, PS5 Pro, and PlayStation Portal. Now the company is raising prices again, this time for...
Nintendo's expected Summer showcase is here, offering up plenty of new announcements and exciting reveals. Let's see what they have in store in the latest Nintendo...
Continuing with the great news of Pokémon Platinum getting a native unofficial PC port just a few days ago, today, yet another classic title from the franchise has...
The latest in a growing number of native PC ports, Paper Mario ReCut got its first pre-release build earlier this week. Based on the N64 recompilation toolchain, the...
When you talk about 3DS emulation, most people would jump to Citra. As the defacto choice since its first release it's seen tremendous success, and even after its...
A whole hour of PlayStation content is on the way, thanks to the latest State of Play showcase. Headlining the stream will be Marvel's Wolverine, alongside a...
For the first time in 13 years, the Call of Duty series will again return to Nintendo's consoles. Set to launch on the 23rd of October, the latest release, Modern...
What once seemed like a far off dream, and after many, many community restarts throughout the years, the elusive Mother 1 / EarthBound Beginnings Remake, which is a...
With very little in the way of announcement, Valve has today increased the price of the Steam Deck but some fairly considerable margins. Both of the available models...
Nintendo's expected Summer showcase is here, offering up plenty of new announcements and exciting reveals. Let's see what they have in store in the latest Nintendo...
Seemingly out of nowhere a PC port for Pokemon Platinum has surfaced online, bundled alongside the source code for those interested in building and developing it for...
Earlier this year, Sony announced major price increases for the PS5, PS5 Pro, and PlayStation Portal. Now the company is raising prices again, this time for...
With rumours circulating about a Nintendo Direct in the coming days and weeks, fans are left speculating and hoping as to what might be included. At the centre of all...
The latest in a growing number of native PC ports, Paper Mario ReCut got its first pre-release build earlier this week. Based on the N64 recompilation toolchain, the...
After much speculation and rumour, the fabled Nintendo Direct is upon us. Set to go live tomorrow, the 9th of June, at 3pm in the UK, it'll feature 50 minutes of...
A whole hour of PlayStation content is on the way, thanks to the latest State of Play showcase. Headlining the stream will be Marvel's Wolverine, alongside a...
For the first time in 13 years, the Call of Duty series will again return to Nintendo's consoles. Set to launch on the 23rd of October, the latest release, Modern...
Following an investigation over misleading commercial practices, today Nintendo has been imposed a fine of 35 million euros related to the controller malfunctions...
