Hacking [Pre-release, WIP] Yet another rxTools?

[d0k3]

Weird issue, have no clue what is the cause If you have any other 1+GB SD card to check it would be better to do so.

I have some toher sd-cards, maybe I will try it tomrrow then. That's really a weired issure. Sorry that i have such problems xD

As for strange SD errors, I'd say get used to it. To add something useful in here... Don't use Easeus Partition Manager (I know that tools is popular), my (and others!) recommendation is to use EmuNAND9 to format even if you don't intend to use an EmuNAND.

 

[duke_srg]

As for strange SD errors, I'd say get used to it. To add something useful in here... Don't use Easeus Partition Manager (I know that tools is popular), my (and others!) recommendation is to use EmuNAND9 to format even if you don't intend to use an EmuNAND.

The weird thing that EmuNand treated as not detected only in case its MBR not located in the first or the last sector, determined by the total size of the SysNAND NCSD partitions. And his MBR have definitely the same partition sizes as mine which works fine. The only possible thing is GW placed MBR deeper inside last partition empty space like for smaller sized NAND and other tools also detect it

@nedron92 There are another option to check, but requires some advanced tasks. If you're able to make a raw SD card dump or just walk through the sectors (with Paragon Partition Manager for example) and check where actual NCSD header is placed. Because EmuNAND dump always makes a plain (aka RedNAND) image with header placed first. It must be placed at physical sector 1953792 of your SD card for your NAND type, if it placed at 1931264 it is for 943Mb flash chips. Could be probably happend if you created your EmuNAND with first GW releases, which could (I actually don't know) have only one smaller offset for NCSD header relocation. It is anyway outside of the actual CTRNAND FAT16 partition, so data is not corrupted, but is inside of the CTR partition, wich is wrong. Other tools could have a predefined NAND size/offset so checking both options. If this is your case I'll just add an autofix to relocate wrong placed NCSD header, which should not interfere with other software but will be technicaly correct.

 

[Segtendo]

Holy crap. rxTools returns!

 

[d0k3]

The weird thing that EmuNand treated as not detected only in case its MBR not located in the first or the last sector, determined by the total size of the SysNAND NCSD partitions. And his MBR have definitely the same partition sizes as mine which works fine. The only possible thing is GW placed MBR deeper inside last partition empty space like for smaller sized NAND and other tools also detect it

@nedron92 There are another option to check, but requires some advanced tasks. If you're able to make a raw SD card dump or just walk through the sectors (with Paragon Partition Manager for example) and check where actual NCSD header is placed. Because EmuNAND dump always makes a plain (aka RedNAND) image with header placed first. It must be placed at physical sector 1953792 of your SD card for your NAND type, if it placed at 1931264 it is for 943Mb flash chips. Could be probably happend if you created your EmuNAND with first GW releases, which could (I actually don't know) have only one smaller offset for NCSD header relocation. It is anyway outside of the actual CTRNAND FAT16 partition, so data is not corrupted, but is inside of the CTR partition, wich is wrong. Other tools could have a predefined NAND size/offset so checking both options. If this is your case I'll just add an autofix to relocate wrong placed NCSD header, which should not interfere with other software but will be technicaly correct.

Okay, understood. The GW EmuNAND setup does not place the NCSD NAND header after the end of the used space, though, but rather at the end of the NAND chip size. Every other tool does it the same way. You can also take a look here. Calling it an MBR is a bit misleading, for two reasons... first, because EmuNAND9 does create an actual (SD card) MBR with a EmuNAND hidden partitition entry (GW does not), second because the NCSD NAND header actually does contain a (encrypted) MBR table for the TWL partitions.

Btw, this giant waste of space (think N3DS with 1.8GB NAND chip sizes) is one of the reasons why I pushed the RedNAND format.

 

[duke_srg]

Okay, understood. The GW EmuNAND setup does not place the NCSD NAND header after the end of the used space, though, but rather at the end of the NAND chip size. Every other tool does it the same way. You can also take a look here. Calling it an MBR is a bit misleading, for two reasons... first, because EmuNAND9 does create an actual (SD card) MBR with a EmuNAND hidden partitition entry (GW does not), second because the NCSD NAND header actually does contain a (encrypted) MBR table for the TWL partitions.

Btw, this giant waste of space (think N3DS with 1.8GB NAND chip sizes) is one of the reasons why I pushed the RedNAND format.

Doesn't the end of the NAND chip size equal to the offset+size of the last NCSD partition?
It's clear for me about MBR/NCSD header differences, sorry for the confusion
So EmuNAND9 also creates the second partiton record on SD card MBR for EmuNAND?

 

[nedron92]

The weird thing that EmuNand treated as not detected only in case its MBR not located in the first or the last sector, determined by the total size of the SysNAND NCSD partitions. And his MBR have definitely the same partition sizes as mine which works fine. The only possible thing is GW placed MBR deeper inside last partition empty space like for smaller sized NAND and other tools also detect it

@nedron92 There are another option to check, but requires some advanced tasks. If you're able to make a raw SD card dump or just walk through the sectors (with Paragon Partition Manager for example) and check where actual NCSD header is placed. Because EmuNAND dump always makes a plain (aka RedNAND) image with header placed first. It must be placed at physical sector 1953792 of your SD card for your NAND type, if it placed at 1931264 it is for 943Mb flash chips. Could be probably happend if you created your EmuNAND with first GW releases, which could (I actually don't know) have only one smaller offset for NCSD header relocation. It is anyway outside of the actual CTRNAND FAT16 partition, so data is not corrupted, but is inside of the CTR partition, wich is wrong. Other tools could have a predefined NAND size/offset so checking both options. If this is your case I'll just add an autofix to relocate wrong placed NCSD header, which should not interfere with other software but will be technicaly correct.

Advanced Tasks? xD That are basic-tasks for me as a computer-scientist ^^.
I have paragon and gparted, I will take a look later today. Or maybe you already have fixed it then

 

[duke_srg]

Advanced Tasks? xD That are basic-tasks for me as a computer-scientist ^^.
I have paragon and gparted, I will take a look later today. Or maybe you already have fixed it then

That's great, you could easily check actual NCSD placement in that case

upd: @nedron92 additional NAND size check was added, if you able to build rxTools you could check the last master commit.

 

[d0k3]

Doesn't the end of the NAND chip size equal to the offset+size of the last NCSD partition?
It's clear for me about MBR/NCSD header differences, sorry for the confusion
So EmuNAND9 also creates the second partiton record on SD card MBR for EmuNAND?

Nope, that's not the same. Take these N3DS with 1.8GB NAND chips - the actual sizes in the NCSD header, and with that the used space, is the same as on a N3DS with a 1.2GB NAND chip, meaning that roughly ~600MB is simply unused (that's on both the NAND chip and for GW type EmuNANDs).

And, yup, EmUNAND9 also creates a SD card MBR partition record (second one in that case) for the EmuNAND. I was trying to push CFW devs to use this, as it would open up certain possibilities (f.e. putting your EmuNAND at the end of the SD card, instead of the beginning, use multi EmuNAND without wasting tons of space, use any imager tool to backup / restore EmuNAND, etc...). As of now, I only know that mid-kid plans on supporting this in CakesFW.

 

[duke_srg]

Nope, that's not the same. Take these N3DS with 1.8GB NAND chips - the actual sizes in the NCSD header, and with that the used space, is the same as on a N3DS with a 1.2GB NAND chip, meaning that roughly ~600MB is simply unused (that's on both the NAND chip and for GW type EmuNANDs).

And, yup, EmUNAND9 also creates a SD card MBR partition record (second one in that case) for the EmuNAND. I was trying to push CFW devs to use this, as it would open up certain possibilities (f.e. putting your EmuNAND at the end of the SD card, instead of the beginning, use multi EmuNAND without wasting tons of space, use any imager tool to backup / restore EmuNAND, etc...). As of now, I only know that mid-kid plans on supporting this in CakesFW.

I also already did so and even 3rd and 4rth partition slot supported. Can't see in your code anything but start sector is used, do you write partition size and type?

 

[d0k3]

I also already did so and even 3rd and 4rth partition slot supported. Can't see in your code anything but start sector is used, do you write partition size and type?

Good to read you're the first one to support this . And, well, got me. I'm not using the MBR for EmuNAND detection, but rather the 'classic' method. I do write the MBR entry when formatting, though.

 

[duke_srg]

Good to read you're the first one to support this . And, well, got me. I'm not using the MBR for EmuNAND detection, but rather the 'classic' method. I do write the MBR entry when formatting, though.

0x1C is for hidden FAT32, which is actually not. AFAIK SD card specifications ignore all but 1st partition, so it is safe to use non-hidden partition. I choosed type 0xD3 as it looks like was not reserved (0x3D was already reserved ), so if you agree we could use the same.
Sorry, did not check your sources throughly - too much hardcode IMO.
Pity GW screwed NSCD header placement. Will check and fix in rxTools. I could autofix it with the end-of-partitions placement, but will be worthless because won't be compatible with GW and existing tools, while plain/RedNAND placed at the flash 1/4Mb boundry should be way faster.

 

[d0k3]

0x1C is for hidden FAT32, which is actually not. AFAIK SD card specifications ignore all but 1st partition, so it is safe to use non-hidden partition. I choosed type 0xD3 as it looks like was not reserved (0x3D was already reserved ), so if you agree we could use the same.
Sorry, did not check your sources throughly - too much hardcode IMO.
Pity GW screwed NSCD header placement. Will check and fix in rxTools. I could autofix it with the end-of-partitions placement, but will be worthless because won't be compatible with GW and existing tools, while plain/RedNAND placed at the flash 1/4Mb boundry should be way faster.

While I know this is not a hidden 0x1C FAT partition, I'd rather not change this now. 0x1C works (doesn't lead to trouble anywhere and is recognized by imager tools) and I don't know how far Aurora and mid-kid are with their support. Maybe at a later point? I'd suggest you don't check partition type in the MBR at all - any partition except the first one must be an EmuNAND, no way around.

Don't autofix - this will do nothing good (space is still wasted), and as you see, incompatibilies will be introduced. As for the placement of the RedNAND - I have to slightly disagree. The first sector that is ever (and rarely) written to (by the 3DS OS) is the TWLN FAT partition (at 0x00012E00). Other than that, the OS reads only NAND header and sector 0x96 (1024 byte combined) from these first sectors, which should not be critical for speed.

EDIT: ... and I have to correct myself. The TWLN partition is not written to for EmuNANDs. The first sector that is ever written to in an EmuNAND is the CTRNAND partition start (at 0x0B930000). Also, sector0x96 is never read for EmuNANDs. For NAND headers, I am unsure.

 

[duke_srg]

While I know this is not a hidden 0x1C FAT partition, I'd rather not change this now. 0x1C works (doesn't lead to trouble anywhere and is recognized by imager tools) and I don't know how far Aurora and mid-kid are with their support. Maybe at a later point? I'd suggest you don't check partition type in the MBR at all - any partition except the first one must be an EmuNAND, no way around.

Don't autofix - this will do nothing good (space is still wasted), and as you see, incompatibilies will be introduced. As for the placement of the RedNAND - I have to slightly disagree. The first sector that is ever (and rarely) written to (by the 3DS OS) is the TWLN FAT partition (at 0x00012E00). Other than that, the OS reads only NAND header and sector 0x96 (1024 byte combined) from these first sectors, which should not be critical for speed.

We could use a dedicated partition type to avoid additional format checks. Anyway it is better to get an unised one, I've checked in different sources, nothing is using 0xD3 for now. Any partition management tools should work with unregestered partition type as an unformetted region so no problem at all.
I won't autofix, it is actually useless. Just added a quick fix for first EmuNAND detection based on size of the NAND chip (while SD MBR partiton autocreate untouched and still need to be fixed for that case).

It's not about the SD reads, should not be a problem with any offset. It's about flash wearing and performance drops for writes non aligned with flash erase page size boundry. Safe way is to use partition first sector displacement like SD formatter does (can't say if it is 1MB or more now, Transcend autoformat uses 4MB). We could investigate further - EmuNAND partitions have 64Kb alignment, which is most probably eMMC module flash erase page size. Also need to check actual CTRNAND first cluster alignment, but I suspect it would be aligned at 64K.

 

[d0k3]

It's not about the SD reads, should not be a problem with any offset. It's about flash wearing and performance drops for writes non aligned with flash erase page size boundry. Safe way is to use partition first sector displacement like SD formatter does (can't say if it is 1MB or more now, Transcend autoformat uses 4MB). We could investigate further - EmuNAND partitions have 64Kb alignment, which is most probably eMMC module flash erase page size. Also need to check actual CTRNAND first cluster alignment, but I suspect it would be aligned at 64K.

Yup, I understand. And, EmuNAND9 is using 4MB for the alignment. Keep in mind that we have only little control over where the FAT table and the actual data are stored (in the CTRNAND partition). We can only assume it is the same for all O3DS and the same for all N3DS. The FAT header itself is @0x0B95CA00 for O3DS and @0x0B95AE00 for N3DS. Also, when considering all this: I know existing standards - if they are just wrong - are a pain (see: how GW handles things, how EmuNAND Tool handles things), but I can't single handedly enforce a new one. The SD card MBR is the first step to make better (aligned) EmuNAND formats possible, but unless all major CFWs (Luma3DS, CakesFW and ReiNAND, too) support using the MBR for finding the EmuNAND, I won't change anything in EmuNAND9's default format options and the placement of the EmuNAND.

 

[duke_srg]

but unless all major CFWs (Luma3DS, CakesFW and ReiNAND) support using the MBR for finding the EmuNAND, I won't change anything in EmuNAND9's default format options and the placement of the EmuNAND.

0x0B95CA00 or 0x0B95AE00 is a FAT start addresses, but more important is data area start address, I did not check it yet. But anyway with 4MB offset it will be not worse that in SysNAND
Well, having all-in-one is on of the cause I refactoring rxTools. Having full multi EmuNAND support is better, since first one could be left GW compatible, while other could be properly aligned.

 

[WhoAmI?]

We could use a dedicated partition type to avoid additional format checks. Anyway it is better to get an unised one, I've checked in different sources, nothing is using 0xD3 for now. Any partition management tools should work with unregestered partition type as an unformetted region so no problem at all.
I won't autofix, it is actually useless. Just added a quick fix for first EmuNAND detection based on size of the NAND chip (while SD MBR partiton autocreate untouched and still need to be fixed for that case).

It's not about the SD reads, should not be a problem with any offset. It's about flash wearing and performance drops for writes non aligned with flash erase page size boundry. Safe way is to use partition first sector displacement like SD formatter does (can't say if it is 1MB or more now, Transcend autoformat uses 4MB). We could investigate further - EmuNAND partitions have 64Kb alignment, which is most probably eMMC module flash erase page size. Also need to check actual CTRNAND first cluster alignment, but I suspect it would be aligned at 64K.

Yup, I understand. And, EmuNAND9 is using 4MB for the alignment. Keep in mind that we have only little control over where the FAT table and the actual data are stored (in the CTRNAND partition). We can only assume it is the same for all O3DS and the same for all N3DS. The FAT header itself is @0x0B95CA00 for O3DS and @0x0B95AE00 for N3DS. Also, when considering all this: I know existing standards - if they are just wrong - are a pain (see: how GW handles things, how EmuNAND Tool handles things), but I can't single handedly enforce a new one. The SD card MBR is the first step to make better (aligned) EmuNAND formats possible, but unless all major CFWs (Luma3DS, CakesFW and ReiNAND, too) support using the MBR for finding the EmuNAND, I won't change anything in EmuNAND9's default format options and the placement of the EmuNAND.

CLEVER!

 

[duke_srg]

Updated preview release. Changelog available
https://github.com/dukesrg/rxTools/releases/tag/2.6-bj-prealpha4

@nedron92 GW NAND detection should be fixed now

 

[nedron92]

faster then I could look with paragon, cause I've written on my editor ^^.
I will try it now

Edit: No chance
sorry to say that. Finally I have to look a time with paragon, dammit.

Edit 2: And on my curent SdCard, the NCSD Header is located at 1953792, plain text with header

 

[Lemon_]

So... does it work faster than OG RxTools?

 

[duke_srg]

faster then I could look with paragon, cause I've written on my editor ^^.
I will try it now

Edit: No chance
sorry to say that. Finally I have to look a time with paragon, dammit.
Edit 2: And on my curent SdCard, the NCSD Header is located at 1953792, plain text with header

I was finally able to reproduce it!