ROM Hack RELEASE NX-Trimmer. An update partition trimmer based in hacbuild

[JRoad]

NX-Trimmer v0.4
This and it's brother XCI_Builder are meant to highlight the excelent work done by LucaFraga with hacbuild which is actually fully functional with a small issue if you now how to use it.
The issue with hacbuild is that it can only build xci files with 4nca in the secure partition. This is probably because a parameter in the header of the xci which points to the number of files in the secure partition. Probably was thought as a fixed value since the normal partition only have more than 4 files in rare occasions. This was solved via a compromise (in these rare occasions, as described bellow, but requires a better approach by identifying the issue and editing hacbuild)
You're wellcome to modify, repack, make a gui.. etcera of this as long as you give credits to at least to LucaFraga, SciresM and CVFireDragon wich made the hardest work, especially Luca and Scires. Mine is just a workflow facilitator of something you can do manually with these programs.
More in https://github.com/julesontheroad/NX-Trimmer

Note: Replaced ztools according to the advice by @Bigjokker. It now features a hacbuild version rebuilt by him. (REV2). All included in rev2 file but keeping Bigjokker file linked as a way of showing gratitude for his help. See bellow posts for more information.

1. Description
This tool is meant to clean the update partition from xci files and to reduce the padding used between partitions.
This is a batch application which serves to automate the workflow between the following programs:
a.) hacbuild: Program meant to create xci files from nca files, made by LucaFraga.
https://github.com/LucaFraga/hacbuild/
b.) hactool: Program which function is give information, decrypt and extract a lot of different kind of files us by the NX System.
Hactool was made by SciresM
https://github.com/SciresM/hactool/
c.) nspBuild: Program meant to create nsp files from nca files.
https://github.com/CVFireDragon/nspBuild/
nspBuild was made by CVFireDragon
NX-Trimmer was also inspired by "A Simple XCI, NCA, NSP Extracting Batch file (Just Drag and Drop) with Titlekey decrypt"
by Bigjokker and published in gbatemp:
https://gbatemp.net/threads/a-simpl...t-drag-and-drop-with-titlekey-decrypt.513300/

2. Changelog
**CHANGED header_key.txt to header_key(optional).txt ***
If you want to use the header_key.txt fill the text file and erase the (optional) mark

header_key(optional).txt > fill the "xci_header_key" value > change name to header_key.txt to use it.

v0.3.0 - Lot of changes made to first version:
I - Added compatibility with games with more than 5 nca. This accomplished by using the following approach.
a) Games with html manual (5 nca). The manual was stripped from the game, this doesn't impede game execution. And game is fully functional.
If you try to access the manual in these kind of games this action won't get any result and you will be able to keep playing.
Games are fully funtional without manual nca and there are few games that includes it to begin with.
b) Games "rev" (revision) that includes updates or even dlcs (Sonic Mania). For this games was decided to strip the updates and dlc files
from the xci.
Additionally nsp files are created with the updates and dlcs files that were stripped from the xci.
For this you'll need to have Python installed in the computer.
II - Output route was moved to "output_nxt" so everything is better organiced and NX-Trimmer can be in the same folder as XCI_Builder,
which now uses el usará "output_xcib" as output folder.
III - Reformed ztools folder, erasing not needed files.
IV - Little mod to hacbuild.exe so it takes "xci_header_key" from ztools. "xci_header_key" it's actually not needed everything works without it
but hacbuild gives a warning wich was ugly for the batch output
V - Added template for keys.txt and header_key.txt in ztools
VI - Added tag system for the files output. It goes as follows:
[nxt] xci trimmed with n NX-Trimmer
[nm] "no manual", manual was erased to get the resulting xci to work
[c1][c2]... In output nsp: order of the content in the original xci. (updates, dlcs ...)
[dlc] In output nsp means the content is a dlc.
[upd] In output nsp means the content is an update.
The output dlc files are ticketless, should be encrypted with cartridge key and only functional with SX OS. To make them functional in other CFW
a false ticket should be needed as 4NXCI does but this defies this program purpose and my current goal.
The updates use common tickets signed by Nintendo, so they should be functional in every CFW.
VII - The "normal" partition content is now erased as it's thought in case the format of the "meta" or/and or "control" nca changes and no current cartridge
really uses it right now. It can serve a purpose in new cartridge type revisions.
Nowadays the nca files in normal are the same as in "secure" so this partition is not necessary and is empty in card 2 types.
VIII - Better use of hactool only extracting the secure partition.
NOTE: If upgrading from previous version replace ztools folder. hactool version was upgraded and an small modification was made in hacbuild code.
v0.3.1 - Added options in batch header. Edit with notepad++ to select them
I - "preservemanual" By default 0. Set at 1 if you want the manual nca to not be deleted. If set at 1 the manual will be sent to output folder.
NOTE1: Is advised to ¡¡NOT INSTALLED the manual .nca!! as the game will give error if you try to access the game manual.
NOTE2: The xci without manual is completely functional and if manual is selected it won't happened anything.
NOTE3: This option is thought for a future xci to nsp converter that will revert the process.
II - "delete_brack_tags" Erase [] tags like [trimmed]- By default at 1
III - "delete_pa_tags" Erase tags () like (USA) - By default at 0. If activated it could also erase [] tags
v0.2.x - Development versions
v0.1.0 - First launch in elotrolado.net

3. Requirements
This tool is thought to clean the update partition from xci files and to reduce the padding used between partitions.
This is a batch application which serves to automate the workflow between the following programs:
a.) hacbuild: Program meant to create xci files from nca files, made by LucaFraga.
https://github.com/LucaFraga/hacbuild
b.) hactool: Program which function is give information, decrypt and extract a lot of different kind of files us by the NX System.
Hactool was made by SciresM
https://github.com/SciresM/hactool
c.) nspBuild: Program meant to create nsp files from nca files.
nspBuild was made by CVFireDragon
https://github.com/CVFireDragon/nspBuild
NX-Trimmer was also inspired by "A Simple XCI, NCA, NSP Extracting Batch file (Just Drag and Drop) with Titlekey decrypt"
by Bigjokker and published in gbatemp:
https://gbatemp.net/threads/a-simpl...t-drag-and-drop-with-titlekey-decrypt.513300/

4. Functions
- Padding reduction, partition cleaning for "update" and "normal" removal of logo partition (not needed).
- All cartridges are built as "card1"
- Stripping of updates and dlcs from rev cartridges.
- Removal of manual nca from cartridges that incudes it (very few of them) without loosing cartridge funcion.
- "game_info" files extraction.

5. Limitations
- xci files only work with SX OS
- Currently hacbuild can only build files with 4nca files in the secure partition.
Now that XCI_Builder and NX-Trimmer are fully update this issue will be investigated. It seems to ne tied to some parameter in the cartrige header
which is undocumented and points to the number of files in the secure partition. Probably was assumed as a fixed value as most of the cartridges
only use program, control, meta and legal nca files which makes 4 nca files.
- It's necessary to remove the manual nca to get the games that incorporate it to work after the rebuild
- For rev games is necessary to extract updates and dlc files in nsp files.
- Currently the program doesn't identify the version for the updates
(Not sure if you can do it with hactool without extracting the nca files)
- Processing times for more than 4gb games is longer than it shoul be proporcionallu. Probably linked to hacbuild fix number 5 regarding the overflow
error in this kind of files. (Could be interesting to investigate if a fastest processing approach is possible)
- The symbol "!" gives error when passed to hacbuild. So rename snipperclips file

6. Use of the application
I.- First fill "keys.txt" in ztools folder so hactool can work properly.
More info: https://github.com/SciresM/hactool
II.- Optionally fill the file header_key.txt in ztools with xci_header_key value
III.- To trim an xci dragg it over "NX-Trimmer_v0.31" and wait till the cmd windows closes itself. You should see a thumps up
IV.- You'll get a folder with the name of the file in output_nxt. Inside it'll be the trimmed file.
V.- In case the cartridge incorporate the updates or dlcs they will be included in the same output folder
V.- Aditionally the game_info.ini file will be placed in the folder "game_info"
VI.- Load the xci in OS.
VII.- Optionally install the updates or dlcs generated by the program.

7. Compatibility
With current changes and accepting the described limitations this method should be compatible with all current xci files.
At least I didn't find any issues.

8. Thanks and credits to
LucaFraga, SciresM and CVFireDragon
Also thanks to all members from gbatemp and elotrolado.net

 

[JRoad]

XCI-Builder moved to the other thread since time limit finally passed

I want to say this:
All people are wellcome to redistribute, modify, add a gui or whatever they want as long as they give credits at least to LucaFraga (hacbuild), SciresM (hactool) and CVFiredragon (nspbuild). Especially LucaFraga who is the creator of hacbuild in which all of this is based.
Mine is a workflow facilitator between the three of them for a process that can be manually achieved. Only to make mine and other people live easier in regards of completing the process.
Also please try to respect the limitations and usage described in the readme. If you do this you're good with me.

Also let me express my gratitude to @Bigjokker as his batch application was to say the least a big inspiration for this one:
https://gbatemp.net/threads/a-simpl...t-drag-and-drop-with-titlekey-decrypt.513300/

All said above is true for NX-Trimmer and XCI-Builder

 

[Vantis]

Does the nsp to xci do batch conversions?

 

[JRoad]

Does the nsp to xci do batch conversions?

Not yet. Planning on that and custom output folder. For batch conversion you could do another bath with a for loop that gives this batch the complete route for the *.xci from a folder. But let me do a couple of changes in 1-2 days.
Also hacbuild would use double the space of the original title at certain point so for games like RE Revelations 2 you'll need a good amount of space in the disk. Planning to move second post to another thread BTW as soon as gbatemp rules allow me to.
Also planned is to make a gui with several options for both programs but i am torn between looking for hacbuild issue about not building xci files with more than 4 files in the secure partition or making these kind of improvements.

 

[Bigjokker]

You say header_key.txt is optional but without it all I get is "root.hfs0" in output folder.

 

[JRoad]

You say header_key.txt is optional but without it all I get is "root.hfs0" in output folder.

Without it I get a warning but it still completes the process. My case at least. It makes the root.hfs0 and then the .xci. LucaFraga said it was obligatory but in my case and other people's case I didn't really needed.
The issue I is that I put the header_key.txt in the ztools folder in the needed format. It probably takes key as 0 or something. If you erase the text file it takes a default value.
So in conclusion the problem was putting the text file in the ztools forder with the correct format but with the key blanc out.
You're right and right now if you don't fillout the key it doesn't complete the xci file.
If you don't want to use the key you need to erase the "header_key.txt" text file. Just tried myself.
I think I'll put it with an (optional) mark and say to release the mark if you want to use the key.

 

[Bigjokker]

Ok that fixed it.

If you want to get rid of that error message and still show what is going on you could add something like this.

echo HACbuild - 1.0.0.0
echo : : Building "%~dp0\nspDecrypted\!filename![xcib].xci"
"%~dp0\ztools\hacbuild.exe" xci_auto "%~dp0\nspDecrypted"  "%~dp0\nspDecrypted\!filename![xcib].xci" >NUL 2>&1

You won't need the header_key.txt anymore.

 

[JRoad]

Ok that fixed it.

If you want to get rid of that error message and still show what is going on you could add something like this.

echo Building "%~dp0\nspDecrypted\!filename![xcib].xci"
"%~dp0\ztools\hacbuild.exe" xci_auto "%~dp0\nspDecrypted"  "%~dp0\nspDecrypted\!filename![xcib].xci" >NUL 2>&1

You won't need the header_key.txt anymore.

Ok, thanks. I'll definitly take your advice

 

[Bigjokker]

Better yet, I removed the [ERR] and [WARN] an compiled it. So no header_key and no errors with this one.

 

[JRoad]

Better yet, I removed the [ERR] and [WARN] an compiled it. So no header_key and no errors with this one.

Yeah, I thought about that one but didn't know exactly why LucaFraga wanted that key. I guess we can omit it.

 

[Arkangelshadow]

Sry to ask, but is this better or have the same function as XCI cutter ?

Or i'm completely lost.

 

[JRoad]

Sry to ask, but is this better or have the same function as XCI cutter ?

Or i'm completely lost.

It's different. XCI-Cutter erases the padding at the end of the xci file (0s). Hacbuild by LucaFraga reduce also padding between partitions and lets you erase files in the partitions. Basically you don't need this:
- update: System updates. There are the files that update your firmware, for example if the games requires 5.1 the xci has the 5.1 firmware in it.
- normal: It's meant in the case control and meta nca format changes in future cartridge format or revisions, so if the ones in secure change you still have the ones on normal and low firmwares. Right now it's not used between current firmware/cartridges and really not needed. In card 2 format is empty.
- logo: For car 2 format. It has start animation and nintendo logo. These are also inside of another one of the nca from secure. So probably it's think with same purpose as old normal partitions. Not really needed.

So well basically you get gain from less padding + empty secure partition from XCI cutter which is relevant for some games like flashback:

Small games like flashback, 88 heroes or Megaman Legacy Collection really benefit from the update stripping.

Catch is right now hacbuild can't incorporate the updates and/or dlcs in the cartridge so they need to be output as dlc. This can be interesting for rev games like zelda or Donkey Kong Tropical Freeze because the update in cartridge is not the las one. Not so much for Sonic Mania that has a dlc + update.
Even so you can install the .nsp files the program gives you in these cases dlc files are ticketless and update files has same ticket as the cartridge a common ticket signed by Nintendo/developper with the update titlekey.

Following are some trimming examples.

 

[Erol]

regarding:

I.- First fill "keys.txt" in ztools folder so hactool can work properly.
More info: https://github.com/SciresM/hactool

where can I get the keys? there is no info on the github

 

[JRoad]

regarding:

where can I get the keys? there is no info on the github

You can google the name of the key. It should get you to some relevant information. try for example with "keyblob_mac_key_source" should get you to something. If you still have troubles I can try to help you.

 

[Erol]

so, when I drag multiple XCI files into the tool, it will only output a single xci... when I drag them all individually it works fine..

 

[JRoad]

so, when I drag multiple XCI files into the tool, it will only output a single xci... when I drag them all individually it works fine..

Yes, it doesn't do batch processing yet

 

[JRoad]

so, when I drag multiple XCI files into the tool, it will only output a single xci... when I drag them all individually it works fine..

I have a batch processing version working, it will include slightly modified versions of XCI_Builder/NX-Trimmer. It takes a folder with xci files and nsp files and process xci with the trimmer and nsp with the builder. I also does files individually.
I'm going to modified a little later in the day cause I want it to take several cases:
1- You only have trimmer in folder
2- You only have builder in folder
3- You have trimmer in ztools folder
4- You have builder in ztools folder

Basically it'll let you either have the 3 bat files in the root folder or only the batch processing one. Batch processing one will do everything automatically. No matter what you throw to it. Current version won't have subfolder processing, i'll look into it later.

Also if you have "update nsp" files in the folder keep in mind it convert those to xci files too. They actually work even if the main game is installed as nsp XD

 

[Vantis]

I have a batch processing version working, it will include slightly modified versions of XCI_Builder/NX-Trimmer. It takes a folder with xci files and nsp files and process xci with the trimmer and nsp with the builder. I also does files individually.
I'm going to modified a little later in the day cause I want it to take several cases:
1- You only have trimmer in folder
2- You only have builder in folder
3- You have trimmer in ztools folder
4- You have builder in ztools folder

Basically it'll let you either have the 3 bat files in the root folder or only the batch processing one. Batch processing one will do everything automatically. No matter what you throw to it. Current version won't have subfolder processing, i'll look into it later.

Also if you have "update nsp" files in the folder keep in mind it convert those to xci files too. They actually work even if the main game is installed as nsp XD

that's great would love to give it a try.

 

[JRoad]

that's great would love to give it a try.

It's more of less done but i want to clean the echos to give the programs a better presentation since they were kinda messy. I'll keep at it later at night, if it's not posted today in the temp and github it'll be tomorrow.

 

[Erol]

It's more of less done but i want to clean the echos to give the programs a better presentation since they were kinda messy. I'll keep at it later at night, if it's not posted today in the temp and github it'll be tomorrow.

do people really care for that? As long as it works, I dont care what the presentation is lol currently, we dont have a functioning batch processing solution