ROM Hack [Release] Various 3DS editing tools

[ObsidianB]

Hey folks,

A while back I started translating Rhythm Tengoku The Best+ (before Nintendo announced the US release -_-) and couldn't find tools that were cross-platform and worked with the specific (newer?) NintendoWare formats found in RTG, so I decided to write my own

https://github.com/ObsidianX/3dstools

These tools are all written in Python and should be cross-platform compatible, though I wrote them all in Linux and haven't tested them in Windows so patches/bug reports are welcome.

bffnt.py

Extract, edit, and re-create fonts in the bffnt format. It outputs a PNG of each texture sheet found in the source bffnt (usually just one) and a JSON manifest that describes each glyph.

bflim.py

Convert bflim images to and from PNG format for editing. Supports conversion of all texture formats but cannot create ETC1 bflim files because I didn't need it at the time and ETC1 compression is a bit more involved than just re-encoding pixels

msbt.py

Converts the binary "message" text files to a JSON object for editing and re-packing. Has minimal support for the color format found in RTG and a couple other games I'd found that use colored text. Fair forewarning: the JSON format is horrible in order to maintain the original binary structure in the event that it matters to the game. I never tested whether I could modify the structure since this worked well enough.

sarc.py

Extracts and creates SARC archives with zlib compression support. In RTG all SARC files are zlib-compressed with the size of the original file prepended to the beginning of the file. This feature may not be needed outside of RTG but it's there The SARC archives were the main thing keeping the game from accepting my modified files until I had figured out the big thing that differentiates them from the other *ARC formats: sorted entires. The files are sorted within the archive based on the hash associated with each file entry. Once that part was figured out the game would boot and everything was spiffy.


Dependencies

bffnt and bflim depend on PyPng:

• pypng - https://github.com/drj11/pypng

bflim will utilize OpenCV, if available, for swizzle support (rotate/flip operations on the image)

• OpenCV
  • Windows - http://www.lfd.uci.edu/~gohlke/pythonlibs/#opencv
  • Linux - use your favorite package manager
  • OS X - total pain in the ass with brew/macports, sorry

If you don't install OpenCV you can perform the de-swizzling yourself with your favorite image editor, but ensure that you re-swizzle images to the orientation they were in before repacking and set the right flag. In RTG at least, the swizzle still needed to be applied, perhaps because of the Layout remembering or something.

Special Thanks/References

Most of the RE work for these file formats were done by other wonderful homebrew developers with only a little extra work needed for the different format versions and missing pieces (The hardest being BFFNT). None of my work is based off of any proprietary NintendoWare code or tools.

• MSBT - http://mk8.tockdom.com/wiki/MSBT_(File_Format)
• BFLIM
  • http://mk8.tockdom.com/wiki/BFLIM_(File_Format)
• BFFNT - http://3dbrew.org/wiki/BCFNT (v4 data added by me)
  • v4 info gleaned from https://github.com/dnasdw/3dsfont
• SARC - http://mk8.tockdom.com/wiki/SARC_(File_Format)
• Texture format and ETC1 decompression sanity checking thanks to EveryFileExplorer
  • https://github.com/Gericom/EveryFileExplorer/blob/master/3DS/GPU/Textures.cs

 

[Blundermann]

This should give the community a headstart into moving away from leaked SDK tools. Good work!

 

[lucaboy]

Good job!

 

[ObsidianB]

This should give the community a headstart into moving away from leaked SDK tools. Good work!

Leaked and out-of-date >_>

 

[Thelostrune]

Perhaps this would be something that interests @ultramario1998

 

[Deleted User]

msbt.py

Converts the binary "message" text files to a JSON object for editing and re-packing. Has minimal support for the color format found in RTG and a couple other games I'd found that use colored text. Fair forewarning: the JSON format is horrible in order to maintain the original binary structure in the event that it matters to the game. I never tested whether I could modify the structure since this worked well enough.

Does this works with .mes files ?

 

[ObsidianB]

Does this works with .mes files ?

Doubtful. If you had a .mes file handy I could see how different it is from .msbt (likely a different format altogether).

 

[Deleted User]

Doubtful. If you had a .mes file handy I could see how different it is from .msbt

 

[MasterLel]

Does your tools support .bcfnt files ?

 

[richardparker]

Good job!

umm may i pls ask whats the 'laziness' version(ntr)

 

[Helmax]

Is that the SaveLoad file from DQM Joker 3?

--------------------- MERGED ---------------------------

Obsidian, I'm using your tools at the moment to unpack a 19 PNG long BFFNT file. It is a japanese game, and none of the glyphs or japanese characters from out in the PNG files.

Using another contributors projects, his only converts BFFNT to BCFNT. When I do this, everything goes correctly.

https://github.com/dnasdw/3dsfont

Any ideas?

 

[Deleted User]

Is that the SaveLoad file from DQM Joker 3?

--------------------- MERGED ---------------------------

Obsidian, I'm using your tools at the moment to unpack a 19 PNG long BFFNT file. It is a japanese game, and none of the glyphs or japanese characters from out in the PNG files.

Using another contributors projects, his only converts BFFNT to BCFNT. When I do this, everything goes correctly.

https://github.com/dnasdw/3dsfont

Any ideas?

Yes it's the SaveLoad.mes

 

[0xffcf]

Does it support wii u bffnt files?
I got
Unknown version: 0x00000003 (expected 0x04000000)

 

[Helmax]

Digging into these tools more, I found it more useful to spit out 1 sheet instead of multiple "sheets", so I slightly refactored it to do so.
But something is majorly off with this tool during rebuild.

I think the Pypng library is injecting extra lines at the top, mostly black. But some with additional color pixels (not sure why).
The original BFFNT doesnt have these hex codes at the start of the TGLP (or whatever its called) section on 0x200000.

I was also having issues with the Green and Alpha values being swapped on the RGBA4 format (when exporting).
After flipping it, i had a properly opaque (alpha 255) background and a white font with some light shading.

I also see a couple times you should probably be using file_.tell() to reassign your current position instead of adding array lengths together.

I have honestly given up for now with the limited time I had to investigate why this wasn't working on my BFFNT file.
I posted it to the reddit post I started @ObsidianB and hopefully hear back from you soon.

--------------------- MERGED ---------------------------

Does it support wii u bffnt files?
I got
Unknown version: 0x00000003 (expected 0x04000000)

This python script has specific code to check the version.
Looks like the WiiU file you are working with is Version 3.0 and this script is written for 4.0
Perhaps if you posted the file, @ObsidianB or myself could see how different it is and maybe integrate a handler for Version 3 files.

until then, no this tool does not support that file you are working with.
But it likely could. It just needs some slight modifications.

 

[0xffcf]

Digging into these tools more, I found it more useful to spit out 1 sheet instead of multiple "sheets", so I slightly refactored it to do so.
But something is majorly off with this tool during rebuild.

I think the Pypng library is injecting extra lines at the top, mostly black. But some with additional color pixels (not sure why).
The original BFFNT doesnt have these hex codes at the start of the TGLP (or whatever its called) section on 0x200000.

I was also having issues with the Green and Alpha values being swapped on the RGBA4 format (when exporting).
After flipping it, i had a properly opaque (alpha 255) background and a white font with some light shading.

I also see a couple times you should probably be using file_.tell() to reassign your current position instead of adding array lengths together.

I have honestly given up for now with the limited time I had to investigate why this wasn't working on my BFFNT file.
I posted it to the reddit post I started @ObsidianB and hopefully hear back from you soon.

--------------------- MERGED ---------------------------


This python script has specific code to check the version.
Looks like the WiiU file you are working with is Version 3.0 and this script is written for 4.0
Perhaps if you posted the file, @ObsidianB or myself could see how different it is and maybe integrate a handler for Version 3 files.

until then, no this tool does not support that file you are working with.
But it likely could. It just needs some slight modifications.

here
https://drive.google.com/open?id=0B_srL3ZAdHWtd0dIdWVrSlFkT00
(It's windwaker hd font)

 

[Helmax]

I made some quick tweaks to the hex. Looks like this file is in big endian, and the version 4 file he was handling was written for little endian.
The python script (when i read over it) has code to support both byte orders, but also has a bunch of hard coded references in it, that are preventing it from breaking down this verion 3 BFFNT file.

I dont have time to look into it now, just putting the info out for anyone who does.

 

[The Koopa Kingdom]

I'm having trouble running bflim.py, whenever I run it, I get:

Traceback (most recent call last):
  File "F:\Wii U Hacking\File Editing\BFLIM.py", line 5, in <module>
    import numpy
ImportError: No module named numpy

I've been trying over and over again, but I just can't get this to work.
Trying to install numpy again:

C:\Users\Kyle>pip install numpy
Requirement already satisfied (use --upgrade to upgrade): numpy in c:\users\kyle\appdata\local\programs\python\python35-32\lib\site-packages

C:\Users\Kyle>pip3.5 install numpy
Requirement already satisfied (use --upgrade to upgrade): numpy in c:\users\kyle\appdata\local\programs\python\python35-32\lib\site-packages

C:\Users\Kyle>pip3.5 install --upgrade numpy
Requirement already up-to-date: numpy in c:\users\kyle\appdata\local\programs\python\python35-32\lib\site-packages

List of installed python modules:

C:\Users\Kyle>pip3.5 list
numpy (1.11.0)
pip (8.1.2)
pypng (0.0.18)
setuptools (18.2)
wheel (0.29.0)

C:\Users\Kyle>pip list
numpy (1.11.0)
pip (8.1.2)
pypng (0.0.18)
setuptools (18.2)
wheel (0.29.0)

C:\Users\Kyle>pip2 list
'pip2' is not recognized as an internal or external command,
operable program or batch file.

The contents of c:\users\kyle\appdata\local\programs\python\python35-32\lib\site-packages:

C:\Users\Kyle>cd c:\users\kyle\appdata\local\programs\python\python35-32\lib\site-packages

c:\Users\Kyle\AppData\Local\Programs\Python\Python35-32\Lib\site-packages>ls
README.txt   easy_install.py         pip                  png.py                  setuptools-18.2.dist-info
__pycache__  numpy                   pip-8.1.2.dist-info  pypng-0.0.18.dist-info  wheel
_markerlib   numpy-1.11.0.dist-info  pkg_resources        setuptools              wheel-0.29.0.dist-info

c:\Users\Kyle\AppData\Local\Programs\Python\Python35-32\Lib\site-packages>

And finally, my environment variables:

Does anyone know why pynum (And pypng, I also tried installing that to no avail) isn't being recognized?

 

[andibad]

hi obsidianX.

I have problem with your sarc.py to extract some file on sos2 game (since is used sarc file). Not all sarc file is extracted correctly, like when i try to extract sarc containing bflim (with ETC-1 compression) and layout file. I just got corrupted bflim file, since i got 65563 byte (randomly increasing) on image data, but it should 65536 byte. But when i using EFE to extract file from sarc, is not have problem with it.

i almost forgot, is seems sarc.py messing file on middle data ... but idk why.

just only on bflim file with ETC-1 compression file (so far).

btw thanks

 

[pikatsu]

ObsidianB i want to unpack and repack a BFLIM file and i get a wrong header message. Here is a sample Bflim file. The file is from a 3ds game

--------------------- MERGED ---------------------------

The message i get is

 

[andibad]

ObsidianB i want to unpack and repack a BFLIM file and i get a wrong header message. Here is a sample Bflim file. The file is from a 3ds game

--------------------- MERGED ---------------------------

The message i get is

is seems bflim.py is not correctly read flim lenght attribute on flim header (since script think file lenght as flim header lenght).

my tool able to convert that (to png), is used etc1a4 (0xB, 11), no rotate, image used blocks (no separate palette file needed),and bflim version is 117571840 :

since image width is 0x80 and alignment 0x80, so no need adjust them.