[Release] Various 3DS editing tools

Discussion in '3DS - ROM Hacking, Translations and Utilities' started by ObsidianB, Apr 25, 2016.

  1. ObsidianB
    OP

    ObsidianB Advanced Member

    Newcomer
    68
    171
    Sep 18, 2007
    United States
    Hey folks,

    A while back I started translating Rhythm Tengoku The Best+ (before Nintendo announced the US release -_-) and couldn't find tools that were cross-platform and worked with the specific (newer?) NintendoWare formats found in RTG, so I decided to write my own :)

    https://github.com/ObsidianX/3dstools

    These tools are all written in Python and should be cross-platform compatible, though I wrote them all in Linux and haven't tested them in Windows so patches/bug reports are welcome.

    bffnt.py

    Extract, edit, and re-create fonts in the bffnt format. It outputs a PNG of each texture sheet found in the source bffnt (usually just one) and a JSON manifest that describes each glyph.

    bflim.py

    Convert bflim images to and from PNG format for editing. Supports conversion of all texture formats but cannot create ETC1 bflim files because I didn't need it at the time and ETC1 compression is a bit more involved than just re-encoding pixels ;)

    msbt.py

    Converts the binary "message" text files to a JSON object for editing and re-packing. Has minimal support for the color format found in RTG and a couple other games I'd found that use colored text. Fair forewarning: the JSON format is horrible in order to maintain the original binary structure in the event that it matters to the game. I never tested whether I could modify the structure since this worked well enough.

    sarc.py

    Extracts and creates SARC archives with zlib compression support. In RTG all SARC files are zlib-compressed with the size of the original file prepended to the beginning of the file. This feature may not be needed outside of RTG but it's there :) The SARC archives were the main thing keeping the game from accepting my modified files until I had figured out the big thing that differentiates them from the other *ARC formats: sorted entires. The files are sorted within the archive based on the hash associated with each file entry. Once that part was figured out the game would boot and everything was spiffy.


    Dependencies

    bffnt and bflim depend on PyPng:

    bflim will utilize OpenCV, if available, for swizzle support (rotate/flip operations on the image)

    If you don't install OpenCV you can perform the de-swizzling yourself with your favorite image editor, but ensure that you re-swizzle images to the orientation they were in before repacking and set the right flag. In RTG at least, the swizzle still needed to be applied, perhaps because of the Layout remembering or something.

    Special Thanks/References

    Most of the RE work for these file formats were done by other wonderful homebrew developers with only a little extra work needed for the different format versions and missing pieces (The hardest being BFFNT). None of my work is based off of any proprietary NintendoWare code or tools.
     
    Last edited by ObsidianB, Apr 25, 2016
  2. Blundermann

    Blundermann GBAtemp Regular

    Member
    173
    124
    Oct 15, 2013
    United States
    This should give the community a headstart into moving away from leaked SDK tools. Good work!
     
    SirBeethoven and Bubsy Bobcat like this.
  3. lucaboy

    lucaboy Advanced Member

    Newcomer
    66
    67
    Jul 13, 2012
    Italy
    Good job! :D
     
  4. ObsidianB
    OP

    ObsidianB Advanced Member

    Newcomer
    68
    171
    Sep 18, 2007
    United States
    Leaked and out-of-date >_>
     
    Blundermann likes this.
  5. Thelostrune

    Thelostrune GBAtemp Advanced Fan

    Member
    545
    128
    Aug 13, 2015
    Norway
  6. Aigle67

    Aigle67 GBAtemp Regular

    Member
    256
    43
    Mar 31, 2015
    Belgium
    Does this works with .mes files ?
     
  7. ObsidianB
    OP

    ObsidianB Advanced Member

    Newcomer
    68
    171
    Sep 18, 2007
    United States
    Doubtful. If you had a .mes file handy I could see how different it is from .msbt (likely a different format altogether).
     
    Aigle67 likes this.
  8. Aigle67

    Aigle67 GBAtemp Regular

    Member
    256
    43
    Mar 31, 2015
    Belgium
     

    Attached Files:

  9. MasterLel

    MasterLel /x69/x73/x73/x6f/x75

    Member
    1,246
    450
    Mar 14, 2015
    France
    France
    Does your tools support .bcfnt files ?
     
  10. richardparker

    richardparker GBAtemp Advanced Fan

    Member
    707
    113
    Oct 10, 2014
    India
    Swallowed up by the ocean!
    umm may i pls ask whats the 'laziness' version(ntr)
     
  11. Helmax

    Helmax GBAtemp Regular

    Member
    188
    276
    Feb 17, 2011
    United States
    Is that the SaveLoad file from DQM Joker 3?

    — Posts automatically merged - Please don't double post! —

    Obsidian, I'm using your tools at the moment to unpack a 19 PNG long BFFNT file. It is a japanese game, and none of the glyphs or japanese characters from out in the PNG files.

    Using another contributors projects, his only converts BFFNT to BCFNT. When I do this, everything goes correctly.

    https://github.com/dnasdw/3dsfont

    Any ideas?
     
  12. Aigle67

    Aigle67 GBAtemp Regular

    Member
    256
    43
    Mar 31, 2015
    Belgium
    Yes it's the SaveLoad.mes :)
     
  13. 0xffcf

    0xffcf Member

    Newcomer
    24
    0
    May 20, 2016
    United States
    Does it support wii u bffnt files?
    I got
    Unknown version: 0x00000003 (expected 0x04000000)
     
  14. Helmax

    Helmax GBAtemp Regular

    Member
    188
    276
    Feb 17, 2011
    United States
    Digging into these tools more, I found it more useful to spit out 1 sheet instead of multiple "sheets", so I slightly refactored it to do so.
    But something is majorly off with this tool during rebuild.

    I think the Pypng library is injecting extra lines at the top, mostly black. But some with additional color pixels (not sure why).
    The original BFFNT doesnt have these hex codes at the start of the TGLP (or whatever its called) section on 0x200000.

    I was also having issues with the Green and Alpha values being swapped on the RGBA4 format (when exporting).
    After flipping it, i had a properly opaque (alpha 255) background and a white font with some light shading.

    I also see a couple times you should probably be using file_.tell() to reassign your current position instead of adding array lengths together.

    I have honestly given up for now with the limited time I had to investigate why this wasn't working on my BFFNT file.
    I posted it to the reddit post I started @ObsidianB and hopefully hear back from you soon.

    — Posts automatically merged - Please don't double post! —

    This python script has specific code to check the version.
    Looks like the WiiU file you are working with is Version 3.0 and this script is written for 4.0
    Perhaps if you posted the file, @ObsidianB or myself could see how different it is and maybe integrate a handler for Version 3 files.

    until then, no this tool does not support that file you are working with.
    But it likely could. It just needs some slight modifications.
     
  15. 0xffcf

    0xffcf Member

    Newcomer
    24
    0
    May 20, 2016
    United States
    here
    https://drive.google.com/open?id=0B_srL3ZAdHWtd0dIdWVrSlFkT00
    (It's windwaker hd font)
     
  16. Helmax

    Helmax GBAtemp Regular

    Member
    188
    276
    Feb 17, 2011
    United States
    I made some quick tweaks to the hex. Looks like this file is in big endian, and the version 4 file he was handling was written for little endian.
    The python script (when i read over it) has code to support both byte orders, but also has a bunch of hard coded references in it, that are preventing it from breaking down this verion 3 BFFNT file.

    I dont have time to look into it now, just putting the info out for anyone who does.
     
  17. The Koopa Kingdom

    The Koopa Kingdom Advanced Member

    Newcomer
    68
    66
    Oct 14, 2015
    United States
    Castle Grounds
    I'm having trouble running bflim.py, whenever I run it, I get:
    Code:
    Traceback (most recent call last):
      File "F:\Wii U Hacking\File Editing\BFLIM.py", line 5, in <module>
        import numpy
    ImportError: No module named numpy
    I've been trying over and over again, but I just can't get this to work.
    Trying to install numpy again:
    Code:
    C:\Users\Kyle>pip install numpy
    Requirement already satisfied (use --upgrade to upgrade): numpy in c:\users\kyle\appdata\local\programs\python\python35-32\lib\site-packages
    
    C:\Users\Kyle>pip3.5 install numpy
    Requirement already satisfied (use --upgrade to upgrade): numpy in c:\users\kyle\appdata\local\programs\python\python35-32\lib\site-packages
    
    C:\Users\Kyle>pip3.5 install --upgrade numpy
    Requirement already up-to-date: numpy in c:\users\kyle\appdata\local\programs\python\python35-32\lib\site-packages
    List of installed python modules:
    Code:
    C:\Users\Kyle>pip3.5 list
    numpy (1.11.0)
    pip (8.1.2)
    pypng (0.0.18)
    setuptools (18.2)
    wheel (0.29.0)
    
    C:\Users\Kyle>pip list
    numpy (1.11.0)
    pip (8.1.2)
    pypng (0.0.18)
    setuptools (18.2)
    wheel (0.29.0)
    
    C:\Users\Kyle>pip2 list
    'pip2' is not recognized as an internal or external command,
    operable program or batch file.
    The contents of c:\users\kyle\appdata\local\programs\python\python35-32\lib\site-packages:
    Code:
    C:\Users\Kyle>cd c:\users\kyle\appdata\local\programs\python\python35-32\lib\site-packages
    
    c:\Users\Kyle\AppData\Local\Programs\Python\Python35-32\Lib\site-packages>ls
    README.txt   easy_install.py         pip                  png.py                  setuptools-18.2.dist-info
    __pycache__  numpy                   pip-8.1.2.dist-info  pypng-0.0.18.dist-info  wheel
    _markerlib   numpy-1.11.0.dist-info  pkg_resources        setuptools              wheel-0.29.0.dist-info
    
    c:\Users\Kyle\AppData\Local\Programs\Python\Python35-32\Lib\site-packages>
    And finally, my environment variables:
    [​IMG]
    Does anyone know why pynum (And pypng, I also tried installing that to no avail) isn't being recognized?
     
  18. andibad

    andibad Soon™

    Member
    701
    90
    Sep 14, 2009
    Indonesia
    hospital
    hi obsidianX.

    I have problem with your sarc.py to extract some file on sos2 game (since is used sarc file). Not all sarc file is extracted correctly, like when i try to extract sarc containing bflim (with ETC-1 compression) and layout file. I just got corrupted bflim file, since i got 65563 byte (randomly increasing) on image data, but it should 65536 byte. But when i using EFE to extract file from sarc, is not have problem with it.

    i almost forgot, is seems sarc.py messing file on middle data ... but idk why.

    just only on bflim file with ETC-1 compression file (so far).

    btw thanks
     
    Last edited by andibad, Jun 28, 2016
  19. pikatsu

    pikatsu GBAtemp Advanced Fan

    Member
    744
    128
    Apr 16, 2014
    Argentina
    ObsidianB i want to unpack and repack a BFLIM file and i get a wrong header message. Here is a sample Bflim file. The file is from a 3ds game

    — Posts automatically merged - Please don't double post! —

    The message i get is
     

    Attached Files:

  20. andibad

    andibad Soon™

    Member
    701
    90
    Sep 14, 2009
    Indonesia
    hospital
    is seems bflim.py is not correctly read flim lenght attribute on flim header (since script think file lenght as flim header lenght).

    my tool able to convert that (to png), is used etc1a4 (0xB, 11), no rotate, image used blocks (no separate palette file needed),and bflim version is 117571840 :
    [​IMG]
    since image width is 0x80 and alignment 0x80, so no need adjust them.
     
    Last edited by andibad, Aug 14, 2016