Nintendo DS/GBA Compressors

Discussion in 'NDS - ROM Hacking and Translations' started by CUE, Nov 11, 2011.

Nov 11, 2011

Nintendo DS/GBA Compressors by CUE at 9:59 AM (3,472 Views / 5 Likes) 14 Comments

    • Newcomer

    CUE New Member

    Member Since:
    Oct 24, 2002
    Message Count:
    53
    Country:
    Spain
    Nintendo DS/GBA - Some of my compression tools (source code included).
    Maybe the best set of compression tools for DS/GBA.

    Support both encode and decode data.
    Code:
    + BIOS compatible compressions (DS and GBA):
    - RLE
    - LZSS (with my own optimized version, 'LZ-CUE')
    - Huffman (this first and only tool to encode properly the 8-bit mode)
    + non BIOS compatible compressions (only DS):
    - BLZ (used in ARM overlay files, RPG Maker DS)
    - LZX (also known as LZ11 and LZ40, used in some games)
    - LZE (used in Luminous Arc 2, Luminous Arc 3)
    Sorry, the 'readme' file is in spanish language (I'm too lazy to translate).
    Run each command-line tool without parameters to view their usage.

    If you use any of my stuff in your work, please include my name, 'CUE', in the credits.
    You may use these tools under the terms of the GNU General Public License (GPL).

    Enjoy ;)

    The links:
    http://www.megaupload.com/?d=PVBSZ4MA
    http://www.bwass.org/bucket/Nintendo_DS_Compressors-CUE.rar
    • Member

    andibad beginner hacker

    Member Since:
    Sep 14, 2009
    Message Count:
    593
    Location:
    in your heart
    Country:
    Indonesia
    link or name you tools? o.oa i didn't find that
    • Member

    shadowmanwkp Your roms are on another rom site

    Member Since:
    Apr 17, 2008
    Message Count:
    487
    Location:
    Vleuten, The Netherlands
    Country:
    Netherlands
    This might sound insanely stupid on my part, but where are the links?
    • Newcomer

    locolol New Member

    Member Since:
    Sep 13, 2009
    Message Count:
    41
    Location:
    Mexicali,BC
    Country:
    Mexico
    @_@ where did the links/download go?
    • Newcomer

    CUE New Member

    Member Since:
    Oct 24, 2002
    Message Count:
    53
    Country:
    Spain
    Links added (my fault, sorry)
    1 people like this.
    • Newcomer

    Barubary New Member

    Member Since:
    Feb 27, 2009
    Message Count:
    28
    Country:
    Netherlands
    While I admit you beat me to an implementation of the LZE format (which I coincidentally started working on yesterday...), you weren't the first with an implementation for the 8-bit Huffman compression, and certainly not the only one. ;]
    (Your handling of the LZ40 flag byte is much more graceful than mine though)

    I'm also wondering why you've chosen to create separate programs instead of one combined program. Now users need to know how to recognize the compression format and run the appropriate program (or just try them all) before they can decompress. (for compression it doesn't matter much, as then the user needs to know which format to compress back in anyway)


    PS: I'm also a bit icky about the use of the word 'best' (and it sounds a bit pretentious tbh), but that may well be because I'm a science student who happened to have developed a similar tool. In what way are you claiming to (maybe) be the 'best'? ;]
    • Newcomer

    CUE New Member

    Member Since:
    Oct 24, 2002
    Message Count:
    53
    Country:
    Spain
    Hi, Barubary

    I known your tool (nice work) but fail with the 8-bit mode. An example: a 256 bytes file, from 0x00 to 0xFF. Try compress and decompress with DSDecmp. The new decompressed file and the original file are not identical!!! :(
    The error is a simple overflow in the 6-bits offset, because some values are greater than 0x3F.
    You can check too all the Huffman files from the "Professor Layton" roms: "Curious Village" / "Pandora's Box".

    Yes, I known, a simple '-' is the solution :D

    Because I'm only interested in the encoding methods and their use does not interest me much. I released the tools so they can be used in other programs (like DSDecmp).

    "Maybe the best". Reasons: 3 of 4 methods used by Nintendo can now be used without problems, or improved:
    - encodes the standard LZSS with a new, better, optimized method (files are smaller)
    - encodes properly the 8-bit Huffman mode
    - encodes properly overlay files
    (RLE is the 4th compression)
    • Newcomer

    Barubary New Member

    Member Since:
    Feb 27, 2009
    Message Count:
    28
    Country:
    Netherlands
    I retract my statement; thanks for pointing out where I erred =)
    I only thought about an easy way to write the huffman tree at the time, but never took into account the limited size of the offset field.

    Do you happen to have sample data in which your tools outperform DSDecmp in terms of compression size using the LZ format(s), when using the optimal-compression options for both? I've done several tests with some game data, and DSDecmp was consistently at least as good as your tools. =)
    I admit that mine is not as fast, but I am certain that my compression routine is optimal if you disregard the flag bytes. (and I believe there is only little gain to be had if the flag bytes have to be taken into account as well)

    I will not contest the other two points; I've never been 100% certain about the overlay compression format anyway.
    • Newcomer

    CUE New Member

    Member Since:
    Oct 24, 2002
    Message Count:
    53
    Country:
    Spain
    DSDecmp isnt fast? Are you sure? The optimal method is veeeeeeeeery slow (4 checks per chunk) and only gain a few bytes (1-2% in 4000 files tested) , but I used the compression in old consoles, and the 'few bytes' is a world ;)
    • Newcomer

    Barubary New Member

    Member Since:
    Feb 27, 2009
    Message Count:
    28
    Country:
    Netherlands
    I am quite sure; my optimal method has at most [number of bytes left to compress] checks for each byte if you choose the optimal compression. ;]
    (the actual number depending on the possible length of the chunk starting at that byte)
    • Newcomer

    SecretChao428 New Member

    Member Since:
    Mar 17, 2009
    Message Count:
    34
    Country:
    United States
    Well what still discourages me on this is that it is all Command Line, something I am not good at since most compression tools to day can not run in GUI(eg. Can not run in Win 32 Mode), but I wonder if this kind of tool can also be used for older consules that use Huffman, mainly for script compression...
    • Member

    Gericom New Member

    Member Since:
    Jun 30, 2011
    Message Count:
    261
    Country:
    Netherlands
    >>CUE
    Can you also make something to compress the arm9 files?
    They have some sections that aren't compressed.
    Maybe something that you have to insert the origional and the uncompressed, edited version. So you can see what is compressed and what not.
    • Member

    Blazer New Member

    Member Since:
    Aug 10, 2008
    Message Count:
    245
    Country:
    United States
    This may sound silly since I'm not a great hacker but are there any tips on how to recognize different compression formats? Like say I have a file and want to know what compression is used on it. I stick it into a hex editor and look into the header--what do I look for? Just anything really basic to lead me in the right direction, if possible. I know the following, so no need to reiterate these:

    - LZ10 - 0x10
    - LZ11 - 0x11
    - RLE? - 0x40 (used in Fire Emblem: Shadow Dragon, I believe, it just compressed strings of '00's and nothing else really, IIRC)

    I used to know some others (I think Huffman and LZSS) but I can't remember them and it's not easy finding this information so if you already know it, that'd be great and quite useful to hackers like me.

    If you can't really help me here (or just don't want to) that's fine and I appreciate your time anyway--this is a good program and I'll be sure to keep it in mind if I ever get stuck on decompressing or compressing files...
    • Reporter

    FAST6191 Techromancer

    Member Since:
    Nov 21, 2005
    Message Count:
    17,180
    Country:
    United Kingdom
    First thanks for these cue- more compression tools are always welcome.

    10,11 and 40 are types of LZ. Alongside this files often have an extension or name that indicates it (often even normal extensions .lz). This is somewhat unique to the DS though (certainly do not go looking for it on the GBA or something) and far from a sure thing (sometimes files start with 10 or one of the others (it being 16 bytes in a pointer after all) or simply lack it). Type 10 is what the GBA and DS BIOS have onboard.
    On top of this it is often quite evident in the hex itself- the simple example is text starts fairly readable (with the odd "random" character in parts) but then deteriorates. Several of us (including me) have worked through the basics of custom LZ for various people before (edit, found one of mine- http://gbatemp.net/topic/298610-file-compression-question/page__view__findpost__p__3736892 ) and there is plenty of source code out there (Barubary has some, gbadev has some and there are others out there). but http://www.romhackin.../documents/281/ is the reference document for most of us and it is occasionally worth a look at the GBA BIOS sections of GBAtek http://nocash.emubas...essionfunctions and you should remember if the BIOS is used (or if you want to go in depth most roms) will have flags sent up (certain reads, DMI, BIOS calls) when decompression happens.
    Generally the file is taken, maybe a header is added, split up into sections with flags at each end and then if a section matches it references back to the previous version. What these flags hold and what the limits are varies with each implementation.
    Type 10 appeared from the start, type 11 appeared later (and still causes trouble with some GBA era tools) and type 40 hit with golden sun and a few games since.
    Many of the bigger hacking programs scan for this as well (crystaltile2 certainly does) and some GBA stuff does in depth scanning (crystaltile2 has the ability)
    The other main thing of note is ARM binary compression for the DS- implementations are out there but generally it starts from the end of the file and works backwards (as opposed to the other way).

    30 was either huffman or RLE. Neither are that common on the DS in my experience although worth knowing about.

    The other was filtering and this is quite common in fonts. It is so simple it is available in tile editors as stuff like NDS 1bpp. I had a nice link I like to give out here but I appear not to have the exact one- http://www.coranac.c...nc/text/toc.htm has it somewhere though.

    As for info if it is not a standard one as mentioned it will usually be custom so a look for university level compression stuff (extensions .edu or .ac.uk.....) are my usual ports of call. Some compression stuff is also available from the more general PC game hacking sites like
    http://wiki.multimed...ry:Game_Formats
    http://wiki.xentax.c..._Format_Central

    Any film time so I am out of here.
    • Member

    Blazer New Member

    Member Since:
    Aug 10, 2008
    Message Count:
    245
    Country:
    United States
    Do you really think this is an LZ compression?

    Especially given that a friend of mine made a program to (de)compress it and he never mentioned LZ to me--at least not that I remember. Said program (compression labeled "FEDS portrait compression FYI") and its source just in case someone wanted to see why these two things use the same header but don't seem to be, to my limited knowledge, the same compression.

    If someone feels I'm derailing the topic by discussing the compressions this program handles and its headers then my apologies--feel free to stop me--I don't know how strict this forum is when it comes to friendly discussions...

SPONSORED LINKS
 

Share This Page