ROM Hack Nintendo DS/GBA Compressors

  • Thread starter CUE
  • Start date
  • Views 31,409
  • Replies 21
  • Likes 8

CUE

Well-Known Member
OP
Newcomer
Joined
Oct 24, 2002
Messages
53
Trophies
0
Location
Pucela
Website
romxhack.esforos.com
XP
290
Country
Nintendo DS/GBA - Some of my compression tools (source code included).
Maybe the best set of compression tools for DS/GBA.

Support both encode and decode data.
Code:
+ BIOS compatible compressions (DS and GBA):
- RLE
- LZSS (with my own optimized version, 'LZ-CUE')
- Huffman (this first and only tool to encode properly the 8-bit mode)
+ non BIOS compatible compressions (only DS):
- BLZ (used in ARM overlay files, RPG Maker DS)
- LZX (also known as LZ11 and LZ40, used in some games)
- LZE (used in Luminous Arc 2, Luminous Arc 3)
Sorry, the 'readme' file is in spanish language (I'm too lazy to translate).
Run each command-line tool without parameters to view their usage.

If you use any of my stuff in your work, please include my name, 'CUE', in the credits.
You may use these tools under the terms of the GNU General Public License (GPL).

Enjoy ;)

The links:
http://www.megaupload.com/?d=PVBSZ4MA
http://www.bwass.org/bucket/Nintendo_DS_Compressors-CUE.rar
 

Barubary

Active Member
Newcomer
Joined
Feb 27, 2009
Messages
28
Trophies
0
Website
github.com
XP
143
Country
United States
Nintendo DS/GBA - Some of my compression tools (source code included).
Maybe the best set of compression tools for DS/GBA.

Support both encode and decode data.
Code:
+ BIOS compatible compressions (DS and GBA):
- RLE
- LZSS (with my own optimized version, 'LZ-CUE')
- Huffman (this first and only tool to encode properly the 8-bit mode)
+ non BIOS compatible compressions (only DS):
- BLZ (used in ARM overlay files, RPG Maker DS)
- LZX (also known as LZ11 and LZ40, used in some games)
- LZE (used in Luminous Arc 2, Luminous Arc 3)
Sorry, the 'readme' file is in spanish language (I'm too lazy to translate).
Run each command-line tool without parameters to view their usage.
While I admit you beat me to an implementation of the LZE format (which I coincidentally started working on yesterday...), you weren't the first with an implementation for the 8-bit Huffman compression, and certainly not the only one. ;]
(Your handling of the LZ40 flag byte is much more graceful than mine though)

I'm also wondering why you've chosen to create separate programs instead of one combined program. Now users need to know how to recognize the compression format and run the appropriate program (or just try them all) before they can decompress. (for compression it doesn't matter much, as then the user needs to know which format to compress back in anyway)


PS: I'm also a bit icky about the use of the word 'best' (and it sounds a bit pretentious tbh), but that may well be because I'm a science student who happened to have developed a similar tool. In what way are you claiming to (maybe) be the 'best'? ;]
 

CUE

Well-Known Member
OP
Newcomer
Joined
Oct 24, 2002
Messages
53
Trophies
0
Location
Pucela
Website
romxhack.esforos.com
XP
290
Country
Hi, Barubary

Barubary said:
... you weren't the first with an implementation for the 8-bit Huffman compression, and certainly not the only one. ;]
I known your tool (nice work) but fail with the 8-bit mode. An example: a 256 bytes file, from 0x00 to 0xFF. Try compress and decompress with DSDecmp. The new decompressed file and the original file are not identical!!! :(
The error is a simple overflow in the 6-bits offset, because some values are greater than 0x3F.
You can check too all the Huffman files from the "Professor Layton" roms: "Curious Village" / "Pandora's Box".

Barubary said:
(Your handling of the LZ40 flag byte is much more graceful than mine though)
Yes, I known, a simple '-' is the solution :D

Barubary said:
I'm also wondering why you've chosen to create separate programs instead of one combined program. Now users need to know how to recognize the compression format and run the appropriate program (or just try them all) before they can decompress. (for compression it doesn't matter much, as then the user needs to know which format to compress back in anyway)
Because I'm only interested in the encoding methods and their use does not interest me much. I released the tools so they can be used in other programs (like DSDecmp).

Barubary said:
PS: I'm also a bit icky about the use of the word 'best' (and it sounds a bit pretentious tbh), but that may well be because I'm a science student who happened to have developed a similar tool. In what way are you claiming to (maybe) be the 'best'? ;]
"Maybe the best". Reasons: 3 of 4 methods used by Nintendo can now be used without problems, or improved:
- encodes the standard LZSS with a new, better, optimized method (files are smaller)
- encodes properly the 8-bit Huffman mode
- encodes properly overlay files
(RLE is the 4th compression)
 

Barubary

Active Member
Newcomer
Joined
Feb 27, 2009
Messages
28
Trophies
0
Website
github.com
XP
143
Country
United States
Barubary said:
... you weren't the first with an implementation for the 8-bit Huffman compression, and certainly not the only one. ;]
I known your tool (nice work) but fail with the 8-bit mode. An example: a 256 bytes file, from 0x00 to 0xFF. Try compress and decompress with DSDecmp. The new decompressed file and the original file are not identical!!! :(
The error is a simple overflow in the 6-bits offset, because some values are greater than 0x3F.
You can check too all the Huffman files from the "Professor Layton" roms: "Curious Village" / "Pandora's Box".
I retract my statement; thanks for pointing out where I erred =)
I only thought about an easy way to write the huffman tree at the time, but never took into account the limited size of the offset field.

Barubary said:
PS: I'm also a bit icky about the use of the word 'best' (and it sounds a bit pretentious tbh), but that may well be because I'm a science student who happened to have developed a similar tool. In what way are you claiming to (maybe) be the 'best'? ;]
"Maybe the best". Reasons: 3 of 4 methods used by Nintendo can now be used without problems, or improved:
- encodes the standard LZSS with a new, better, optimized method (files are smaller)
- encodes properly the 8-bit Huffman mode
- encodes properly overlay files
(RLE is the 4th compression)
Do you happen to have sample data in which your tools outperform DSDecmp in terms of compression size using the LZ format(s), when using the optimal-compression options for both? I've done several tests with some game data, and DSDecmp was consistently at least as good as your tools. =)
I admit that mine is not as fast, but I am certain that my compression routine is optimal if you disregard the flag bytes. (and I believe there is only little gain to be had if the flag bytes have to be taken into account as well)

I will not contest the other two points; I've never been 100% certain about the overlay compression format anyway.
 

CUE

Well-Known Member
OP
Newcomer
Joined
Oct 24, 2002
Messages
53
Trophies
0
Location
Pucela
Website
romxhack.esforos.com
XP
290
Country
Barubary said:
...I admit that mine is not as fast, but I am certain that my compression routine is optimal if you disregard the flag bytes...
DSDecmp isnt fast? Are you sure? The optimal method is veeeeeeeeery slow (4 checks per chunk) and only gain a few bytes (1-2% in 4000 files tested) , but I used the compression in old consoles, and the 'few bytes' is a world ;)
 

Barubary

Active Member
Newcomer
Joined
Feb 27, 2009
Messages
28
Trophies
0
Website
github.com
XP
143
Country
United States
Barubary said:
...I admit that mine is not as fast, but I am certain that my compression routine is optimal if you disregard the flag bytes...
DSDecmp isnt fast? Are you sure? The optimal method is veeeeeeeeery slow (4 checks per chunk) and only gain a few bytes (1-2% in 4000 files tested) , but I used the compression in old consoles, and the 'few bytes' is a world ;)
I am quite sure; my optimal method has at most [number of bytes left to compress] checks for each byte if you choose the optimal compression. ;]
(the actual number depending on the possible length of the chunk starting at that byte)
 

SecretChao428

Active Member
Newcomer
Joined
Mar 17, 2009
Messages
34
Trophies
0
XP
123
Country
United States
Barubary said:
...I admit that mine is not as fast, but I am certain that my compression routine is optimal if you disregard the flag bytes...
DSDecmp isnt fast? Are you sure? The optimal method is veeeeeeeeery slow (4 checks per chunk) and only gain a few bytes (1-2% in 4000 files tested) , but I used the compression in old consoles, and the 'few bytes' is a world ;)
Well what still discourages me on this is that it is all Command Line, something I am not good at since most compression tools to day can not run in GUI(eg. Can not run in Win 32 Mode), but I wonder if this kind of tool can also be used for older consules that use Huffman, mainly for script compression...
 

Gericom

Well-Known Member
Member
Joined
Jun 30, 2011
Messages
1,379
Trophies
2
Age
25
XP
4,658
Country
Netherlands
>>CUE
Can you also make something to compress the arm9 files?
They have some sections that aren't compressed.
Maybe something that you have to insert the origional and the uncompressed, edited version. So you can see what is compressed and what not.
 

Blazer

Well-Known Member
Member
Joined
Aug 10, 2008
Messages
281
Trophies
0
XP
470
Country
United States
This may sound silly since I'm not a great hacker but are there any tips on how to recognize different compression formats? Like say I have a file and want to know what compression is used on it. I stick it into a hex editor and look into the header--what do I look for? Just anything really basic to lead me in the right direction, if possible. I know the following, so no need to reiterate these:

- LZ10 - 0x10
- LZ11 - 0x11
- RLE? - 0x40 (used in Fire Emblem: Shadow Dragon, I believe, it just compressed strings of '00's and nothing else really, IIRC)

I used to know some others (I think Huffman and LZSS) but I can't remember them and it's not easy finding this information so if you already know it, that'd be great and quite useful to hackers like me.

If you can't really help me here (or just don't want to) that's fine and I appreciate your time anyway--this is a good program and I'll be sure to keep it in mind if I ever get stuck on decompressing or compressing files...
 

FAST6191

Techromancer
Editorial Team
Joined
Nov 21, 2005
Messages
36,798
Trophies
3
XP
28,284
Country
United Kingdom
First thanks for these cue- more compression tools are always welcome.

10,11 and 40 are types of LZ. Alongside this files often have an extension or name that indicates it (often even normal extensions .lz). This is somewhat unique to the DS though (certainly do not go looking for it on the GBA or something) and far from a sure thing (sometimes files start with 10 or one of the others (it being 16 bytes in a pointer after all) or simply lack it). Type 10 is what the GBA and DS BIOS have onboard.
On top of this it is often quite evident in the hex itself- the simple example is text starts fairly readable (with the odd "random" character in parts) but then deteriorates. Several of us (including me) have worked through the basics of custom LZ for various people before (edit, found one of mine- http://gbatemp.net/topic/298610-file-compression-question/page__view__findpost__p__3736892 ) and there is plenty of source code out there (Barubary has some, gbadev has some and there are others out there). but http://www.romhackin.../documents/281/ is the reference document for most of us and it is occasionally worth a look at the GBA BIOS sections of GBAtek http://nocash.emubas...essionfunctions and you should remember if the BIOS is used (or if you want to go in depth most roms) will have flags sent up (certain reads, DMI, BIOS calls) when decompression happens.
Generally the file is taken, maybe a header is added, split up into sections with flags at each end and then if a section matches it references back to the previous version. What these flags hold and what the limits are varies with each implementation.
Type 10 appeared from the start, type 11 appeared later (and still causes trouble with some GBA era tools) and type 40 hit with golden sun and a few games since.
Many of the bigger hacking programs scan for this as well (crystaltile2 certainly does) and some GBA stuff does in depth scanning (crystaltile2 has the ability)
The other main thing of note is ARM binary compression for the DS- implementations are out there but generally it starts from the end of the file and works backwards (as opposed to the other way).

30 was either huffman or RLE. Neither are that common on the DS in my experience although worth knowing about.

The other was filtering and this is quite common in fonts. It is so simple it is available in tile editors as stuff like NDS 1bpp. I had a nice link I like to give out here but I appear not to have the exact one- http://www.coranac.c...nc/text/toc.htm has it somewhere though.

As for info if it is not a standard one as mentioned it will usually be custom so a look for university level compression stuff (extensions .edu or .ac.uk.....) are my usual ports of call. Some compression stuff is also available from the more general PC game hacking sites like
http://wiki.multimed...ry:Game_Formats
http://wiki.xentax.c..._Format_Central

Any film time so I am out of here.
 

Blazer

Well-Known Member
Member
Joined
Aug 10, 2008
Messages
281
Trophies
0
XP
470
Country
United States
Do you really think this is an LZ compression?

Especially given that a friend of mine made a program to (de)compress it and he never mentioned LZ to me--at least not that I remember. Said program (compression labeled "FEDS portrait compression FYI") and its source just in case someone wanted to see why these two things use the same header but don't seem to be, to my limited knowledge, the same compression.

If someone feels I'm derailing the topic by discussing the compressions this program handles and its headers then my apologies--feel free to stop me--I don't know how strict this forum is when it comes to friendly discussions...
 

SilverTheGamer

Active Member
Newcomer
Joined
Sep 27, 2016
Messages
34
Trophies
0
Age
32
XP
126
Country
Poland
Nintendo DS/GBA - Some of my compression tools (source code included).
Maybe the best set of compression tools for DS/GBA.

Support both encode and decode data.
Code:
+ BIOS compatible compressions (DS and GBA):
- RLE
- LZSS (with my own optimized version, 'LZ-CUE')
- Huffman (this first and only tool to encode properly the 8-bit mode)
+ non BIOS compatible compressions (only DS):
- BLZ (used in ARM overlay files, RPG Maker DS)
- LZX (also known as LZ11 and LZ40, used in some games)
- LZE (used in Luminous Arc 2, Luminous Arc 3)
Sorry, the 'readme' file is in spanish language (I'm too lazy to translate).
Run each command-line tool without parameters to view their usage.

If you use any of my stuff in your work, please include my name, 'CUE', in the credits.
You may use these tools under the terms of the GNU General Public License (GPL).

Enjoy ;)

The links:
http://www.megaupload.com/?d=PVBSZ4MA
http://www.bwass.org/bucket/Nintendo_DS_Compressors-CUE.rar
Um...... i copypaste the link and it doesn't lead me anywhere.........
 

1198901110

New Member
Newbie
Joined
Dec 19, 2018
Messages
1
Trophies
0
Age
36
XP
52
Country
China
PS C:\Windows\system32> C:\Users\11989\339435-Nintendo_DS_Compressors_v1.4-CUE\rle.exe -d C:\Users\11989\33943
5-Nintendo_DS_Compressors_v1.4-CUE\2.gba

RLE - (c) CUE 2011
RLE coding for Nintendo GBA/DS

- decoding 'C:\Users\11989\339435-Nintendo_DS_Compressors_v1.4-CUE\2.gba', WARNING: file is not RLE encoded!

Done

2.gba is zelda minish cop
How to use this program
less.exe and huffman.exe also have WARNING: file is not RLE encoded!
 

FAST6191

Techromancer
Editorial Team
Joined
Nov 21, 2005
Messages
36,798
Trophies
3
XP
28,284
Country
United Kingdom
GBA games have all the things within it in one file. These programs merely attempt to decompress the thing you tell them to using the known formats.

You will have to slice out the section you want decompressed if you want to use these, can't remember offhand if they have offset support. What you tried to do there was present it a whole file and it obviously had no clue how to handle it.

There are some tools that will attempt to scan GBA ROMs and decompress things they think is compression. I am not a big fan of those and find them somewhat unreliable.
There are also some other tools that will take logs from VBA (the BIOS contains decompression functions and it is easy to log those, which also tell you what type and where it is). When these work they work well but not all games will use the BIOS calls, and you also have to be at a point where the data you want is fetched (if you have to wait for the end boss to be loaded you need to find a save just before it or play to that point, though cheats can either speed things up or maybe force a boss fight).

I cover some of the basics of all this in https://gbatemp.net/threads/gbatemp-rom-hacking-documentation-project-new-2016-edition-out.73394/ if you want.
 

Site & Scene News

Popular threads in this forum

General chit-chat
Help Users
    S @ salazarcosplay: until they add pokemon mainline games