ROM Hack Codec LZSS DS released

[FAST6191]

Browsing around the internet I came across this and as we are often in need of capable LZSS decompression tools I thought I would link it up.

For those not familiar the GBA and DS are fairly powerful consoles and as such can afford to use compression fairly extensively and as such they make extensive use of it.
Probably the most popular compression type is LZSS and there are several variations upon it usually noted by a flag/hexadecimal magic stamp in the initial bytes of compression or just before.

type 10 is the most common and oldest- if I am not mistaken it is the method that the BIOS SWI calls can handle. Most LZSS compression tools you will find geared towards GBA and DS stuff use this method.
type 11 appeared some time back and was incompatible with type 10, crystaltile2 was among the first applications to support it.
type 40 according to the NFO at least appeared recently in the DS release of Golden Sun.

It supports both compression and decompression of these compression types.

There are other types out there (Sylvanian Families had a writeup of the custom compression somewhere) and you should note that even if it is BIOS compatible compression the SWI calls will not always be used (they are a bit slow on occasion).

I have not had time to put it through much but a new tool is always nice to have, if it saves having to load up and beat the options of crystaltile2 into shape that is even better- naturally it is fairly limited in scope so if you have say compressed files within an archive (I have seen several NARC files like this in the past and some of the big package formats used by some games also use it) but the header for the file is standard/uncompressed you might run into trouble- it is not hard to slice it up and feed it to the tool but if you were coming from some of the compression searching tools or are trying to use it on the GBA it might take a bit of getting used to.
According to the NFO however it should support standard ARM9.bin and overlay compression methods.

Basic screenshot

Readme (In French)

Codec LZSS DS


Auteur : Loki

Version du : 26 / 12 / 2010


Ce codec peut compresser et décompresser des fichiers avec les algorithmes LZSS de
type 0x10, 0x11 et 0x40. Il peut aussi décompresser les fichiers arm9.bin et overlay.bin.

Le type 0x40 n'est autre que la protection apparue pour la première fois le 28 octobre 2010
dans la version japonaise du jeu Golden Sun: Dark Dawn / Obscure Aurore.


Bonne traduction...

Filetrip download
http://filetrip.net/f23640-Codec-LZSS-DS-1-0.html

My source
http://nintendo-ds.logic-sunrise.com/news-...ichiers-ds.html
Author/project homepage
http://myth-project.legtux.org/

 

[Poryhack]

Codec LZSS DS

Author: Loki

Version: 26 / 12 / 2010


This codec can compress and decompress files using the algorithms of LZSS
type 0x10, 0x11 and 0x40. It can also decompress files and arm9.bin overlay.bin.

0x40 is the type other than the protection appeared for the first time October 28, 2010
in the Japanese version of the game Golden Sun: Dark Dawn / Aurora Dark.


Good translation ...

Thanks for posting this FAST, it's good to finally find something other than CrystalTile2 to do overlay decompression with. I wonder if it would be possible to get the source; I have a friend who I think would like to implement something like this on Linux.

 

[loco365]

Codec LZSS DS

Author: Loki

Version: 26 / 12 / 2010


This codec can compress and decompress files using the algorithms of LZSS
type 0x10, 0x11 and 0x40. It can also decompress files and arm9.bin overlay.bin.

0x40 is the type other than the protection appeared for the first time October 28, 2010
in the Japanese version of the game Golden Sun: Dark Dawn / Aurora Dark.


Good translation ...

Thanks for posting this FAST, it's good to finally find something other than CrystalTile2 to do overlay decompression with. I wonder if it would be possible to get the source; I have a friend who I think would like to implement something like this on Linux.

If I could get the source, I could make an attempt to translate this to English for you all.

 

[FAST6191]

There is not really any point in a translation- this is the main screen

, actually I might edit that into the first post.

Only thing to note is it seems to want files with the extension LZSS or LZ (or you type *.* in the name box) but that is nothing new to rom hacking (many times do I force things past extension checks).

As for source for porting purposes afraid I am not aware of any, still http://www.gbadev.org/tools.php?showinfo=56 has some source (mainly for type 10/GBA bios style), there is http://www.romhacking.net/utils/431/ , http://www.romhacking.net/utils/495 , http://www.romhacking.net/utils/511/ (this one has a measure of source and seems to be built around .net) and http://www.romhacking.net/docs/281/ (a nice document on the basics of LZ)

 

[Barubary]

If you just want to decompress, you can also use my tool, DSDecmp. All three LZ-formats are supported, as well as RLE and Huffman compression.
It is command-line (or drag&drop) only, but the source is freely available.

 

[FAST6191]

Well this is embarrassing, I had forgotten about your tool Barubary (despite actually using it on several occasions).

 

[Poryhack]

As for source for porting purposes afraid I am not aware of any, still http://www.gbadev.org/tools.php?showinfo=56 has some source (mainly for type 10/GBA bios style), there is http://www.romhacking.net/utils/431/ , http://www.romhacking.net/utils/495 , http://www.romhacking.net/utils/511/ (this one has a measure of source and seems to be built around .net) and http://www.romhacking.net/docs/281/ (a nice document on the basics of LZ)
I didn't figure there was any but from the looks of it it wouldn't be too hard to contact the author and ask. I might do just that.

QUOTE(Barubary @ Jan 14 2011, 04:52 PM) If you just want to decompress, you can also use my tool, DSDecmp. All three LZ-formats are supported, as well as RLE and Huffman compression.
It is command-line (or drag&drop) only, but the source is freely available.

I didn't realize that tool was yours, excellent work! I've used it many times and I was about to bring that up in my earlier post until I saw that this new tools supports compressed ARM9 and ARM9 overlays. If you could add support for that in your tool it would be amazing.

 

[Phoenix Goddess]

If I could hug you for this FAST, I would.
Thank you

Barubary: This is the first time I'm seeing your tool and I wish I had spotted it sooner T_T
Thank you for your work! It's amazing

 

[Barubary]

If you just want to decompress, you can also use my tool, DSDecmp. All three LZ-formats are supported, as well as RLE and Huffman compression.
It is command-line (or drag&drop) only, but the source is freely available.

I didn't realize that tool was yours, excellent work! I've used it many times and I was about to bring that up in my earlier post until I saw that this new tools supports compressed ARM9 and ARM9 overlays. If you could add support for that in your tool it would be amazing.

If anyone has a specification of the compression used in the ARM9 and overlay files, I'd gladly incorporate it.
I know the compression used is the plain LZ (0x10?) but in reverse (read and write from end to start). However since only a part of the files are compressed I'll need to know the compression header format. I could try to figure it out myself, but it'd be much faster if there was already some documentation available =)

 

[Naxito]

What about 0x30 in some files on Desktop Tower Defense?

 

[Poryhack]

If you just want to decompress, you can also use my tool, DSDecmp. All three LZ-formats are supported, as well as RLE and Huffman compression.
It is command-line (or drag&drop) only, but the source is freely available.

I didn't realize that tool was yours, excellent work! I've used it many times and I was about to bring that up in my earlier post until I saw that this new tools supports compressed ARM9 and ARM9 overlays. If you could add support for that in your tool it would be amazing.

If anyone has a specification of the compression used in the ARM9 and overlay files, I'd gladly incorporate it.
I know the compression used is the plain LZ (0x10?) but in reverse (read and write from end to start). However since only a part of the files are compressed I'll need to know the compression header format. I could try to figure it out myself, but it'd be much faster if there was already some documentation available =)

Turns out this tool was written in C#, so I decompiled it. You can download the source here. All the interesting stuff is happening in Form1.cs FYI.

 

[Barubary]

What about 0x30 in some files on Desktop Tower Defense?
0x30 can indicate that the file has RLE compression, which is not a variant of LZ. The 'Codec LZSS DS' does not support this, DSDecmp does.

QUOTE(Barubary @ Jan 14 2011, 04:52 PM) If you just want to decompress, you can also use my tool, DSDecmp. All three LZ-formats are supported, as well as RLE and Huffman compression.
It is command-line (or drag&drop) only, but the source is freely available.

I didn't realize that tool was yours, excellent work! I've used it many times and I was about to bring that up in my earlier post until I saw that this new tools supports compressed ARM9 and ARM9 overlays. If you could add support for that in your tool it would be amazing.

If anyone has a specification of the compression used in the ARM9 and overlay files, I'd gladly incorporate it.
I know the compression used is the plain LZ (0x10?) but in reverse (read and write from end to start). However since only a part of the files are compressed I'll need to know the compression header format. I could try to figure it out myself, but it'd be much faster if there was already some documentation available =)

Turns out this tool was written in C#, so I decompiled it. You can download the source here. All the interesting stuff is happening in Form1.cs FYI.

Your link is a 404, but I found a decompiler and did it myself. Since the format was so similar to 0x10 LZ, I've added support for it right away. =)

EDIT: whoops, I was too eager. One or more small bugs are still present, which is/are currently being fixed.

 

[FAST6191]

Nice work both of you.

I probably should kick this to another thread but
Been doing a lot of reading today on various aspects of rom hacking, things have moved on a lot in recent months not to mention all the Chinese hacking work/docs that have appeared. In many ways it seems I have been caught sleeping and frankly I could not be more exited about it.

Today's reading
http://www.romhacking.net/forum/index.php/topic,8407 (I have linked it many times but a lot will has happened in the last couple of months).
http://bbs.cntgb.com/read.php?tid=541 a bit but check the surrounding site.

Also new version of crystaltile2 up on filetrip http://filetrip.net/f23649-CrystalTile2-2010-09-06.html , it seems some vestiges of the "registered" version were not removed from the initial return to normalcy, this should correct that (frankly though despite being one of my go to tools I had not noticed).

Oh and if you all are still wanting compression stuff it is an old one but a good one
http://members.iinet.net.au/~freeaxs/gbaco...sion%20on%20GBA

Oh and for those mourning the loss of tahaxan's image format stuff http://llref.emutalk.net/docs/

Short version- I need to do a lot more work on my rom hacking docs to have them stand side by side with current stuff.

 

[Barubary]

I've fixed the bugs in my decompression routine (at least the ones that I could find). There's also a 'hack' for files that have a value out of range. It may be that this hack is actually the proper way to do it (the files I've checked that have that problem do seem to come out properly), but I can't be sure. When using the command line, there will be a message 'DISP is too large [...]' if this happens. 'Codec LZSS DS' just crashes when that problem arises.
Anyway, the project link is still the same.

 

[Poryhack]

Your link is a 404, but I found a decompiler and did it myself. Since the format was so similar to 0x10 LZ, I've added support for it right away. =)

EDIT: whoops, I was too eager. One or more small bugs are still present, which is/are currently being fixed.

Oh sorry about that, not exactly sure how it happened but it got deleted from my Dropbox folder. It's back now in case anyone is still interested. Very glad to hear that you implemented it so quickly Barubary.

 

[Naxito]

Any program from RLE? :S

 

[Barubary]

If you just want to decompress, you can also use my tool, DSDecmp. All three LZ-formats are supported, as well as RLE and Huffman compression.
It is command-line (or drag&drop) only, but the source is freely available.

QUOTE(Naxito @ Jan 15 2011, 02:37 PM) What about 0x30 in some files on Desktop Tower Defense?

0x30 can indicate that the file has RLE compression, which is not a variant of LZ. The 'Codec LZSS DS' does not support this, DSDecmp does.

 

[Naxito]

I'm trying to descompres with option RLE but only copy the file :/

Command: dscecmp -nr file43.ncgr "./descompressfolder"

 

[Barubary]

I'm trying to descompres with option RLE but only copy the file :/

Command: dscecmp -nr file43.ncgr "./descompressfolder"
You should not provide the -nr flag, as it will prevent RLE from being used (not RLE); by default all recognized formats are attempted to be decompressed.
The command you want to use is:
CODEdsdecmp file43.ncgr "./descompressfolder"

(although you don't need the "-s unless the path contains a space)

 

[Naxito]

Thx!. And to recompress? ^^''