Guitar Hero for DS (Extracting NoteCharts)

Discussion in 'NDS - ROM Hacking and Translations' started by codemann8, Feb 27, 2011.

Feb 27, 2011
  1. codemann8
    OP

    Newcomer codemann8 Newbie

    Joined:
    Feb 27, 2011
    Messages:
    5
    Country:
    United States
    Hi,
    I am new here, someone I know referred me to this site to discuss various hacks with the DS.

    I have a website called PathHero. The site displays a good portion of the songs in the Guitar Hero and Rock Band series of games. Here you can visually see what notes will be coming up throughout any given song, think of it like a map, only we call it a notechart or chart. In addition, users of my site will submit what they think is the best way to achieve the optimal score (by means of showing where you should activate starpower/multiplier) in any given song/difficulty/instrument combo. Anywho, I'll stop advertising [​IMG]

    I have been able to extract chart information from GH:On Tour as the source files are in plaintext, QGM files. As you can see on the site, you can see those charts. But ever since GH:On Tour Decades, the files have been encrypted and I cannot make out any data. No progress really had been made on this until about a week ago. I had the bright idea of loading the ROM into an emulator, DemSuME to be exact (it seemed to have good dev tools and also had the Guitar Grip built-in). When I ran it, I selected Dirty Little Secret, the first song, and paused right before the notes started coming out. I did a memory dump and searched for "00 00 00 00 10 01", a common sequence of bytes inside QGM files (but isn't guaranteed, its been awhile but I think 00 00 10 01 means a plain green note). There were multiple instances, but I eliminated those where that sequence above ends on an even 4-byte word. Although its not guaranteed to eliminate it to 1 instance like I did, you should be able to see a repeating structure in increments of 8 bytes somewhere in the few instances.

    So I traced backward until I found the start, not hard since I recognize QGM files when I see them, each note in a song takes up 8 bytes, like so:
    Code:
    Plaintext QGM File Structure
    - - - - - - - - - - - - - - - - - -
    4-bytes - Start of note in milliseconds
    2-bytes - Duration of note
    2-bytes - Information about note (color/star/etc)
    ...and voila, I found the decompressed version!

    Then I discovered that in DS memory, there are parts that repeat in this format:
    Code:
    4-bytes - Length of data (x bytes)
    x-bytes - Data
    So, before the start of the data, it shows the length of the data. I also found that the length of the decompressed file matches what it says in the header of the compressed version, the header is as follows:
    Code:
    Compressed QGM File Header
    - - - - - - - - - - - - - - - - - - - - - 
    5-bytes - Magic Number "4C 08 00 00 45"
    2 bytes - Length of compressed file
    2 bytes - Always "00 10", possibly indicating the size in chunks in which it is compressed
    2 bytes - Length of decompressed file
    It does seem strange that the header seems to be only 11 bytes, but its possible the length of decompressed file is 3 bytes, but its in little endian so I cannot make that determination.

    So I tried this again for a different song. And it worked, I was able to find it, but its in a completely different memory location, which shouldn't surprise me, but it makes it difficult. So I've come to the conclusion that using this method to extract the charts cannot be done systematically and will take a long time (8 times per song, 2 instruments * 4 difficulties).

    My original intention was to debug and step through the process of it decompressing it to figure out what algorithm it is doing, but I think I don't have the tools to exactly do that. This is where you all come into play, I was hoping if someone could help me out with more DS-specific general experience, like with developing emulators.

    I would be willing to provide sample compressed & decompressed files, although if you have the ROMS for GHOT and GHOTD, you can see the source files for yourself. GHOT has a GOB file, which you have to ungob, I have a perl script for that if you need it, idk if GOBs are typical...and GHOTD doesn't use that, the files are just listed at the root. The only thing you won't have right out of the box is the decompressed files I extracted, but I provided the tutorial above to help.

    EDIT: Tutorial to extract contents of GHOT GOB file
     
  2. Forstride

    Member Forstride The rudder moves when I turn the wheel

    Joined:
    May 6, 2008
    Messages:
    2,763
    Location:
    Pittsburgh, PA
    Country:
    United States
    Oh wow, it's been forever since anyone brought up GHOT hacking...That's why I originally came to this site back when I did.

    I remember back when everyone stopped working towards custom songs, we stopped because we couldn't recompile the GOB files. It uses a special type of GOB compression that I'm sure only the devs have access to or whatever.
     
  3. codemann8
    OP

    Newcomer codemann8 Newbie

    Joined:
    Feb 27, 2011
    Messages:
    5
    Country:
    United States
    Ah yes, customs, well, if we can figure out the compression algorithm used for the QGM files, customs would be possible, there are no GOBs on GHOTD and GHOTMH
     
  4. habababa

    Newcomer habababa Advanced Member

    Joined:
    Nov 24, 2010
    Messages:
    60
    Country:
    Philippines
    Customs?
    You can make custom band hero songs by replacing the *.ogg, *.hwas and *.qgm files with smaller files from GHOT and GHOTMH.
    I did it once and it worked flawlessly.


    @codemann8
    is your gob unpacker different from the other command line gob unpacker?
    have you tried band hero ds?
    it's like the game uses two copies of fat.bin so you can't unpack and repack it with dslazy.
     
  5. ~Stenny

    Member ~Stenny GBAtemp Regular

    Joined:
    Sep 17, 2009
    Messages:
    116
    Country:
    Brazil
    really good as i'm getting my guitar grip today.

    so, i know that there's a DeSmuMe emulator for devs. why don't you try it out?
     
  6. codemann8
    OP

    Newcomer codemann8 Newbie

    Joined:
    Feb 27, 2011
    Messages:
    5
    Country:
    United States
    I used it to find the plaintext files in the memory dump....but I have no way to systematically do this for all the files in one swoop...

    But ideally, the goal is to see if I can find what compression method/algorithm is being used, but that's what I'm looking to get help with. I don't believe its open source code, so I cannot set up breakpoints to see exactly when the plaintext file is loaded into memory...I'm just hoping for some help with this.
     
  7. Barubary

    Newcomer Barubary Member

    Joined:
    Feb 27, 2009
    Messages:
    28
    Country:
    United States
    The compression scheme is quite simple, as it is just the built-in LZ-0x10 compression. The first 8 bytes are a custom header (5 bytes magic and 3 bytes file length), everything after that has the same format as any other LZ-0x10 compressed file on DS and GBA games.

    I've made a quick mod of my decompression tool DSDecmp that always skips the first 8 bytes in the given file. Get it here. If you're batch-processing, you may want to separate the files you want to decompress from the others, as it may crash halfway on a file that's almost like a compressed file if you skip the first 8 bytes. (those first 8 aren't checked for the proper QGM format).
     
  8. codemann8
    OP

    Newcomer codemann8 Newbie

    Joined:
    Feb 27, 2011
    Messages:
    5
    Country:
    United States
    You sir, are a gentleman and a scholar! This perfectly solves my problem.

    See I don't know much about typical compression methods other than their names. I know of LZ but if I saw a file, I wouldn't be able to tell. If I wanted to learn more about compression/encryption, do you know of any sites I should visit? I have a hard time googling things like "compression file structures" and getting what I want. I want to get to the point where I just look at a file and know what type of compression it is (that is by looking at headers of files and such). Know any good resources?

    Now if I understand this right, LZ has this header?:
    Code:
    1 byte - type of compression 0x10 0x11 or 0x40
    3 bytes - length of plaintext data
    Also...where can I find documentation about the LZ-0x10 algorithm? Again Google isn't getting me anywhere. If this moves to an effort to make customs, we will also need to reverse the process by encrypting homemade files.
     
  9. codemann8
    OP

    Newcomer codemann8 Newbie

    Joined:
    Feb 27, 2011
    Messages:
    5
    Country:
    United States
    EDIT: Sorry for the double post, I tried to delete...it seems I can't do that
     
  10. Barubary

    Newcomer Barubary Member

    Joined:
    Feb 27, 2009
    Messages:
    28
    Country:
    United States
    Yes, but only for the implementation used for the DS (and Wii). It's also not LZ-specific, as Huffman- and Run-Length-Encoding -compressed files also have this header (with type {0x24 or 0x28} and 0x30 respectively) (again also only for the implementation used for the DS).



    * Not entirely true, but close enough; any others I know are other implementations of the same technique.
     
  11. basher11

    Member basher11 GBAtemp's Official Vocaloid Lover

    Joined:
    Jun 29, 2009
    Messages:
    4,382
    Country:
    United States
    i tried changing band hero songs with smaller files from guitar hero modern hits and it crashed the game.
    unless you can tell me what you changed.
     
  12. habababa

    Newcomer habababa Advanced Member

    Joined:
    Nov 24, 2010
    Messages:
    60
    Country:
    Philippines
    Use a hex editor.Find the offset of the files you want to replace then copy paste.
    Then repoint the file list in arm9.bin if the files won't fit.
    You can't use CrystalTile2 and NitroExplorer to reinsert files since the game uses 2 copies of fat.bin.
    And there's another file index in the file fsindex-figs.bin for the EUR version and fsindex-us.bin for the US version.
    I'm not sure of the correct filenames though.
     

Share This Page