arm9.bin compression issue

Discussion in 'NDS - ROM Hacking and Translations' started by Nagato, Aug 20, 2015.

  1. Nagato
    OP

    Nagato GBAtemp Advanced Fan

    Member
    533
    513
    Jul 15, 2011
    United States
    For the Nanashi no Game: Me project, I'm running into some issues with compression of the arm9.bin file that is causing the game to completely break on a real DS. Even with the original Japanese version, you can make it crash by applying some patches to the code and recompressing, or even just trying to use the regular uncompressed arm9.bin in place of the original arm9.bin. Is anyone here knowledgeable about how the compression works and can give it a try?

    Here's everything I've tried:
    1) Patching 0xba0 with decompressed ARM9 size - 0x18, 0xba4 with decompressed ARM9 size, 0xbb4 with the compressed ARM9 size
    2) Setting the compressed ARM9 size field to 0 as if it was a regular uncompressed arm9.bin
    3) Not patching any fields but leaving it uncompressed
    4) Letting CrystalTile2 handle it for me

    I've already verified that it's nothing I edited since I tested it with a clean ROM with nothing changed except replacing the arm9.bin with an uncompressed version or with a modified compressed version.

    I also noticed that the compressed version was slightly larger than the original compressed arm9.bin and thought that maybe it was a size issue, so I used -eo9 on blz which shaved a few kb off the size, but even that still broke.

    I'm truly out of ideas at this point. It works fine in most emulators it seems. The latest no$gba has some issues with the modified ROMs like a real DS, but it's not as crashy.

    Sometimes the game can load to a certain point before it'll break. Other times if the compression is completely messed up then it'll just error on start. On a real DS, it seems to consistently crash either when you open your TS menu from a save or on the first level when you open the door in the bathroom.
     
    Last edited by Nagato, Aug 20, 2015 - Reason: Forgot to add that I tried CrystalTile2 as well
  2. FAST6191

    FAST6191 Techromancer

    pip Reporter
    23,363
    9,165
    Nov 21, 2005
  3. Nagato
    OP

    Nagato GBAtemp Advanced Fan

    Member
    533
    513
    Jul 15, 2011
    United States
    I honestly didn't try messing with Crystaltile2 until the very end, but as far as I can tell it wasn't doing anything weird. I have scripted setup using ndstool to build my ROMs, so I know for sure that at least the ROMs that I was building had things compressed or decompressed. So I don't think the issue is with CrystalTile2.
     
  4. jjjewel

    jjjewel GBAtemp Maniac

    Member
    1,009
    293
    Dec 17, 2009
    United States
    Did you change a lot of things in arm9.bin for the English version?

    As far as I tried compressing the unaltered arm9 file with CT2 (decompressed arm9.bin and then compressed it back), the new compressed file is smaller.
    But my card reader for DS broke so I couldn't test whether the newly compressed file worked on the real DS or not.

    So far for games with arm9 compression, I always used CT2 and never ran into problem.
     
  5. Nagato
    OP

    Nagato GBAtemp Advanced Fan

    Member
    533
    513
    Jul 15, 2011
    United States
    I don't think it's not a problem with any changes I made in the arm9.bin. Right now my current testing is with an unmodified but uncompressed arm9.bin and the original Japanese ROM. The only difference between the original Japanese ROM and my modified one right now is the compression on the arm9.bin. All of the game data is exactly the same. The game just does not like its arm9.bin to be changed in size at all from what I can tell.

    I've honestly never encountered this problem before when I needed to modify compressed arm9.bin files before. I'm trying to debug the decompression part of the startup routine in no$gba v2.8b, but I think all of that happens at the bios level long before I can start debugging in no$gba, so I'm at a loss right now. :(
    Edit: Actually, I think I found the decompression code now. My breakpoints just weren't getting it. It reads the decompressed arm9.bin size at 0200089C and then calls a decompression function? at 2000970.

    Edit: It just occurred to me... did Nanashi no Game: Me have any AP? I don't know if AP could cause these errors though. Previously, no$gba was complaining about bad instructions and such when it would crash even though it worked fine in DeSmuME.
     
    Last edited by Nagato, Aug 20, 2015
  6. jjjewel

    jjjewel GBAtemp Maniac

    Member
    1,009
    293
    Dec 17, 2009
    United States
    Yeah, tried it with no$gba v2.8b and it really crashed with the newly compressed arm9.bin even though it was unmodified. I have no clue why either.

    Edit: There's one overlay file overlay9_0000.bin. Could it have anything to do with arm9?
     
  7. Lycanroc

    Lycanroc Midnight Wolf

    Member
    2,007
    2,394
    May 11, 2014
    Germany
    Ibbenb├╝ren
    I thought of that too when the game froze, but a quick google search told me that the first game didn't have any anti-piracy measures, so maybe this one doesn't either. Also, if it were anti-piracy, the clean rom would most likely have the same crashes, which it doesn't.
     
  8. Nagato
    OP

    Nagato GBAtemp Advanced Fan

    Member
    533
    513
    Jul 15, 2011
    United States
    I have no idea. I'm stepping through all of the boot code that handles the compression and something else, and it looks like it's loading in properly. The version with an uncompressed arm9.bin and the compressed size field cleared out should also work fine, but it's not... It's weird. :\

    Edit: Playing around some more. I can't tell what's doing it, but for some reason ITCM is getting overwritten and it's immediately causing errors.
     
    Last edited by Nagato, Aug 20, 2015
  9. jjjewel

    jjjewel GBAtemp Maniac

    Member
    1,009
    293
    Dec 17, 2009
    United States
    I don't know much about debugging but the game seemed to crash where the bytes in original compressed arm9 and the newly compressed arm9 started to differ. Mine breaks at 02003FFC.

    (Pic under spoiler)
    Warning: Spoilers inside!
     
    Last edited by jjjewel, Aug 20, 2015
  10. Nagato
    OP

    Nagato GBAtemp Advanced Fan

    Member
    533
    513
    Jul 15, 2011
    United States
    Wait, what? Why would that be different? Is the compression algorithm in CrystalTile2 broken or something?
     
  11. jjjewel

    jjjewel GBAtemp Maniac

    Member
    1,009
    293
    Dec 17, 2009
    United States
    Usually different programs give different results in compressed files, but they all should be the same after decompressed again. If you try other programs, I think they will give different compressed arm9 too. (Ex. Sometimes you have different algorithms that take more time but give you smaller file, etc.)

    But the problem is this game seems to expect an exact compression algorithm, so anything besides the original file doesn't work. (But I don't know ASM hacking or debugging so this is merely a guess.)

    Edit: Have you tried this program?
    http://www.romhacking.net/utilities/826/
    I'm not sure if it'll make any difference or not but it might worth trying.
     
  12. Nagato
    OP

    Nagato GBAtemp Advanced Fan

    Member
    533
    513
    Jul 15, 2011
    United States
    I think it might be something else. Everything up to 0x4000 is uncompressed, however in your modified game it wrote some random data on the very last uncompressed instruction.

    For what it's worth, my latest test was this: I used blz's -eo9 to make a compressed arm9.bin that's even smaller than the original, then I manually pasted it over the original arm9.bin (plus padded until it filled the space of the entire old arm9.bin) and tried to load that. The game seems to always crash at the same point no matter what if it's not using the original arm9.bin. I also double checked using a memory dump that the decompressed arm9.bin was the same as the original decompressed arm9.bin in memory and it was, so I think the data is getting decompressed properly. So now I know that it's not something that happened during recreating the ROM, since literally the entire ROM except the arm9.bin part is exactly the same.

    This is seriously confusing me. :(



    Edit: Alright, so I decided to test the original, unmodified ROM... and that crashed too. Maybe It's something else crashing wrong with my DS or the save file I'm using to test.
    Edit 2: Save file ruled out. Reformat of microsd coming up. :\
     
    Last edited by Nagato, Aug 21, 2015
  13. Nagato
    OP

    Nagato GBAtemp Advanced Fan

    Member
    533
    513
    Jul 15, 2011
    United States
    Alright, so I think I figured out the problem. I think at first it might've been a real issue since no$gba was displaying the same behavior that NikolaMiljevic was talking about, and I also was able to get that problem to show up on my DS. Now everything seems fine after working out more details about how the compression stuff works, and now I'm patching the compressed length field to 0 so that it thinks it's an uncompressed arm9.bin and it's working fine. It probably would work too if I used compression, but this way is easier and quicker to put in my build chain.

    So, the cause of *my* problems. At the end of my last post I edited in that for some reason now my original, untouched ROM wasn't even working even with a fresh save. So I reformatted the microsd and tried again, but it still crashed exactly the same. Now, this was with the same exact ROM that I played 4-5 years ago, so I know it works on my card usually. So then I considered that it might be my card or my DS dying, so I got out the blue card from my Gateway that I haven't had a reason to use yet, and it worked! So I tried the patched version (described in first paragraph), and no more crashes! I'm still waiting for NikolaMiljevic to try out the latest patch so I cannot say if it works for him too, but I can say one thing for sure:

    RIP CycloDS
    2008(ish)-2015
    We shared many great memories together and you never left my DS for years, and you were there for me during the Nanashi no Game translation project. Kind of fitting that you died on the 7th day since I seriously started working on the Nanashi no Game Me translation project. May you sleep well in your tin case. RIP.

    :(

    (This thread can probably be closed now.)
     
    Lycanroc likes this.
  14. Auryn

    Auryn GBAtemp Advanced Fan

    Member
    537
    348
    Jul 21, 2011
    Swaziland
    Hi Nagato,
    if you need to test a bunch of flashcards, you can send me the patch. I have 5-6different types laying around (starting from the old orignal R4 to Acekard to DStwo to the usually sold in Thailand last week :) )