Hacking Looking for advices or help to decode files for Yakuza PSP

Aegehn

Well-Known Member
OP
Newcomer
Joined
Mar 14, 2009
Messages
71
Trophies
0
XP
755
Country
France
Hello everyone,

I'm back asking for your help, for the last time I think (and hope). I need your input on this file format, magic word KSEQ, I've included some examples of different size attached to this post. They contains all the in-games dialogues, inner thoughts, etc. cf. my post with "Tatsuya" above (link).
Because of the way they are stored in the game, I'm pretty those files are self-sufficient, meaning they contains there own table of content. But I can't figure out how they work, and especially how the strings are reference in the file. As you can see on my screenshot above, by remplacing "龍也" by "Tatsuya", we can deduce that only the beginning of the string is referenced, and the game reads it until a null character. So the name "Tatsuya" appears complete, but because it's longer than 2 kanjis, it moved all the following strings, and the text in the box starts in the middle of Tatsuya, up to the next null char.

Here's what I guessed of the structure by comparing multiple files (this is 01_xxx.kseq) :
oCLr3GO.png


The legend :
red : magic word; orange: file size; green: id?; blue : null
purple : number of 12 bytes entries; pink : number of 8 bytes entries in pink; red : always 0x28000000; green : number of 8 bytes entries in orange
orange : offset of the 8 bytes entries in orange; red : always 0x6a610000;
pink 8 bytes entries
orange 8 bytes entries
purple 12 bytes entries
????? I don't know, but it contains strings at the end (in unicode);


Thanks for your help!
 

Attachments

  • main_seq01.zip
    15.9 KB · Views: 131
  • Like
Reactions: Hargrun

Romsstar

Operation Decoded
Member
Joined
Sep 14, 2008
Messages
1,100
Trophies
2
XP
1,489
Country
Germany
By putting Tatsuya you changed the location of the strings (which were all shifted) but you didn't change the pointers accordingly to it.
Read about Pointers and how they work, otherwise this won't get you anywhere....
Basically any document about Pointers on RDHN should do.
The Encoding of the dialogues is Unicode 16 LE, in 01_3512382442_00000B18.kseq for example it starts at 0x544. This is where the dialogue sequences start.
Now here are a couple of things that will need to be determined:
- Long/Short Pointers
- LE/BE
- Do they go by Start Offset+Size or by StartOffset+EndOffset

If you're new to this do yourself a favor and read the Documents section at RDHN. You'll have many more questions in the process and this site has the answers.
 

Attachments

  • yakuza.png
    yakuza.png
    19 KB · Views: 261
  • Like
Reactions: Hargrun

Aegehn

Well-Known Member
OP
Newcomer
Joined
Mar 14, 2009
Messages
71
Trophies
0
XP
755
Country
France
Thanks, but I know what pointers are, and where the text start (in your screen its at 0x510 I think, with the five first strings being the identifiers of who is speaking). The thing that there are no obvious pointer in the file for any of the strings, so I think the game starts "somewhere" then goes from string to string by offset, given by one of the tables. But I can't figure which one.
 

Romsstar

Operation Decoded
Member
Joined
Sep 14, 2008
Messages
1,100
Trophies
2
XP
1,489
Country
Germany
No 510-52C aren't names but the identifier, in this case amb_underground_LR.
The names start at 0x52C.

There are cases in which the game follows the script string by string without any pointers.
In this case the end tags would serve as the pointers.
After 龍也 (Tatsuya) we have 8 bytes of zeroes (00 00 00 00) that separate the name from the next:
沙紀 (Saki) and another 8 bytes from 泰山 (TaiShan, not sure without context).
Have you made sure that you still had 8 bytes of zeroes after you put Tatsuya? If you did and you still ended up with that result, there are definitely pointers there, or at least one pointer that tells the game where to start reading (but if that was the case you wouldn't have ended up with the cut-off sentence, which indicates pointer problems)
 

Attachments

  • identifier.png
    identifier.png
    10.5 KB · Views: 303

Aegehn

Well-Known Member
OP
Newcomer
Joined
Mar 14, 2009
Messages
71
Trophies
0
XP
755
Country
France
I'll check tonight, but I think I've left the 8 null bytes. But even if there's just one pointer (and then the game reads strings after strings), I can't find it in the file. Plus I think it needs a table of some sort to distinguish what's identifiers and what's dialogues, because each identifier is written only once at the start of the text area, but in-game they're used multiple times, switching back and forth between characters.
 

Aegehn

Well-Known Member
OP
Newcomer
Joined
Mar 14, 2009
Messages
71
Trophies
0
XP
755
Country
France
I've checked : letting the 8 null bytes does not prevent cut-off strings. Therefore there must be a table with pointer for each string, or offsets from a starting.
 

thexyz

Active Member
Newcomer
Joined
Jan 8, 2014
Messages
40
Trophies
0
Age
54
XP
180
Country
Serbia, Republic of
If you look at your "purple 12-bytes entries", you can find ones that start with 4C 00 00 00. At offset base + 0x8, where base is the start of the entry, they have a half. Multiply this half by 4, and that's the offset of your string relative to file start. I've only checked some dialogues from the beginning of the game, but I think other stuff should work in the same way.

Example (from 07_*)
iDEuZZ2.png


If you multiply 0xE5 by 4 you get 0x394 which is where the string "(チッ……
 天馬も雄介も、ヘタレどもが……)" lives.

PS
It'd be cool if you could provide more info when asking for help next time, I had to spend some time trying to connect these files to where they're displayed in the game, same for string.
 
  • Like
Reactions: Hargrun and Aegehn

Aegehn

Well-Known Member
OP
Newcomer
Joined
Mar 14, 2009
Messages
71
Trophies
0
XP
755
Country
France
That seems right! And the two bytes before accounts for the string the name of the speaker (here, 龍也, Tatsuya). I'm testing this right now. Thank you very much. This morning I realised that those 4C... lines increased with the number of strings, but I couldn't figure out how to link them.

And I'm sorry If I didn't provide enough informations. It's not simple to guess what's useful and what's not. Again, thank you very much.

EDIT : That works.
WIkRBXX.png

y35tejk.png

Thanks!
 
  • Like
Reactions: Hargrun and cearp

Nebelwurfer

New Member
Newbie
Joined
Oct 3, 2014
Messages
3
Trophies
0
Age
62
XP
51
Country
United States
Aegehn,

I am investigating translating a Japanese PSP wargame recently released by SystemSoft.

I encountered a problem similar to yours where I have an archive of .GIM files, each of which start with
a "MIG.00.1PSP" header. I can extract the .GIM images with TextER, convert them to .png files with
GIMConv, and edit the .png files with a paint program.

Unfortunately, when it comes time to reconvert the .png into GIM files with GIMconv, the new GIM file is
about 7 times bigger than the original .GIM file. It also begins the GIM file with the header "GIM1.00.PSP".
The command line that I used to reconvert was:
gimconv.exe modified_file.png -o modified_file.gim.

I am using GIMconv 1.2 that I got from Sony's Texture Modification workshop. Are you using a newer
version of GIMconv? Are you using different arguments to the GIMconv tool to get it to produce a
MIG.00.1PSP format GIM file? I don't have much in the way of documentation for the GIMconv tool.

Any help you could provide would be appreciated.
 

Aegehn

Well-Known Member
OP
Newcomer
Joined
Mar 14, 2009
Messages
71
Trophies
0
XP
755
Country
France
Hi Nebelwurfer,

To convert a png into a gim, I proceed in two stages :
First, I use PNGoo (thanks FShadow for pointing it to me), it's a tool shrinks pngs size without loosing too much detail. The size of the output file depend of the size of the palette, set in the display "colour settings".

Then, I use the following command : "GimConv.exe pic.png --format_style psp --format_endian little --pixel_order normal --image_format index4 -o pic.gim"
You can change index4 to index8 or more, until you find the right size for your files.

Hope it helps!
 

Nebelwurfer

New Member
Newbie
Joined
Oct 3, 2014
Messages
3
Trophies
0
Age
62
XP
51
Country
United States
Thanks,

Using those settings did produce a GIM file with the proper MIG.00.1PSP heading.

However the resulting file size is ballooning from the original 198K to 794K.
The converted .jpg file is only 127K.

I also got a warning about index color conversion not being supported.

I wonder if instead of storing each pixel, the original .GIM file is just storing the .PNG file
content, or a compressed version of the .PNG file content with a notation about what
the width and height of the graphic is.

I don't have any knowledge of PSP graphics formats.

Thanks again for your help!
 

Nebelwurfer

New Member
Newbie
Joined
Oct 3, 2014
Messages
3
Trophies
0
Age
62
XP
51
Country
United States
I played around some more. Using pngquant to reduce the color palette of the modified jpg to 256, and then specifying
--image_format index8 to gimconv produced a .GIM file that was 196K in size, rather than the original GIM from the
archive which was 198K in size. Also GIMconv didn't flag a index color conversion not being supported.

I will try padding the file to match the original length, repacking it in the archive with TextEx and then replacing
the modified Archive file in the ISO to see if this process works for one file. If so, I have a working method to proceed with.

Again, thanks for your help.
 

Aegehn

Well-Known Member
OP
Newcomer
Joined
Mar 14, 2009
Messages
71
Trophies
0
XP
755
Country
France
Hello all,

I'm back with another issue I can't solve by myself. It seems the game can't handle the fact that files are growing beyond a certain margin.

First, a quick description of how files are storred :
One big file, kurohyou.cpk contains 25 .cpk files, each related to specific part the game (cabaret, fights, menu, city, etc.).
One of those file, globals.cpk, decompress as an ELPK file containing lots of files (other ELPKs, textures, etc.), amongst them the big text file with more than 5000 entries. Translating it cause it to grow, but until now the game worked without issue. But as we progress, we must have crossed a threshold and now the game hangs at the start while loading files. It displays the loading wheel but hangs.

Anyway I've done a simple test to make sure this is not related to that file in particular, but due to the fact that any file growing cause the issue. I've add padding at the end of the original globals.bin. If it's only a few kos of zeros, game works. More, then game hangs. The issue happens in both ppsspp and jcpsp, and of course on a real PSP where it freezes the console.

I tried to look at the log of both emulators, but messages aren't really explicit as to what cause this issue, and I don't have the asm knowledge to understand what's happening at assembly level.

That's why I'm asking for your help, if you've faced something similar, or if you have the knowledge to debug asm, or any help is welcome.

On a side note, looking at original cpk files, I can see in the header "CPKMC.2.15.00;DDL.2.75.00". The tool I use, wether FileMajik or CPK Repacker, seem to be of lower version. I don't know if it's related, but if you have access to more recent tools, please let me know by PM.

Thanks!
 

Aegehn

Well-Known Member
OP
Newcomer
Joined
Mar 14, 2009
Messages
71
Trophies
0
XP
755
Country
France
So the more I look at it, the more it seems related to the fact that the .cpk files we are repacking are done using a much older version of the cpk format (namely 1.36.00). As the game use the CRIware filesystem API to load the cpk, it may expect a certain format/version. And I don't know how to correct this. If you've got any packer/unpacker for cpk 2.15, please tell me.

Thanks.
 

Abu_Senpai

Well-Known Member
Member
Joined
Jul 13, 2014
Messages
1,515
Trophies
0
XP
1,186
Country
Syria
Hwy this is a theory a friend has about your problem

It sounds like making the file bigger overwrites fixed position data.
Depending on the game it may not have a solution, say for example that your text file grows over the PSP RAM capacity.
The idea is that if you have two contiguous files A and B; if A grows two much, it will overwrite the start of B (B being fixed in memory).
B could be moved, but then you would need enough RAM to place it in a different spot. This happened in Type-0, but there was no RAM left ).
Normally game system data, such as native assemblies are stored in fixed positions, one would have to check if that data is being overwritten. The fact that the game hangs at boot time seems to corroborate this, but still a theory.


Also

if it was the version, shouldn't an updated pack freeze always, regarless of its size?
 

Riku

Well-Known Member
Member
Joined
May 3, 2009
Messages
288
Trophies
0
XP
1,490
Country
United States
So the more I look at it, the more it seems related to the fact that the .cpk files we are repacking are done using a much older version of the cpk format (namely 1.36.00). As the game use the CRIware filesystem API to load the cpk, it may expect a certain format/version. And I don't know how to correct this. If you've got any packer/unpacker for cpk 2.15, please tell me.

Thanks.
CPK version has nothing to do with it, trust me. Most likely memory allocation issue.
 

Aegehn

Well-Known Member
OP
Newcomer
Joined
Mar 14, 2009
Messages
71
Trophies
0
XP
755
Country
France
If that's the case, what is the usual way to solve this issue ? There's only so much we can do by trying to be concise in our translation.
 

Riku

Well-Known Member
Member
Joined
May 3, 2009
Messages
288
Trophies
0
XP
1,490
Country
United States
Pray... we can only pray for savior to come, my friend. It's not so common and can only be fixed through asm debugging.
 
  • Like
Reactions: cearp

Aegehn

Well-Known Member
OP
Newcomer
Joined
Mar 14, 2009
Messages
71
Trophies
0
XP
755
Country
France
Before resorting to praying I think I'll try make sure that memory allocation is really the issue here, and try thing to see if I can mitigate it.
 

Site & Scene News

Popular threads in this forum

General chit-chat
Help Users
  • No one is chatting at the moment.
    K3Nv2 @ K3Nv2: Sorry for accidentally bending over