- Status
i would love to help or learn to do it myself. if anyone can point me in the direction to material i would be glad to learn and help any way i can.
So I decided to take a short break this afternoon from working on LPR and look at the files for Phantom Kingdom Portable for the PSP. (Wanted a break from bug checking and proofreading LRP. I felt like some coding this afternoon.)
The good news is that I was able to extract most of the text from the PS2 English version of Makai Kingdom using the LPR tools that I created.
I am currently having issues extracting the data from DATA.dat. My files are not lining up right.
Also I can see that the START.KS4 may be compress and encrypted in a slightly different format than the LZS in LPR and also in the PS2 version of Makai Kingdom.
I think getting this compression/ encryption algorithm figured out will enable this translation to go forward. I post back if I have any luck.
The good news is that I was able to extract most of the text from the PS2 English version of Makai Kingdom using the LPR tools that I created.
I am currently having issues extracting the data from DATA.dat. My files are not lining up right.
Also I can see that the START.KS4 may be compress and encrypted in a slightly different format than the LZS in LPR and also in the PS2 version of Makai Kingdom.
I think getting this compression/ encryption algorithm figured out will enable this translation to go forward. I post back if I have any luck.
Alright... I got DATA.dat unpacked fine now. It was a stupid issues.
The format is:
16 byte header:
0x0-0x7: SPFS_V1
0x8-0xB: Number of entries in header table
0xC-0xF: Nothing
32 byte header per entry in header table:
0x00-0x13: Filename
0x14-0x18: decompressed file size (if needed) [little endian]
0x18-0x1B: file size [little endian]
0x1C-0x1F: pointer - location in DATA.dat (No offset...absolute position) [little endian]
So START.KS4 is compressed and cannot get any further until I figure out how the format works.
The format is:
16 byte header:
0x0-0x7: SPFS_V1
0x8-0xB: Number of entries in header table
0xC-0xF: Nothing
32 byte header per entry in header table:
0x00-0x13: Filename
0x14-0x18: decompressed file size (if needed) [little endian]
0x18-0x1B: file size [little endian]
0x1C-0x1F: pointer - location in DATA.dat (No offset...absolute position) [little endian]
So START.KS4 is compressed and cannot get any further until I figure out how the format works.
I am really stumped with the KS4 compression for START.KS4.
Here is a screenshot of the KS4 files and what I know the stream decodes to.
I know the code needs to decode to the second screenshot with 5 more 0x00 and ANM0000.DAT.
The marker 0x0B at 0x14 says it has 11 bytes uncompressed data which gets the header DSARC FL14 .
The next encoding symbols is 91 A3 8D this needs to be decoded as 5 0x00 and A. I cannot figure out how the dictionary is being population based upon the previous bytes in 0x15 - 0x0F. There has to be a 0x0041, 0x000041 or 0x00000041 in the dictionary and that substring is not in the previous string of bytes.
My current thinking is the dictionary is being populated by some combination of the previous chars which are: 0x44, 0x53, 0x41, 0x52, 0x43, 0x20, 0x46, 0x4C, 0x14, 0x00 and 0x00. I think each encoding symbol is a location in the dictionary since the next encoding symbol is just 0x91 at 0x28 and has to decode to 0x3030.
I am completely stumped what is going on. There is so many different ways you could populate the dictionary and I cannot think of one that works for more than one or two cases. Also there is a random 0x04 at location 0x08 in the header and may be key to how the dictionary is being populated.
Anyone have any ideas or seen this type of compression?
-ChepChep
Here is a screenshot of the KS4 files and what I know the stream decodes to.
I know the code needs to decode to the second screenshot with 5 more 0x00 and ANM0000.DAT.
The marker 0x0B at 0x14 says it has 11 bytes uncompressed data which gets the header DSARC FL14 .
The next encoding symbols is 91 A3 8D this needs to be decoded as 5 0x00 and A. I cannot figure out how the dictionary is being population based upon the previous bytes in 0x15 - 0x0F. There has to be a 0x0041, 0x000041 or 0x00000041 in the dictionary and that substring is not in the previous string of bytes.
My current thinking is the dictionary is being populated by some combination of the previous chars which are: 0x44, 0x53, 0x41, 0x52, 0x43, 0x20, 0x46, 0x4C, 0x14, 0x00 and 0x00. I think each encoding symbol is a location in the dictionary since the next encoding symbol is just 0x91 at 0x28 and has to decode to 0x3030.
I am completely stumped what is going on. There is so many different ways you could populate the dictionary and I cannot think of one that works for more than one or two cases. Also there is a random 0x04 at location 0x08 in the header and may be key to how the dictionary is being populated.
Anyone have any ideas or seen this type of compression?
-ChepChep
This is something I found from observing the input file and ram dump output. I'm not sure if it can apply to all files or not, though.
YKCMP format
0x00 chars(8) YKCMP_V1
0x08 uInt32 unknown (0x00000004)
0x0C uInt32 compressed file size
0x10 uInt32 decompressed file size
After 0x14 are compressed data.
From 0x14
Read 1 byte (I'll call it b1.)
If b1 < 0x80 Read and output b1 bytes.
If b1 >= 0x80 there will be 2 indicators which are number of bytes to move back (Y) and number of bytes to read (X).
If b1 >= 0x80 but < 0xC0 split it to XY.
If b1 >= 0xC0 but < 0xE0 read 1 more byte (I'll call it b2) and split it to XXYY
If b1 >= 0xE0 read 2 more bytes and split it to XXXYYY
Note: The values will be something like move back Y+1 bytes, and then for X, you have to subtract from its base first. (Ex. If XY then X-8, if XXYY, then XX-0xC0, if XXXYYY then XXX-0xE00)
Ex. From your compressed file example, starting from 0x14.
@0x14: b1 = 0B, so read 0B bytes and output.
@0x20: b1 = 91, so X = 9, Y = 1. Move back 1+1= 2 bytes. Read 9-8+1= 2 bytes and output. (You got 00 00.)
@0x21: b1 = A3, Move back 3+1= 4 bytes. Read A-8+1 = 3 bytes and output. (00 00 00)
@0x22: b1 = 8D, Move back D+1= 0xE bytes. Read 8-8+1= 1 byte and output. (41)
@0x23: b1 = 04, read 04 bytes and output. (4E 4D 30 30)
@0x28: b1 = 91. Move back 2 bytes, read 2 bytes. (30 30)
@0x29: b1 = 04, read 04 bytes and output. (2E 44 41 54)
@0x2E: b1 = C5, b2= 11. Move back 0x11+1= 0x12 bytes. Read C5-C0+2=7 byte and output. (00 00 00 00 00 00 00)
Ex. for 0xE0 base is b1 = FF, b2 = F3, b3 = FF. So you get FFF3FF. XXX = FFF, YYY = 3FF. Move back 3FF+1 bytes. Read FFF-E00=1FF+3 bytes.
(I think if it's 0xC0 base, you add 2 to number of byte read, and if it's 0XE00 base, you add 3 to number of byte read, but I'm not 100% sure. You can try to experiment with these numbers.)
YKCMP format
0x00 chars(8) YKCMP_V1
0x08 uInt32 unknown (0x00000004)
0x0C uInt32 compressed file size
0x10 uInt32 decompressed file size
After 0x14 are compressed data.
From 0x14
Read 1 byte (I'll call it b1.)
If b1 < 0x80 Read and output b1 bytes.
If b1 >= 0x80 there will be 2 indicators which are number of bytes to move back (Y) and number of bytes to read (X).
If b1 >= 0x80 but < 0xC0 split it to XY.
If b1 >= 0xC0 but < 0xE0 read 1 more byte (I'll call it b2) and split it to XXYY
If b1 >= 0xE0 read 2 more bytes and split it to XXXYYY
Note: The values will be something like move back Y+1 bytes, and then for X, you have to subtract from its base first. (Ex. If XY then X-8, if XXYY, then XX-0xC0, if XXXYYY then XXX-0xE00)
Ex. From your compressed file example, starting from 0x14.
@0x14: b1 = 0B, so read 0B bytes and output.
@0x20: b1 = 91, so X = 9, Y = 1. Move back 1+1= 2 bytes. Read 9-8+1= 2 bytes and output. (You got 00 00.)
@0x21: b1 = A3, Move back 3+1= 4 bytes. Read A-8+1 = 3 bytes and output. (00 00 00)
@0x22: b1 = 8D, Move back D+1= 0xE bytes. Read 8-8+1= 1 byte and output. (41)
@0x23: b1 = 04, read 04 bytes and output. (4E 4D 30 30)
@0x28: b1 = 91. Move back 2 bytes, read 2 bytes. (30 30)
@0x29: b1 = 04, read 04 bytes and output. (2E 44 41 54)
@0x2E: b1 = C5, b2= 11. Move back 0x11+1= 0x12 bytes. Read C5-C0+2=7 byte and output. (00 00 00 00 00 00 00)
Ex. for 0xE0 base is b1 = FF, b2 = F3, b3 = FF. So you get FFF3FF. XXX = FFF, YYY = 3FF. Move back 3FF+1 bytes. Read FFF-E00=1FF+3 bytes.
(I think if it's 0xC0 base, you add 2 to number of byte read, and if it's 0XE00 base, you add 3 to number of byte read, but I'm not 100% sure. You can try to experiment with these numbers.)
jjjewel,
Thanks for looking at the files. One minor issue with your scheme and one I came up (It is very similar), is that it chokes when you try to go compress the file. There is no reason to ever compress the A in ANM0000.DAT with this scheme. You would just have the code 0x05 and then have ANM00. It will always return a code that is just as long as the item you are encoding. If I try this coding scheme on the file it breaks down and starts to spit out junk later in the file. I do think the first part about splitting the bye into 0x0 and 0x0 to do different things is correct. The A encoding is just making me doubt this.
There has to be a 0x0041, 0x000041 or 0x00000041 in the dictionary or compression does not work. I need to be able to decompress and also compress the scheme in the exact same way in order to be able to change things in the files and get it back working on the PSP.
Thanks for looking at the files. One minor issue with your scheme and one I came up (It is very similar), is that it chokes when you try to go compress the file. There is no reason to ever compress the A in ANM0000.DAT with this scheme. You would just have the code 0x05 and then have ANM00. It will always return a code that is just as long as the item you are encoding. If I try this coding scheme on the file it breaks down and starts to spit out junk later in the file. I do think the first part about splitting the bye into 0x0 and 0x0 to do different things is correct. The A encoding is just making me doubt this.
There has to be a 0x0041, 0x000041 or 0x00000041 in the dictionary or compression does not work. I need to be able to decompress and also compress the scheme in the exact same way in order to be able to change things in the files and get it back working on the PSP.
hope you can do this really wanting to play this never got to on the ps2 but would like to on my psp good luck
Well it took me about 6-8 months to get LPR from start to playable in English. My guess it will take similar amount of time for this game as long as the technical hurdles are overcome.
ChepChep I also found the part where it read the letter "A" quite odd but it seemed to generate some readable output. Anyway, as I said, you probably have to experiment with it.
After you get the decompressed file, it might be okay to test if the game allows uncompressed data, so you don't have to worry about making a re-compression program. (Which is usually much more difficult than a decompressor.) But the ISO might be much bigger then. (I haven't dealt with many PSP hacking. Usually in NDS hacking, it doesn't make that much difference, like, just a few MB bigger. But it's probably not the same in PSP.)
After you get the decompressed file, it might be okay to test if the game allows uncompressed data, so you don't have to worry about making a re-compression program. (Which is usually much more difficult than a decompressor.) But the ISO might be much bigger then. (I haven't dealt with many PSP hacking. Usually in NDS hacking, it doesn't make that much difference, like, just a few MB bigger. But it's probably not the same in PSP.)
ChepChep, just for your information, you might want to check out Tinke's plugin for Hetalia for info about image encryption, etc. I have been wondering why the file extensions in Makai Kingdom seem familiar and I found out they were similar to files in Hetalia for NDS. I haven't checked if they are exactly the same format but since they use the same unique image extensions (IMY, MAP), you might find something useful.
jjjewel,
Thanks for your help I appreciate it. I will get them out and see it helps figure out the compression.
-ChepChep
Thanks for your help I appreciate it. I will get them out and see it helps figure out the compression.
-ChepChep
So I played around with the compression for KS4 a bit more today. It turned out that I had a bug in my code that prevented the KS4 files from being uncompressed properly. The algorithm that jjewel outlines above works with a few minor tweaks. I wrote some python code to uncompress the DAT and KS4 format in PKP.
Here is a link to the code I wrote:
(download)
The good news is that many of the files in START.KS4 are very similar to the LPR files. Many are exactly the same format but some are just slightly different.
My next step is to write code that will encode the dat and KS4 files so that the PSP can use them. This is going to be a much harder task if my experience from LPR for LZS format is similar.
Here is a link to the code I wrote:
(download)
The good news is that many of the files in START.KS4 are very similar to the LPR files. Many are exactly the same format but some are just slightly different.
My next step is to write code that will encode the dat and KS4 files so that the PSP can use them. This is going to be a much harder task if my experience from LPR for LZS format is similar.
Well I am really looking forward to this and good luck and hope you can get past this and I know I have said this before but thank you for every thing your doing. 
Thanks for the support... it will be slow going but hopefully something will come of this.
- Status
Similar threads
-
BigOnYa
-
BakerMan
I rather enjoy a life of taking it easy. I haven't reached that life yet though. -
Xdqwerty
what are you looking at?
-
-
-
-
@
BakerMan:
I- AURORA BOREALIS? AT THIS TIME OF YEAR, AT THIS TIME OF DAY, IN THIS PART IF THE COUNTRY, LOCALIZED ENTIRELY WITHIN YOUR KITCHEN?+1
-
@
BakerMan:
jokes aside, anyone else who saw the northern lights tonight, what did you think, i thought they were beautiful for a while, before it went to a vague pink+1
maybe later they'll pick up again -
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
@
BakerMan:
sadly, the clouds are setting in now
hey BigOnYa the clouds are coming from the south, maybe check again -
-
-
