Hacking Code a *.fpk packer

[WiiShizzza]

As you might now, fpk is a container file somehow like an arc, but with a sort of PRS compression. It's used in some games. (ex. TvC UAS)
tbu from the game research forum wrote an extractor for those

Spoiler

CODE#include
#include
#include
#include
#include
#include
#include


typedef unsigned intÂÂ u32;
typedef unsigned short u16;
typedef unsigned charÂÂu8;


int endian=1;

u32 BE32(u32 data)
{
ÂÂ if(endian)
ÂÂÂÂÂÂreturn ( (data24) );
ÂÂ else
ÂÂÂÂÂÂreturn data;
}

int blen;
int fbuf;

/* PRS get bit form lsb to msb, FPK get it form msb to lsb */
int get_bits(int n, char *sbuf, int *sptr)
{
ÂÂ int retv;

ÂÂ retv = 0;
ÂÂ while(n){
ÂÂÂÂÂÂretv

 

[giantpune]

packer is easy. just run through code through a reverse compiler.

 

[WiiShizzza]

lol...pune!
you are a code guru, I understand nothing about that.

Would you be so kind and use your kung fu to compile me the fpk packer ?

 

[giantpune]

it would take me a while. ive never even messed with any compression/decompression algorithms besides just using what others have written.

 

[Slimmmmmm]

lol...pune!
you are a code guru, I understand nothing about that.

Would you be so kind and use your kung fu to compile me the fpk packer ?

He's a ninja

 

[vashgs]

You have all of the decompression algorithms. Write them out in pseudo-code, create the opposite for each step of the way, write your "opposite" pseudo into whatever language you like and compile. The end.

 

[ClockWorK]

If nobody else steps up, I can probably do this.

I would want a sample .FPK to test with, along with the unpacked contents.

First, I'd re-write the unpacker to get familiar with the format and structure.

Then, the packer.

I would probably want 2 or 3 weeks. I already have a full time job as a programmer, and I have a wife, so fitting in things like this isn't easy.

I'm sure somebody with more free time could probably do it quicker.

Also, if I do it, it'll be done in VB.NET (with source code provided)

When I first looked at the code, it made me think it's using a Run-Length encoding compression. I'm not familiar with .FPK files or their contents, but RLE compression is usually only used in simple image formats.

If you want to PM me, I'll take a look.

 

[smf]

You have all of the decompression algorithms. Write them out in pseudo-code, create the opposite for each step of the way, write your "opposite" pseudo into whatever language you like and compile. The end.

It's generally a lot harder to write a compression than decompression, because you want the decompression to be fast and memory efficient. All the hard work is done when you compress.

Not saying it's impossible, because I've done similar things before. I wouldn't bother with pseudo code though, just code it in something like c# or java.

 

[DrThunder]

That unpacker fails on Monster Hunter G's single fpk file at 59.9MB extracted, every time. I don't know whether it's a problem with the program itself, the decompression algorithm it uses, or a certain file in the fpk, but I think the problem should be worked out before anyone spends too much time working with that code... at least, if it's going to be an fpk packer and not just a TvC packer.

 

[WiiShizzza]

that unpacker works for the fpk files from TvC UAS. For Monster Hunters you can find a different unpacker somewhere on here.

 

[RupertAvery]

I think I'll have a shot at this...

It looks to be some sort of dictionary encoding scheme.

[EDIT] Actually it more resembles LZ77

I may be a bit wrong, please let me know if I missed anything or interpreted the code incorrectly...

First off the general file format:

0x00 headers???
0x04 [no of files]
0x10 [directory] = [directory entry] x [no of files]

[directory entry] - 48 bytes
0x00 file name
0x24 [file data] offset
0x28 file length
0x2C (compressed) data length

[file data] - starts after last directory entry? (compressed)

Compression algorithm

[UPDATED]

Woke up this morning with an itch to code. Had a bit of trouble at first compiling in VC6 (didn't realize it was for *NIX, doh!), but finally managed to get it running.
My test data is fpack/chr/ryu/0000.fpk.

The data looks to be compression CONTROL codes (encode in a bit stream) interspersed with DATA bytes. For example:

00000340h: 9E 00 FF 02 04 3F FB 10 00 07 55 A0 0C FC 53 20
00000350h: 23 FC 50 F8 4E 18 F8 4B C6 F8 49 F8 46 31 F8 44

The first byte will always be a CONTROL code.

9E = 10011110

We read the first bit, see its a 1, so the NEXT available byte 0x00, is copied verbatim.

Our next bit is 0, it's compressed data. It is type A (0), so the next two bits will decode the length 3 + 2 = 5.
The next available byte 0xFF encodes the distance in the output stream, which is -1.

Since our current output stream contains only 0x00, we will copy it five times.

The next two bits are both 1, so the next two bytes are written as-is, 02 and 04.

We're down to our last bit 0, which is of course compressed data, but we run out of bit information here, so we grab the next CONTROL byte

3F = 0 01 1 1 1 1 1

Continuing our bit reading, the next bit is 0 (type A), and the length bits is 1 + 2 = 3
The distance is 0xFB or -5. This will copy 3 0x00's to the output buffer.

The next 5 bits are all 1's so data is just copied and pointers are advanced.

If the next two bits are 01, then the next two available bytes encode the distance and length as so:

000003c0h: 50 86 D0 FC 43 50 78 59 06 3C DC CC 31 [b]FA 6C[/b] 06
000003d0h: 29 31 7C 06 1E AC 8C 17 FC 10 61 DC 09 DC 05 86

FA6C = 1111101001101 100

pos = ff4d = -179, length = 4 + 2 = 6

First 13 bits encode distance and the last 3 encode length. If length = 0, then the next available byte encodes the length.

The window size is therefore 8192 bytes or 8kb, and the maximum length is 256.

 

[WiiShizzza]

this sounds great! Thank you for your try. I wonder f you could provide an compiled windows exe.
I could test it out, then.

 

[Tonitonichopchop]

I'd like to see this too. I liking the enthusiasm this game is getting.

 

[RupertAvery]

I'm still getting a grip on the compression scheme, and implementing a proper compression routine will take a bit more time.

I've never written a real compression routine before (well I did have a go at a Huffman implementation, but it was long ago and only worked on text files for some reason)

Hopefully I'll be able to write a compression routine that will compress the unmodified back to original state, and since the window sizes and distances are inherent in the decoding, using those same parameters in encoding should produce the same input.

I've updated the compression algorithm if anyone wants to take a look.

 

[RupertAvery]

[UPDATE]

I've managed to get compression working to a point where the uncompressed data looks just like the original. In my test below, I compress a file, then use tba's decompression routine to unpack the data.

Using FileAlyzer I verified that the uncompressed and original data were exact copies.

My only concern now is that my compressed data is not 100% the same as the original (compressed) data in the fpk. My compressed data was a bit larger than the original by around 1KB with chr\cmn000.mot

My 602,112-byte test file chr\ryu000.brres took a little under a minute to compress (yeah, slow). The compressed file was 307,002 bytes. CRC-32 is FBA86BE6

Also tested with chr\cmn000.mot with similar results.

I do reuse the input buffer to uncompress but I make sure to clear it first.

Spoiler

CODE// CompTest.cpp : Defines the entry point for the console application.
//

#include "stdio.h"
#include "string.h"
#include "malloc.h"
#include "io.h"

#define u8 unsigned char

int stop()
{
ÂÂÂÂreturn 0;
}

int blen;
int fbuf;

int get_bits(int n, char *sbuf, int *sptr)
{
ÂÂ int retv;

ÂÂ retv = 0;
ÂÂ while(n){
ÂÂÂÂÂÂretv 5))
ÂÂÂÂÂÂÂÂÂÂÂÂ{
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂwrite_comp_b(dbuf,length,pos);
ÂÂÂÂÂÂÂÂÂÂÂÂ} else
ÂÂÂÂÂÂÂÂÂÂÂÂ{
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂwrite_comp_a(dbuf,length,pos);
ÂÂÂÂÂÂÂÂÂÂÂÂ}

ÂÂÂÂÂÂÂÂÂÂÂÂ//write_comp_a(dbuf,length,pos);
ÂÂÂÂÂÂÂÂÂÂÂÂsptr += length;
ÂÂÂÂÂÂÂÂ} else {
ÂÂÂÂÂÂÂÂÂÂÂÂif (check_window(sbuf,sptr,&length,&pos))
ÂÂÂÂÂÂÂÂÂÂÂÂ{
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ//printf("COPY %02X %02X\n", length, (u8) pos);
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂif((pos > 255) || (length > 5))
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ{
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂwrite_comp_b(dbuf,length,pos);
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ} else
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ{
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂwrite_comp_a(dbuf,length,pos);
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ}

ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂsptr += length;
ÂÂÂÂÂÂÂÂÂÂÂÂ} else {
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ//printf("RAW %02X\n", (u8) sbuf[sptr]);
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂwrite_nocomp(dbuf,sbuf,sptr);
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂsptr++;
ÂÂÂÂÂÂÂÂÂÂÂÂ}
ÂÂÂÂÂÂÂÂ}
ÂÂÂÂ}

ÂÂÂÂprintf("Compression complete!\n");

ÂÂÂÂint dlen;
ÂÂÂÂdlen = data_ptr;
ÂÂÂÂfp = fopen("C:\\compressed.bin", "wb");
ÂÂÂÂ//fp = fopen("C:\\file2.fpk", "wb");
ÂÂÂÂfwrite(dbuf, dlen, 1, fp);
ÂÂÂÂfclose(fp);

ÂÂÂÂ// clear out buffer
ÂÂÂÂmemset(sbuf,0, slen);
ÂÂÂÂuncomp(sbuf,slen, dbuf, dlen);

ÂÂÂÂfp = fopen("C:\\uncompressed.bin", "wb");
ÂÂÂÂfwrite(sbuf, slen, 1, fp);
ÂÂÂÂfclose(fp);

ÂÂÂÂprintf("Wrote C:\\compressed.bin and C:\\uncompressed.bin\n");
}

int main(int argc, char* argv[])
{
ÂÂÂÂ//process_file("C:00_fpk\\chr\\cmn00.mot");
ÂÂÂÂprintf("FPK compression test\n");
ÂÂÂÂif(argc==2)
ÂÂÂÂ{
ÂÂÂÂÂÂÂÂprintf("Compressing file %s\n",argv[1]);
ÂÂÂÂÂÂÂÂprocess_file(argv[1]);
ÂÂÂÂ} else {
ÂÂÂÂÂÂÂÂprintf("Incorrect parameters\n");
ÂÂÂÂ}
ÂÂÂÂreturn 0;
}

 

[RupertAvery]

I've compiled the above test program and uploaded it. Link below.

FPKer windows binary

Please try and let me know if you have any problems running it. Syntax is

CODEfpker.exe

You can also drag and drop a file onto it. The program will generate a compressed.bin and uncompressed.bin file in C:\

Please let me know if it crashes while compressing a certain file, or if the uncompressed.bin and original file checksums don't match. Use FileAlyzer or your favorite checksum generating program for testing.

Meanwhile, I will fix this up to actually generate the FPK files, and finally do some testing with unFPK.

Cheers!

 

[jackharvest]

Hey thanks. This will really help us at Kitty Corp: Meow Mix to get Full Metal Alchemist models into Brawl.

 

[jaybee]

I made a packer on the Amiga once but never released it. I think I called it Fudge! xD