Code a *.fpk packer

Discussion in 'Wii - Hacking' started by WiiShizzza, Jan 31, 2010.

Jan 31, 2010

Code a *.fpk packer by WiiShizzza at 12:28 AM (4,136 Views / 0 Likes) 17 replies

  1. WiiShizzza
    OP

    Member WiiShizzza Graphics juggler

    Joined:
    Oct 10, 2008
    Messages:
    1,201
    Country:
    Germany
    As you might now, fpk is a container file somehow like an arc, but with a sort of PRS compression. It's used in some games. (ex. TvC UAS)
    tbu from the game research forum wrote an extractor for those [​IMG]

    Warning: Spoilers inside!
     
  2. giantpune

    Member giantpune GBAtemp Addict

    Joined:
    Apr 10, 2009
    Messages:
    2,860
    Country:
    United States
    packer is easy. just run through code through a reverse compiler. [​IMG]
     
  3. WiiShizzza
    OP

    Member WiiShizzza Graphics juggler

    Joined:
    Oct 10, 2008
    Messages:
    1,201
    Country:
    Germany
    lol...pune!
    you are a code guru, I understand nothing about that. [​IMG]

    Would you be so kind and use your kung fu to compile me the fpk packer ?
     
  4. giantpune

    Member giantpune GBAtemp Addict

    Joined:
    Apr 10, 2009
    Messages:
    2,860
    Country:
    United States
    it would take me a while. ive never even messed with any compression/decompression algorithms besides just using what others have written.
     
  5. Slimmmmmm

    Member Slimmmmmm GBAtemp MoNkEeE

    Joined:
    Nov 1, 2007
    Messages:
    1,687
    Location:
    the land of lol
    Country:
    United Kingdom

    He's a ninja [​IMG]
     
  6. vashgs

    Member vashgs GBAtemp Regular

    Joined:
    Feb 1, 2008
    Messages:
    236
    Country:
    United States
    You have all of the decompression algorithms. Write them out in pseudo-code, create the opposite for each step of the way, write your "opposite" pseudo into whatever language you like and compile. The end.
     
  7. ClockWorK

    Member ClockWorK GBAtemp Regular

    Joined:
    May 10, 2009
    Messages:
    150
    Country:
    United States
    If nobody else steps up, I can probably do this.

    I would want a sample .FPK to test with, along with the unpacked contents.

    First, I'd re-write the unpacker to get familiar with the format and structure.

    Then, the packer.

    I would probably want 2 or 3 weeks. I already have a full time job as a programmer, and I have a wife, so fitting in things like this isn't easy.

    I'm sure somebody with more free time could probably do it quicker.

    Also, if I do it, it'll be done in VB.NET (with source code provided)

    When I first looked at the code, it made me think it's using a Run-Length encoding compression. I'm not familiar with .FPK files or their contents, but RLE compression is usually only used in simple image formats.

    If you want to PM me, I'll take a look.
     
  8. smf

    Member smf GBAtemp Advanced Fan

    Joined:
    Feb 23, 2009
    Messages:
    836
    Country:
    United Kingdom
    It's generally a lot harder to write a compression than decompression, because you want the decompression to be fast and memory efficient. All the hard work is done when you compress.

    Not saying it's impossible, because I've done similar things before. I wouldn't bother with pseudo code though, just code it in something like c# or java.
     
  9. DrThunder

    Newcomer DrThunder Member

    Joined:
    Aug 7, 2009
    Messages:
    15
    Country:
    United States
    That unpacker fails on Monster Hunter G's single fpk file at 59.9MB extracted, every time. I don't know whether it's a problem with the program itself, the decompression algorithm it uses, or a certain file in the fpk, but I think the problem should be worked out before anyone spends too much time working with that code... at least, if it's going to be an fpk packer and not just a TvC packer.
     
  10. WiiShizzza
    OP

    Member WiiShizzza Graphics juggler

    Joined:
    Oct 10, 2008
    Messages:
    1,201
    Country:
    Germany
    that unpacker works for the fpk files from TvC UAS. For Monster Hunters you can find a different unpacker somewhere on here.
     
  11. RupertAvery

    Newcomer RupertAvery Member

    Joined:
    Feb 1, 2010
    Messages:
    16
    Country:
    Philippines
    I think I'll have a shot at this...

    It looks to be some sort of dictionary encoding scheme.

    [EDIT] Actually it more resembles LZ77

    I may be a bit wrong, please let me know if I missed anything or interpreted the code incorrectly...

    First off the general file format:

    0x00 headers???
    0x04 [no of files]
    0x10 [directory] = [directory entry] x [no of files]

    [directory entry] - 48 bytes
    0x00 file name
    0x24 [file data] offset
    0x28 file length
    0x2C (compressed) data length

    [file data] - starts after last directory entry? (compressed)

    Compression algorithm

    [UPDATED]

    Woke up this morning with an itch to code. Had a bit of trouble at first compiling in VC6 (didn't realize it was for *NIX, doh!), but finally managed to get it running.
    My test data is fpack/chr/ryu/0000.fpk.

    The data looks to be compression CONTROL codes (encode in a bit stream) interspersed with DATA bytes. For example:

    Code:
    00000340h: 9E 00 FF 02 04 3F FB 10 00 07 55 A0 0C FC 53 20
    00000350h: 23 FC 50 F8 4E 18 F8 4B C6 F8 49 F8 46 31 F8 44

    The first byte will always be a CONTROL code.

    9E = 10011110

    We read the first bit, see its a 1, so the NEXT available byte 0x00, is copied verbatim.

    Our next bit is 0, it's compressed data. It is type A (0), so the next two bits will decode the length 3 + 2 = 5.
    The next available byte 0xFF encodes the distance in the output stream, which is -1.

    Since our current output stream contains only 0x00, we will copy it five times.

    The next two bits are both 1, so the next two bytes are written as-is, 02 and 04.

    We're down to our last bit 0, which is of course compressed data, but we run out of bit information here, so we grab the next CONTROL byte

    3F = 0 01 1 1 1 1 1

    Continuing our bit reading, the next bit is 0 (type A), and the length bits is 1 + 2 = 3
    The distance is 0xFB or -5. This will copy 3 0x00's to the output buffer.

    The next 5 bits are all 1's so data is just copied and pointers are advanced.

    If the next two bits are 01, then the next two available bytes encode the distance and length as so:

    Code:
    000003c0h: 50 86 D0 FC 43 50 78 59 06 3C DC CC 31 [b]FA 6C[/b] 06
    000003d0h: 29 31 7C 06 1E AC 8C 17 FC 10 61 DC 09 DC 05 86
    FA6C = 1111101001101 100

    pos = ff4d = -179, length = 4 + 2 = 6

    First 13 bits encode distance and the last 3 encode length. If length = 0, then the next available byte encodes the length.

    The window size is therefore 8192 bytes or 8kb, and the maximum length is 256.
     
  12. WiiShizzza
    OP

    Member WiiShizzza Graphics juggler

    Joined:
    Oct 10, 2008
    Messages:
    1,201
    Country:
    Germany
    this sounds great! Thank you for your try. I wonder f you could provide an compiled windows exe.
    I could test it out, then.
     
  13. Tonitonichopchop

    Member Tonitonichopchop GBAtemp Maniac

    Joined:
    Dec 9, 2009
    Messages:
    1,210
    Country:
    United States
    I'd like to see this too. I liking the enthusiasm this game is getting.
     
  14. RupertAvery

    Newcomer RupertAvery Member

    Joined:
    Feb 1, 2010
    Messages:
    16
    Country:
    Philippines
    I'm still getting a grip on the compression scheme, and implementing a proper compression routine will take a bit more time.

    I've never written a real compression routine before (well I did have a go at a Huffman implementation, but it was long ago and only worked on text files for some reason)

    Hopefully I'll be able to write a compression routine that will compress the unmodified back to original state, and since the window sizes and distances are inherent in the decoding, using those same parameters in encoding should produce the same input.

    I've updated the compression algorithm if anyone wants to take a look.
     
  15. RupertAvery

    Newcomer RupertAvery Member

    Joined:
    Feb 1, 2010
    Messages:
    16
    Country:
    Philippines
    [UPDATE]

    I've managed to get compression working to a point where the uncompressed data looks just like the original. In my test below, I compress a file, then use tba's decompression routine to unpack the data.

    Using FileAlyzer I verified that the uncompressed and original data were exact copies.

    My only concern now is that my compressed data is not 100% the same as the original (compressed) data in the fpk. My compressed data was a bit larger than the original by around 1KB with chr\cmn000.mot

    My 602,112-byte test file chr\ryu000.brres took a little under a minute to compress (yeah, slow). The compressed file was 307,002 bytes. CRC-32 is FBA86BE6

    Also tested with chr\cmn000.mot with similar results.

    I do reuse the input buffer to uncompress but I make sure to clear it first.

    Warning: Spoilers inside!
     
  16. RupertAvery

    Newcomer RupertAvery Member

    Joined:
    Feb 1, 2010
    Messages:
    16
    Country:
    Philippines
    I've compiled the above test program and uploaded it. Link below.

    FPKer windows binary

    Please try and let me know if you have any problems running it. Syntax is

    CODEfpker.exe

    You can also drag and drop a file onto it. The program will generate a compressed.bin and uncompressed.bin file in C:\

    Please let me know if it crashes while compressing a certain file, or if the uncompressed.bin and original file checksums don't match. Use FileAlyzer or your favorite checksum generating program for testing.

    Meanwhile, I will fix this up to actually generate the FPK files, and finally do some testing with unFPK.

    Cheers!
     
  17. jackharvest

    Newcomer jackharvest Member

    Joined:
    May 12, 2009
    Messages:
    10
    Country:
    United States
    Hey thanks. This will really help us at Kitty Corp: Meow Mix to get Full Metal Alchemist models into Brawl.
     
  18. jaybee

    Member jaybee GBAtemp Regular

    Joined:
    Jan 29, 2010
    Messages:
    206
    Country:
    United Kingdom
    I made a packer on the Amiga once but never released it. I think I called it Fudge! xD
     

Share This Page