Phantasy Star Portable 2 Infinity - English Translation

Discussion in 'PSP - Hacking & Homebrew' started by JamRules, Jul 5, 2014.

Loading...
  1. JamRules
    OP

    JamRules .....

    Member
    7
    Jan 9, 2014
    United States
    Since you possibly don’t need to rebuild the fpb and use it, I would recommend using essen tools to split up the fpb, this basically does a brute force split but should suffice if you are only interested in getting at the models.

    Edit: that said, PSU doesn’t have a fpb file I thought

    Edit:*gasetools
    https://github.com/essen/gasetools
     
    Last edited by JamRules, Sep 3, 2019
    Kion likes this.
  2. Kion

    Kion Member

    Newcomer
    1
    Sep 2, 2019
    Japan
    Fuji-shi, Shizuoka
    Thanks for the link to the tools! It compiles just fine but I get a segmentation fault when I try and run:

    Code:
    ./fpb file.fpb
    I tried it both for the "file.fpb" file in Phantasy Star Portable and PSP2i with no luck. Is this tool for the boost.fpb? Should I be using the code from release and not cloned from the repository directly? Or is there some patch that needs to be added for it to work?

    Derp. My brain grouped the PSU-like games. Specifically I'm looking to try and dig into any of the Phantasy Star Portable games on PSP.


    Edit:
    I think the reason for the segmentation fault was because my laptop didn't have enough memory for all of the files (it looks like over 10,000!??). Each one of the files looks like a .nbl archive. So I'm guessing that's an archive format compressed with prs. Is there anything else needed to unpack the individual files from the .nbl archive?
     
    Last edited by Kion, Sep 5, 2019
    JamRules likes this.
  3. JamRules
    OP

    JamRules .....

    Member
    7
    Jan 9, 2014
    United States
    Essen’s code for nbl should work in the first instance for getting started. He also posted the following

    http://psumods.co.uk/archives/wiki/files%3Anbl.html

    Edit:the guys over a psumods or clementine possibly know more, I have seen a few examples of models exported or converted
     
    Last edited by JamRules, Sep 5, 2019
    Kion likes this.
  4. Kion

    Kion Member

    Newcomer
    1
    Sep 2, 2019
    Japan
    Fuji-shi, Shizuoka
    Okay, I was able to extract the nbl files from the fpb and the files from the nbl files using essen's tools. For some reason's I kept getting a segment fault on the fpb tool, but it happens after the files get written, so it seems to work. And then for the nbl tool, the compiler gives an error for strict mode, but after fixing the nbl extractor seems to work just fine.

    I'll start digging on on psumods to see if there's any hints or information. Right now the two things I'm wondering about are how many different file extensions are there in PSP2i? Which extension is the model format? And is the an nbl file that contains the common items like item boxes or meseta?
     
    JamRules likes this.
  5. JamRules
    OP

    JamRules .....

    Member
    7
    Jan 9, 2014
    United States
    I think models are unj files, not sure though as I’ve mainly been interested in the text.
    I’m not sure where everything is to be honest.
     
  6. Kion

    Kion Member

    Newcomer
    1
    Sep 2, 2019
    Japan
    Fuji-shi, Shizuoka
    I was able to deform a mesh in psp2i using ppsspp. I guess I'll write a quick summary of how I got here, even though it's not that much. Looks like the game stores all of the assets in a single file named 'file.fbp' on the psp2i disk. To unpack this you can use essen's gasetools, specifically the fpb tool to extract the nbl archives. And from there you can use the nbl archive to extract the raw files from the nbl archives. And then as JamRules mentioned the .unj "universe ninja" file seem to be the model files.

    NPJH50332_00000.

    Going in order one nbl file worth mentioning is "nmll-08894" as it seems to contain a lot of the basic and common models used in the game.

    Code:
    eff_beast_wing.unj
    eff_change_flarelight.unj
    eff_st_change_all_down.unj
    eff_st_change_all.unj
    eff_th_state_ice_brk.unj
    eff_th_state_ice.unj
    NullModel.unj
    ob_xxx_beginner.unj
    ob_xxx_bikkuri.unj
    ob_xxx_bombset.unj
    ob_xxx_bombtimer.unj
    ob_xxx_drop_blue.unj
    ob_xxx_drop_calorie.unj
    ob_xxx_drop_green.unj
    ob_xxx_drop_meseta.unj
    ob_xxx_drop_orange.unj
    ob_xxx_drop_purple.unj
    ob_xxx_drop_rainbow.unj
    ob_xxx_drop_red.unj
    ob_xxx_drop_white.unj
    ob_xxx_drop_yellow.unj
    wp_bul_bo_00.unj
    
    Now that we have the raw files, we can go ahead and edit the source code for ppsspp. Specifically ppsspp compresses memory, and we don't want it to do that. So the way to do that is to clone the repository and then edit the "Common/ChunkFile.cpp" file, and then we have to make two small changes to the file

    Screenshot from 2019-09-08 15-53-03. Screenshot from 2019-09-08 15-53-53.

    In the "CChunkFileReader::SaveFile" function we pretend that the compressed buffer was unable to be allocated, and in "CChunkFileReader::LoadFile", we ignore the Header and force the savestate to load uncompressed. What that does is now our save states won't be 14MB files, they will be the full 35MB of uncompressed information. What that means is that we can take files and look for them in memory, we can take memory and try find what's loaded from the files. And we can also abuse the hell out of the system by saving, editing, loading, editing loading to make changes to memory and seeing what the effect is.

    So I guess the next step is since the save state doesn't align the memory to 4 byte boundaries I can make a copy of the meseta model, edit that and then write a script to copy it into the save state and then test to try and work out the boundaries of each triangle and then each vertex respectively.
     

    Attached Files:

    JamRules and SCOTT0852 like this.
  7. JamRules
    OP

    JamRules .....

    Member
    7
    Jan 9, 2014
    United States
    Interesting notes.

    I would normally just use the memory viewer and then the dump option from there.
    Also if you use the ge debugger you’ll be able to see the vertex format (the files store the data in the same order it is used if I remember correctly so it shouldn’t be too bad). Main things to find is how to tell where each mesh starts and ends.
     
    Kion likes this.
  8. HomeShineBrew

    HomeShineBrew Member

    Newcomer
    2
    Apr 15, 2018
    Canada
    Does this get access to any textures in the game? If so I'm willing to try diving into the files to attempt upscaling the game with Gigapixel AI, Waifu 2x, ESRGAN, etc.

    Sent from my PH-1 using Tapatalk
     
  9. JamRules
    OP

    JamRules .....

    Member
    7
    Jan 9, 2014
    United States
    Unless you know how to do the programming I would say you're better off playing the game and letting PPSSPP dump the files. Otherwise you need to figure out all the container formats and the conversion of the textures from their raw formats into a usable format such as png. And also work on a process to calculate the hash from the raw binary data that matches PPSSPP.

    It's also worth noting that it would be good to pull the textures from Phantasy Star Universe as they are double the resolution, of course not all the content exists but for the content that does it'll look better.

    Edit:
    Example of textures from PSP, PC and then PC after Waifu2x
    [​IMG]
    [​IMG]
    Whether there is any noticable difference in action is to be seen to be honest
     
    Last edited by JamRules, Sep 8, 2019
  10. HomeShineBrew

    HomeShineBrew Member

    Newcomer
    2
    Apr 15, 2018
    Canada
    Yeah I've tried the whole playing games to dump textures but it just makes way too many copies of the same texture under different names. If there was a program that could grab all the names of the copies so I could paste them into the texture ini and match them all to one hash I might be willing to try again. Just being hopeful lol

    Sent from my PH-1 using Tapatalk
     
  11. JamRules
    OP

    JamRules .....

    Member
    7
    Jan 9, 2014
    United States
    The name is based on the address, CLUT and pixel data hashes so you'll need code that mimics PPSSPP's hashing for the CLUT and pixel data and then remove the address requirement.
    You never know, someone might write something eventually...
     
    Last edited by JamRules, Sep 8, 2019
  12. HomeShineBrew

    HomeShineBrew Member

    Newcomer
    2
    Apr 15, 2018
    Canada
    I'm waiting for someone who's borderline obsessed like the guy behind N64 Revived to start dumping and figuring out everything. If there isn't anyone like that before the translation releases I'll probably just end up playing through it and dumping the normal way lol

    Sent from my PH-1 using Tapatalk
     
    JamRules likes this.
  13. JamRules
    OP

    JamRules .....

    Member
    7
    Jan 9, 2014
    United States
    Manually doing it is the way I would go at to start with. The textures that are calculated at 512x512 are a pain though.
     
  14. Kion

    Kion Member

    Newcomer
    1
    Sep 2, 2019
    Japan
    Fuji-shi, Shizuoka
    Managed to track nail down the vertex format, and managed to export the meseta model. Though to be complete, should probably add vertex color to the faces for the complete look.

    Screenshot-2019-9-9 Burning Rangers.

    At this point all of my computers are running Debian. I set up a windows machine as a virtual client on the network running on a Latte Panda, but I continue to forget that it exists. What I like about using save states is that it allows for writing scripts for searching inside or writing to memory. For example, for testing to find out if the vertex color alpha uses argb1555 format or not, I wrote this short script:

    Screenshot from 2019-09-09 19-40-07.

    Code:
    const meseta = "meseta.unj";
    const state = "/home/kion/.config/ppsspp/PSP/PPSSPP_STATE/NPJH50332_1.02_0.ppst";
    
    const needle = fs.readFileSync(meseta).subarray(0x180, 0x360);
    const haystack = fs.readFileSync(state);
    
    let vertices = [];
    let colors = [];
    let lookup = {
        'f7ee' : 0b1111110000000000, // red
        'f0ad' : 0b1000001111100000, // green
        'f0bd' : 0b1000000000011111, // blue
        'f0cf' : 0b0111111111111111 // white with 50% alpha?
    }
    
    for(let i = 0; i < needle.length; i += 0x10) {
    
        console.log(i.toString(16));
    
        const c = needle.readUInt16LE(i);
        let color = c.toString(16);
        needle.writeUInt16LE(lookup[color], i)
        const x = needle.readFloatLE(i + 4).toFixed(2);
        const y = needle.readFloatLE(i + 8).toFixed(2);
        const z = needle.readFloatLE(i + 12).toFixed(2);
    
    }
    
    let index = 0x138a171;
    needle.copy(haystack, index);
    

    meseta_unj.PNG

    My current notes look something like this. The file starts with a normal magic number and length. Following that is a pointer towards the end of the file, followed by the int value of 3 (or at least in the files I've looked at so far). The top of the file looks like it's the bone. The four 1.0f floats in a diagonal is pretty telling as a 4x4 transformation matrix. It seems to have the three 1.0f floats above it. And then files with multiple bones have 0x20 of information above and below the matrix, so that's how I segmented that off.

    For the materials, this is easier to look at with models with multiple materials. The patterns starting with 02000100 and ending with FFFFFFFF seems to repeat in model. So i think that's about the right length, though I can't be sure if it's materials and what each byte does without more comparison and testing.

    After that there seems to be 0x20 of space between the materials and vertex list. I'm guessing these are draw calls. Just a guess, but the general idea is that you try to identify structs, make hypothesis about what each struct does, test, and refine. So I'll try removing that value or changing it to see what happens.

    After this is the vertex list. For meseta the format ended up being really easy. It's argb1555 for the vertex color, the space following that could potentially be the weight index (zero, since we only have one bone), followed by float values for x, y, z. I expect most models to follow a similar format, likely with the addition of normals and uv-diffuse when applicable. Also now that I know how many vertices there are, I can scan the bytes in the file to look for where the vertex list length is declared.

    Following the vertex list are what I'm guessing are uniforms. I've only changed one of the values so far. Changing the bit flags had different effects such as changing the strip order, putting the model into normal debug mode and turning off vertex colors. So it's my guess that these are a series of values that are passed into the shader to describe how to render the model. A side note is that there don't seem to be any indices, so I think the models are using direct buffer geometry.

    Screenshot from 2019-09-09 08-34-12. Screenshot from 2019-09-09 08-31-29.

    And then after that is the end of the file, which is pointed to by the header. Since there's no other place to put it, i would expect this area to be used for describing the model attributes like how many bones there are, how many materials, or how many vertices there are. One thing I find pretty quirky about this model format is the complete lack of pointers. It looks like they really went out of their way to design a file format that stack attributes and offsets are probably managed by the program by the offset of the start of the model, plus the size and count of the structs to skip over to read the attribute they want to work with.
     
    Manana and JamRules like this.
  15. JamRules
    OP

    JamRules .....

    Member
    7
    Jan 9, 2014
    United States
    Good that you seem to making some decent progress :yaypsp:

    Out of interest what prompted you to start looking at the models.
    Just for fun or are you aiming to export the models for use somewhere else, or perhaps even create mods for the game itself?
     
    Manana and Kion like this.
  16. Kion

    Kion Member

    Newcomer
    1
    Sep 2, 2019
    Japan
    Fuji-shi, Shizuoka
    A little bit. I thought it was weird that the model files don't seem to have any internal pointers. But looking at some of these values seems to relative to the PSP memory, and are potentially baked in. So I guess I need to dump the PSP memory and see if the offsets match up.

    Originally I was thinking of looking into AOTI, but I don't like having to set up Wine. So I ended up going for the Phantasy Star Portable series.

    I'm generally on the "dump and document" side of things. Modding is satisfying at the expense of being time consuming.

    Edit:

    I looked into the offset problem some more. The meseta model is only 0x440 bytes in length, and doesn't seem to have any pointers. That is until I looked at the model in memory where pointers had been adjusted to the offsets inside memory. So I think the pointers aren't set relative to the file, but relative to their position in the nbl, and the whole unpacked nbl is copied directly into memory. Here's a traced out spreedsheet of the meseta file: https://drive.google.com/file/d/1z4rGt5XPrAlgdjeFigZA0BkOorj6A_t6/view?usp=sharing
     
    Last edited by Kion, Sep 11, 2019
    JamRules likes this.
  17. nanei6

    nanei6 Newbie

    Newcomer
    1
    Sep 11, 2019
    China
    Simply using Waifu2x doesn't make the game look better.But we can completely replace the texture of the environment, like some other games that have been dealt with.As for reseting the weapons’ texture,it is a a major project that can only be done by hand.
    2febc08065380cd7bfa113c1ae44ad34588281f0. ae38c1ef76094b369161afc1accc7cd98c109d44.
    The material in the picture comes from MonsterHunterPortable3rdHDRemake on github (sorry i can not post hyperlinks).I am using it here just for demonstration.
     
  18. Azumablade

    Azumablade Member

    Newcomer
    1
    Sep 14, 2017
    Philippines
    Phantasy Star fans looking forward for it's completion keep it up :)
     

    Attached Files:

    Kion likes this.
  19. JamRules
    OP

    JamRules .....

    Member
    7
    Jan 9, 2014
    United States
    Thanks for the cool demonstration.
    I agree, just using Waifu2x alone maybe doesn’t have the best results.
    Just that it would be quite time consuming to create proper high quality replacements.

    Definitely a good thought though.
     
    Manana and Kion like this.
  20. RisingHopperZeroOne

    RisingHopperZeroOne Member

    Newcomer
    1
    Jul 31, 2019
    Australia
    Is it possible for you to share tools for extracting and repacking the fpb to replace audio? There was an undub attempted for 1 but it wasn't fully working.
     
    FlorinEmil likes this.
Loading...