Hacking Phantasy Star Portable 2 Infinity - English Translation

Kion

Member
Newcomer
Joined
Sep 2, 2019
Messages
18
Trophies
0
Age
35
Location
Fuji-shi, Shizuoka
Website
dashgl.com
XP
143
Country
Japan
Since you possibly don’t need to rebuild the fpb and use it, I would recommend using essen tools to split up the fpb, this basically does a brute force split but should suffice if you are only interested in getting at the models.

Edit:*gasetools (gbatemp won't let me reply with link)

Thanks for the link to the tools! It compiles just fine but I get a segmentation fault when I try and run:

Code:
./fpb file.fpb

I tried it both for the "file.fpb" file in Phantasy Star Portable and PSP2i with no luck. Is this tool for the boost.fpb? Should I be using the code from release and not cloned from the repository directly? Or is there some patch that needs to be added for it to work?

Edit: that said, PSU doesn’t have a fpb file I thought

Derp. My brain grouped the PSU-like games. Specifically I'm looking to try and dig into any of the Phantasy Star Portable games on PSP.


Edit:
I think the reason for the segmentation fault was because my laptop didn't have enough memory for all of the files (it looks like over 10,000!??). Each one of the files looks like a .nbl archive. So I'm guessing that's an archive format compressed with prs. Is there anything else needed to unpack the individual files from the .nbl archive?
 
Last edited by Kion,
  • Like
Reactions: JamRules

JamRules

.....
OP
Member
Joined
Jan 9, 2014
Messages
527
Trophies
1
XP
2,204
Country
United States
Thanks for the link to the tools! It compiles just fine but I get a segmentation fault when I try and run:

Code:
./fpb file.fpb

I tried it both for the "file.fpb" file in Phantasy Star Portable and PSP2i with no luck. Is this tool for the boost.fpb? Should I be using the code from release and not cloned from the repository directly? Or is there some patch that needs to be added for it to work?



Derp. My brain grouped the PSU-like games. Specifically I'm looking to try and dig into any of the Phantasy Star Portable games on PSP.


Edit:
I think the reason for the segmentation fault was because my laptop didn't have enough memory for all of the files (it looks like over 10,000!??). Each one of the files looks like a .nbl archive. So I'm guessing that's an archive format compressed with prs. Is there anything else needed to unpack the individual files from the .nbl archive?

Essen’s code for nbl should work in the first instance for getting started. He also posted the following

http://psumods.co.uk/archives/wiki/files%3Anbl.html

Edit:the guys over a psumods or clementine possibly know more, I have seen a few examples of models exported or converted
 
Last edited by JamRules,
  • Like
Reactions: Kion

Kion

Member
Newcomer
Joined
Sep 2, 2019
Messages
18
Trophies
0
Age
35
Location
Fuji-shi, Shizuoka
Website
dashgl.com
XP
143
Country
Japan
Essen’s code for nbl should work in the first instance for getting started.

Okay, I was able to extract the nbl files from the fpb and the files from the nbl files using essen's tools. For some reason's I kept getting a segment fault on the fpb tool, but it happens after the files get written, so it seems to work. And then for the nbl tool, the compiler gives an error for strict mode, but after fixing the nbl extractor seems to work just fine.

Edit:the guys over a psumods or clementine possibly know more, I have seen a few examples of models exported or converted

I'll start digging on on psumods to see if there's any hints or information. Right now the two things I'm wondering about are how many different file extensions are there in PSP2i? Which extension is the model format? And is the an nbl file that contains the common items like item boxes or meseta?
 
  • Like
Reactions: JamRules

JamRules

.....
OP
Member
Joined
Jan 9, 2014
Messages
527
Trophies
1
XP
2,204
Country
United States
I think models are unj files, not sure though as I’ve mainly been interested in the text.
I’m not sure where everything is to be honest.
 

Kion

Member
Newcomer
Joined
Sep 2, 2019
Messages
18
Trophies
0
Age
35
Location
Fuji-shi, Shizuoka
Website
dashgl.com
XP
143
Country
Japan
I was able to deform a mesh in psp2i using ppsspp. I guess I'll write a quick summary of how I got here, even though it's not that much. Looks like the game stores all of the assets in a single file named 'file.fbp' on the psp2i disk. To unpack this you can use essen's gasetools, specifically the fpb tool to extract the nbl archives. And from there you can use the nbl archive to extract the raw files from the nbl archives. And then as JamRules mentioned the .unj "universe ninja" file seem to be the model files.

NPJH50332_00000.jpg

Going in order one nbl file worth mentioning is "nmll-08894" as it seems to contain a lot of the basic and common models used in the game.

Code:
eff_beast_wing.unj
eff_change_flarelight.unj
eff_st_change_all_down.unj
eff_st_change_all.unj
eff_th_state_ice_brk.unj
eff_th_state_ice.unj
NullModel.unj
ob_xxx_beginner.unj
ob_xxx_bikkuri.unj
ob_xxx_bombset.unj
ob_xxx_bombtimer.unj
ob_xxx_drop_blue.unj
ob_xxx_drop_calorie.unj
ob_xxx_drop_green.unj
ob_xxx_drop_meseta.unj
ob_xxx_drop_orange.unj
ob_xxx_drop_purple.unj
ob_xxx_drop_rainbow.unj
ob_xxx_drop_red.unj
ob_xxx_drop_white.unj
ob_xxx_drop_yellow.unj
wp_bul_bo_00.unj

Now that we have the raw files, we can go ahead and edit the source code for ppsspp. Specifically ppsspp compresses memory, and we don't want it to do that. So the way to do that is to clone the repository and then edit the "Common/ChunkFile.cpp" file, and then we have to make two small changes to the file

Screenshot from 2019-09-08 15-53-03.png Screenshot from 2019-09-08 15-53-53.png

In the "CChunkFileReader::SaveFile" function we pretend that the compressed buffer was unable to be allocated, and in "CChunkFileReader::LoadFile", we ignore the Header and force the savestate to load uncompressed. What that does is now our save states won't be 14MB files, they will be the full 35MB of uncompressed information. What that means is that we can take files and look for them in memory, we can take memory and try find what's loaded from the files. And we can also abuse the hell out of the system by saving, editing, loading, editing loading to make changes to memory and seeing what the effect is.

So I guess the next step is since the save state doesn't align the memory to 4 byte boundaries I can make a copy of the meseta model, edit that and then write a script to copy it into the save state and then test to try and work out the boundaries of each triangle and then each vertex respectively.
 

Attachments

  • Screenshot from 2019-09-08 15-53-53.png
    Screenshot from 2019-09-08 15-53-53.png
    88.2 KB · Views: 163

JamRules

.....
OP
Member
Joined
Jan 9, 2014
Messages
527
Trophies
1
XP
2,204
Country
United States
Interesting notes.

I would normally just use the memory viewer and then the dump option from there.
Also if you use the ge debugger you’ll be able to see the vertex format (the files store the data in the same order it is used if I remember correctly so it shouldn’t be too bad). Main things to find is how to tell where each mesh starts and ends.
 
  • Like
Reactions: Kion

HomeShineBrew

Active Member
Newcomer
Joined
Apr 15, 2018
Messages
27
Trophies
0
Age
24
XP
292
Country
Canada
I was able to deform a mesh in psp2i using ppsspp. I guess I'll write a quick summary of how I got here, even though it's not that much. Looks like the game stores all of the assets in a single file named 'file.fbp' on the psp2i disk. To unpack this you can use essen's gasetools, specifically the fpb tool to extract the nbl archives. And from there you can use the nbl archive to extract the raw files from the nbl archives. And then as JamRules mentioned the .unj "universe ninja" file seem to be the model files.

View attachment 178846

Going in order one nbl file worth mentioning is "nmll-08894" as it seems to contain a lot of the basic and common models used in the game.

Code:
eff_beast_wing.unj
eff_change_flarelight.unj
eff_st_change_all_down.unj
eff_st_change_all.unj
eff_th_state_ice_brk.unj
eff_th_state_ice.unj
NullModel.unj
ob_xxx_beginner.unj
ob_xxx_bikkuri.unj
ob_xxx_bombset.unj
ob_xxx_bombtimer.unj
ob_xxx_drop_blue.unj
ob_xxx_drop_calorie.unj
ob_xxx_drop_green.unj
ob_xxx_drop_meseta.unj
ob_xxx_drop_orange.unj
ob_xxx_drop_purple.unj
ob_xxx_drop_rainbow.unj
ob_xxx_drop_red.unj
ob_xxx_drop_white.unj
ob_xxx_drop_yellow.unj
wp_bul_bo_00.unj

Now that we have the raw files, we can go ahead and edit the source code for ppsspp. Specifically ppsspp compresses memory, and we don't want it to do that. So the way to do that is to clone the repository and then edit the "Common/ChunkFile.cpp" file, and then we have to make two small changes to the file

View attachment 178848 View attachment 178849

In the "CChunkFileReader::SaveFile" function we pretend that the compressed buffer was unable to be allocated, and in "CChunkFileReader::LoadFile", we ignore the Header and force the savestate to load uncompressed. What that does is now our save states won't be 14MB files, they will be the full 35MB of uncompressed information. What that means is that we can take files and look for them in memory, we can take memory and try find what's loaded from the files. And we can also abuse the hell out of the system by saving, editing, loading, editing loading to make changes to memory and seeing what the effect is.

So I guess the next step is since the save state doesn't align the memory to 4 byte boundaries I can make a copy of the meseta model, edit that and then write a script to copy it into the save state and then test to try and work out the boundaries of each triangle and then each vertex respectively.
Does this get access to any textures in the game? If so I'm willing to try diving into the files to attempt upscaling the game with Gigapixel AI, Waifu 2x, ESRGAN, etc.

Sent from my PH-1 using Tapatalk
 

JamRules

.....
OP
Member
Joined
Jan 9, 2014
Messages
527
Trophies
1
XP
2,204
Country
United States
Does this get access to any textures in the game? If so I'm willing to try diving into the files to attempt upscaling the game with Gigapixel AI, Waifu 2x, ESRGAN, etc.

Sent from my PH-1 using Tapatalk

Unless you know how to do the programming I would say you're better off playing the game and letting PPSSPP dump the files. Otherwise you need to figure out all the container formats and the conversion of the textures from their raw formats into a usable format such as png. And also work on a process to calculate the hash from the raw binary data that matches PPSSPP.

It's also worth noting that it would be good to pull the textures from Phantasy Star Universe as they are double the resolution, of course not all the content exists but for the content that does it'll look better.

Edit:
Example of textures from PSP, PC and then PC after Waifu2x
comparison-1.png

comparison-2.png

Whether there is any noticable difference in action is to be seen to be honest
 
Last edited by JamRules,

HomeShineBrew

Active Member
Newcomer
Joined
Apr 15, 2018
Messages
27
Trophies
0
Age
24
XP
292
Country
Canada
Unless you know how to do the programming I would say you're better off playing the game and letting PPSSPP dump the files. Otherwise you need to figure out all the container formats and the conversion of the textures from their raw formats into a usable format such as png. And also work on a process to calculate the hash from the raw binary data that matches PPSSPP.

It's also worth noting that it would be good to pull the textures from Phantasy Star Universe as they are double the resolution, of course not all the content exists but for the content that does it'll look better.
Yeah I've tried the whole playing games to dump textures but it just makes way too many copies of the same texture under different names. If there was a program that could grab all the names of the copies so I could paste them into the texture ini and match them all to one hash I might be willing to try again. Just being hopeful lol

Sent from my PH-1 using Tapatalk
 

JamRules

.....
OP
Member
Joined
Jan 9, 2014
Messages
527
Trophies
1
XP
2,204
Country
United States
Yeah I've tried the whole playing games to dump textures but it just makes way too many copies of the same texture under different names. If there was a program that could grab all the names of the copies so I could paste them into the texture ini and match them all to one hash I might be willing to try again. Just being hopeful lol

Sent from my PH-1 using Tapatalk

The name is based on the address, CLUT and pixel data hashes so you'll need code that mimics PPSSPP's hashing for the CLUT and pixel data and then remove the address requirement.
You never know, someone might write something eventually...
 
Last edited by JamRules,

HomeShineBrew

Active Member
Newcomer
Joined
Apr 15, 2018
Messages
27
Trophies
0
Age
24
XP
292
Country
Canada
I'm waiting for someone who's borderline obsessed like the guy behind N64 Revived to start dumping and figuring out everything. If there isn't anyone like that before the translation releases I'll probably just end up playing through it and dumping the normal way lol

Sent from my PH-1 using Tapatalk
 
  • Like
Reactions: JamRules

JamRules

.....
OP
Member
Joined
Jan 9, 2014
Messages
527
Trophies
1
XP
2,204
Country
United States
I'm waiting for someone who's borderline obsessed like the guy behind N64 Revived to start dumping and figuring out everything. If there isn't anyone like that before the translation releases I'll probably just end up playing through it and dumping the normal way lol

Sent from my PH-1 using Tapatalk

Manually doing it is the way I would go at to start with. The textures that are calculated at 512x512 are a pain though.
 

Kion

Member
Newcomer
Joined
Sep 2, 2019
Messages
18
Trophies
0
Age
35
Location
Fuji-shi, Shizuoka
Website
dashgl.com
XP
143
Country
Japan
Managed to track nail down the vertex format, and managed to export the meseta model. Though to be complete, should probably add vertex color to the faces for the complete look.

Screenshot-2019-9-9 Burning Rangers.png

Interesting notes.

I would normally just use the memory viewer and then the dump option from there.
Also if you use the ge debugger you’ll be able to see the vertex format (the files store the data in the same order it is used if I remember correctly so it shouldn’t be too bad). Main things to find is how to tell where each mesh starts and ends.

At this point all of my computers are running Debian. I set up a windows machine as a virtual client on the network running on a Latte Panda, but I continue to forget that it exists. What I like about using save states is that it allows for writing scripts for searching inside or writing to memory. For example, for testing to find out if the vertex color alpha uses argb1555 format or not, I wrote this short script:

Screenshot from 2019-09-09 19-40-07.png

Code:
const meseta = "meseta.unj";
const state = "/home/kion/.config/ppsspp/PSP/PPSSPP_STATE/NPJH50332_1.02_0.ppst";

const needle = fs.readFileSync(meseta).subarray(0x180, 0x360);
const haystack = fs.readFileSync(state);

let vertices = [];
let colors = [];
let lookup = {
    'f7ee' : 0b1111110000000000, // red
    'f0ad' : 0b1000001111100000, // green
    'f0bd' : 0b1000000000011111, // blue
    'f0cf' : 0b0111111111111111 // white with 50% alpha?
}

for(let i = 0; i < needle.length; i += 0x10) {

    console.log(i.toString(16));

    const c = needle.readUInt16LE(i);
    let color = c.toString(16);
    needle.writeUInt16LE(lookup[color], i)
    const x = needle.readFloatLE(i + 4).toFixed(2);
    const y = needle.readFloatLE(i + 8).toFixed(2);
    const z = needle.readFloatLE(i + 12).toFixed(2);

}

let index = 0x138a171;
needle.copy(haystack, index);


meseta_unj.PNG

My current notes look something like this. The file starts with a normal magic number and length. Following that is a pointer towards the end of the file, followed by the int value of 3 (or at least in the files I've looked at so far). The top of the file looks like it's the bone. The four 1.0f floats in a diagonal is pretty telling as a 4x4 transformation matrix. It seems to have the three 1.0f floats above it. And then files with multiple bones have 0x20 of information above and below the matrix, so that's how I segmented that off.

For the materials, this is easier to look at with models with multiple materials. The patterns starting with 02000100 and ending with FFFFFFFF seems to repeat in model. So i think that's about the right length, though I can't be sure if it's materials and what each byte does without more comparison and testing.

After that there seems to be 0x20 of space between the materials and vertex list. I'm guessing these are draw calls. Just a guess, but the general idea is that you try to identify structs, make hypothesis about what each struct does, test, and refine. So I'll try removing that value or changing it to see what happens.

After this is the vertex list. For meseta the format ended up being really easy. It's argb1555 for the vertex color, the space following that could potentially be the weight index (zero, since we only have one bone), followed by float values for x, y, z. I expect most models to follow a similar format, likely with the addition of normals and uv-diffuse when applicable. Also now that I know how many vertices there are, I can scan the bytes in the file to look for where the vertex list length is declared.

Following the vertex list are what I'm guessing are uniforms. I've only changed one of the values so far. Changing the bit flags had different effects such as changing the strip order, putting the model into normal debug mode and turning off vertex colors. So it's my guess that these are a series of values that are passed into the shader to describe how to render the model. A side note is that there don't seem to be any indices, so I think the models are using direct buffer geometry.

Screenshot from 2019-09-09 08-34-12.png Screenshot from 2019-09-09 08-31-29.png

And then after that is the end of the file, which is pointed to by the header. Since there's no other place to put it, i would expect this area to be used for describing the model attributes like how many bones there are, how many materials, or how many vertices there are. One thing I find pretty quirky about this model format is the complete lack of pointers. It looks like they really went out of their way to design a file format that stack attributes and offsets are probably managed by the program by the offset of the start of the model, plus the size and count of the structs to skip over to read the attribute they want to work with.
 
  • Like
Reactions: Manana and JamRules

JamRules

.....
OP
Member
Joined
Jan 9, 2014
Messages
527
Trophies
1
XP
2,204
Country
United States
Good that you seem to making some decent progress :yaypsp:

Out of interest what prompted you to start looking at the models.
Just for fun or are you aiming to export the models for use somewhere else, or perhaps even create mods for the game itself?
 
  • Like
Reactions: Manana and Kion

Kion

Member
Newcomer
Joined
Sep 2, 2019
Messages
18
Trophies
0
Age
35
Location
Fuji-shi, Shizuoka
Website
dashgl.com
XP
143
Country
Japan
Good that you seem to making some decent progress :yaypsp:

A little bit. I thought it was weird that the model files don't seem to have any internal pointers. But looking at some of these values seems to relative to the PSP memory, and are potentially baked in. So I guess I need to dump the PSP memory and see if the offsets match up.

Out of interest what prompted you to start looking at the models.

Originally I was thinking of looking into AOTI, but I don't like having to set up Wine. So I ended up going for the Phantasy Star Portable series.

Just for fun or are you aiming to export the models for use somewhere else, or perhaps even create mods for the game itself?

I'm generally on the "dump and document" side of things. Modding is satisfying at the expense of being time consuming.

Edit:

I looked into the offset problem some more. The meseta model is only 0x440 bytes in length, and doesn't seem to have any pointers. That is until I looked at the model in memory where pointers had been adjusted to the offsets inside memory. So I think the pointers aren't set relative to the file, but relative to their position in the nbl, and the whole unpacked nbl is copied directly into memory. Here's a traced out spreedsheet of the meseta file: https://drive.google.com/file/d/1z4rGt5XPrAlgdjeFigZA0BkOorj6A_t6/view?usp=sharing
 
Last edited by Kion,
  • Like
Reactions: JamRules

nanei6

New Member
Newbie
Joined
Sep 11, 2019
Messages
1
Trophies
0
Age
28
XP
69
Country
China
Simply using Waifu2x doesn't make the game look better.But we can completely replace the texture of the environment, like some other games that have been dealt with.As for reseting the weapons’ texture,it is a a major project that can only be done by hand.
2febc08065380cd7bfa113c1ae44ad34588281f0.png
ae38c1ef76094b369161afc1accc7cd98c109d44.png

The material in the picture comes from MonsterHunterPortable3rdHDRemake on github (sorry i can not post hyperlinks).I am using it here just for demonstration.
 

JamRules

.....
OP
Member
Joined
Jan 9, 2014
Messages
527
Trophies
1
XP
2,204
Country
United States
Simply using Waifu2x doesn't make the game look better.But we can completely replace the texture of the environment, like some other games that have been dealt with.As for reseting the weapons’ texture,it is a a major project that can only be done by hand.
View attachment 179211 View attachment 179212
The material in the picture comes from MonsterHunterPortable3rdHDRemake on github (sorry i can not post hyperlinks).I am using it here just for demonstration.

Thanks for the cool demonstration.
I agree, just using Waifu2x alone maybe doesn’t have the best results.
Just that it would be quite time consuming to create proper high quality replacements.

Definitely a good thought though.
 
  • Like
Reactions: Manana and Kion

JamRules

.....
OP
Member
Joined
Jan 9, 2014
Messages
527
Trophies
1
XP
2,204
Country
United States
Is it possible for you to share tools for extracting and repacking the fpb to replace audio? There was an undub attempted for 1 but it wasn't fully working.

One again, as asked previously, there are no plans to distribute this at any point.
Also the .fpb structure for 1 is different if I remember correctly.

Seems like an undub would be a pain to make.
Out of interest have you got a link to the project?
 

Site & Scene News

Popular threads in this forum

General chit-chat
Help Users
  • No one is chatting at the moment.
    Psionic Roshambo @ Psionic Roshambo: @SylverReZ, Indeed lol