Gaming Xenoblade - Unused/Beta area in game?

[Helsionium]

At the end of the game, there is a cutscene that explains a lot of the game's background story/past events. For a few seconds, an area can be seen that I've never seen before in the game.

The map consists of two plains, stretched over the left shoulder of Bionis, his head can clearly be seen in the distance. A seemingly artifical tunnel and a natural cave connect the two plains. Two destroyed towers are facing Mechonis. It appears as if a colony once stood on the lower plain. There is a lake with two rivers on the upper plain (note the bridge near C in the 4th image).

Intrigued, I was looking through the game's data and surprisingly found that this area is actually (in theory at least) fully explorable, like any of the other areas, but for some reason it appears to be unused in the game (the cutscene I was talking about is pre-rendered, so the area appears to be technically unused). It is the map ma0801.

I also looked through all the other map files and I can safely say that there are no other unused maps, since I could identify all the other map files. This map is placed between the Makna Forest and the Eruyt Sea areas. The maps are otherwise in exactly the same order as they are visited in the game. This map appears to have two exits, both in the direction of Makna Forest/Eruyt Sea (towards Bionis' center)

For illustration purposes, a few screenshots of the model viewer I used - unfortunately it doesn't support rendering the textures, but the basic structure of the map can still be recognized.

View from Bionis' head: http://files.helsionium.eu/xb-map0801-1.png
View from the side: http://files.helsionium.eu/xb-map0801-2.png
View from behind: http://files.helsionium.eu/xb-map0801-3.png
Closeup of the upper plain from A: http://files.helsionium.eu/xb-map0801-4.png
Closeup of the platforms at E: http://files.helsionium.eu/xb-map0801-5.png

A - Watchtower on upper plain
B - Watchtower and Colony Ruins on lower plain
C - Upper plain
D - Colony wall (like at Colony 6)
E - Platforms accessible by bridges from the higher plain

Note that the graphical quality and level of detail of the map is actually the same as the rest of the game, it's just that the programs I used for converting and rendering are somewhat buggy.

The three exits - all caves - are marked in red: One near D, one on the lower plain near E, one on the higher plain near C. The road from B to E is marked in blue.

The map consists of two plains, stretched over the left shoulder of Bionis, his head can clearly be seen in the distance. A seemingly artifical tunnel and a natural cave connect the two plains. Two destroyed towers are facing Mechonis. It appears as if a colony once stood on the lower plain. There is a lake with two rivers on the upper plain (note the bridge near C in the 4th image).

So I want to ask you, is this area used in other cutscenes? Or is there even a way to reach it in normal gameplay or with cheat codes? It just seems a waste of time to construct such a beautiful map (similar to Bionis' Legs and Colony 6).

 

[Helsionium]

*** UPDATE ***

I was at last able to enter that stage in-game. And so I could share it with everyone, I decided to upload a video to YouTube.

Have a look at one of the most beautiful maps in the game:
http://www.youtube.com/watch?v=l86GCaqZuxY

Be sure to select at least 480p for acceptable quality and display the annotations if you want to read some of my comments.

What a waste not using this in the actual game...

 

[robbywu]

damn thats a cool map ^^ great job finding it

 

[Kodama]

@Helsionium

Can you give a quick rundown on what needs to be extracted and changed?

Tried extracting map.pkb but ended up with a bunch of *.dap & *.lod files which I don't know what to do with.

 

[Helsionium]

You don't even have to touch the map.pkb file for what I did. But I'll still explain the contents of the map.pkb file:

It consists of *.dap and *.lod files. Each *.dap file has a corresponding *.lod file, it is the number that directly follows the *.dap file. The *.lod files contain the textures for its corresponding *.dap file. It can be viewed with brresviewer, for example.The *.dap files are archives themselves. You can extract files from them with another BMS script. Each map is made up from several files, which I will explain briefly:

bdat.bin - contains various metadata, such as item and enemy lists
*.id - unknown, maybe some sort of object table
*.lgt - unknown, likely some lighting information
*.map - the actual map model. This is a standard BRRES file. Can be opened with brresviewer.
*.mpf - absolutely no idea
*.occ - unknown, maybe occlusion file for rendering optimations?
*.kp - a pack of object models, may contain animations as well. Can be opened with brresviewer.

The file I edited was "\dvddata\common\jp\bdat.bin" in static.arc. It contains several metadata tables, and I just switched two map names around in the map table. "ma0801" is the name of the beta map. I originally switched it with Hienter Graveyard ("ma0901") but you had to use the moonjump code to reach the beta map this way. The GameFAQs user DarkKamonG found out that it was better to switch with Eruyt Sea ("ma1001") instead and use the seventh warp point to access the beta map safely.

 

[Kodama]

Thanks for the quick reply. Works nicely.

It's a damn shame they never used this in the game.

A few screenshots;

Spoiler

 

[Griffion]

Every thread I go to is old. I know this stuff was looked at before localization was confirmed, but hopefully I can get some sort of help. Now with a NTSC-U copy of Xenoblade, I have extracted the game's contents into a folder on my PC. I was able to extract the character models from a .pkb compressed file using quickbms and Falo's script. But now I am trying to get the map models similar to Helsionium's pictures. I understand that there is a difference between the japanese and US versions of the games contents (don't know what), but if it is just as true that I don't even need to touch the map.pkb files to do what is seen above, I'd love the help immensely!

 

[Cyan]

The files format and location should be the same for all regions.
filename could change, but I suppose it's rare as they would have to edit a lot of source codes instead of just translating/localizing the script.

if you want to extract and export the map models, I can't help you. Never did it.
if you only want to visit that map ingame, there are Ocarina cheat codes to do that, no need to swap map files. (but you already read it)

 

[Griffion]

The files format and location should be the same for all regions.
filename could change, but I suppose it's rare as they would have to edit a lot of source codes instead of just translating/localizing the script.

if you want to extract and export the map models, I can't help you. Never did it.
if you only want to visit that map ingame, there are Ocarina cheat codes to do that, no need to swap map files. (but you already read it)

I was able to do it, FINALLY! I only found one person who said it. Maybe other people had, but it wasn't as clear. I wasn't aware you HAD to have the pkh file that goes with it's corresponding pkb file. And then that was compressed in static.arc. Yeah, I've already been on the map ingame with cheats. I'm thinking about a content restoration mod! But I'm a noob at this stuff. Now, I'm stuck with .bat files in one folder I don't know what to do with them. And it seems one pkb files has multiple pkh files and they all just exract the same thing, so I don't know why there are so many pkh files.

I was hoping to find the map images (with the landmarks and whatnot) to hopefully use it as a template for a map for the Bionis Left Shoulder.

Nonetheless, thanks for the reply on such an old thread. I think I'm stepping into territory nobody gone with this game, I'm not sure. I guess if I need additional help, I could find some modders who have already made expansions to Wii games.

 

[Griffion]

Well I'm stuck again if anybody happens upon this to help!

The file I edited was "\dvddata\common\jp\bdat.bin" in static.arc. It contains several metadata tables

As Helsionium said, he was able to edit the metadata, which I'm guessing is basically source code. But I'm not sure how to open this type of file (.bin). I need a compiler right? I've heard some people mention devkitPPC to do this. I guess I just want to know if I'm leaning in the right direction or not?

 

[Arras]

Well I'm stuck again if anybody happens upon this to help!



As Helsionium said, he was able to edit the metadata, which I'm guessing is basically source code. But I'm not sure how to open this type of file (.bin). I need a compiler right? I've heard some people mention devkitPPC to do this. I guess I just want to know if I'm leaning in the right direction or not?

A bin file is not just something you can normally open. bin is short for binary, and bin files can contain just about anything. Compilers/devkitPPC are for something else entirely (making a basically standalone homebrew program that runs on the Wii) afaik. You'll probably need poke around the file with a hex editor and see if you can figure out how it's structured etc. From what Helsionium said, that file contains (among other things) a list of the maps that are used in the game, and he just replaced one name with the unused one. Actually adding a completely new area without replacing any existing content would be a ton of work (and honestly, probably out of your league).

 

[Griffion]

A bin file is not just something you can normally open. bin is short for binary, and bin files can contain just about anything. Compilers/devkitPPC are for something else entirely (making a basically standalone homebrew program that runs on the Wii) afaik. You'll probably need poke around the file with a hex editor and see if you can figure out how it's structured etc. From what Helsionium said, that file contains (among other things) a list of the maps that are used in the game, and he just replaced one name with the unused one. Actually adding a completely new area without replacing any existing content would be a ton of work (and honestly, probably out of your league).

Thanks for the quick reply. Yeah, I had a feeling devkitPPC would be irrelevant to me. I was just doing some quick research and that is something that popped up a bit. Is there any recommended place to get this hex editor?

Actually adding a completely new area without replacing any existing content would be a ton of work (and honestly, probably out of your league).

xD What a blunt way to put it! I know it's going to be tough, but I think a content restoration mod is something I want to do. I mean, other people have obviously made great stuff like Project M, and Super Mario Galaxy 2.5. I don't know if my vision of this mod is in a league of it's own compared to those, but I'll try. I'm just getting started... literally. I am new to this stuff completely. The closest I'll be to actually learning anything helpful will be in a programming class I'll take along side my normal high school classes next year. Even then, I know a lot of it probably won't help me here. It's a dream right now yes, but nonetheless it's been cool seeing how this game was put together!

 

[Arras]

Thanks for the quick reply. Yeah, I had a feeling devkitPPC would be irrelevant to me. I was just doing some quick research and that is something that popped up a bit. Is there any recommended place to get this hex editor?

xD What a blunt way to put it! I know it's going to be tough, but I think a content restoration mod is something I want to do. I mean, other people have obviously made great stuff like Project M, and Super Mario Galaxy 2.5. I don't know if my vision of this mod is in a league of it's own compared to those, but I'll try. I'm just getting started... literally. I am new to this stuff completely. The closest I'll be to actually learning anything helpful will be in a programming class I'll take along side my normal high school classes next year. Even then, I know a lot of it probably won't help me here. It's a dream right now yes, but nonetheless it's been cool seeing how this game was put together!

It probably wouldn't be that far off from those two, but a lot of research has gone into those two games before said projects were a thing AFAIK. Brawl modding is maybe the biggest Wii game modding scene of them all and a lot of people spent a ton of time researching how that game works and how to edit it without breaking everything. For Mario Galaxy 2, someone wrote a full 3D level editor IIRC (Whitehole?). For Xenoblade there's nothing even near that level (again, afaik), so you'd have to make any tools you need yourself, figure out the way the data is stored, then add to it without breaking the entire game. It's probably not impossible, but it would require a very large time investment - especially considering you will also need to add enemies and quests to the new area if you don't want it to be completely empty, and make sure those are balanced and adjust the rest of the game for the exp you'd get there and everything.
For a hex editor, I honestly don't know. There's quite a few different ones out there. I've heard good things about HxD, but I have no idea which one would be the best for this kind of purpose.

 

[Griffion]

It probably wouldn't be that far off from those two, but a lot of research has gone into those two games before said projects were a thing AFAIK. Brawl modding is maybe the biggest Wii game modding scene of them all and a lot of people spent a ton of time researching how that game works and how to edit it without breaking everything. For Mario Galaxy 2, someone wrote a full 3D level editor IIRC (Whitehole?). For Xenoblade there's nothing even near that level (again, afaik), so you'd have to make any tools you need yourself, figure out the way the data is stored, then add to it without breaking the entire game. It's probably not impossible, but it would require a very large time investment - especially considering you will also need to add enemies and quests to the new area if you don't want it to be completely empty, and make sure those are balanced and adjust the rest of the game for the exp you'd get there and everything.
For a hex editor, I honestly don't know. There's quite a few different ones out there. I've heard good things about HxD, but I have no idea which one would be the best for this kind of purpose.

Well I plan to take my time with this, but I hope 5 years is enough considering I thought it be cool to have it out by the game's 10th anniversary. I've looked all over the place and know there is nothing to help me do this. NOTHING... but I was hoping I could get some others that love this game to help. I just don't know where to share it. That and I thought if I shared it too early someone would do it first, but I see it is more work than I thought. Now I just don't know where to share it. I probably only should once I know more myself.

I've mostly got the game conceptual stuff down like enemies, npcs, quests and more. I used HxD and I think I can work with it. Now to just figure out how to read this stuff. Yes I came unprepared ;P but you gotta start somewhere. I was hoping to start here in the overall process of learning game design for a career.

 

[Cyan]

If you understand the map format, you could make an editor (like a Regie editor).
To see and edit enemies placement, elements, grass, trees, etc. and you will have warp and cutscene activators too.
creating a map editor is probably the best way to add new areas.

Currently, you can extract or edit the 3D map, but not the texture/elements on it.


Bin files are either raw binary data (a picture, a sound, etc.), or a format created by the developers for their own games and doesn't have any program to open or edit them. the game knows how to read them, but that's all. without the game sources, you will have to understand the format yourself.
The binary data could be a single file with multiple files concatened :
"header+file1+file2+file3".bin
the header generally contains a Magic (static data, signature for that file that tell that this binary is what you expect), and information like "number of files inside that .bin" "size for each files" etc.

Exemple of binary file you could encounter:
MAG003001000100005aaaaaaaaaaaaaaaabbbbbbbbbbbbbbbbccccc
MAG0 = the magic which tells that this file is of type "mag0", and your program knows that mag0 files have :
2 bytes = number of files
following the number of file, you have 4bytes x number of files = size for each files
followed by the files themselves (texture, sound, etc.)

03 = 3 files
0010 = 16 bytes for the first file
0010 = 16 bytes for the second file
0005 = 5 bytes for the third file

then starts the actual file data (16 byte of a, 16 bytes of b, 5 bytes of c)


You have to understand the binary format, it's content to extract the files, and re-create a new binary file after you edited your map.
By editing that file you will probably make the binary a different file size that it was, you should ask yourself if that file size is not referenced in another file for the game to know how much data it has to buffer etc.


As for hexeditor, the two usual programs are : HxD and Hexworkshop.
But for binary structure like the exemple I gave above, 010 hexeditor is what you want.
unfortunately, only HxD is free.

 

[Griffion]

I'll... try and wrap my head around that! I think right now I'll use HxD to view the binary and check out 010 hexeditor when I'm ready. Do you know a good place I could learn about this stuf, I guess starting with reading and understanding binary format and construction?

 

[Cyan]

learning C/C++ is maybe useful.
check Structures, class, etc.
understanding data type, size and Struct will be useful as it's the way binary are read/written by the programs.

for example, data are stored in 2^x bit sizes.
8, 16, 32 or 64 bits.

in binary (in your hexeditor) you will see this:
8 bits : 00 to FF (in binary : 00000000 to 11111111 <- 8 bits)
16 bits : 0000 to FFFF
32 bits : 0000 0000 to FFFF FFFF
etc.
When looking for something in hex editor, you will never find a data like this "000", as size is either 2 (8 bit) or 4 (16bit), not 3 (12bit).

You will also need to understand the difference between little and big endian (the endianness)

 

[Griffion]

I've looked into C/C++ as languages to learn in game design since a lot of people mention them first. But I guess first I should learn to read it? Something like this:
http://www.instructables.com/id/Learn-Binary-The-easy-way-01000001-00000001/
or is there a different/easier way to start?

 

[Cyan]

I wouldn't recommend that link to "learn" how binary system is working. (not a binary file)

in file, binary refers as "raw" data.
in numeral system, binary refers as a two value (0 to 1) numeral system, as opposed to 10 values decimal (0 to 9) or 8 values octal (0 to 7) or 16 values hexadecimal system (0 to 15), or why not a 4 value system if you like it (0 to 3).

if you understand how decimal is working, all other systems are easy to understand.

In decimal :
hundreds | tenths | units
col 2 | col 1 | col 0
col 2 is the most significant bit (100 is more than 001)
col 0 is the least significant bit (1 is very low compare to other bits)

0 0 1 increment the unit and 1 becomes 2
0 0 2
etc.
0 0 9 up to 9
when you reach the max number of the unit, you increment the tenths (the adjacent and stronger significant digit next/left to the unit) and go back to 0 with the units (

think of it like a turn-counter on the bicycle or cars.
after one turn, you increment by one the wheel next to the smaller one, etc.)

0 1 0
0 1 1
etc.


To decode a base 10 (decimal) number:
You learned to count in decimal so there's no decoding to do for you, but it works like this:

The least significant bit (the unit) has less power than the other. it has 0 power.
the next one (tenths) has 10 times more power, because we are in a decimal counting system
so you can set each column as a power of 10.

10 power 2 (hundreds) + 10 power 1 (tenths) + 10 power 0 (units)

hundreds * 102 + tenth * 101 + unit * 100
245 (decimal) = (2 * 102 ) + (4 * 101) + (5 * 100) = (2 * 100) + (4 * 10) + (5 * 1) = 200 + 40 + 5 = 245 (decimal)
101 (decimal) = 1 * 102 + 0 x 101 + 1 * 100 = (1 * 100) + (0 * 10) + (1 * 1) = 100 + 0 + 1 = 101 (decimal)

in binary:
0 0 0
0 0 1 (you increment the lowest significant bit (the unit) and you reached the max value (which is 1)
so your next increment, you will also increment the adjacent and stronger significant bit next to the unit, and set the unit back to 0
0 1 0 <-- increment the second, go back to 0 with the unit
0 1 1
1 0 0
etc.

in binary, it's "2 possibilities", so you use 2x instead of 10x ( )
column2 * 22 + column1 * 21 + column0 * 20
101 (binary) = (1*22 ) + (0*21) + (1*20) = (1 * 4) + (0 * 2) + (1 * 1) = 4 + 0 + 1 = 5 (decimal)

column value * systemcolumn number


edit:
it's probably better explained here:
http://en.wikipedia.org/wiki/Binary_number#Decimal_counting

 

[Griffion]

learning C/C++ is maybe useful.
check Structures, class, etc.
understanding data type, size and Struct will be useful as it's the way binary are read/written by the programs.

for example, data are stored in 2^x bit sizes.
8, 16, 32 or 64 bits.

in binary (in your hexeditor) you will see this:
8 bits : 00 to FF (in binary : 00000000 to 11111111 <- 8 bits)
16 bits : 0000 to FFFF
32 bits : 0000 0000 to FFFF FFFF
etc.
When looking for something in hex editor, you will never find a data like this "000", as size is either 2 (8 bit) or 4 (16bit), not 3 (12bit).

You will also need to understand the difference between little and big endian (the endianness)

You confused me a "bit" here. Pun not intended. I don't understand the 2^x bit sizes. I can understand that those actual sizes are 8, 16, 32, etc. And also how you wrote 00 to FF (in binary: 00000000 to 11111111), is the 00 to FF supposed to be hex if 000... and 111... are binary?

There are 16 bits in total between:
8 here (0)(0)(0)(0)(0)(0)(0)(0) and
8 here (1)(1)(1)(1)(1)(1)(1)(1)
and 2 bytes (00000000) (11111111)
So wouldn't it be 8bits x 2bytes = 16 bit not just 8 bit
It seem that you only consider 1 byte either 00000000 or 11111111.

I understand your second comment about understanding decimal and binary numerical values and placement. I'm just lost on the bits part.