Xenoblade X replace music

[masagrator]

Thank you for these tools, masagrator! I've been following your steps for bgm replacement, I tried to use bnk_extract.py in cmd, and it would not create the bgm folder. I would have to create the folder manually and then the script works.

Another thing, in HIRC.json, some hashes for songs like 00D5ED49 (Wir fliegen) aren't in the file. Would that mean it's unreplaceable with custom bgm?

Some songs are played via different instance that has loop turned on by default. Try to check first if you really need changing anything.

 

[Linh_Tran]

Thank you @masagrator.Work very well.

 

[NumaNuma]

Thanks masagrator for your efforts. I got it working after some troubleshooting now. Two things worth noting:

• vgmstream-cli will consider the wem files generated by NXAenc.exe (Convert.py) as invalid WWISE: wrong expected size (re-rip?). The game loads, plays and loops them fine however, so ignore the error message.
  
  
• pck_nonvoice_pack.py uses glob.glob, which returns files in a nonpredictible order (sorted by the OS), if you're lucky, this will match the order from the original .pck, if not, the generated pack will be invalid and the game will not play any music. On OSes with case-sensitive filesystems (Linux, MacOS) the latter is the case.
  @masagrator Attached is a git patch for your repo

 

[masagrator]

- Fixed bnk_extract.py to create automatically folder in case of detecting HIPC section
- Fixed Convert.py to properly calculate sizes, so now they pass vgmstream-cli checks
- Added sorting names in pck_nonvoice_pack.py, though if this is really an issue, it's not really fixed in other pck files than Music.pck since for its banks we have all names (imo sorting by names shouldn't be an issue, non sorting its hashes may create sometimes issues if it's using ordered set)

Thx @NumaNuma for pointing errors

Edit: Updated pck_nonvoice_pack.py to save informations about banks hashes in numerical order, which gives us the same order in which are stored in original pck files. Only difference is that it seems there is no fullproof logic in how bank files are put into pck as f.e. original System.pck stores data of second bank file first in blob.

Edit2: Updated bnk_extract.py to align hash names to be always 8-digit.

Another thing, in HIRC.json, some hashes for songs like 00D5ED49 (Wir fliegen) aren't in the file. Would that mean it's unreplaceable with custom bgm?

It was there, but stored as "0xd5ed49". Now bnk_extract.py properly stores name "0x00d5ed49"

Spoiler

    {
        "HircType": "MusicTrack",
        "Data1": "4a7b360900",
        "Sources": [
            "010011000249edd50074e8000008"
        ],
        "PlayListItems": [
            {
                "TrackID": 0,
                "Hash_filename": "0x00d5ed49",
                "EventID": 0,
                "PlayAt": 0.0,
                "BeginTrimOffset": 0.0,
                "EndTrimOffset": 0.0,
                "SrcDuration": 284984.9791666667
            }
        ],
        "Data2": "0100000000000000000000000000000000",
        "MusicSegmentID": 373300327,
        "Data3": "0001070000c84200000000000000000100000000000000000064000000"
    },
    {
        "HircType": "MusicSegment",
        "ID": 373300327,
        "Data1": "0600000000008dac99180000000000",
        "Props": [
            "070000c842"
        ],
        "Props>": [],
        "Data2": "03080000000000000100000000",
        "StateProps": [],
        "StateGroups": [],
        "RTPC": [],
        "Data3": "010000004a7b3609000000000040bf4000000000000000000000f042040401",
        "Stingers": [],
        "Duration": 284984.979166667,
        "Markers": [
            {
                "ID": 43573010,
                "Position": 0.0,
                "Name": ""
            },
            {
                "ID": 1539036744,
                "Position": 284984.979166667,
                "Name": ""
            }
        ]
    },

 

[NumaNuma]

Only difference is that it seems there is no fullproof logic in how bank files are put into pck as f.e. original System.pck stores data of second bank file first in blob.

The banks and streams are put into the .pck ordered by their hashes in ascending order.
I added logging code to pck_unpack.py and pck_unpack.py (see attachements).

The banks are mapped to names, so the repacker script retrieves the bank hashes and sorts them -> correct.
However, the streams are directly stored with their hashes as names. The problem is that glob.glob returns the stream files in arbitrary order. (gbatemp won't allow me to post url links, but just search How are glob.glob()'s return values ordered?)

On my system, they are returned in reverse (maybe sorted by modification date... - whatever), thus the generated .pck is invalid. (See diff unpack.txt repack.txt)

Long story short: glob.glob needs to be sorted(), else the repacker script relies on luck (actually the OS).

pck_nonvoice_pack.py : 15
stream_files = sorted(glob.glob("%s/STREAMS/*.wav" % os.path.normpath(sys.argv[1])))

 

[masagrator]

Ok, so streams are also important. I have done it different way because we don't want to sort it by name, but by hash. And when eventually we start to figure out names, it will be easier to add support for hashing names this way.
Also removed unnecesary code when sorting bank hashes to make loop faster.

 

[solomani]

Noob question - is this at a point it can be turned into a mod or not yet?

 

[masagrator]

Yes for

Noob question - is this at a point it can be turned into a mod or not yet?

Yes for BGM that is not used exclusively in cutscenes.

They have info in separate bank file that may affect some stuff. If someone is interested in that and there are issues after replacing Stream file, I can help with that.

 

[auradj]

There is something strange going on with looping.
I assigned all the right variables and the looping points seem to work in the game, but the full track is still playing.
The game simply shifts the beginning forward, so that the start of the track is offset by the loop start. And then it plays that over the already playing track.
(Barracks from my looping mod on gamebanana)

It also plays the track beyond the loop end point. I tried switching the beginning of the loop with the loop in the audio while using "BeginTrimOffset" to start from the beginning part, but loop end was only used to start the track again and not to stop the currently playing track.

 

[masagrator]

Explain to me exactly what you are doing, name of songs with their hashes (the one you replace and the one you are replacing with), data you are putting, etc. From what I got here it's only that it doesn't end loop when it should - which is not enough to help me find the issue. There is ton of data for which I'm not adding proper names for the sake of simplicity, so there may be something in those data I'm omitting necessary for what you are doing.

And do you know BGM that does exactly the thing you are trying to accomplish?

 

[auradj]

For barracks (3960FE96) in particular:
Length and Loopend is the same
I changed in HIRC.json:

{
    "ID": 43573010,
    "Position": 1244.416666667, (originally 0.0)
    "Name": ""
},

I replaced it in both "c0d5932300" and "6c905e0f00" Data1
Converted and packaged with your tools (most up to date version)
The Loopstart point was converted from 59732 samples.

If you want to look at the files directly or test it yourself, you can find them in the mod I posted on gamebanana.

I don't think there is a single track in this game that doesn't loop form start to end. The new tracks by Misaki Umase just hide it better by being designed to loop.

 

[masagrator]

Well, uncharted territory can be annoying.

Updated bnk_extract and bnk_packer:

• Add support for "MusicRanSeqCntr" entry type that is used by some MusicSegments for some reason. If MusicSegment has value stored in "MusicRanSeqCntrID" other than 0, it means it has this entry.
  • This entry has section called "Rules" from which one option is called "PlayPostExit", all of them are set to "true"

Dunno if this will help at all.

And if somebody wants to go into rabbit hole, here is full dump of bgm.bnk
You can open this in any browser, be aware that it may take few minutes before it becomes responsive. I recommend to use Edge as it seems Chrome has issues with this.
This dump was made by tool "wwiser", it's only for reading data, it doesn't support editing them.

 

[auradj]

That the intro plays before the loop point does, seem to happen to some other people:
https://www.audiokinetic.com/qa/1332/problem-using-music-playlist-container-to-loop-music
And even seems to be expected behavior:
https://www.audiokinetic.com/en/cou...approach_positioning_entry_and_exit_cues#read

Could you please expose "PlayPreEntry" from "AkMusicTransDstRule" similarily to how you exposed "PlayPostExit"? I want to see, if changing those two values can accomplish something.

 

[cucholix]

Black Tar (no lyrics)

 

[auradj]

I found an easier solution to the looping problem. Respect the current functionality and rework the audio so that it can still loop correctly with just the loop points.
I only had to rework 1/3 of the 40 tracks I looped and the rest can use the original audio file with just my looping points.
It can be downloaded from the Gamebanana page for Xenoblade X DE

 

[masagrator]

I doubt PlayPreEntry really will help, so I guess I will restore old scripts in repo so there won't be so many worthless data.

 

[auradj]

I don't really know why I did it, but here is the full track list of the music files:
https://docs.google.com/spreadsheets/d/1sW4VepE0UQVhOh5o0WbG_26iF9wfIyAvHyxzKFyuLjI

 

[masagrator]

It seems like if we replace Music.pck with an empty archive, it tries to find files outside of pck file.
In case of STREAM files it expects it as "romfs/sound/%u.wem" where %u is file hash as unsigned integer, in case of BANK files it's "romfs/sound/%s.bnk" and %s is the name of bank file.

So here is script that unpacks Streams with unsigned integers instead of hexadecimals.

https://github.com/masagrator/NXGameScripts/blob/main/XenobladeChroniclesX/pck_unpack_digits.py

And in attachments you have empty Music.pck
Remember to copy ALL .wem and .bnk files from STREAM and BANKS folders to "romfs/sound/"

 

[CrysDark]

so....still no colony 9 in NLA?

 

[NumaNuma]

so....still no colony 9 in NLA?

Here you go: xcxde_colony9.xd3
This is the delta you need to apply to the original Music.pck.
E.g. xdelta3.exe -d -s Music.pck xcxde_colony9.xd3 Music_mod.pck
Then copy Music_mod.pck romfs/sound/Music.pck
Or here's a GUI application you could use instead of xdelta3.