Hacking Weird findings from that DQ trilogy switch port

[Rotted_Fish]

We'll start with some basics here:
The games were released on switch in 2019 to coincide with DQ11S. They were ports of the mobile versions with updated sprites that clashed with the backgrounds.
As of right now, the only mods available for it replace the character and enemy sprites with their mobile version counterparts.

Onto the topic at hand, a few days ago I decided to try my hand at messing with the music in these games, since the current versions use some pretty low quality midi tracks. At first, I was pleasantly surprised that all the sound files are in .wav format. The pleasantries unfortunately end there...
I couldn't open these files in any sound editor. Not Wavosaur, not Audacity (Except as Raw Data, but that results in audio that is...incorrect, we'll say) and none of the hex indicated any magic.
I decided to run this by someone else, and as you can see below--

,FileName,MAGIC,Extension,Matches,Time
0,/home/jetblack/Downloads/Sound/RDQ1_bgm01_overture.wav,RIFF****WAVE,.wav,False,2024-04-27 14:44:49.822259
1,/home/jetblack/Downloads/Sound/RDQ_me02_cursed.wav,RIFF****WAVE,.wav,False,2024-04-27 14:44:49.822259
2,/home/jetblack/Downloads/Sound/RDQ_me06_win.wav,RIFF****WAVE,.wav,False,2024-04-27 14:44:49.822259
3,/home/jetblack/Downloads/Sound/RDQ1_se_09.wav,RIFF****WAVE,.wav,False,2024-04-27 14:44:49.822259
4,/home/jetblack/Downloads/Sound/RDQ1_bgm02_name.wav,RIFF****WAVE,.wav,False,2024-04-27 14:44:49.822259
5,/home/jetblack/Downloads/Sound/RDQ1_se_56.wav,RIFF****WAVE,.wav,False,2024-04-27 14:44:49.822259
6,/home/jetblack/Downloads/Sound/RDQ1_bgm08_chika.wav,RIFF****WAVE,.wav,False,2024-04-27 14:44:49.822259
7,/home/jetblack/Downloads/Sound/RDQ1_bgm04_machi.wav,RIFF****WAVE,.wav,False,2024-04-27 14:44:49.822259
8,/home/jetblack/Downloads/Sound/RDQ_me03_death.wav,RIFF****WAVE,.wav,False,2024-04-27 14:44:49.822259
9,/home/jetblack/Downloads/Sound/RDQ1_bgm03_radatomu.wav,RIFF****WAVE,.wav,False,2024-04-27 14:44:49.822259
10,/home/jetblack/Downloads/Sound/RDQ1_se_25.wav,RIFF****WAVE,.wav,False,2024-04-27 14:44:49.822259
11,/home/jetblack/Downloads/Sound/RDQ2_bgm11_hokora.wav,RIFF****WAVE,.wav,False,2024-04-27 14:44:49.822259
12,/home/jetblack/Downloads/Sound/RDQ1_se_35.wav,RIFF****WAVE,.wav,False,2024-04-27 14:44:49.822259
13,/home/jetblack/Downloads/Sound/RDQ1_bgm14_finale.wav,RIFF****WAVE,.wav,False,2024-04-27 14:44:49.822259
14,/home/jetblack/Downloads/Sound/RDQ1_se_05.wav,RIFF****WAVE,.wav,False,2024-04-27 14:44:49.822259
15,/home/jetblack/Downloads/Sound/RDQ1_se_07.wav,RIFF****WAVE,.wav,False,2024-04-27 14:44:49.822259
16,/home/jetblack/Downloads/Sound/RDQ1_se_58.wav,RIFF****WAVE,.wav,False,2024-04-27 14:44:49.822259
17,/home/jetblack/Downloads/Sound/RDQ1_se_06.wav,RIFF****WAVE,.wav,False,2024-04-27 14:44:49.822259
18,/home/jetblack/Downloads/Sound/RDQ1_se_1d.wav,RIFF****WAVE,.wav,False,2024-04-27 14:44:49.822259
19,/home/jetblack/Downloads/Sound/RDQ1_se_13.wav,RIFF****WAVE,.wav,False,2024-04-27 14:44:49.822259
20,/home/jetblack/Downloads/Sound/RDQ1_bgm05_unknown.wav,RIFF****WAVE,.wav,False,2024-04-27 14:44:49.822259
21,/home/jetblack/Downloads/Sound/RDQ_me04_Inn1.wav,RIFF****WAVE,.wav,False,2024-04-27 14:44:49.822259
22,/home/jetblack/Downloads/Sound/RDQ_me09_silver.wav,RIFF****WAVE,.wav,False,2024-04-27 14:44:49.822259
23,/home/jetblack/Downloads/Sound/RDQ1_se_01.wav,RIFF****WAVE,.wav,False,2024-04-27 14:44:49.822259
24,/home/jetblack/Downloads/Sound/RDQ_me07_fairy.wav,RIFF****WAVE,.wav,False,2024-04-27 14:44:49.822259
25,/home/jetblack/Downloads/Sound/RDQ1_se_1c.wav,RIFF****WAVE,.wav,False,2024-04-27 14:44:49.822259
26,/home/jetblack/Downloads/Sound/RDQ1_se_1e.wav,RIFF****WAVE,.wav,False,2024-04-27 14:44:49.822259
27,/home/jetblack/Downloads/Sound/RDQ_me05_level.wav,RIFF****WAVE,.wav,False,2024-04-27 14:44:49.822259
28,/home/jetblack/Downloads/Sound/RDQ_me10_rainbow.wav,RIFF****WAVE,.wav,False,2024-04-27 14:44:49.822259
29,/home/jetblack/Downloads/Sound/RDQ1_se_53.wav,RIFF****WAVE,.wav,False,2024-04-27 14:44:49.822259
30,/home/jetblack/Downloads/Sound/RDQ1_se_22.wav,RIFF****WAVE,.wav,False,2024-04-27 14:44:49.822259
31,/home/jetblack/Downloads/Sound/RDQ1_se_4a.wav,RIFF****WAVE,.wav,False,2024-04-27 14:44:49.822259
32,/home/jetblack/Downloads/Sound/RDQ2_bgm08_requiem.wav,RIFF****WAVE,.wav,False,2024-04-27 14:44:49.822259
33,/home/jetblack/Downloads/Sound/RDQ1_bgm13_king.wav,RIFF****WAVE,.wav,False,2024-04-27 14:44:49.822259
34,/home/jetblack/Downloads/Sound/RDQ_me11_item.wav,RIFF****WAVE,.wav,False,2024-04-27 14:44:49.822259
35,/home/jetblack/Downloads/Sound/RDQ1_se_19.wav,RIFF****WAVE,.wav,False,2024-04-27 14:44:49.822259
36,/home/jetblack/Downloads/Sound/RDQ_me08_princess.wav,RIFF****WAVE,.wav,False,2024-04-27 14:44:49.822259
37,/home/jetblack/Downloads/Sound/RDQ1_se_0a.wav,RIFF****WAVE,.wav,False,2024-04-27 14:44:49.822259
38,/home/jetblack/Downloads/Sound/RDQ1_se_48.wav,RIFF****WAVE,.wav,False,2024-04-27 14:44:49.822259
39,/home/jetblack/Downloads/Sound/RDQ1_se_16.wav,RIFF****WAVE,.wav,False,2024-04-27 14:44:49.822259
40,/home/jetblack/Downloads/Sound/RDQ1_se_08.wav,RIFF****WAVE,.wav,False,2024-04-27 14:44:49.822259
41,/home/jetblack/Downloads/Sound/RDQ1_se_51.wav,RIFF****WAVE,.wav,False,2024-04-27 14:44:49.822259
42,/home/jetblack/Downloads/Sound/RDQ_me01_church.wav,RIFF****WAVE,.wav,False,2024-04-27 14:44:49.822259
43,/home/jetblack/Downloads/Sound/RDQ1_se_4f.wav,RIFF****WAVE,.wav,False,2024-04-27 14:44:49.822259
44,/home/jetblack/Downloads/Sound/RDQ1_se_0b.wav,RIFF****WAVE,.wav,False,2024-04-27 14:44:49.822259
45,/home/jetblack/Downloads/Sound/RDQ1_se_04.wav,RIFF****WAVE,.wav,False,2024-04-27 14:44:49.822259
46,/home/jetblack/Downloads/Sound/RDQ1_se_49.wav,RIFF****WAVE,.wav,False,2024-04-27 14:44:49.822259
47,/home/jetblack/Downloads/Sound/RDQ1_bgm07_battle.wav,RIFF****WAVE,.wav,False,2024-04-27 14:44:49.822259
48,/home/jetblack/Downloads/Sound/RDQ1_se_1b.wav,RIFF****WAVE,.wav,False,2024-04-27 14:44:49.822259
49,/home/jetblack/Downloads/Sound/RDQ1_se_03.wav,RIFF****WAVE,.wav,False,2024-04-27 14:44:49.822259
50,/home/jetblack/Downloads/Sound/RDQ1_se_59.wav,RIFF****WAVE,.wav,False,2024-04-27 14:44:49.822259

Nothing.

According to some of their deeper looking, there's no magic here.
Their exact words were--

most these files don't have any magic associated with them
99% of the time magic is at the start of the file
all I can say for certain is that those files are not wav
or maybe they wrote their own wav format :^) which sounds absolutely unhinged

So there you go. I'm generally at a loss on what to do here outside of toss it to the broader community in the hopes someone has a clue of what may be going on.

 

[masagrator]

Send few files somewhere and give links here.

 

[Rotted_Fish]

Send few files somewhere and give links here

once my account gets upgraded to Newcomer, I'll send that link

 

[masagrator]

You got Newcomer from the start. Restrictions were lifted when you sent 5th post.

Post automatically merged: Apr 30, 2024

Beside you can just cheat site by changing link so site cannot recognize it as link.

 

[Rotted_Fish]

You got Newcomer from the start. Restrictions were lifted when you sent 5th post.

Post automatically merged: Apr 30, 2024

Beside you can just cheat site by changing link so site cannot recognize it as link.

Forgive me, I'm just a boi

https://cdn.discordapp.com/attachme...c20593d2ab331972671fe8c024ec15771cf6254f9fb1&

 

[masagrator]

vgmstream works fine with them if you want to convert them to real WAV.
According to it, it's Nintendo DSP 4-bit ADPCM with custom DSP header used in Square Enix games.

Post automatically merged: May 1, 2024

So .wav files they use are separated to two types:
- type 0 uses Nintendo DSP 4-bit ADPCM format with custom header format used only in this trilogy (used in short files)
- type 1 uses Nintendo OPUS format with also custom header above it (used in long files)

I have done this to produce type 0 files:
https://github.com/masagrator/NXGameScripts/tree/main/Dragon Quest 1+2+3 Collection

It converts RIFF WAV. It works only with Windows.
Download all files to the same folder. In cmd:

python Convert_RIFF-WAV_to_SE-WAV.py path_to_file.wav

It will produce new file to "Converted" folder. It should work even if you replace with them type 1 files. Only con of using type 0 file over type 1 in long files is that they are few times bigger than if you would use type 1 (but don't look at size of original files, for example RDQ1_bgm08_chika.wav is type 1 and has bitrate around 24 kbps, so it's too low for anything more ambitious than midi. Difference is around 3:1 if you would use 128 kbps for OPUS)

 

[Rotted_Fish]

vgmstream works fine with them if you want to convert them to real WAV.
According to it, it's Nintendo DSP 4-bit ADPCM with custom DSP header used in Square Enix games.

Post automatically merged: May 1, 2024

So .wav files they use are separated to two types:
- type 0 uses Nintendo DSP 4-bit ADPCM format with custom header format used only in this trilogy (used in short files)
- type 1 uses Nintendo OPUS format with also custom header above it (used in long files)

I have done this to produce type 0 files:
https://github.com/masagrator/NXGameScripts/tree/main/Dragon Quest 1+2+3 Collection

It converts RIFF WAV. It works only with Windows, requires installing "pydub" from pip.
Download all files to the same folder. In cmd:

python Convert_RIFF-WAV_to_SE-WAV.py path_to_file.wav

It will produce new file to "Converted" folder. It should work even if you replace with them type 1 files. Only con of using type 0 file over type 1 in long files is that they are few times bigger than if you would use type 1 (but don't look at size of original files, for example RDQ1_bgm08_chika.wav is type 1 and has bitrate around 24 kbps, so it's too low for anything more ambitious than midi. Difference is around 3:1 if you would use 128 kbps for OPUS)

well I'll be damned...
Thanks! I'll see what I can crank out here

 

[sylandro]

vgmstream works fine with them if you want to convert them to real WAV.
According to it, it's Nintendo DSP 4-bit ADPCM with custom DSP header used in Square Enix games.

Post automatically merged: May 1, 2024

So .wav files they use are separated to two types:
- type 0 uses Nintendo DSP 4-bit ADPCM format with custom header format used only in this trilogy (used in short files)
- type 1 uses Nintendo OPUS format with also custom header above it (used in long files)

I have done this to produce type 0 files:

It converts RIFF WAV. It works only with Windows.
Download all files to the same folder. In cmd:

python Convert_RIFF-WAV_to_SE-WAV.py path_to_file.wav

It will produce new file to "Converted" folder. It should work even if you replace with them type 1 files. Only con of using type 0 file over type 1 in long files is that they are few times bigger than if you would use type 1 (but don't look at size of original files, for example RDQ1_bgm08_chika.wav is type 1 and has bitrate around 24 kbps, so it's too low for anything more ambitious than midi. Difference is around 3:1 if you would use 128 kbps for OPUS)

Thanks for this, it works great except for looping tracks.

Can type 0 files loop, or is loop info stored in the type 1 header? I tried adding loop points to each channel's dsp in VGAudio (-l x:x) into your script, but the audio doesn't loop properly in game. If it is exclusive to type 1, could you please provide some guidance on how the header is structured?

 

[masagrator]

Can type 0 files loop, or is loop info stored in the type 1 header?

It's not stored in custom header, it's stored in Nintendo OPUS header which belongs to type 1. I don't understand what custom header in type 1 represents at offsets 0x10, 0x14, 0x18, 0x1C, and 0x20, so I can't help make converter.

type 1 header looks like this:
0x0 int32: 0x00000001
0x4 int32: number of channels
0x8 int32: size of nintendo opus file (which is always = size_of_file - header_size)
0xC int32: size of header (always 0x40)
0x10 int32: unk
0x14 int32: unk
0x18 int32: unk
0x1C int32: unk
0x20 int32: unk
0x24 int32: sampling rate (it's always 48000)
up to 0x40: nulls
0x40: Nintendo opus file

 

[sylandro]

It's not stored in custom header, it's stored in Nintendo OPUS header which belongs to type 1. I don't understand what custom header in type 1 represents at offsets 0x10, 0x14, 0x18, 0x1C, and 0x20, so I can't help make converter.

type 1 header looks like this:
0x0 int32: 0x00000001
0x4 int32: number of channels
0x8 int32: size of nintendo opus file
0xC int32: size of header (always 0x40)
0x10 int32: unk
0x14 int32: unk
0x18 int32: unk
0x1C int32: unk
0x20 int32: unk
0x24 int32: frequency (it's always 48000)
up to 0x40: nulls
0x40: Nintendo opus file

I just found out that vgmstream can read this header type

Looking at the vgmstream repo (I can't post links because i'm new), in the file src/meta/opus.c , lines 497 to 517:

0x1C= total number of samples
0x14 = loop start sample
0x18 = loop end sample

not sure about the others though!

/* Square Enix variation [Dragon Quest I-III (Switch)] */
VGMSTREAM* init_vgmstream_opus_sqex(STREAMFILE* sf) {
    off_t offset = 0;
    int num_samples = 0, loop_start = 0, loop_end = 0, loop_flag;

    /* checks */
    if (read_u32be(0x00, sf) != 0x01000000)
        goto fail;

    /* .wav: original */
    if (!check_extensions(sf, "wav,lwav"))
        goto fail;
    /* 0x04: channels */
    /* 0x08: data_size */
    offset = read_32bitLE(0x0C, sf);
    num_samples = read_32bitLE(0x1C, sf);

    loop_flag = read_32bitLE(0x18, sf);
    if (loop_flag) {
        loop_start = read_32bitLE(0x14, sf);
        loop_end = read_32bitLE(0x18, sf);
    }

    return init_vgmstream_opus(sf, meta_OPUS, offset, num_samples, loop_start, loop_end);
fail:
    return NULL;
}

EDIT: By comparing files it looks like 0x10 is a looping flag (vgmstream is mistakenly reading 0x18): it always has 1 if the file has loop points, 0 if it is not looping. 0x20 seems to be consistently 40, maybe it's the header size again?

 

[masagrator]

0x1C= total number of samples

This doesn't match with my tools. For example RDQ1_bgm08_chika has 18'567'299 samples according to RIFF wav produced by vgmstream, but according to that info it has 9'815'675 samples.

 

[sylandro]

This doesn't match with my tools. For example RDQ1_bgm08_chika has 18'567'299 samples according to my tool, but according to that info it has 9'815'675 samples.

The audio data unnecessarily has two loops in it, so maybe they truncated it by setting a smaller sample num despite what the real one is (the loop end point comes before that, after all)

 

[masagrator]

Nvm, it seems like vgmstream applies loop by default to be repeated once. It is necessary to add "-i" argument to not apply loop.

Post automatically merged: May 4, 2024

And because of that I was wrong with bitrate. It's 48 kbps originally.

Here's package for testing. pydub required. It relies on libopus that I have installed permanently on Windows and it's possible that it may not detect included libopus-0.dll, that's why for now for tests. By default it uses the same bitrate which is 48 kbps. If you want to change bitrate, edit 70th line of Python file where is "120", this is frame size in bytes. By doubling the size you will get double the bitrate, etc.

python Convert_RIFF-WAV_to_NOPUS.py file.wav start_loop_in_samples end_loop_in_samples

If you don't want loop to be applied, write two zeroes at the end.
Otherwise you must provide start loop and end loop in hexadecimal, f.e.

python Convert_RIFF-WAV_to_NOPUS.py RDQ1_bgm08_chika.wav 0x100000 0x200000

 

[sylandro]

Nvm, it seems like vgmstream applies loop by default to be repeated once. It is necessary to add "-i" argument to not apply loop.

Post automatically merged: May 4, 2024

And because of that I was wrong with bitrate. It's 48 kbps originally.

Here's package for testing. pydub required. It relies on libopus that I have installed permanently on Windows and it's possible that it may not detect included libopus-0.dll, that's why for now for tests. By default it uses the same bitrate which is 48 kbps. If you want to change bitrate, edit 70th line of Python file where is "120", this is frame size in bytes. By doubling the size you will get double the bitrate, etc.

python Convert_RIFF-WAV_to_NOPUS.py file.wav start_loop_in_samples end_loop_in_samples

If you don't want loop to be applied, write two zeroes at the end.
Otherwise you must provide start loop and end loop in hexadecimal, f.e.

python Convert_RIFF-WAV_to_NOPUS.py RDQ1_bgm08_chika.wav 0x100000 0x200000

It works mostly fine, if I increase the frame size higher than 120, the game crashes right after the loop point (I wonder if the 0x04 magic number could be related?). Also forgot to mention that for non-looping files, loop_end needs to have the same value as sample_num, otherwise the music never plays.

 

[masagrator]

I think it may be related to other value.
Check this script. Also updated loop_end to match sample_count if loop_end equals 0.

 

[sylandro]

I think it may be related to other value.
Check this script. Also updated loop_end to match sample_count if loop_end equals 0.

Still crashes unfortunately!

I also tested using VgAudioCli to encode an Nxopus file (VgAudioCli.exe -c file.opus file.lopus), then I manually added the header. It sounds fine the first time and crashes after loop - similar to your script.

 

[masagrator]

Try now this. I have remembered that one game had issues with playing audio if there was null after frame size, I don't know what this value does, but I was adding there completely random value and fixed the issue.

 

[sylandro]

Try now this. I have remembered that one game had issues with playing audio if there was null after frame size, I don't know what this value does, but I was adding there completely random value and fixed the issue.

Still crashes after looping when frame size is higher than 120.

Another thing I noticed is that even though I set the exact same loop points as the source ogg file I am converting, the song loops at a different point...

 

[masagrator]

Then I can only assume that's an issue with their engine. Dunno why they have decided to implement their own loop logic when Nintendo OPUS supports it natively.

 

[sylandro]

Then I can only assume that's an issue with their engine. Dunno why they have decided to implement their own loop logic when Nintendo OPUS supports it natively.

I found how to fix it: 0x4A should have the frame size + 8, the script was hardcoding it with 128. It looks like the game's code is using that value to seek where the sample is after looping.