Hacking DSPADPCM (.dsp audio) Encoding Made Easy!

[jackoalan]

Hello everyone!

I'm a game reverse-engineering and modding enthusiast focusing on games by Retro Studios (i.e. the Metroid Prime games and recent Donkey Kong titles). For some time, I've been wanting to insert custom music and SFX by implementing a DSPADPCM encoder (which many titles seem to use for audio compression).

Nintendo provides developers with DSPADPCM.EXE and dsptool.dll to perform the encoding. This tool and codec is available for GameCube, Wii and Wii U systems. The codec is based on an IEEE-defined ADPCM predictor combined with 8 coefficient-pairs selected via filtered discrete-fourier-transform. The result is 8-byte ADPCM frames with 1-byte header at 4 bits-per-sample. A mono 16-bit PCM stream ends up being ~28.5% the original size in DSPADPCM.

@delroth pointed me to this implementation in BrawlTools which appears to contain reverse-engineered versions of the original algorithms:
https://code.google.com/p/brawltools2/source/browse/trunk/BrawlLib/Wii/Audio/AudioConverter.cs

I spent much time trying to grok the algorithms, identifying the mathematical basis and transforming the reverse-engineered code into something more readable and optimized:
https://github.com/jackoalan/gc-dspadpcm-encode/blob/master/grok.c

I'm pleased with the results! Generated streams sound great when injected into actual game images. To make the encoding process easier, I've created a fork of Audacity to import/export .dsp files as well as .dsp derivatives used by Retro. Part of the .dsp format includes metadata for a loop-region as well; this is visualized as a label-track in Audacity. The encoder can easily be applied to layouts used by other games as well, so long as they derive from DSPADPCM:

Check it out here:
https://github.com/jackoalan/audacity/releases

 

[Antidote]

Nice work jack DSPADPCM turned out to be surprisingly easy once we had an idea where to go with it.

 

[jackoalan]

Ah yes! Due thanks to @Antidote and Parax from M2K2 for supplying test material and plenty of background information on relevant file formats.

 

[Shrinefox]

This is awesome, made my life a lot easier modding sound files for Gamecube games.

However, Audacity (seemingly at random) decides not to load anything when I click on a dsp sometimes, even though it's listed in the recent files after I do so.
I don't know if this is an Audacity glitch or maybe something wrong/incompatible with the files (maybe too short in length?)

I even tried going into my appdata and temp audacity folders and clearing it out but it made no difference trying to open some of them.

 

[jackoalan]

If you notice it happen consistently for certain individual files, then you might be dealing with a different container type.

Not all .dsp files are made with Nintendo's usual utility. In most cases the same codec is used, but it's also arbitrarily split for stereo or other multi-track usage. For instance Metroid Prime 2 comes with .dsp files that begin with 'RS03' and uses ~2sec per-channel interleaving for fluid use with the DVD -> ARAM DMA mechanic in the GameCube specifically.

This fork is specifically made for the containers used by Retro Studios (found in the Prime Trilogy and both new DKC games).

Depending on the game you're dealing with, the container will need to be added to the fork.

 

[koz]

Does your DSPADPCM code allow streaming to a wii remote for sound output.

This would be a great achievement if you can make it work.

 

[jackoalan]

According to this http://wiibrew.org/wiki/Wiimote#Sound_Data_Format, the Wiimote uses Yamaha ADPCM.

It should be noted that DSPADPCM is not well-suited for real-time encoding (which is how the Wiimote stream works). The entire music/sfx track needs to be mixed and encoded ahead of time, using some rather complex computations.

 

[Shrinefox]

If you notice it happen consistently for certain individual files, then you might be dealing with a different container type.

What's odd is that they all came from the same file in the same game, though (Thousand Year Door). It'd be strange if they made different sound effects with a different utility, so maybe there's just something variable about that specific type of the format that only breaks sometimes. Thanks for the heads up though, didn't know it was made for a specific game.

 

[jackoalan]

Yea, it's common for SFX and other event cues to be short, standard mono .dsp files

On the other hand, GameCube music will often be a fully-produced stereo mix, requiring a stereo container of some sort (not standardised by Nintendo)

 

[Antidote]

Retro cheated a little by having both left and right channels as separate files in MP1.

 

[jackoalan]

New Version!!

https://github.com/jackoalan/audacity/releases

This updates Audacity to the current dev branch and fixes a bug discovered with the help of @Shrinefox: .dsp files with less than 1024 samples are now correctly identified as valid input.

 

[TGE]

jackoalan, would you mind implementing the Paper Mario: The Thousand Year Door's .stm stereo dspadpcm container? It's a pretty basic container without much added to it.
The structure is as follows:

// size = 64 bytes
struct STMHeader
{
    u16 field00;                // 0x00, always 512, possibly a version number of 2.00
    u16 sampleRate;             // 0x02, usually 32000
    u32 numChannels;            // 0x04, usually 2 (could also be initial offset)
    u32 adpcmData2Offset;       // 0x08, byte offset to the adpcm data of the second channel (relative to the end of the this header + all channel headers)
    u32 adpcmLoopStartOffset;        // 0x0C, byte loop start offset, equal to 0xFFFFFFFF if not used (relative to the start of the data itself, not the headers)
    u32 adpcmData2OffsetAux1;   // 0x10, same value as adpcmData2Offset
    u32 adpcmData2OffsetAux2;   // 0x14, same value as adpcmData2Offset
    u32 adpcmLoopStartOffsetAux1;    // 0x18, if adpcmLoopOffset != 0xFFFFFFFF then same value else value is 0
    u32 adpcmLoopStartOffsetAux2;    // 0x1C, same as adpcmData2OffsetAux1
    u8  padding[32];            // 0x20, 32 bytes unused
}

// after this numChannels * standard dspadpcm header follows
// right after that the adpcm data of the first channel starts
// if there's a second channel, then you must seek to the adpcmData2Offset to get the data for the second channel
// there appears to be 32 byte alignment between the adpcm data blocks
// and some unknown alignment (or just a fixed amount of padding) at the end of the file
// what's missing from this is which value indicates the loop end offset
// so far, all the files in the game loop from the loop start point to the end so
// there's a chance that adpcmData2OffsetAux1 or adpcmData2OffsetAux2 is the loop end offset

The game seems to completely ignore the looping contexts of the embedded dsps and instead uses the values specified in the stm header.
Samples (includes a looped and non-looped one) http://puu.sh/nVH16.7z

 

[jackoalan]

Here's what i've got so far for .stm support:
https://github.com/jackoalan/audacity/releases/tag/v2.1.3-dev-stmtest

I'm a bit confused by the two files though. Both seem to set 'adpcmLoopStartOffset' as 0xffffffff. I'm assuming this means that both of these are non-looping files.

Judging by each channel's .dsp header, 'btl_die1_32k.stm' loops, but it's the whole file as the loop region.

I implemented looping from the .stm based on your struct comments. Let me know if this functions correctly with files that definitely loop.

 

[TGE]

Ah, great work. Thanks a lot.
Sorry about them both not looping, I accidentally put the wrong file in the archive. Even still, it works perfectly for actually looped tracks.
One weird issue though. Testing it ingame gave me some trouble. Basically, if adpcmData2OffsetAux1 & 2 aren't exactly equal to adpcmData2Offset, the channels will desync in stereo and produce garbage audio in mono. Same goes for the other 2 aux variables.
Seeing as how none of the tracks in the game have the loop end point not set to the end I'm not sure if this is a format limitation.
But aside from that, the files work perfectly ingame.

 

[jackoalan]

OK, changed those aux fields to match

let me know if this works correctly, and I'll make it part of the general release:
https://github.com/jackoalan/audacity/releases/tag/v2.1.3-dev-stmtest

 

[libertyernie]

I'm looking at the differences between your grok.c and AudioConverter.cs (mostly as a way for me to practice C/C++, honestly) and I noticed this part in particular:

        /* Set initial scale */
        for (scale=0 ; (scale<=12) && ((distance>7) || (distance<-8)); scale++, distance>>=1)
            scale = (scale <= 1) ? -1 : scale - 2;

BrawlLib has a semicolon right after the for loop, so the loop gets run until it stops, and then it runs the line afterwards. Your code treats that next line as the body of the loop. In comparing the audio files, I don't see much of a difference, so I was wondering which is correct (or at least better.)

Unrelated to the above: I also converted this code back to C# and slotted it into BrawlLib - see https://github.com/libertyernie/brawltools/commit/e1b40ddc383dfa487320e294e172d8b8d1406d92 for the side-by-side comparison.
Trying it out, it seems like there's more difference between the original and encoded audio in this version than in the original BrawlLib code, but I can't tell by listening - only by subtracting the original waveform from the encoded-and-then-decoded waveform in Audacity (Invert on one, then Mix and Render) and comparing them.

EDIT: I think I've got it working better now. Here's the changes I made: https://github.com/libertyernie/brawltools/commit/d6708a356d3cc3b9d133c37e94bfb5721e8593e2
I'm guessing the pass-by-reference issue was when I ported it to C# and implanted tvec as a struct, and the pcmHistBuffer issue was also a change I made since I couldn't allocate arrays in the same way (so your code is probably fine).

 

[jackoalan]

Ah cool! Thanks for the eagle-eyed catch! I'll merge the changes in my using repos soon

 

[Elijah6133]

for some reason it wont open on my laptop it keeps giving the following error The application was unable to start correctly (0xc000007b) Ill keep trying to figure it out, but if you know of a way to fix this could you let me know?

 

[Elijah6133]

I still cant get it to work. is it because i have a x64 operating system?

 

[Elijah6133]

ok got it to open now, but is there any chance you could make a tutorial on youtube showing all the different featuresÉ