ROM Hack RELEASE Smash Ultimate (Partial) Data.arc Extractor/Inject Tool

[FIX94]

So I may not have a switch, but I do like smash music so I felt like taking a look at whatever came out those days, and when you extract the romfs of the game, you'll be greeted by one big data.arc. I have been working on this tool to start dealing with that file:
https://github.com/FIX94/ssbu-arc-extract
See the README at the bottom for details and the "releases" tab for a compiled executable.
If you just want the music and videos, you just have to drop into the same folder as the data.arc file of the game, run it, and wait for it to put lots of files into separate "lopus" and "webm" folders, 1357 .lopus files and 97 .webm files to be exact. The .lopus music files can be played back with any player that has the vgmstream plugin installed.
The .webm video files should just work in any standard video player, I tried it in mpv and they all seem to work just fine. Fun fact about those video files, some actual contain multiple audio tracks for both japanese and english voices that you can switch in whatever player you're using. I have no clue where names are located in the data.arc for the .webm files so they'll just be numbered sequentially by my tool.
If you want to extract other files, then for now you will have to hand over the tool a full filepath to extract, as there right now is no way that I have found to get the filenames directly out of the file because they are all hashed, so this tool takes your inputted filename, hashes it and extracts it based on that. Also, when you want to then later replace a file, you can call the tool and use "replace" in front of that path and it will try to inject it into the data.arc. See the github README for some example paths and much more details on the modes.

 

[Small Kat]

Very nice tool!

So theres still no exact way to mod SSBU with layeredfs right? Since you might need to pack the modded files back into that data.arc file.

 

[Gren22]

Very nice tool!

So theres still no exact way to mod SSBU with layeredfs right? Since you might need to pack the modded files back into that data.arc file.

Xenoblade 2 had a similar system with an ARD and ARH file, but we ended up coming up with a way to inject modded files back into them. Also when DLC came out we found the DLC titles had the files bare inside of the NCA instead of in an archive and we can just use LayeredFS on the DLC title. Who knows if it will be the same

 

[AlternativeZero]

Just tested and your tool works great, do you think it's possible to allow the extraction of images and other assets?

 

[Mat37]

Xenoblade 2 had a similar system with an ARD and ARH file, but we ended up coming up with a way to inject modded files back into them. Also when DLC came out we found the DLC titles had the files bare inside of the NCA instead of in an archive and we can just use LayeredFS on the DLC title. Who knows if it will be the same

Wait you can mod Xenoblade 2 ? Have you a link or something ?

 

[FIX94]

Just tested and your tool works great, do you think it's possible to allow the extraction of images and other assets?

well as it is right now I only parse one of many file tables, this particular table deals with the first 2gb of the file, those first 2gb contain the music and video files.
Though I am looking deeper into it and I am starting to understand a little more about the whole structure (headers, how big an element is, hashes, names etc.), it seems like the names arent directly in the data.arc but instead full paths and folder names are hashed in crc32 so there is a sort-of tree you can go through, like for example the file "sound/bank/enemy/se_enemy_yellowdevil.nus3bank" has a hash for that full name, then a hash for "nus3bank", another for "se_enemy_yellowdevil.nus3bank" and yet another for "sound/bank/enemy/", but the name itself is not in the arc file like that but I did figure that out by looking a bit at the game executable itself.

 

[BruhLookAtThis]

Very nice tool!

So theres still no exact way to mod SSBU with layeredfs right? Since you might need to pack the modded files back into that data.arc file.

I used a hex editor to copy my new .webm video & replace those same hex numbers in the data.arc file. Worked 80%ish, the audio is finicky but it works.

 

[FIX94]

just pushed this up to my github cause I think it may belong on there, also started adding lots of details into the main.c about how the file works in a bit more detail:
https://github.com/FIX94/ssbu-av-extract

 

[minibar]

I've got this if you want to take a look at it:
https://github.com/Thealexbarney/SmashNxTools

 

[FIX94]

Well, I just did even more documentation and added in some more features to the executable itself, you can now also give the executable a filepath as argument via command line and it will try and locate and then extract that file, compared to just running the tool without arguments which only extracts the first 2gb of the file, when you give a filename in it can actually get that file from anywhere, so it should allow pretty much full extraction if you know the filenames.
Also, if that file you gave in as argument is compressed, it will also attempt to automatically decompress it. The only thing left now really is to figure out where the filepaths are stored to allow actually easy extraction of everything rather than try and guess filenames. Still, progress is progress I guess

 

[Cozimo]

Well, I just did even more documentation and added in some more features to the executable itself, you can now also give the executable a filepath as argument via command line and it will try and locate and then extract that file, compared to just running the tool without arguments which only extracts the first 2gb of the file, when you give a filename in it can actually get that file from anywhere, so it should allow pretty much full extraction if you know the filenames.
Also, if that file you gave in as argument is compressed, it will also attempt to automatically decompress it. The only thing left now really is to figure out where the filepaths are stored to allow actually easy extraction of everything rather than try and guess filenames. Still, progress is progress I guess

Any progress with this? I'm hoping to get the raw nus3audio for my organised collection and I can't live with low-quality opus

 

[FIX94]

Any progress with this? I'm hoping to get the raw nus3audio for my organised collection and I can't live with low-quality opus

there is no conversion of the music going on, the nus3audio files are just another container on top of the opus files, thats just the internal format they used this time. all my tool does is extract them from the nus3audio container to make them actually playable with vgmstream.

 

[Cozimo]

there is no conversion of the music going on, the nus3audio files are just another container on top of the opus files, thats just the internal format they used this time. all my tool does is extract them from the nus3audio container to make them actually playable with vgmstream.

Oh so the nus3audio has no audio data discrepancies? No quality or sample rate differences?

Wow that's actually disappointing, they're using significantly lower quality audio this time around...

Is the 67kbps bitrate of the opus files configurable when you change the container? How is the bitrate determined here?

 

[FIX94]

Is the 67kbps bitrate of the opus files configurable when you change the container? How is the bitrate determined here?

Well it is a non-standard nintendo header that vgmstream deals with, I think that whatever math it does is not quite right, I think the actual bitrate of the file is more around 96kbps judging by the amount of samples in the file and its filesize.

 

[Cozimo]

Well it is a non-standard nintendo header that vgmstream deals with, I think that whatever math it does is not quite right, I think the actual bitrate of the file is more around 96kbps judging by the amount of samples in the file and its filesize.

I'm not sure if this is the case, I tested one of the returning tracks from Wii U and converted it from the raw format to a 67kbps opus and it ended up with the same number of samples and spectral image as the SSBU counterpart, if not ever so slightly more detailed.

67kbps opus is equivalent to about a 112kpbs MP3, which of course is an unfortunately low standard.

Thanks for your responses btw, I really appreciate the work you did on this tool as well.

 

[FIX94]

I'm not sure if this is the case, I tested one of the returning tracks from Wii U and converted it from the raw format to a 67kbps opus and it ended up with the same number of samples and spectral image as the SSBU counterpart, if not ever so slightly more detailed.

it would be rather strange though if you look at filesize, I did just for convert one file to wav, cut that at <STREAM_TOTAL_SAMPLES> as given by vgmstream (it loops the track once by default so keep that in mind) and then converted that cut file back to opus at 96kbps and that ends up with about the same size as the lopus file again.

 

[Cozimo]

it would be rather strange though if you look at filesize, I did just for convert one file to wav, cut that at <STREAM_TOTAL_SAMPLES> as given by vgmstream (it loops the track once by default so keep that in mind) and then converted that cut file back to opus at 96kbps and that ends up with about the same size as the lopus file again.

I think the audible differences are in the frequency range. The Wii U audio spectragraph shows a cap at 22kHz while the SSBU opus files are displaying a hard cut at 20kHz which leaves a noticeable loss in clarity. Though that's just the limitation of Opus.

96kbps would be certainly much better as it's practically transparent so I'll run some more personal checks for peace of mind, thanks again.

 

[FIX94]

96kbps would be certainly much better as it's practically transparent so I'll run some more personal checks for peace of mind, thanks again.

well I did run some more myself now cause it was so weird to me and it seems like for some reason, that 67kbps may actually be correct, and the file for some reason is followed by some more OPUS blocks right behind it that have a slighly different size and start like one minute into the song already (or are the last minute of the song, again?), I have no clue as to why this is even a thing to be honest, doesnt even seem to be played by vgmstream, I just hacked that together with a hex editor, so confusing...

 

[Cozimo]

well I did run some more myself now cause it was so weird to me and it seems like for some reason, that 67kbps may actually be correct, and the file for some reason is followed by some more OPUS blocks right behind it that have a slighly different size and start like one minute into the song already (or are the last minute of the song, again?), I have no clue as to why this is even a thing to be honest, doesnt even seem to be played by vgmstream, I just hacked that together with a hex editor, so confusing...

Would you be able to configure and force the bitrate higher during the container change process?

Is this something you will look more into?

 

[FIX94]

Would you be able to configure and force the bitrate higher during the container change process?

Sadly you cant just get more data out of something that was given to us like this, you cant make missing data suddenly appear out of thin air, that'd be pretty great eh After cutting the file to its true internal length it does very much appear to only be 67kbps, the math now sadly checks out.