Hacking Command-line tools for Sega VC injection (U8, CCF, WTE)

libertyernie

Well-Known Member
OP
Member
Joined
Apr 6, 2011
Messages
112
Trophies
0
XP
334
Country
United States
So when I do a Virtual Console injection, I want everything to be perfect - icons, names, down to the manual if I can. In the past I've done a lot of this with the HowardC tools - they don't run on my Windows 10 install, but I have a Windows XP virtual machine I can use. The bigger problem, though, is that every time I make a change to the manual, I have to recompile a CCF, recompile another CCF, and recompile a U8, and then repack the WAD. It gets time-consuming when I just want to test small changes in Dolphin and I'm afraid I'll miss something.

So in order to make the system mostly automated, I wrote a set of ~9 .NET console apps, which you can get at https://github.com/libertyernie/brawllib-wit/releases. Most of these are built on BrawlLib.dll from BrawlCrate - the library is used to handle U8 archives, TEX0 and TPL textures, and endian conversions. It also relies on memory-mapped files and often assumes pointers are 32-bit so it's pretty much Windows-only at the moment, as far as I know.
  • u8pack - packs everything in the current directory into a U8 archive
  • u8unpack - unpacks everything in a U8 archive into the current directory (not recursively - archives inside archives just give you another U8 file)
  • texextract - converts a TEX0 texture to PNG
  • texreplace - replaces the texture in a TEX0 file with the contents of a PNG (brings up a conversion dialog)
  • wteconvert - converts between WTE (found in Sega's misc.ccf) and TEX0 by performing a header change
  • imgoverlay - overlays one image onto another image and then saves it to a third file, using the .NET System.Drawing namespace.
  • openingextract - extracts one of the three .arc files found in 00000000.app.
  • openingimet - extracts the name of a game in the specified language from 00000000.app.
  • bannercfgfromtxt - replaces the contents of Sega's banner.cfg.txt with strings read in from other files in the current directory.
I've also forked ccf-tools to fix a couple "bugs" (that were preventing the injection from working) and to allow it to compile with Visual Studio and use .NET's DeflateStream instead of zlib (because I don't know how to link libraries properly on Windows - I'm a .NET developer, and when you have a hammer, everything looks like a nail).

All of these apps are built on .NET Framework 4.7.2, which has been included with Windows for the past couple years at least.

What these apps can't do:
  • Replace the game's title, title ID, or Wii Menu icons (CustomizeMii can do all this, though)
  • Replace the game's title, release year, or number of players in the banner (I still recommend HowardC's tools for this - if you can't run them on your own OS, and you want your injects to look good, it's worth setting up a VM to do it)
Check SaulFabre's guide for how to use CustomizeMii and vcbrlyt.

Here's a sample batch file / the project that inspired me to do this in the first place: injecting Sonic 3D Blast Director's Cut into the original Sonic 3D Blast WAD.

Code:
@echo Before running this batch file:
@echo 1. Extract the Sonic 3D Blast (North American region) WAD file.
@echo 2. Open the WAD file with CustomizeMii and extract banner.brlyt from the
@echo    Layout tab.
@echo 3. Use HowardC's VCbrlyt9.0 to edit the banner.brlyt file, and fill in
@echo    the appropriate title (Sonic 3D Blast: Director's Cut) and release
@echo    year (2017) for all languages. If vcbrlyt.exe does not run on your
@echo    computer, try installing Windows XP (or even Windows 98) to a virtual
@echo    machine and running it there.
@echo 4. Go back to CustomizeMii's Layout tab and replace the banner.brlyt
@echo    with the one you've created. Also update the titles on the Title tab
@echo    (but keep them short - maybe use something like "Sonic 3D Blast DX")
@echo    and the Title ID on the Options tab (this batch file will assume you
@echo    have chosen M3DE as your Title ID). In the Banner and Icon tabs,
@echo    replace VCPic and IconVCPic with a new screenshot of the game's title
@echo    screen (CustomizeMii should resize the images for you). Save the
@echo    resulting WAD in this folder as "input.wad".
@echo.
@echo You should also look through the batch file before running it to make
@echo sure you have any required programs downloaded first (or edit the file
@echo to skip those sections if you'd prefer).
@echo.
@echo To quit out early, press Ctrl+C.

pause

@REM Unpack the input WAD into the "wadunpack" folder.
@REM https://github.com/libertyernie/wad-tools/releases
wadunpacker.exe input.wad wadunpack

@REM Extract the game's titles in various languages (as shown in CustomizeMii) to text files.
@REM This will be used for the save data title later. bannercfgfromtxt expects files with these particular names.
@REM https://github.com/libertyernie/brawllib-wit/releases
mkdir names
openingimet.exe wadunpack\00000000.app ja > names\JP.txt
openingimet.exe wadunpack\00000000.app en > names\EN.txt
openingimet.exe wadunpack\00000000.app de > names\GE.txt
openingimet.exe wadunpack\00000000.app fr > names\FR.txt
openingimet.exe wadunpack\00000000.app es > names\SP.txt
openingimet.exe wadunpack\00000000.app it > names\IT.txt
openingimet.exe wadunpack\00000000.app nl > names\DU.txt

@REM Extract VCIcon from banner.arc so we can use it to make new save data icons later on.
openingextract.exe wadunpack\00000000.app banner banner.arc
mkdir banner
cd banner
..\u8unpack.exe ..\banner.arc
cd ..
del banner.arc
texextract.exe banner/arc/timg/VCPic.tpl VCPic.png
rmdir /s banner

@REM Extract 00000005, which contains all of the data we'll need to update.
mkdir 00000005_contents
cd 00000005_contents
..\u8unpack.exe ..\wadunpack\00000005.app
cd ..

@REM Unpack the data.ccf file found inside 00000005. The CCF format is used in some of Sega's releases.
@REM https://github.com/libertyernie/ccf-tools/releases
mkdir ccf_contents
cd ccf_contents
..\ccfex.exe ..\00000005_contents\data.ccf
cd ..

@REM Use xdelta to apply the Director's Cut patch to the Sonic 3D Blast ROM image found inside data.ccf.
@REM https://github.com/jmacd/xdelta-gpl/releases/tag/v3.0.11
@REM https://www.romhacking.net/hacks/3810/
xdelta3-3.0.11-i686.exe -d -s ccf_contents\Sonic3D.SGD SONICDX.xdelta SONICDX.SGD
del ccf_contents\Sonic3D.SGD
move SONICDX.SGD ccf_contents\Sonic3D.SGD

@REM Grab a modified version of the manual from libertyernie's github.
@REM Wii Virtual Console manuals are just websites written in HTML and they're easy enough to modify once you figure out how to extract and replace them.
@REM Feel free to modify or delete this section if you don't have Git installed.
git clone https://github.com/libertyernie/custom-wii-vc-operations-guides custom-wii-vc-operations-guides --depth 1
cd "custom-wii-vc-operations-guides\Sonic 3D Blast Director's Cut"
del ..\..\ccf_contents\man.arc
..\..\u8pack.exe ..\..\ccf_contents\man.arc
cd ..\..
rmdir /s custom-wii-vc-operations-guides

@REM Extract misc.ccf found inside data.ccf.
mkdir misc_contents
cd misc_contents
..\ccfex.exe ..\ccf_contents\misc.ccf
cd ..

@REM Convert the save file banner to PNG, overlay the VCIcon screenshot onto it, then convert it back.
@REM The PNG to TEX0 conversion will show a dialog - just hit OK, the default settings should be acceptable.
wteconvert.exe misc_contents\banner.wte banner.tex0
texextract.exe banner.tex0 banner1.png
imgoverlay.exe banner1.png VCPic.png 8 8 64 48 banner2.png
texreplace.exe banner.tex0 banner2.png
wteconvert.exe banner.tex0 misc_contents\banner.wte
del banner.tex0 banner1.png banner2.png

@REM Convert the save file icon to PNG, overlay the VCIcon screenshot onto it, then convert it back.
@REM The PNG to TEX0 conversion will show a dialog - just hit OK, the default settings should be acceptable.
@REM To save effort, we'll also replace the other frames of the icon's animation (the one that usually goes back and forth between the game's logo and the console's name).
wteconvert.exe misc_contents\01.wte icon.tex0
texextract.exe icon.tex0 icon1.png
imgoverlay.exe icon1.png VCPic.png 2 9 44 31 icon2.png
texreplace.exe icon.tex0 icon2.png
wteconvert.exe icon.tex0 misc_contents\01.wte
wteconvert.exe icon.tex0 misc_contents\02.wte
wteconvert.exe icon.tex0 misc_contents\03.wte
wteconvert.exe icon.tex0 misc_contents\04.wte
wteconvert.exe icon.tex0 misc_contents\05.wte
wteconvert.exe icon.tex0 misc_contents\06.wte
del icon.tex0 icon1.png icon2.png

del VCPic.png

@REM Modify banner.cfg.txt using the names we extracted earlier.
@REM banner.cfg.txt needs to be big-endian UTF-16. bannercfgfromtxt takes care of that for us, though.
cd names
..\bannercfgfromtxt.exe ..\misc_contents\banner.cfg.txt
cd ..
rmdir /s names

@REM Recompile misc.ccf.
@REM misc.ccf needs to be rebuilt without any compression (unlike data.ccf, where LZ77 compression is allowed).
cd misc_contents
..\ccfarcraw.exe 01.wte 02.wte 03.wte 04.wte 05.wte 06.wte banner.cfg.txt banner.wte
cd ..
del ccf_contents\data.ccf
move misc_contents\out.ccf ccf_contents\misc.ccf
rmdir /s misc_contents

@REM Recompile data.ccf.
cd ccf_contents
..\ccfarc.exe Opera.arc Sonic3D.SGD config home.csv man.arc misc.ccf patch
cd ..
del 00000005_contents\data.ccf
move ccf_contents\out.ccf 00000005_contents\data.ccf
rmdir /s ccf_contents

@REM Recompile 00000005.
del wadunpack\00000005.app
cd 00000005_contents
..\u8pack.exe ..\wadunpack\00000005.app
cd ..
rmdir /s 00000005_contents

@REM Recompile the WAD.
copy common-key.bin wadunpack\common-key.bin
cd wadunpack
..\wadpacker.exe 000100014d334445.tik 000100014d334445.tmd 000100014d334445.cert ..\output.wad
cd ..
rmdir /s wadunpack

@echo File written to output.wad.
pause
 
Last edited by libertyernie,
  • Like
Reactions: SaulFabre

libertyernie

Well-Known Member
OP
Member
Joined
Apr 6, 2011
Messages
112
Trophies
0
XP
334
Country
United States
Similar to what I did with ccf-tools (see first post), I've also tracked down the source code to (a couple different versions of) wadpacker and wadunpacker, and rebuilt them as C++/CLI applications. The versions I'd been using had been built against a twelve-year-old version of Cygwin (although, to be fair, they did still work) -- these versions are built against Visual C++ 2019 and .NET Framework 4.7.2 instead. I rewrote the hash and crypto functions so they call out to .NET instead of OpenSSL, so they should work on most Windows 10 systems (and possibly earlier) without needing any extra DLLs that might cause conflicts.

https://github.com/libertyernie/wad-tools/releases
 
  • Like
Reactions: SaulFabre

SaulFabre

Wii scene/hacking & Wii VC well-known guy
Member
Joined
Feb 6, 2019
Messages
1,105
Trophies
1
Age
22
Location
Ecuador
Website
saulfabreg-wiivc.blogspot.com
XP
2,432
Country
Ecuador
So when I do a Virtual Console injection, I want everything to be perfect - icons, names, down to the manual if I can. In the past I've done a lot of this with the HowardC tools - they don't run on my Windows 10 install, but I have a Windows XP virtual machine I can use. The bigger problem, though, is that every time I make a change to the manual, I have to recompile a CCF, recompile another CCF, and recompile a U8, and then repack the WAD. It gets time-consuming when I just want to test small changes in Dolphin and I'm afraid I'll miss something.

So in order to make the system mostly automated, I wrote a set of ~9 .NET console apps, which you can get at https://github.com/libertyernie/brawllib-wit/releases. Most of these are built on BrawlLib.dll from BrawlCrate - the library is used to handle U8 archives, TEX0 and TPL textures, and endian conversions. It also relies on memory-mapped files and often assumes pointers are 32-bit so it's pretty much Windows-only at the moment, as far as I know.
  • u8pack - packs everything in the current directory into a U8 archive
  • u8unpack - unpacks everything in a U8 archive into the current directory (not recursively - archives inside archives just give you another U8 file)
  • texextract - converts a TEX0 texture to PNG
  • texreplace - replaces the texture in a TEX0 file with the contents of a PNG (brings up a conversion dialog)
  • wteconvert - converts between WTE (found in Sega's misc.ccf) and TEX0 by performing a header change
  • imgoverlay - overlays one image onto another image and then saves it to a third file, using the .NET System.Drawing namespace.
  • openingextract - extracts one of the three .arc files found in 00000000.app.
  • openingimet - extracts the name of a game in the specified language from 00000000.app.
  • bannercfgfromtxt - replaces the contents of Sega's banner.cfg.txt with strings read in from other files in the current directory.
I've also forked ccf-tools to fix a couple "bugs" (that were preventing the injection from working) and to allow it to compile with Visual Studio and use .NET's DeflateStream instead of zlib (because I don't know how to link libraries properly on Windows - I'm a .NET developer, and when you have a hammer, everything looks like a nail).

All of these apps are built on .NET Framework 4.7.2, which has been included with Windows for the past couple years at least.

What these apps can't do:
  • Replace the game's title, title ID, or Wii Menu icons (CustomizeMii can do all this, though)
  • Replace the game's title, release year, or number of players in the banner (I still recommend HowardC's tools for this - if you can't run them on your own OS, and you want your injects to look good, it's worth setting up a VM to do it)
Check SaulFabre's guide for how to use CustomizeMii and vcbrlyt.

Here's a sample batch file / the project that inspired me to do this in the first place: injecting Sonic 3D Blast Director's Cut into the original Sonic 3D Blast WAD.

Code:
@echo Before running this batch file:
@echo 1. Extract the Sonic 3D Blast (North American region) WAD file.
@echo 2. Open the WAD file with CustomizeMii and extract banner.brlyt from the
@echo    Layout tab.
@echo 3. Use HowardC's VCbrlyt9.0 to edit the banner.brlyt file, and fill in
@echo    the appropriate title (Sonic 3D Blast: Director's Cut) and release
@echo    year (2017) for all languages. If vcbrlyt.exe does not run on your
@echo    computer, try installing Windows XP (or even Windows 98) to a virtual
@echo    machine and running it there.
@echo 4. Go back to CustomizeMii's Layout tab and replace the banner.brlyt
@echo    with the one you've created. Also update the titles on the Title tab
@echo    (but keep them short - maybe use something like "Sonic 3D Blast DX")
@echo    and the Title ID on the Options tab (this batch file will assume you
@echo    have chosen M3DE as your Title ID). In the Banner and Icon tabs,
@echo    replace VCPic and IconVCPic with a new screenshot of the game's title
@echo    screen (CustomizeMii should resize the images for you). Save the
@echo    resulting WAD in this folder as "input.wad".
@echo.
@echo You should also look through the batch file before running it to make
@echo sure you have any required programs downloaded first (or edit the file
@echo to skip those sections if you'd prefer).
@echo.
@echo To quit out early, press Ctrl+C.

pause

@REM Unpack the input WAD into the "wadunpack" folder.
@REM https://github.com/libertyernie/wad-tools/releases
wadunpacker.exe input.wad wadunpack

@REM Extract the game's titles in various languages (as shown in CustomizeMii) to text files.
@REM This will be used for the save data title later. bannercfgfromtxt expects files with these particular names.
@REM https://github.com/libertyernie/brawllib-wit/releases
mkdir names
openingimet.exe wadunpack\00000000.app ja > names\JP.txt
openingimet.exe wadunpack\00000000.app en > names\EN.txt
openingimet.exe wadunpack\00000000.app de > names\GE.txt
openingimet.exe wadunpack\00000000.app fr > names\FR.txt
openingimet.exe wadunpack\00000000.app es > names\SP.txt
openingimet.exe wadunpack\00000000.app it > names\IT.txt
openingimet.exe wadunpack\00000000.app nl > names\DU.txt

@REM Extract VCIcon from banner.arc so we can use it to make new save data icons later on.
openingextract.exe wadunpack\00000000.app banner banner.arc
mkdir banner
cd banner
..\u8unpack.exe ..\banner.arc
cd ..
del banner.arc
texextract.exe banner/arc/timg/VCPic.tpl VCPic.png
rmdir /s banner

@REM Extract 00000005, which contains all of the data we'll need to update.
mkdir 00000005_contents
cd 00000005_contents
..\u8unpack.exe ..\wadunpack\00000005.app
cd ..

@REM Unpack the data.ccf file found inside 00000005. The CCF format is used in some of Sega's releases.
@REM https://github.com/libertyernie/ccf-tools/releases
mkdir ccf_contents
cd ccf_contents
..\ccfex.exe ..\00000005_contents\data.ccf
cd ..

@REM Use xdelta to apply the Director's Cut patch to the Sonic 3D Blast ROM image found inside data.ccf.
@REM https://github.com/jmacd/xdelta-gpl/releases/tag/v3.0.11
@REM https://www.romhacking.net/hacks/3810/
xdelta3-3.0.11-i686.exe -d -s ccf_contents\Sonic3D.SGD SONICDX.xdelta SONICDX.SGD
del ccf_contents\Sonic3D.SGD
move SONICDX.SGD ccf_contents\Sonic3D.SGD

@REM Grab a modified version of the manual from libertyernie's github.
@REM Wii Virtual Console manuals are just websites written in HTML and they're easy enough to modify once you figure out how to extract and replace them.
@REM Feel free to modify or delete this section if you don't have Git installed.
git clone https://github.com/libertyernie/custom-wii-vc-operations-guides custom-wii-vc-operations-guides --depth 1
cd "custom-wii-vc-operations-guides\Sonic 3D Blast Director's Cut"
del ..\..\ccf_contents\man.arc
..\..\u8pack.exe ..\..\ccf_contents\man.arc
cd ..\..
rmdir /s custom-wii-vc-operations-guides

@REM Extract misc.ccf found inside data.ccf.
mkdir misc_contents
cd misc_contents
..\ccfex.exe ..\ccf_contents\misc.ccf
cd ..

@REM Convert the save file banner to PNG, overlay the VCIcon screenshot onto it, then convert it back.
@REM The PNG to TEX0 conversion will show a dialog - just hit OK, the default settings should be acceptable.
wteconvert.exe misc_contents\banner.wte banner.tex0
texextract.exe banner.tex0 banner1.png
imgoverlay.exe banner1.png VCPic.png 8 8 64 48 banner2.png
texreplace.exe banner.tex0 banner2.png
wteconvert.exe banner.tex0 misc_contents\banner.wte
del banner.tex0 banner1.png banner2.png

@REM Convert the save file icon to PNG, overlay the VCIcon screenshot onto it, then convert it back.
@REM The PNG to TEX0 conversion will show a dialog - just hit OK, the default settings should be acceptable.
@REM To save effort, we'll also replace the other frames of the icon's animation (the one that usually goes back and forth between the game's logo and the console's name).
wteconvert.exe misc_contents\01.wte icon.tex0
texextract.exe icon.tex0 icon1.png
imgoverlay.exe icon1.png VCPic.png 2 9 44 31 icon2.png
texreplace.exe icon.tex0 icon2.png
wteconvert.exe icon.tex0 misc_contents\01.wte
wteconvert.exe icon.tex0 misc_contents\02.wte
wteconvert.exe icon.tex0 misc_contents\03.wte
wteconvert.exe icon.tex0 misc_contents\04.wte
wteconvert.exe icon.tex0 misc_contents\05.wte
wteconvert.exe icon.tex0 misc_contents\06.wte
del icon.tex0 icon1.png icon2.png

del VCPic.png

@REM Modify banner.cfg.txt using the names we extracted earlier.
@REM banner.cfg.txt needs to be big-endian UTF-16. bannercfgfromtxt takes care of that for us, though.
cd names
..\bannercfgfromtxt.exe ..\misc_contents\banner.cfg.txt
cd ..
rmdir /s names

@REM Recompile misc.ccf.
@REM misc.ccf needs to be rebuilt without any compression (unlike data.ccf, where LZ77 compression is allowed).
cd misc_contents
..\ccfarcraw.exe 01.wte 02.wte 03.wte 04.wte 05.wte 06.wte banner.cfg.txt banner.wte
cd ..
del ccf_contents\data.ccf
move misc_contents\out.ccf ccf_contents\misc.ccf
rmdir /s misc_contents

@REM Recompile data.ccf.
cd ccf_contents
..\ccfarc.exe Opera.arc Sonic3D.SGD config home.csv man.arc misc.ccf patch
cd ..
del 00000005_contents\data.ccf
move ccf_contents\out.ccf 00000005_contents\data.ccf
rmdir /s ccf_contents

@REM Recompile 00000005.
del wadunpack\00000005.app
cd 00000005_contents
..\u8pack.exe ..\wadunpack\00000005.app
cd ..
rmdir /s 00000005_contents

@REM Recompile the WAD.
copy common-key.bin wadunpack\common-key.bin
cd wadunpack
..\wadpacker.exe 000100014d334445.tik 000100014d334445.tmd 000100014d334445.cert ..\output.wad
cd ..
rmdir /s wadunpack

@echo File written to output.wad.
pause
Similar to what I did with ccf-tools (see first post), I've also tracked down the source code to (a couple different versions of) wadpacker and wadunpacker, and rebuilt them as C++/CLI applications. The versions I'd been using had been built against a twelve-year-old version of Cygwin (although, to be fair, they did still work) -- these versions are built against Visual C++ 2019 and .NET Framework 4.7.2 instead. I rewrote the hash and crypto functions so they call out to .NET instead of OpenSSL, so they should work on most Windows 10 systems (and possibly earlier) without needing any extra DLLs that might cause conflicts.

https://github.com/libertyernie/wad-tools/releases
Excellent work with that tool set for SEGA Mega Drive injection basing on my tutorial!!! :D
I will download it and keep it, it's amazing :D
 

SaulFabre

Wii scene/hacking & Wii VC well-known guy
Member
Joined
Feb 6, 2019
Messages
1,105
Trophies
1
Age
22
Location
Ecuador
Website
saulfabreg-wiivc.blogspot.com
XP
2,432
Country
Ecuador
Thanks! I figured the individual parts of this could be useful to people even if they don't want to do the whole batch file thing :)
Hi @libertyernie I'm using your tools for managing CCF archive files for make a simple SEGA Master System and SEGA Genesis / SEGA Mega Drive Wii VC iNJECTORs. :D
 

SaulFabre

Wii scene/hacking & Wii VC well-known guy
Member
Joined
Feb 6, 2019
Messages
1,105
Trophies
1
Age
22
Location
Ecuador
Website
saulfabreg-wiivc.blogspot.com
XP
2,432
Country
Ecuador
General chit-chat
Help Users
    Psionic Roshambo @ Psionic Roshambo: But Apple did it best? lol (not really)