Hacking WIA = Wii ISO Archive

Wiimm

Developer
OP
Member
Joined
Aug 11, 2009
Messages
2,292
Trophies
1
Location
Germany
Website
wiimmfi.de
XP
1,519
Country
Germany
vexing said:
Remfin said:
Until you realize the lookup table for a disc stored in 8k chunks is 4 megs (4 bytes per entry, which may not be feasible).

you could have a two layer table, maybe 4700 entries stored in memory, each entry would cover a 1mb area, then the second lookup table could be stored in the 1mb area because the hard disk is going to be reading that area and caching it anyway.
This was my first thought too. See my post above (near bottom).


WiiPower said:
Also i'm asking because i have a new .iso tool for GC games in mind, and that would require defrag code. Would you be interested in writing tools for GC games?
Surprise (after ~70 min of work)
CODE
# wit d pool/gc/mkdd.iso --show all

Dump of file pool/gc/mkdd.iso

ÂÂReal path:ÂÂÂÂÂÂ /disk/ide1/m740av/work/pool/gc/mkdd.iso
ÂÂFile size:ÂÂÂÂÂÂÂÂ57058000/hex = 1459978240 = 1392 MiB
ÂÂDisc type:ÂÂÂÂÂÂ GameCube
ÂÂID & file type:ÂÂGM4P01, GC-ISO
ÂÂDisc name:ÂÂÂÂÂÂ Mario Kart Double Dash!
ÂÂDB title:ÂÂÂÂÂÂÂÂMario Kart: Double Dash!!
ÂÂRegion:ÂÂÂÂÂÂÂÂÂÂPAL [PAL ]
ÂÂRegion setting:ÂÂ0 / 00 00 00 00ÂÂ00 00 00 00
ÂÂDirectories:ÂÂÂÂÂÂÂÂ 35
ÂÂFiles:ÂÂÂÂÂÂÂÂÂÂÂÂÂÂ334
ÂÂUsed ISO blocks:ÂÂ18540 * 32 KiB = 579 MiB


ÂÂ35 directories with 334 files, disk usage 579 MiB:

ÂÂÂÂÂÂ offsetÂÂÂÂ sizeÂÂÂÂ size
ÂÂÂÂÂÂÂÂÂÂhexÂÂÂÂÂÂhexÂÂÂÂÂÂdecÂÂpath + file
ÂÂÂÂ-------------------------------------------------------------------------------
ÂÂÂÂÂÂÂÂÂÂÂÂ-ÂÂÂÂÂÂÂÂ- N=5ÂÂÂÂÂÂ DATA/sys/
ÂÂÂÂÂÂÂÂÂÂÂÂ0+ÂÂÂÂ 440ÂÂÂÂ 1088ÂÂDATA/sys/boot.bin
ÂÂÂÂÂÂÂÂÂÂ440+ÂÂÂÂ2000ÂÂÂÂ 8192ÂÂDATA/sys/bi2.bin
ÂÂÂÂÂÂÂÂ 2440+ÂÂÂÂÂÂ20ÂÂÂÂÂÂ 32ÂÂDATA/sys/apploader.img
ÂÂÂÂÂÂÂÂ7dc00+ÂÂ372200ÂÂ3613184ÂÂDATA/sys/main.dol
ÂÂÂÂÂÂ e46400+ÂÂÂÂ920cÂÂÂÂ37388ÂÂDATA/sys/fst.bin
ÂÂÂÂÂÂÂÂÂÂÂÂ-ÂÂÂÂÂÂÂÂ- N=354ÂÂÂÂ DATA/files/
ÂÂÂÂ155df2480+ÂÂ55d460ÂÂ5624928ÂÂDATA/files/ARAM.arc
ÂÂÂÂÂÂÂÂÂÂÂÂ-ÂÂÂÂÂÂÂÂ- N=54ÂÂÂÂÂÂDATA/files/AudioRes/
ÂÂÂÂ cc23e0d0+ÂÂ 5ab00ÂÂ 371456ÂÂDATA/files/AudioRes/GCKart.baa
ÂÂÂÂ cc3a8cd0+ÂÂÂÂ 700ÂÂÂÂ 1792ÂÂDATA/files/AudioRes/GCKart.bsft
ÂÂÂÂ cc3aa8d0+ÂÂÂÂ94c0ÂÂÂÂ38080ÂÂDATA/files/AudioRes/GCKart.bstn
ÂÂÂÂ cc3cfbd0+ÂÂÂÂ e6dÂÂÂÂ 3693ÂÂDATA/files/AudioRes/GCKart.daa
....
ÂÂÂÂ cf6f2840+ÂÂÂÂ8bb8ÂÂÂÂ35768ÂÂDATA/files/StaffGhosts/Wario.ght
ÂÂÂÂ cf715720+ÂÂÂÂ8bb8ÂÂÂÂ35768ÂÂDATA/files/StaffGhosts/Yoshi.ght


ÂÂISO Usage Map:

ÂÂÂÂÂÂÂÂÂÂÂÂ0: A..AAAAAAAAAAAAA AAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAA
ÂÂÂÂÂÂ 200000: AAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAA AAAA............
ÂÂÂÂÂÂ 400000: ................ ................ ................ ................ *ÂÂ406
ÂÂÂÂ 33000000: ................ .AAAAAAAAAAAAAAA AAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAA
ÂÂÂÂ 33200000: AAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAA *ÂÂ287
ÂÂÂÂ 57000000: AAAAAAAAAAA
Thanx to giantpune for the crash course ...
 

Remfin

Well-Known Member
Newcomer
Joined
Mar 5, 2010
Messages
65
Trophies
0
XP
75
Country
United States
I didn't mean to be that glib, but I kinda had to point out that shrinking the buffers just creates new problems
smile.gif


I'm not sure it's worth even bothering thinking about until someone actually makes a CIOS that loads WIA files. I'm assuming most disc partitions end up with only 2 entries in the map: the stuff at the beginning of the partition (header, fst, dol) and the stuff at the end (all the data)? Simply implementing supporting a plain old WIA file would provide what should be a small space saving (1/32nd) and let people use the exact same format for USB loading and archival with whatever compression they like. You should see if people will get on board with that before worrying about a more complicated format that relies on testing the limits of the ARM processor/IOS memory space.
 

Wiimm

Developer
OP
Member
Joined
Aug 11, 2009
Messages
2,292
Trophies
1
Location
Germany
Website
wiimmfi.de
XP
1,519
Country
Germany
We talk about saving 20-40 percent disc space for a usb loader compatible and complex format. The loading time increases and the developers have to spend much time in this project. On the other side hard drives becomes larger and larger and cheaper and cheaper.

The advantage is minimal and only pirates with multiple hundreds of games have a real advantage, because they can store all 1000 games on 1 hard drive.

My question: Is it all worth it?
(This includes the already ready WIA)
 

vexing

Well-Known Member
Newcomer
Joined
Nov 16, 2009
Messages
65
Trophies
0
XP
27
Country
United States
Wiimm said:
We talk about saving 20-40 percent disc space for a usb loader compatible and complex format. The loading time increases and the developers have to spend much time in this project. On the other side hard drives becomes larger and larger and cheaper and cheaper.

The advantage is minimal and only pirates with multiple hundreds of games have a real advantage, because they can store all 1000 games on 1 hard drive.

My question: Is it all worth it?
(This includes the already ready WIA)

i like it mainly as a conceptual problem, and it could become the new standard for images.
yes, it'd primarily benefit pirates, but what doesn't?
 

W hat

Rhythm Heaven Fan
Member
Joined
Feb 28, 2007
Messages
632
Trophies
1
XP
697
Country
United States
vexing said:
i like it mainly as a conceptual problem, and it could become the new standard for images.
yes, it'd primarily benefit pirates, but what doesn't?
DIOS MIOS! Good thing I have 28+ real GC discs.
 

WiiPower

Well-Known Member
Member
Joined
Oct 17, 2008
Messages
8,165
Trophies
0
XP
345
Country
Gambia, The
Wiimm said:
[...]
WiiPower said:
Also i'm asking because i have a new .iso tool for GC games in mind, and that would require defrag code. Would you be interested in writing tools for GC games?
Surprise (after ~70 min of work)
CODE# wit d pool/gc/mkdd.iso --show all

Dump of file pool/gc/mkdd.iso

ÂÂReal path:ÂÂÂÂÂÂ /disk/ide1/m740av/work/pool/gc/mkdd.iso
ÂÂFile size:ÂÂÂÂÂÂÂÂ57058000/hex = 1459978240 = 1392 MiB
ÂÂDisc type:ÂÂÂÂÂÂ GameCube
ÂÂID & file type:ÂÂGM4P01, GC-ISO
ÂÂDisc name:ÂÂÂÂÂÂ Mario Kart Double Dash!
ÂÂDB title:ÂÂÂÂÂÂÂÂMario Kart: Double Dash!!
ÂÂRegion:ÂÂÂÂÂÂÂÂÂÂPAL [PAL ]
ÂÂRegion setting:ÂÂ0 / 00 00 00 00ÂÂ00 00 00 00
ÂÂDirectories:ÂÂÂÂÂÂÂÂ 35
ÂÂFiles:ÂÂÂÂÂÂÂÂÂÂÂÂÂÂ334
ÂÂUsed ISO blocks:ÂÂ18540 * 32 KiB = 579 MiB


ÂÂ35 directories with 334 files, disk usage 579 MiB:

ÂÂÂÂÂÂ offsetÂÂÂÂ sizeÂÂÂÂ size
ÂÂÂÂÂÂÂÂÂÂhexÂÂÂÂÂÂhexÂÂÂÂÂÂdecÂÂpath + file
ÂÂÂÂ-------------------------------------------------------------------------------
ÂÂÂÂÂÂÂÂÂÂÂÂ-ÂÂÂÂÂÂÂÂ- N=5ÂÂÂÂÂÂ DATA/sys/
ÂÂÂÂÂÂÂÂÂÂÂÂ0+ÂÂÂÂ 440ÂÂÂÂ 1088ÂÂDATA/sys/boot.bin
ÂÂÂÂÂÂÂÂÂÂ440+ÂÂÂÂ2000ÂÂÂÂ 8192ÂÂDATA/sys/bi2.bin
ÂÂÂÂÂÂÂÂ 2440+ÂÂÂÂÂÂ20ÂÂÂÂÂÂ 32ÂÂDATA/sys/apploader.img
ÂÂÂÂÂÂÂÂ7dc00+ÂÂ372200ÂÂ3613184ÂÂDATA/sys/main.dol
ÂÂÂÂÂÂ e46400+ÂÂÂÂ920cÂÂÂÂ37388ÂÂDATA/sys/fst.bin
ÂÂÂÂÂÂÂÂÂÂÂÂ-ÂÂÂÂÂÂÂÂ- N=354ÂÂÂÂ DATA/files/
ÂÂÂÂ155df2480+ÂÂ55d460ÂÂ5624928ÂÂDATA/files/ARAM.arc
ÂÂÂÂÂÂÂÂÂÂÂÂ-ÂÂÂÂÂÂÂÂ- N=54ÂÂÂÂÂÂDATA/files/AudioRes/
ÂÂÂÂ cc23e0d0+ÂÂ 5ab00ÂÂ 371456ÂÂDATA/files/AudioRes/GCKart.baa
ÂÂÂÂ cc3a8cd0+ÂÂÂÂ 700ÂÂÂÂ 1792ÂÂDATA/files/AudioRes/GCKart.bsft
ÂÂÂÂ cc3aa8d0+ÂÂÂÂ94c0ÂÂÂÂ38080ÂÂDATA/files/AudioRes/GCKart.bstn
ÂÂÂÂ cc3cfbd0+ÂÂÂÂ e6dÂÂÂÂ 3693ÂÂDATA/files/AudioRes/GCKart.daa
....
ÂÂÂÂ cf6f2840+ÂÂÂÂ8bb8ÂÂÂÂ35768ÂÂDATA/files/StaffGhosts/Wario.ght
ÂÂÂÂ cf715720+ÂÂÂÂ8bb8ÂÂÂÂ35768ÂÂDATA/files/StaffGhosts/Yoshi.ght


ÂÂISO Usage Map:

ÂÂÂÂÂÂÂÂÂÂÂÂ0: A..AAAAAAAAAAAAA AAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAA
ÂÂÂÂÂÂ 200000: AAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAA AAAA............
ÂÂÂÂÂÂ 400000: ................ ................ ................ ................ *ÂÂ406
ÂÂÂÂ 33000000: ................ .AAAAAAAAAAAAAAA AAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAA
ÂÂÂÂ 33200000: AAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAA *ÂÂ287
ÂÂÂÂ 57000000: AAAAAAAAAAA
Thanx to giantpune for the crash course ...

Nice. So now, here's what i think makes sense(yes fstfix is very close to that already...):
- Align all files by 32KB within the .iso. This is a requirement for audio streams, if an audio stream is not aligned at 32KB, then the game will most likely crash when trying to play the stream. .adp files should always be audio streams, but .trk and .snd can be audio streams too. Just to be sure align all files by 32KB and everything will work. For disc loading with NeoGamma aligning all files at least at 2KB results in a better read speed as it's only able to read sectors directly, so 32KB for everything is good for this too.
- Shrink .isos. GC .isos can have any size, so it should be possible to move all data to the beginning and just cut the .iso.(that's what i think all other shrinkers do)
- Expand .iso size for optimised reads for disc loading. There you some different options would be nice, optimised for 1/2/3 games on disc. If a single game would have a .iso with 4.37 GB and all the data is at the end, the reads are faster than a 1:1 copy.
- Optional: Check if the file is fragmented. SD-Boot requires that .isos are in one fragment.

Well extract/replace files is also nice, but the above is what i think most people will need. Well that is if you want to write this at all, i don't want to push you or something.
 

Wiimm

Developer
OP
Member
Joined
Aug 11, 2009
Messages
2,292
Trophies
1
Location
Germany
Website
wiimmfi.de
XP
1,519
Country
Germany
@WiiPower

All commands (DUMP, LIST, COPY, DIFF, EDIT, MOVE, RENAME, EXTRACT, FILES, ...) works with gamecube discs and only for some I need minimal changes and a little bit fine tuning. That means, extracting, patching, scrubbing ans converting into WDF, CISO or WBFS (but not WIA) is available. Adding to a WBFS partition is also possible.

For your aligning I have to implement a relocation module into the wiidisc module. This will take a while.

P.S.: Beta version comes tomorrow.
 

FenrirWolf

Well-Known Member
Member
Joined
Nov 19, 2008
Messages
4,347
Trophies
1
Location
Sandy, UT
XP
615
Country
United States
WiiBricker said:
W hat said:
vexing said:
i like it mainly as a conceptual problem, and it could become the new standard for images.
yes, it'd primarily benefit pirates, but what doesn't?
DIOS MIOS! Good thing I have 28+ real GC discs.
Tomorrow should start the open beta of DIOS MIOS.
September's a long month. Keep in mind he never gave a specific date.
 

WiiPower

Well-Known Member
Member
Joined
Oct 17, 2008
Messages
8,165
Trophies
0
XP
345
Country
Gambia, The
Wiimm said:
@WiiPower

All commands (DUMP, LIST, COPY, DIFF, EDIT, MOVE, RENAME, EXTRACT, FILES, ...) works with gamecube discs and only for some I need minimal changes and a little bit fine tuning. That means, extracting, patching, scrubbing ans converting into WDF, CISO or WBFS (but not WIA) is available. Adding to a WBFS partition is also possible.

For your aligning I have to implement a relocation module into the wiidisc module. This will take a while.

P.S.: Beta version comes tomorrow.

Please don't make tools that create something else than .iso file for gamecube games. Gamecube games are not encrypted, and there are no hashes to throw away, so if you shrink the .iso, it's nearly as small as possible(32KB alignment increases the size a little). And .iso files can just be burned, put into multi game discs, using existing tools, or can be played via SD-Boot. If you see any advantage in another format for gamecube games, please take you time and make it one format that does everything you want. Also if you create .wbfs files, people will bug around why the games don't work in their usb loaders.

PS: You don't have any relocation tool yet? Hmm, it would be nice if you could do something like this:
join game.iso foldername
And this overwrites all files in game.iso with files it finds in foldername. If there's no matching file the folder, it takes the file from the .iso. And since the sizes can be different, it relocates all files of course. I don't know if wii games have any specific requirements for alignment, so 2KB* as default and 32KB as optional alignment would sound good to me. Maybe that's why file replacment doesn't always work with current tools?

*Backup loaders that use discs use the dvd video read command, which should only be able to read sectors, so the aligment should be at least 2KB
PPS: When relocating files in wii .isos then of couse make sure the "hole" is at the beginning of the disc. And yes, i will try to think first, and then write next time.
 

Wiimm

Developer
OP
Member
Joined
Aug 11, 2009
Messages
2,292
Trophies
1
Location
Germany
Website
wiimmfi.de
XP
1,519
Country
Germany
WDF, WBFS, CISO and WIA are only containers (like tar or zip). They are handled in the file layer of my tools. The commands don't see the details and read from/write to the virtual file. So support for all formats is automatically available (WIA needs only a little extension).

Sorry
wink.gif
 

keine

Well-Known Member
Member
Joined
Aug 29, 2010
Messages
431
Trophies
1
Website
Visit site
XP
1,435
Country
United States
Just want to say thanks to Wiimm for the hard work. ANY ANY development at all for the WII (heaven forbid new effective containers and compressions schemes) should be treasured and appreciated to the nth degree. Thanks
 

wishmasterf

Well-Known Member
Member
Joined
Apr 20, 2009
Messages
167
Trophies
0
Location
Austria
Website
Visit site
XP
92
Country
Australia
Thanks for releasing wit with WIA support. What are your plans with that format. When do you think it is stable and will not be changed?

Do you plan a support for WIA in usb-loaders?
 

hack_10299

New Member
Newbie
Joined
Sep 4, 2010
Messages
1
Trophies
0
XP
1
Country
United States
wishmasterf said:
Thanks for releasing wit with WIA support. What are your plans with that format. When do you think it is stable and will not be changed?

Do you plan a support for WIA in usb-loaders?


i was stupid and accidentally installed 4.3U!!!! when i try to run WAD Manager 1.7 my wiimote goes off, when it gets to nand device (disable)
i dont know whats wrong.... and also usb loader wont even come up when i click on it
 

Wiimm

Developer
OP
Member
Joined
Aug 11, 2009
Messages
2,292
Trophies
1
Location
Germany
Website
wiimmfi.de
XP
1,519
Country
Germany
wishmasterf said:
Thanks for releasing wit with WIA support. What are your plans with that format. When do you think it is stable and will not be changed?

Do you plan a support for WIA in usb-loaders?
WIA is only an backup archive format. USB loaders will never support it because of the complex data structure.

EDIT: forgotten stable answer:
I think is is already stable. I'm waiting for test with a big endian machine to declare it as stable.



@hack_10299
You are off topic here. Don't expect an answer.
 

Wiimm

Developer
OP
Member
Joined
Aug 11, 2009
Messages
2,292
Trophies
1
Location
Germany
Website
wiimmfi.de
XP
1,519
Country
Germany
The test with a big endian machine are positive. It seems that the current format is the final one. In the next days I start a script to convert to wia and back and compare if there are no differences. I will post the bash script if it is ok.
 

Site & Scene News

Popular threads in this forum

General chit-chat
Help Users
  • No one is chatting at the moment.
    SylverReZ @ SylverReZ: https://www.youtube.com/watch?v=hke2YUirpf4 +1