Hacking [WIP] Wii U game info database

dojafoja · Dec 1, 2016

Quantumcat said:
Haha yeah I should have :-) All cleaned up now.

Having my program send data to the database when users download stuff might become redundant. @dojafoja has a way of fetching file sizes in his Funkii-UI program. I just need to learn Python to fully understand what it's doing, then I can steal the logic bwahaha

I piggy backed code from @cearp in FunKiiU because I was to lazy to see what was actually happening :-P

Quantumcat · Dec 1, 2016

bennyman123abc said:
I know Python. Send me a link to this program and I will check it out

https://github.com/dojafoja/FunKii-UI/blob/master/gui.py
It's in the build_database method. It looks like the tmd is used to work out which numbered files will belong. And then it does something (?) and gets the size of each file.

bennyman123abc · Dec 1, 2016

Quantumcat said:
https://github.com/dojafoja/FunKii-UI/blob/master/gui.py
It's in the build_database method. It looks like the tmd is used to work out which numbered files will belong. And then it does something (?) and gets the size of each file.

It appears that everything you need is in this code.

baseurl = 'http://ccs.cdn.c.shop.nintendowifi.net/ccs/download/{}'.format(tid)
if not fnku.download_file(baseurl + '/tmd', 'title.tmd', 1):
print('ERROR: Could not download TMD...')
else:
with open('title.tmd', 'rb') as f:
tmd = f.read()
content_count = int(binascii.hexlify(tmd[TK + 0x9E:TK + 0xA0]), 16)
total_size = 0
for i in range(content_count):
c_offs = 0xB04 + (0x30 * i)
c_id = binascii.hexlify(tmd[c_offs:c_offs + 0x04]).decode()
total_size += int(binascii.hexlify(tmd[c_offs + 0x08:c_offs + 0x10]), 16)
sz = fnku.bytes2human(total_size)
os.remove('title.tmd')

Exactly what the hell it's doing however, I could not tell you. It's doing something with hex and downloading a tmd file.

EDIT: I mean, I know what it's doing but how to do it in Java, I couldn't tell you

bennyman123abc · Dec 2, 2016

Were you able to use the code I directed you to @Quantumcat?

Quantumcat · Dec 2, 2016

bennyman123abc said:
Were you able to use the code I directed you to @Quantumcat?

Yeah, I'll probably learn Python file I/O, put this function on a separate script, and get it to write to a JSON file instead of updating the database file. Then the JSON file will be easy to use to update my database with.

dojafoja · Dec 2, 2016

Quantumcat said:
Yeah, I'll probably learn Python file I/O, put this function on a separate script, and get it to write to a JSON file instead of updating the database file. Then the JSON file will be easy to use to update my database with.

If you are still talking about the code above that I stole from cearp then I have a few thoughts. It opens title.tmd in binary mode and, using some offset magic, calculates the content count. Then it loops the tmd instance x amount of times, x being content count. Each loop it downloads a .app file and then blindly attempts to download an associated .h3 file, ignoring 404 errors. So without spending too much brain power thinking about it, seems to me like we would also need to track IF a .h3 file was in fact downloaded (no 404 error) and get the size of that .h3 file after it's downloaded and subtract it from that iterations content size. That way you could track every single files size. Because in my mind if it calculates the size before downloading the actual file and doesn't know if an .h3 actually exists for that iteration then we need to subtract the size of any .h3 downloaded for that content count. I hope I make sense LOL. Example: It calculates a total content count of 2. c_id 1 is 1024 bytes and c_id 2 is 2048 bytes. If c_id 2 also contains an associated .h3 (00000004.app and 00000004.h3) that's 1000 bytes then the .app file is 1048 and the .h3 file is 1000 for a total of 2048 bytes that was the calculated content size for that content count. OK now I've just confused myself lol I need to test that I'm actually correct about my theory but I think I am. Of course, I'm probably the only one who understands what the hell I just tried explaining! FunkiiU seems unaware of .h3 files and just goes after them blindly.

Quantumcat · Dec 2, 2016

dojafoja said:
If you are still talking about the code above that I stole from cearp then I have a few thoughts. It opens title.tmd in binary mode and, using some offset magic, calculates the content count. Then it loops the tmd instance x amount of times, x being content count. Each loop it downloads a .app file and then blindly attempts to download an associated .h3 file, ignoring 404 errors. So without spending too much brain power thinking about it, seems to me like we would also need to track IF a .h3 file was in fact downloaded (no 404 error) and get the size of that .h3 file after it's downloaded and subtract it from that iterations content size. That way you could track every single files size. Because in my mind if it calculates the size before downloading the actual file and doesn't know if an .h3 actually exists for that iteration then we need to subtract the size of any .h3 downloaded for that content count. I hope I make sense LOL. Example: It calculates a total content count of 2. c_id 1 is 1024 bytes and c_id 2 is 2048 bytes. If c_id 2 also contains an associated .h3 (00000004.app and 00000004.h3) that's 1000 bytes then the .app file is 1048 and the .h3 file is 1000 for a total of 2048 bytes that was the calculated content size for that content count. OK now I've just confused myself lol I need to test that I'm actually correct about my theory but I think I am. Of course, I'm probably the only one who understands what the hell I just tried explaining! FunkiiU seems unaware of .h3 files and just goes after them blindly.

Ah I think I see, that's good information! It's kind of good if something original is needed rather than copying and pasting code. Opportunity to learn, it's more interesting, etc etc.

dojafoja · Dec 2, 2016

Quantumcat said:
Ah I think I see, that's good information! It's kind of good if something original is needed rather than copying and pasting code. Opportunity to learn, it's more interesting, etc etc.

I agree. I was more heavily focused on my app features at the time and FunKiiU already had the code to calculate title sizes so I just re used it to build a db that I could use. I don't always have a ton of time either so writing code to do something that was already there seemed like bad use of time lol. I have a lot on my plate and just do this for fun because it keeps me sane between all the insane moments. Good luck on your project and if I can help in any way just reach out to me.

Quantumcat · Dec 7, 2016

Probably nobody cares, but ........

I used some of @dojafoja's gui.py to product a text file containing all the games and titleids
Then I used some of @Maschell's JNUSTool's classes to download the tmd's using those titleids and work out all the encrypted files and their sizes
Then I used the program I already wrote for @bennyman123abc but using what I found above for the file names and sizes instead of going through the folders present on the machine looking for files, turned the info into a JSON and sent that off to my database

And it worked!! For some reason the updates didn't go through, there are a few with blank names, and the names all have a mysterious character at the front and end that translated to question marks, but I nearly have a listing of all the games and sizes!!

dojafoja · Dec 7, 2016

Quantumcat said:
Probably nobody cares, but ........

I used some of @dojafoja's gui.py to product a text file containing all the games and titleids
Then I used some of @Maschell's JNUSTool's classes to download the tmd's using those titleids and work out all the encrypted files and their sizes
Then I used the program I already wrote for @bennyman123abc but using what I found above for the file names and sizes instead of going through the folders present on the machine looking for files, turned the info into a JSON and sent that off to my database

And it worked!! For some reason the updates didn't go through, there are a few with blank names, and the names all have a mysterious character at the front and end that translated to question marks, but I nearly have a listing of all the games and sizes!!

You're dealing with Unicode issues, those are the mysterious chars probably. The blank names are because there are literally entries on that key site with no names lol. I ignore entries with blank names, but I parse on every program start up because that key site often updates entries and such so names could be added soon. FYI If you want to parse it all yourself you can download a JSON directly from https://thatkeysite/json it doesn't any size or file info though.

CosmoCortney · Dec 7, 2016

Quantumcat said:
Since I had trouble finding a list of game and update file sizes from NUS on the web, I thought I'd create my own. A very bare-bones display is at http://quantumc.at/gamelist.php (specific suggestions for improvements on looks or useability welcome). There isn't much there right now (it is only stuff I downloaded for testing, and my internet is slow so I tried to stick to updates I thought would be small) but there will be more in the future, and I hope one day it will have everything!

My little downloader program sends data about the game or update you downloaded to my website, which then adds it to the database. It also uses info from the database to display the size in the list. The progress bars currently are based on the output from NUSGrabber (thanks @Moquedami for the suggestion), but later it will be based on the file sizes from the database (and can have an accurate total % complete). I compiled a new copy which I have uploaded to quantumc.at if anyone wants to try it (repository at https://github.com/quantumcat1/Wii-U-Downloader).

What I am hoping is that the authors of the other downloader programs (that can do more than mine and are more popular) might consider also sending data about what their users have downloaded to my website, so the database can grow much more quickly. They might also like to use the webservice I made for requesting game sizes from the database to also display sizes in their programs. If their programs are in Java I can provide the code for sending the http POST requests but if not they'll have to work it out themselves :-p

If you think this is a good idea could you please tag a dev :-) All I can think of right now is @cearp for FunKiiU.

View attachment 70003 View attachment 70004

would you mind adding hashes (crc, md5) of discs to the list?

bennyman123abc · Dec 7, 2016

CosmoCortney said:
would you mind adding hashes (crc, md5) of discs to the list?

That wouldn't be a bad idea but one question. How would the end user like me who is supposed to be the one contributing to this database going to det the crc and md5 hashes of the discs?

Quantumcat · Dec 7, 2016

CosmoCortney said:
would you mind adding hashes (crc, md5) of discs to the list?

When you say hashes of the disks - do you mean like of their images?
If you know where to find the info, then sure!

CosmoCortney · Dec 8, 2016

bennyman123abc said:
That wouldn't be a bad idea but one question. How would the end user like me who is supposed to be the one contributing to this database going to det the crc and md5 hashes of the discs?

Quantumcat said:
When you say hashes of the disks - do you mean like of their images?
If you know where to find the info, then sure!

You can find them in the txt that wudump creates. it provides the crc32, md5 and sha hashes.
if gamehacking.org will ever add wii u support to their cheat code database they'd need the hashes to give some more details about the games.

Quantumcat · Dec 11, 2016

Big update, I edited the OP :-)
I will look at that hash stuff you mentioned @CosmoCortney, and probably add it. If it isn't possible to get the data automatically I will add user submission for it.

CosmoCortney · Dec 11, 2016

Quantumcat said:
Big update, I edited the OP :-)
I will look at that hash stuff you mentioned @CosmoCortney, and probably add it. If it isn't possible to get the data automatically I will add user submission for it.

I have created a thread where you can find a lot of hashes

https://gbatemp.net/threads/mind-sharing-hashes-of-disc-images.452697/

dojafoja · Dec 11, 2016

Nice progress! Something like this could render my size db useless for FunKii-UI. I will definitely consider fetching data from this db in a future release. Is it self updating, and always up to date? Or do you feel like it's complete enough that from here we could just rely on user submitted data? It would be awesome if things matured to the point that we have ALL info about wii u titles remotely. Sizes,hashes,serials,descriptions etc. would be awesome to just fetch it all remotely.

bennyman123abc · Dec 11, 2016

dojafoja said:
Nice progress! Something like this could render my size db useless for FunKii-UI. I will definitely consider fetching data from this db in a future release. Is it self updating, and always up to date? Or do you feel like it's complete enough that from here we could just rely on user submitted data? It would be awesome if things matured to the point that we have ALL info about wii u titles remotely. Sizes,hashes,serials,descriptions etc. would be awesome to just fetch it all remotely.

Yea. An API for this database would be AMAZING and an easier way other than the JSON method of pushing data to the database would be great. Why not use SQL?

Quantumcat · Dec 11, 2016

dojafoja said:
Nice progress! Something like this could render my size db useless for FunKii-UI. I will definitely consider fetching data from this db in a future release. Is it self updating, and always up to date? Or do you feel like it's complete enough that from here we could just rely on user submitted data? It would be awesome if things matured to the point that we have ALL info about wii u titles remotely. Sizes,hashes,serials,descriptions etc. would be awesome to just fetch it all remotely.

Not self-updating yet unfortunately (I have to download a new titlekeys.json and send an update), but it will be if I can find out how to run a .jar on my server :-) I learned about cron jobs from a previous project.

Hacking [WIP] Wii U game info database

life elevated

Dead and alive

Well-Known Member

Well-Known Member

Dead and alive

life elevated

Dead and alive

life elevated

Dead and alive

life elevated

i snack raw pasta and chew lollipops

Well-Known Member

Dead and alive

i snack raw pasta and chew lollipops

Dead and alive

i snack raw pasta and chew lollipops

life elevated

Well-Known Member

Dead and alive

Similar threads

Popular threads in this forum