[WIP] Wii U game info database

Discussion in 'Wii U - Hacking & Backup Loaders' started by Quantumcat, Nov 23, 2016.

  1. Quantumcat
    OP

    Quantumcat Dead and alive

    Member
    11,851
    6,368
    Nov 23, 2014
    Australia
    Canberra, Australia
    Update 11th December:
    Now nearly all games, updates, DLC and demos are in the database. If you know one that's missing please let me know and I'll add it manually. Also any suggestions welcome. You can also search (since there's so many now). And you can click on the name of a game to see the individual files and their sizes that belong to it.

    Screenshot:
    Capture.PNG
    If you have an application or homebrew that needs to know details about games, like individual files, sizes of the files or of the whole game etc, I can provide a php page for you to send a POST request to which will return this data for you! Just ask, I like helping people!




    ..... original post:

    Since I had trouble finding a list of game and update file sizes from NUS on the web, I thought I'd create my own. A very bare-bones display is at http://quantumc.at/gamelist.php (specific suggestions for improvements on looks or useability welcome). There isn't much there right now (it is only stuff I downloaded for testing, and my internet is slow so I tried to stick to updates I thought would be small) but there will be more in the future, and I hope one day it will have everything!

    My little downloader program sends data about the game or update you downloaded to my website, which then adds it to the database. It also uses info from the database to display the size in the list. The progress bars currently are based on the output from NUSGrabber (thanks @Moquedami for the suggestion), but later it will be based on the file sizes from the database (and can have an accurate total % complete). I compiled a new copy which I have uploaded to quantumc.at if anyone wants to try it (repository at https://github.com/quantumcat1/Wii-U-Downloader).

    What I am hoping is that the authors of the other downloader programs (that can do more than mine and are more popular) might consider also sending data about what their users have downloaded to my website, so the database can grow much more quickly. They might also like to use the webservice I made for requesting game sizes from the database to also display sizes in their programs. If their programs are in Java I can provide the code for sending the http POST requests but if not they'll have to work it out themselves :-p

    If you think this is a good idea could you please tag a dev :-) All I can think of right now is @cearp for FunKiiU.

    Capture.PNG Capture2.PNG
     
    Last edited by Quantumcat, Dec 11, 2016
  2. pietempgba

    pietempgba GBAtemp Advanced Fan

    Member
    816
    134
    Jun 9, 2016
    United States
    i wish that title key site had file sizes
     
  3. bennyman123abc

    bennyman123abc Master of the Script Kiddies

    Member
    698
    327
    Mar 21, 2013
    United States
    Training some more Script Kiddies
    I can give you exact file sizes to a few games, updates, and dlcs if you need me to. I'm not going to re-download them because my internet speed is 4 mbps down on average so it takes FOREVER to download anything these days. I remember when 4 mbps was f*cking fast too lol
     
    Quantumcat likes this.
  4. Quantumcat
    OP

    Quantumcat Dead and alive

    Member
    11,851
    6,368
    Nov 23, 2014
    Australia
    Canberra, Australia
    I'll make a mini program tonight for you that will send through the data on what you have already (saves typing in all the .app and .h3 names!). I was going to do this anyway as my recycle bin is full of stuff I previously downloaded while testing.

    Thanks!!
     
  5. bennyman123abc

    bennyman123abc Master of the Script Kiddies

    Member
    698
    327
    Mar 21, 2013
    United States
    Training some more Script Kiddies
    Could you possibly just have it to where I can enter the file size manually? For one of the games that I have (Mario Kart 8 USA), I have a DDD dump which is accurate to a range of 10 MB typically (Which shouldn't be that big of a deal. Is it?). But also having the option for the .app and .h3 names is great too because I have a few other games in that format. Thanks for letting me help out with this project btw!
     
    Quantumcat likes this.
  6. Irastris

    Irastris Just another fag with a suggestive avatar

    Member
    1,005
    570
    May 3, 2015
    United States
    Sorting alphabetically on the bare-bones display page would be nice.
     
  7. Moquedami

    Moquedami GBAtemp Regular

    Member
    194
    35
    Nov 16, 2006
    Argentina
    Ey, this is a neat update!
    I dont know about the download size database though. Is it really necessary? I mean, i think other programs get the download size directly from Nintendo servers when loading the game list, but i´m not sure that´s how it works.
    If you are doing it this way just to experiment, that´s cool too. But I think it would take a community effort that would have cause more benefit in the beginning. By this time everybody already downloaded whatever they were interested in, and it´s not really probable that they would download again for the sake of it.
     
    Last edited by Moquedami, Nov 24, 2016
  8. bennyman123abc

    bennyman123abc Master of the Script Kiddies

    Member
    698
    327
    Mar 21, 2013
    United States
    Training some more Script Kiddies
    Well if you haven't noticed at this point, the community is starting to grow for this project.
     
  9. Moquedami

    Moquedami GBAtemp Regular

    Member
    194
    35
    Nov 16, 2006
    Argentina
    yeah, and i'm glad. But that's not the point i'm asking for.
     
  10. bennyman123abc

    bennyman123abc Master of the Script Kiddies

    Member
    698
    327
    Mar 21, 2013
    United States
    Training some more Script Kiddies
    What are you asking then? This is just a project to quickly and easily get a game's file size before you buy it and realize you have to wait 2 weeks for it to download.
     
  11. Moquedami

    Moquedami GBAtemp Regular

    Member
    194
    35
    Nov 16, 2006
    Argentina
    This is the part i´m asking: "i think other programs get the download size directly from Nintendo servers when loading the game list, but i´m not sure that´s how it works."
    If this is the case, why would we need acommunity effort just to get the data thats its already there?
    And again, if that´s for the sake of experimating, I´m super cool with that. I just don´t get the practical use
     
    Last edited by Moquedami, Nov 25, 2016
  12. dojafoja

    dojafoja life elevated

    Member
    471
    315
    Jan 2, 2014
    @Quantumcat I have a full sqlite database of every title size on nus. The only ones missing are those which I could not download the tmd with FunKiiU. I am including it in th next release of my GUI for FunKiiU. You want a copy of it?

    — Posts automatically merged - Please don't double post! —

    Here it is http://www.mediafire.com/file/807f7scw0v51gcy/data
    This database contains the title id, file size in bytes, and a human readable size for every single title on NUS except those which the tmd couldn't be downloaded.
    I also have a script to update this database against the title id's on that key site.
    Its up do date as of yesterday morning but new titles get added all the time.

    If you don't know how to query an SQL database, I can help. I use Python because it's super easy. I could technically write this to a file line by line if its easier

    if the title size in the database is 0 that means that I couldn't download the tmd for that title

    Forgot to mention, the db table is called "titles" the title id field is called "title_id" and there are two fields with size info.
    "size" field has the human readable size and "raw_size" field has the actual title size in bytes

    I'd love to help you out so if you can use the db as is, great! If it's easier to parse them into your program by other means ie: json,csv, etc.let me know and I will send you that output
     
    Last edited by dojafoja, Nov 25, 2016
  13. Quantumcat
    OP

    Quantumcat Dead and alive

    Member
    11,851
    6,368
    Nov 23, 2014
    Australia
    Canberra, Australia
    Done (and can sort by size)
     
  14. Irastris

    Irastris Just another fag with a suggestive avatar

    Member
    1,005
    570
    May 3, 2015
    United States
    Sweet, thanks.
     
  15. bennyman123abc

    bennyman123abc Master of the Script Kiddies

    Member
    698
    327
    Mar 21, 2013
    United States
    Training some more Script Kiddies
    Did you ever finish the other program that allows the file sizes to be entered without downloading the title?
     
  16. Quantumcat
    OP

    Quantumcat Dead and alive

    Member
    11,851
    6,368
    Nov 23, 2014
    Australia
    Canberra, Australia
    I'm working on it :-) What is your folder structure? I am currently assuming game name/titleid/app and h3 files but if different let me know. It is working but there seems to be a bug in my update.php because it created some rows with blank titleids :-(
     
  17. Quantumcat
    OP

    Quantumcat Dead and alive

    Member
    11,851
    6,368
    Nov 23, 2014
    Australia
    Canberra, Australia
    Have a look at what's in the database now. Hilarious :-) I will clean it up when I get home.
     
  18. bennyman123abc

    bennyman123abc Master of the Script Kiddies

    Member
    698
    327
    Mar 21, 2013
    United States
    Training some more Script Kiddies
    You should've screenshot the image and posted it here :rofl2:

    Oh lord I'm a fucking idiot :rofl2:
     
  19. Quantumcat
    OP

    Quantumcat Dead and alive

    Member
    11,851
    6,368
    Nov 23, 2014
    Australia
    Canberra, Australia
    Haha yeah I should have :-) All cleaned up now.

    Having my program send data to the database when users download stuff might become redundant. @dojafoja has a way of fetching file sizes in his Funkii-UI program. I just need to learn Python to fully understand what it's doing, then I can steal the logic bwahaha
     
  20. bennyman123abc

    bennyman123abc Master of the Script Kiddies

    Member
    698
    327
    Mar 21, 2013
    United States
    Training some more Script Kiddies
    I know Python. Send me a link to this program and I will check it out