WIP NSP Organizer and mass downloader

Discussion in 'Switch - Emulation, Homebrew & Software Projects' started by blawar, Jul 27, 2018.

  1. blawar
    OP

    blawar GBAtemp Advanced Fan

    Member
    9
    Nov 21, 2016
    United States
    Nut
    This is a program that automatically downloads all games from the CDN, and organizes them on the file system.

    It will also download any titles you do not have a key for (for archiving), by enabling sansTitleKey in nut.json. These titles are saved with the .nsx file extension, and can be unlocked at a later time when a title key is found.

    If you only wish to rename / organize files, and not download anything, edit nut.json and set all downloading options to false.

    Organization
    It saves the titles in the following format by default:

    base games: titles/{name}[{id}][v{version}].nsp

    DLC: titles/DLC/{baseName}/{name}[{id}][v{version}].nsp

    Updates: titles/updates/{baseName}/{name}[{id}][v{version}].nsp

    Demos: titles/demos/{name}[{id}][v{version}].nsp

    Demo Updates: titles/demos/updates/{baseName}/{name}[{id}][v{version}].nsp

    Titlelist
    This program will load any titlekeys files named *.titlekeys.txt including titlekeys.txt

    Whitelist
    Place title id's that you want to download in whitelist.txt, separated with a newline.

    If you want to download all games, leave the file empty.

    Blacklist
    Place title id's that you do not want to download in blacklist.txt, separated with a newline.

    Installation
    Place all of these files in your already configured CDNSP directory if you want it to download from the CDN.

    If all you want is to organize your existing NSP's, just place all of these files in a dir that contains your NSP files (or sub directory) with the titleid in the filename in brackets.

    run nut.py

    https://github.com/blawar/nut
     
    Last edited by blawar, Jul 28, 2018
  2. Dfroberts86

    Dfroberts86 Advanced Member

    Newcomer
    2
    Jun 27, 2018
    United States
    Thank you!

    Sent from my BND-L24 using Tapatalk
     
  3. masterspike52

    masterspike52 GBAtemp Regular

    Member
    3
    Jan 25, 2017
    United States
    @blawar can you show me via a video of sorts that this will work?

    well i rewrote some of it and fixed the errors but it still does the same thing cdnsp does and asks to check the cert
     
    Last edited by masterspike52, Jul 27, 2018
  4. DeuX

    DeuX GBAtemp Regular

    Member
    4
    Jun 14, 2018
    Switzerland
    Would it be possible to organize the titles already downloaded? I already have most of the titles in a external hard drive so I would need to organize them before starting pulling the missing ones from CDN
     
  5. masterspike52

    masterspike52 GBAtemp Regular

    Member
    3
    Jan 25, 2017
    United States
    from what it looks like, you just blacklist the ones you own by id and then leave the whitelist blank to download the rest
     
  6. DeuX

    DeuX GBAtemp Regular

    Member
    4
    Jun 14, 2018
    Switzerland
    Still would need to check one by one what titles I have and add them to the blacklist. It would take a ton of time considering I have near 800 NSP files lol
     
  7. Poketrekker

    Poketrekker Member

    Newcomer
    3
    Mar 3, 2016
    Awesome work! Like @DeuX I have over 1000 NSP's already and it would be wonderful if we could get it working with existing .NSP's as I may have some dupes from different CDNSP versions :)
     
  8. blawar
    OP

    blawar GBAtemp Advanced Fan

    Member
    9
    Nov 21, 2016
    United States
    just, it scans the current directory (and sub directories for existing NSP's and will organize them too. the filename just needs to have the titleid somewhere in brackets.

    — Posts automatically merged - Please don't double post! —

    no, it will pick up existing NSP's as well, as long as they have the title id in the filename in brackets like Mario Kart 8 Deluxe[0100152000022000].nsp

    — Posts automatically merged - Please don't double post! —

    You do not @DeuX , just make sure the NSP's are in the same directory, or a sub directory under the script and it will automatically pick them up as long as they have the title id in brackets in the filename like Mario Kart 8 Deluxe[0100152000022000].nsp
     
  9. DeuX

    DeuX GBAtemp Regular

    Member
    4
    Jun 14, 2018
    Switzerland
    That’s neat! So if I’m storing my games in an external harddrive I should execute the script from that drive?
     
  10. gene0915

    gene0915 GBAtemp Regular

    Member
    5
    Apr 6, 2008
    United States
    How much space does that take up?
     
  11. blawar
    OP

    blawar GBAtemp Advanced Fan

    Member
    9
    Nov 21, 2016
    United States
    Correct. But remember it needs to be in a valid CDNSP directory with a valid titlekeys.txt. It uses the titlekeys.txt to rename / move / organize. Any title id's found that arent in titlekeys.txt get ignored.

    If you do not want it to download anything, just organize, put some garbage in whitelist.txt and save it. Then it will effectively reorganize only. I'm going to clean it up a lot. But like a lot of you, I had 1000's of NSP's tha twere not consistently named and management was a nightmare :(

    — Posts automatically merged - Please don't double post! —

    1.21 TB if you download everything, but i have a few DLC and updates in there.
     
    Don Jon likes this.
  12. DeuX

    DeuX GBAtemp Regular

    Member
    4
    Jun 14, 2018
    Switzerland
    I'm using CDNSP GUI by BOB, I'll check the regular CDNSP
     
  13. blawar
    OP

    blawar GBAtemp Advanced Fan

    Member
    9
    Nov 21, 2016
    United States
    I use CDNSP GUI by bob too, it works. The bob gui I use, also has the regular CDNSP.py script it in. Just place these files in that directory and it will work.

    You will have to add one line of code to the top of the CDNSP.py script because the people who wrote that script have a bug where they import one library (tqdm) in their main routine, however I am using the script as a library / module so that module never gets imported and errors out when they try to call it. Unforuntately, there is no github repo for me to submit the bug report.
     
    Last edited by blawar, Jul 27, 2018
  14. DeuX

    DeuX GBAtemp Regular

    Member
    4
    Jun 14, 2018
    Switzerland
    Didn't work for me. I put your script inside the CDNSP GUI, added "from tqdm import tqdm" to the top of CDNSP.py and executed your script. It opened a window fast, created a cache folder and closed. Didn't have much time to tinker with it yet, so I guess I did something wrong
     
  15. blawar
    OP

    blawar GBAtemp Advanced Fan

    Member
    9
    Nov 21, 2016
    United States
    I just updated the repo, do not copy the titlekeys.txt from the repo if you are using CDNSP as it has all of the keys zero'd out. I also added the CDNSP.py file since I do not believe that file on its own violates any rules. If the mod take this down, i'll remove it.

    I added the ability to organize your existing NSP's only without CDNSP. If that is what you want, extract all of these files on their own (except CDNSP.py). Make sure the NSP files are in or under the directory you extract the files.

    https://github.com/blawar/nut
     
    morrison22 and DeuX like this.
  16. DeuX

    DeuX GBAtemp Regular

    Member
    4
    Jun 14, 2018
    Switzerland
    Worked Perfectly now! They are organized and looking good :D
    Now to see if I can get it to work with CDNSP GUI
    Thank you!
     
  17. blawar
    OP

    blawar GBAtemp Advanced Fan

    Member
    9
    Nov 21, 2016
    United States
    You're welcome, I am glad you got it working!

    It will work with the GUI, that is what I use.

    I am going to add a configuration file / options to, among other things, specify different formats for how it structures the files. I think some people would want all of the base files in the same directory, and then squirrel away updates and DLC in sub directories perhaps?
     
    morrison22 likes this.
  18. DeuX

    DeuX GBAtemp Regular

    Member
    4
    Jun 14, 2018
    Switzerland
    Yeah, it's a good idea, not everyone likes's to keep files the same way as the neighbor.
    My CDNSP GUI has no CDNSP.py, so I'm a bit unsure how to do it. Maybe you can make it work with CDNSP GUI py file instead? Since I believe more people use the GUI version by BOB than the regular one
     
  19. blawar
    OP

    blawar GBAtemp Advanced Fan

    Member
    9
    Nov 21, 2016
    United States
    Look at the repo, I added a CDNSP.py, just extract it to your CDNSP GUI folder. And it probably would work with your CDNSP GUI if you renamed it to CDNSY.py

    https://github.com/blawar/nut
     
  20. morrison22

    morrison22 GBAtemp Advanced Fan

    Member
    8
    Nov 26, 2005
    United States
    This looks very promising! Thank you. I will be testing it later.

    The readme states the following:

    It saves the games in the following dir structure: titles/{name}/{name}[{title_id}].nsp
    DLC are saved to: titles/{name}/DLC/{name}[{title_id}].nsp
    Updates are saved to: titles/{name}/updates/{name}[{title_id}].nsp


    If I do not want folders created for each NSP and instead, want all NSPs in one folder with the naming scheme shown below, what would need to be done to make this happen?

    Game Name [BASE] [TitleID] [v].nsp
    Game Name [UPD] [TitleID] [v].nsp
    Game Name [DLC] [TitleID] [v].nsp
    Game Name [DEMO] [BASE] [TitleID] [v].nsp
    Game Name [DEMO] [UPD] [TitleID] [v].nsp


    Also #1, How does the renaming handle invalid characters such as ":" for example? Can it be specified to replace ":" with "-"?
    Also #2, Once a title is blacklisted, can it still be set to download updates only for that title when available? (Updates share the same TitleID as the Base Title, so I'm not sure how this would work.)

    This is an excellent project that has come to life and is very much needed! Thank you for doing this! A+
     
    Last edited by morrison22, Jul 28, 2018
Loading...