ROM Hack sky3ds diskwriter python clone (for linux & osx)

[NoSmokingBandit]

for the root stuff maybe take a look at unetbootin, they are doing something to get root privs, maybe you can copy that

I hate to bump this thread so much, and I am making progress in rebuilding my gui, but I found this interesting line of code while poking around in unetbootin (btw, they get sudo access via applescript on macs. Super ugly, I've found a slightly less ugly way to do it, but more on that later).

QString diskutilList = callexternapp("diskutil", "list");
QStringList usbdevsL = diskutilList.split("\n").filter(QRegExp("(FAT|Microsoft)")).join(" ").split(" ").filter("disk");

They use 'diskutil list' to get disks on a mac as well. They don't need to return a size, just the mount point, but the process is almost identical to what I came up with. They could probably have more readable code if they parsed a plist like I do, but I don't think they care as much as I do about the code being human-readable.

 

[lukas_2511]

I hate to bump this thread so much, and I am making progress in rebuilding my gui, but I found this interesting line of code while poking around in unetbootin (btw, they get sudo access via applescript on macs. Super ugly, I've found a slightly less ugly way to do it, but more on that later).

[...]

They use 'diskutil list' to get disks on a mac as well. They don't need to return a size, just the mount point, but the process is almost identical to what I came up with. They could probably have more readable code if they parsed a plist like I do, but I don't think they care as much as I do about the code being human-readable.

Yea I don't know, I wouldn't take unetbootin as code reference, it was really buggy when i used it the last time on osx... was just refering to that tool because of the root/sudo-thing. Wouldn't recommend looking into it any further.

I'll try to get my hands on a osx machine, maybe will install it in VirtualBox or something. Don't want to implement shell-wrappers completely blind...

 

[NoSmokingBandit]

Yea I don't know, I wouldn't take unetbootin as code reference, it was really buggy when i used it the last time on osx... was just refering to that tool because of the root/sudo-thing. Wouldn't recommend looking into it any further.

I'll try to get my hands on a osx machine, maybe will install it in VirtualBox or something. Don't want to implement shell-wrappers completely blind...

I was actually using unetbootin to make a usb drive to install elementary. Didn't work and I had to use something else. But I had to be nosy and poke around the source. I don't have linux installed anywhere so I need to get that going to test a few things. If I get motivated I might have a gui put together by the end of the week. It is taking longer that I'd like because 1) I don't really know what I'm doing and 2) the aforementioned motivation dwindles quicker than I'd like after work and getting stuff done at home. But I will finish it, I promise.

I don't know C++ very well, but unetbootin's code looks super messy. Very hard to read, but everything is hard to read compared to python.

Surely you've heard of osx86? I've been running osx on my pc since Tiger. Tiger only barely worked, but since Leopard I've been 100% compatible with a little bit of work. http://www.insanelymac.com/

Do you care much about window support with this project? I haven't looked too hard but it seems like getting a list of disks in windows (ie not partitions like C:\ etc.) requires importing modules I'd rather not bother with and isn't terribly reliable anyway. I'm not going to implement anything windows-specific unless anyone really wants it.

 

[lukas_2511]

Surely you've heard of osx86? I've been running osx on my pc since Tiger. Tiger only barely worked, but since Leopard I've been 100% compatible with a little bit of work. http://www.insanelymac.com/

[...]

Do you care much about window support with this project?

I'd suggest just installing some linux distro in VirtualBox.

Sure i know my hackintoshs But I don't want to spend to much time to first install an older version of osx to prepare files to install a newer version, and I really want to make sure that stuff is working on the newest official release.

On Windows support... I actually didn't try if my code is working on there, but I mean, on Windows you could just use the normal DiskWriter tool... sure my code now has some features that normal diskwriter doesn't have, but most of them can be accomplished by editing and viewing a few files manually, mostly template.txt... so not really going to invest a lot of time there. Also windows has PhysicalDisk0 identifiers or something like that, not just the visible "C:\" stuff.

 

[lukas_2511]

Just had a look at the features I added today (well, yesterday, ohgowd5amgn8). Looked like i forgot to recalculate the checksum for the header data after injection and sky3ds didn't recognize the rom this way.
Added checksum calculation, imported my own backup of Pokemon Y, and at least my Card2 savegame is now loading perfectly, without any manual modifications

Can't test any online features atm, have no internet connections on sysnand and probably not the latest update for the game anyway. May test it in emunand classicmode as soon as my gateway3ds arrives.

 

[DjoeN]

Just had a look at the features I added today (well, yesterday, ohgowd5amgn8). Looked like i forgot to recalculate the checksum for the header data after injection and sky3ds didn't recognize the rom this way.
Added checksum calculation, imported my own backup of Pokemon Y, and at least my Card2 savegame is now loading perfectly, without any manual modifications

Can't test any online features atm, have no internet connections on sysnand and probably not the latest update for the game anyway. May test it in emunand classicmode as soon as my gateway3ds arrives.

time to install Linux on an old Laptop, just for the extra features your tool has that makes life easier

 

[NoSmokingBandit]

I was finally able to get linux installed (windows 8 was throwing a hissy fit about me resizing a partition, so I had to run dskchk twice to get it to do anything useful) so I can test a few things. It all looks good thus far, so I'll be able to work a little quicker now. Tk looks like complete garbage in linux (oddly it looks ok enough in osx) so I'll probably convert everything to ttk. It ends up being a few more lines, but its worth it so the gui doesnt look like windows 95.

 

[lukas_2511]

Soooooo new things:

• Unfinished automatic template.txt data generator (probably will never work, I have the feeling that the unknown 16 bytes are a cryptographic hash... that's probably the reason why template.txt exists... everything but those 16 bytes can be generated from game dump)
• Savegame Backups should now contain all data required by normal DiskWriter (untested)
• Savegame Restores now should write the unique id from savegame backup back to disk (kinda tested, but not really)
• Added option (-Z) to backup all savegames on disk, stored in "savegames" subdirectory of sky3ds.py data directory (as savegames/gamename (if available)/product_code-date.sav)
• Some options now can be more verbose, just add "-v" and it will show the unique id and card id in list, and will show used template data on write_rom.

Also some bugfixes:

• When using the kinda undocumented header.bin feature it will now only read the unique id, otherwise the game won't work
• Fixed reading template data from template.txt (failed on some entries)
• Raise error if template data is not 0x200 (512) bytes long

 

[NoSmokingBandit]

Fyi, I'm making progress as well. Not near as much as Lukas, who is a machine, but I'm getting there. GUIs are a pain to make. And I really don't know what I'm doing all that well. I've gotten almost all of the hard stuff done, its mostly just attaching functions to buttons and making sure everything works on every platform.

I finally looked at what you did with the progress bar calls. Pretty smart stuff there, very clever.

I'd post a screenshot but it looks identical to the last one.

If I could make a request, don't print template data when writing a rom unless one uses -v like you do for dumping a rom. In order to get all of your warnings to the user through gui widgets I had to send stdout into a messagebox widget. The problem is that there is no way (that I know of) to filter these things. This way I'll be able to use your errors like "Error: Not enough free continous blocks" and the report from updating titles without a giant popup of template data.

 

[Dax_Fame]

Wow good stuff! Just popped up and recents and as a person who primarily uses Linux it would be nice to not have to boot into windows every time I wanted to manage my cart. Looking forward to it, thanks!

I am not a sky3ds owner just yet, but by the looks of things I will be soon.

Now, to read the pages of comments for potential progress spoilers

 

[lukas_2511]

If I could make a request, don't print template data when writing a rom unless one uses -v like you do for dumping a rom. In order to get all of your warnings to the user through gui widgets I had to send stdout into a messagebox widget. The problem is that there is no way (that I know of) to filter these things. This way I'll be able to use your errors like "Error: Not enough free continous blocks" and the report from updating titles without a giant popup of template data.

Actually it only shows template data in verbose mode, or at least it should behave this way.

I may add a simple message-class, and integrate it similar to the progressbar stuff. Would only be for notices (like "Using template data from .3dz"), since Exceptions end the execution of the function anyway and you can catch those easily, and print them as error message. Here have some dummycode:

try:
    write_rom(bla)
except Exception as e:
    errorbox(e)

Edit: Also you should use "try" around every function of the sky3ds class you call, and show an error message if it throws an exception, otherwise the gui would just close without any further notices.

 

[NoSmokingBandit]

I have exceptions handled easily enough, tk has a method for sending exceptions anywhere you want. Its really easy to catch prints as well, but thats different altogether.

Only exceptions in the main loop kill the whole thing. Any exception in a separate class will just kill that instance of the class and keep the tkinter loop running as usual. I like to set it up so that all the tkinter loop does is draw widgets. Then I have a class that gathers and validates whatever the user inputs and passes that along to the parts that do the actual work. Since everything is completely separate its kind of hard to kill the whole thing. Part of that is why it takes a while to make a gui; you want to handle exceptions and crashes in a way that lets the user know what is happening while causing the least amount of interruption.

Heres a simple example. That exception will get printed and will kill that instance of the controller class, but everything else works as if nothing happened.

from Tkinter import *
root = Tk()
 
class View(Frame):
    def __init__(self, root):
        Frame.__init__(self, root, background="white")
 
        self.root = root
        self.root.title("**Title**")
 
        self.create_widgets()
 
    def create_widgets(self):
        a = Button(root, text = 'button 1', command = controller.button_1)
        a.pack()
 
        b = Button(root, text = 'button 2', command = controller.button_2)
        b.pack()
 
        self.label_text = StringVar()
        l = Label(root, textvariable = self.label_text)
        l.pack()
 
class Controller:
    def __init__(self):
        self.num = 1
 
    def button_1(self): 
        view.label_text.set(self.num)
        self.num += 1
 
    def button_2(self):
        raise Exception("OH NO!")
 
if __name__ == '__main__':
    controller = Controller()
    view = View(root)
    root.mainloop()

edit:

I did notice that the two print statements I'm instested in catching are:

if start_block == 0:
    print("Error: Not enough free continous blocks")
    return

And:

if free_slot == -1:
    print("Error: No free slot found. There can be a maximum of %d games on one card." % int(position_header_length / 0x8))
    return

both in disk.py

Since you are ending teh script anyway why not just make them exceptions? Then I wouldn't care at all what prints out and when.

 

[lukas_2511]

I have exceptions handled easily enough, tk has a method for sending exceptions anywhere you want. Its really easy to catch prints as well, but thats different altogether.

Ok.

I did notice that the two print statements I'm instested in catching are:
[...]
both in disk.py

Since you are ending teh script anyway why not just make them exceptions? Then I wouldn't care at all what prints out and when.

I guess you are right. I'll replace those types of messages with exceptions soon. I think those two were one of the first two error messages I implemented, and then it was just for debugging and in the middle of a big piece of code, but yea, they should now be exceptions.

 

[lukas_2511]

Other thing:

Has anybody a Pokemon Y (EUR) savegame dump from DiskWriter and/or sky3ds.py (ideally both) lying around? I want to test the unique id import thingy I build.

 

[benseab23]

Hi guys,

I just wondered if anyone would be kind enough to write a quick 'how-to' guide for those of us new Sky3DS owners with only a Mac, hoping to use Lukas' disk writer clone, but with no previous experience of Python.

I know a big newbie question, but would be a wonderful and really helpful resource to many people I think.

Thanks for any help!

Hey there,

I'm currently working on a Python version of DiskWriter to be able to use it on Linux and OSX machines.
Not sure if anyone attempted this before, I wasn't able to find much information on how DiskWriter stores data to the sd-card, so I had to figure out most by just looking at hexdumps of my sd-card.

Edit:

For new users to this thread: The tool is actually already working. Under OSX it can't detect the disksize (yet), but I am working on that. If you know how to do things in python you can hardcode the size and this tool will work!

Things I still want to improve: 3dz unique-id import (instead of using the data from template.txt), unique-id re-import on savegame import. Both should be relatively easy, I just didn't really need those features yet, so I didn't do that work.

 

[NoSmokingBandit]

Give me a few more days and I'll have the gui done that I'm working on. I'm trying to make it as user-friendly as possible. So much that it is taking me a lot of time testing things and trying to break stuff so other users don't run into errors.

But for now...

Python is pretty easy to use since it comes with osx.

Open terminal.

type "diskutil list"

Find the drive that is your sd card. It will be '/dev/disk#'. Remember that.

Type "cd " and drag the sky3ds.py folder into the terminal window. Hit enter.

Figure out what you want to do. We'll just list roms for now.

"sudo python sky3ds.py -d /dev/disk# -l"

replace '/dev/disk#' with the disk you found earlier. This can change depending on what you have attached to your computer, so always check first.

From now on all of your commands will start with "sudo python sky3ds.py -d /dev/disk#" That tells the script what disk to run the commands on.

To write a rom add "-w " and drag a rom into the terminal window.

Everything else follows the same pattern.


edit:
I'm going to need someone with a mac and/or linux to test out some stuff for me. I don't have a sky3ds card (yet) so I can't make sure the save write/dump parts are working how I expect them to. The gui is pretty close to done, I just want to make sure everything works. I can only think of so many ways to break things. PM me if you want to help out.

edit 2:
It does exist!


Oh god the linux widgets are so ugly! Shield your eyes!


The problem is that switching over to ttk makes the mac version look like ass. I'll eventually get there. As far as I can tell everything functions well.

 

[lukas_2511]

Give me a few more days and I'll have the gui done that I'm working on. I'm trying to make it as user-friendly as possible. So much that it is taking me a lot of time testing things and trying to break stuff so other users don't run into errors.
[...]
The problem is that switching over to ttk makes the mac version look like ass. I'll eventually get there. As far as I can tell everything functions well.

Hey, looks nice.

I've changed some things in code, there now is a function wrapping around diskutil to get disk size on osx (using natively parsed plist output, so should be kinda clean), also i changed the "print"s i used to show information to "logging.info" and "logging.warning", so you can just set up a new logging handler in your script (that shows messages boxes or logs to a console window or whatever).

Also I've added two parameters to disk.py, diskfp and disk_size, so you can give those directly to the class, instead of relying on its internal functions.

Can you try to get your code working with the newest version, and upload it somewhere?

 

[Poison Shroom]

This is really awesome. Thanks for all the hard work.
The GUI looks great as well, can't wait until that's available

I'm trying to use this for the first time and running into trouble. First time I started the script it asked me to put the template file in my library folder. I moved and renamed it, and relaunched the program. It told me it couldn't find the disk, which I expected since it doesn't know where to look. Now, whenever I give it a command I get the error command not found. I tried defining the SD card path and help command but kept getting the error for everything I tried.


What the heck am I doing wrong? I'm not great at terminal/unix but it's not my first time. I'm using OS X 10.10 and download Python 3.4.2

 

[lukas_2511]

This is really awesome. Thanks for all the hard work.
The GUI looks great as well, can't wait until that's available

I'm trying to use this for the first time and running into trouble. First time I started the script it asked me to put the template file in my library folder. I moved and renamed it, and relaunched the program. It told me it couldn't find the disk, which I expected since it doesn't know where to look. Now, whenever I give it a command I get the error command not found. I tried defining the SD card path and help command but kept getting the error for everything I tried.


What the heck am I doing wrong? I'm not great at terminal/unix but it's not my first time. I'm using OS X 10.10 and download Python 3.4.2

Sorry, I don't have an osx machine here so development for that system is going kinda slow.

Does "ERROR" appear in big letters?

 

[Awesomeslayerg]

Can you make it possible to edit NAND type like x/y/oras saves to work with PkHex without having to buy a cybergadget save editor?