ROM Hack sky3ds diskwriter python clone (for linux & osx)

[Poison Shroom]

Sorry, I don't have an osx machine here so development for that system is going kinda slow.

Does "ERROR" appear in big letters?

Here's a copy of my terminal window.


Last login: Mon Feb 16 13:42:25 on ttys000
Benjamins-MBP:~ Ben$ cd '/Users/Ben/Desktop/sky3ds.py-master/' && '/usr/bin/pythonw' '/Users/Ben/Desktop/sky3ds.py-master/sky3ds.py' && echo Exit status: $? && exit 1
Found updated template.txt. Converting...
No disk specified.
Benjamins-MBP:sky3ds.py-master Ben$ -d sdcard /dev/disk1
-bash: -d: command not found
Benjamins-MBP:sky3ds.py-master Ben$ -d
-bash: -d: command not found
Benjamins-MBP:sky3ds.py-master Ben$ -h
-bash: -h: command not found
Benjamins-MBP:sky3ds.py-master Ben$ --help
-bash: --help: command not found

Rereading it the echo exit status thing looks worrisome but I have no idea what that actually means.

 

[Tyrantt]

So how do you download this? i only see the links to github, which i know nothing about python to contribute

 

[Poison Shroom]

So how do you download this? i only see the links to github, which i know nothing about python to contribute

On the github page there's a save .zip button on the right side, it's the last button. The third party plugins didn't download for me (I don't know git enough to know if that's normal.) So click the third_party folder, and open the git pages for appdirs and progressbar, download them and throw them in the third_party folder. Google Python, download Python 3 for your OS and then use Python to launch the script.

 

[Tyrantt]

okay, so i did everything except the launching the script. how do i launch the script? Ive never used python before

 

[lukas_2511]

Here's a copy of my terminal window.


Last login: Mon Feb 16 13:42:25 on ttys000
Benjamins-MBP:~ Ben$ cd '/Users/Ben/Desktop/sky3ds.py-master/' && '/usr/bin/pythonw' '/Users/Ben/Desktop/sky3ds.py-master/sky3ds.py' && echo Exit status: $? && exit 1
Found updated template.txt. Converting...
No disk specified.
Benjamins-MBP:sky3ds.py-master Ben$ -d sdcard /dev/disk1
-bash: -d: command not found
Benjamins-MBP:sky3ds.py-master Ben$ -d
-bash: -d: command not found
Benjamins-MBP:sky3ds.py-master Ben$ -h
-bash: -h: command not found
Benjamins-MBP:sky3ds.py-master Ben$ --help
-bash: --help: command not found

Rereading it the echo exit status thing looks worrisome but I have no idea what that actually means.

Uhm... instead of trying the command "-d /dev/diskN" you maay want to try "python sky3ds.py -d /dev/diskN", but from what I see you should maaybe wait until the gui version is finished...

 

[lukas_2511]

okay, so i did everything except the launching the script. how do i launch the script? Ive never used python before

Sorry, but maybe you also should wait for the gui-version.

 

[lukas_2511]

Can you make it possible to edit NAND type like x/y/oras saves to work with PkHex without having to buy a cybergadget save editor?

No, I don't think that's possible.

 

[Poison Shroom]

Uhm... instead of trying the command "-d /dev/diskN" you maay want to try "python sky3ds.py -d /dev/diskN", but from what I see you should maaybe wait until the gui version is finished...

That worked perfectly, now I can't figure out why it will not access the disk. I read the earlier post about needing sudo access, and when I try sudo or su it prompts me for my password and then just says "su: sorry". Really confusing me I tried sudo before the commands relating to the python script (i.e. su python sky3ds.py -d /dev/disk). I tried opening a new window and just doing su, and I get the same error. I've done that in the past to remove hidden files and can't figure out why it's denying me su access. Even changed my password

Another question if I ever figure this out. I tried the diskutil command. disk0 is a recovery system on my boot drive, disk1 is my boot drive, disk2 is my second hard drive. I see no disk3, should I be worried my computer doesn't even read the micro SD card?

I'd like to get this to work but I may just wait for the GUI version haha

 

[NoSmokingBandit]

Hey, looks nice.

I've changed some things in code, there now is a function wrapping around diskutil to get disk size on osx (using natively parsed plist output, so should be kinda clean), also i changed the "print"s i used to show information to "logging.info" and "logging.warning", so you can just set up a new logging handler in your script (that shows messages boxes or logs to a console window or whatever).

Also I've added two parameters to disk.py, diskfp and disk_size, so you can give those directly to the class, instead of relying on its internal functions.

Can you try to get your code working with the newest version, and upload it somewhere?

Had I known you were going to use diskutil for sizes I'd have given you my method for it. Plist parsing is a pita, but diskutil is surprisingly useful.

I got a PM from Cliemacfr, but I think I'll just post everything I have. Permanent open beta. First I'm going to download the latest commit from Lukas, make sure my stuff works, then post a link to my stuff.

Sidenote:
ttk doesn't look any better on linux. There just aren't any tkinter widget sets that look ok. I really don't want to go with wxpython since some distros don't come with it (i know for a fact that osx does not), so we'll just deal with it being a little ugly.

 

[lukas_2511]

Had I known you were going to use diskutil for sizes I'd have given you my method for it. Plist parsing is a pita, but diskutil is surprisingly useful.

I mean, this is more or less a temporary solution, and the plist parsing wasn't a problem, 3 lines of code or so, mostly copied from an older project.

What I kinda still want is a feature to check if a partition on a disk is mounted, as some kind of security feature, so people don't try to erase their system disks...

 

[NoSmokingBandit]

OSX won't let the script format a drive that is connected iirc, so I had to put in a method that unmounts all volumes from the chosen disk before i send it to your format() method. It won't/can't unmount a partition in use, so its a half-assed failsafe. I'm certainly not going to try it and found out.

edit:
Hmmmm...

$ sudo python sky3ds.py -d '/dev/disk3' -l
ERROR: Couldn't get disksize, will not continue.

This is a slightly modified version of what I use to get disk sizes (since I need to loop through disks and convert bytes to something human-readable I took out all of the unnecessary things):

                size_command = ['diskutil', 'info', '-plist', disk]
           
                try:
                    output = subprocess.check_output(size_command)
                except subprocess.CalledProcessError as err:
                    raise Exception("Could not get disk information. \nError: %s" % err)
       
                size_plist = plistlib.readPlistFromString(output)
           
                return size_plist["TotalSize"]

If you run diskutil info -plist /dev/disk0 you get a very neat little output for just that physical disk.

 

[lukas_2511]

If you run diskutil info -plist /dev/disk0 you get a very neat little output for just that physical disk.

I only had the output of the "diskutil list -plist" available, guessed it would be the same with disk as parameter, but seems i was wrong. Will fix it soon (uh dangerous word).

 

[NoSmokingBandit]

$ diskutil info -plist /dev/disk3
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Bootable</key>
    <false/>
    <key>BusProtocol</key>
    <string>USB</string>
    <key>CanBeMadeBootable</key>
    <false/>
    <key>CanBeMadeBootableRequiresDestroy</key>
    <false/>
    <key>Content</key>
    <string></string>
    <key>DeviceBlockSize</key>
    <integer>512</integer>
    <key>DeviceIdentifier</key>
    <string>disk3</string>
    <key>DeviceNode</key>
    <string>/dev/disk3</string>
    <key>DeviceTreePath</key>
    <string>IODeviceTree:/PCI0@0/USBE@1D,7</string>
    <key>Ejectable</key>
    <true/>
    <key>FreeSpace</key>
    <integer>0</integer>
    <key>GlobalPermissionsEnabled</key>
    <false/>
    <key>IOKitSize</key>
    <integer>4258267136</integer>
    <key>Internal</key>
    <false/>
    <key>LowLevelFormatSupported</key>
    <false/>
    <key>MediaName</key>
    <string>SMI USB DISK Media</string>
    <key>MediaType</key>
    <string>Generic</string>
    <key>MountPoint</key>
    <string></string>
    <key>OS9DriversInstalled</key>
    <false/>
    <key>ParentWholeDisk</key>
    <string>disk3</string>
    <key>RAIDMaster</key>
    <false/>
    <key>RAIDSlice</key>
    <false/>
    <key>SMARTStatus</key>
    <string>Not Supported</string>
    <key>SupportsGlobalPermissionsDisable</key>
    <false/>
    <key>SystemImage</key>
    <false/>
    <key>TotalSize</key>
    <integer>4258267136</integer>
    <key>VolumeName</key>
    <string></string>
    <key>WholeDisk</key>
    <true/>
    <key>Writable</key>
    <true/>
    <key>WritableMedia</key>
    <true/>
    <key>WritableVolume</key>
    <false/>
</dict>
</plist>

I actually prefer diskutil to any tools I've found in linux. lsblk got the job done for me in this project, but diskutil is just so easy to manipulate.

Edit:
I'm a moron and posted the wrong output. Its fixed now.

I've considered running a check for busprotocol so that internal hdds don't show up. Maybe some day. I wonder if lsblk can get something similar...

 

[lukas_2511]

I actually prefer diskutil to any tools I've found in linux. lsblk got the job done for me in this project, but diskutil is just so easy to manipulate.

diskutil better... pah!

On Linux you just go into /sys/class/block and baam, there are all devices that are recognized as blockdevices, containing files with information about basically anything you may ever want to know about a disk, including what physical device it is, was size it is, if it's a removable disk, how many data is currently in-flight, ...was written, ...was read, sub-block-devices (like sda1, sda2, ... for sda) everything.

Also i uploaded a new version, I think it should work now.

 

[NoSmokingBandit]

Pfffft. I like what I choose to like.

Still not working, but I've found out why. The plistlib module creates really weird, deep dictionaries.

in get_disk_size() disk_plist is:

{'AllDisks': ['disk3'], 'WholeDisks': ['disk3'], 'AllDisksAndPartitions': [{'Content': '', 'DeviceIdentifier': 'disk3', 'Size': 4258267136}], 'VolumesFromDisks': []}

So in order to verify the device id you need to drill down like this:

disk_plist['AllDisksAndPartitions'][0]['DeviceIdentifier']

I don't like it either, but thats the only way to get to the 'disk3' string you need to verify. That stuff is too easy to miss.

Edit:
Noticed you'd need this too:

self.disk_size = disk_plist['AllDisksAndPartitions'][0]['Size']

And you also seem to have missed a 'return self.disk_size' at the end of that method.

    def get_disk_size(self):
        """Get sdcard size in bytes
 
        This currently is an ugly workaround. It seeks to the end of the sdcard
        and reads how many bytes were skipped. This should be replaced with
        something more clean."""
 
        if sys.platform == 'darwin':
            # meh
            if not re.match("^\/dev\/disk[0-9]+$", self.disk_path):
                raise Exception("Disk path must be in format /dev/diskN")
            try:
                diskname = os.path.basename(self.disk_path)
                diskutil_output = subprocess.check_output(["diskutil", "list", "-plist", self.disk_path])
                if sys.version_info.major == 3:
                    disk_plist = plistlib.loads(bytearray(diskutil_output, 'utf-8'))
                else:
                    disk_plist = plistlib.readPlistFromString(diskutil_output)
             
                if not disk_plist['AllDisksAndPartitions'][0]['DeviceIdentifier'] == diskname:
                    raise Exception("DeviceIdentifier doesn't match, won't continue.")
 
                self.disk_size = disk_plist['AllDisksAndPartitions'][0]['Size']
             
                return self.disk_size
             
            except Exception as e:
                raise Exception("Can't get disk size from diskutil :(\nError was: %s" % e)
 
            raise Exception("Sorry, disk size detection for osx is currently not working.")
        else:
            self.diskfp.seek(0, os.SEEK_END)
            disk_size = self.diskfp.tell()
            disk_size = disk_size - disk_size % 0x2000000
            if disk_size == 0:
                raise Exception("0 byte disk?!")
            self.disk_size = disk_size

... gives me this:

sudo python sky3ds.py -d /dev/disk3 -l
| Slot | Start  | Size  | Type  | Code      | Title                                              |
|    0 | 160 MB | 512 MB | Card1 | CTR-P-ACCE | Pokemon Rumble Blast                              |
|    1 | 672 MB | 256 MB | Card1 | CTR-P-AVTE | Adventure Time: The Secret of the Nameless Kingdom |
 
Disk Size: 4061 MB | Free space: 3232 MB | Largest free continous space: 3104 MB

 

[lukas_2511]

Pfffft. I like what I choose to like.

Still not working, but I've found out why. The plistlib module creates really weird, deep dictionaries.

in get_disk_size() disk_plist is:

{'AllDisks': ['disk3'], 'WholeDisks': ['disk3'], 'AllDisksAndPartitions': [{'Content': '', 'DeviceIdentifier': 'disk3', 'Size': 4258267136}], 'VolumesFromDisks': []}

So in order to verify the device id you need to drill down like this:

disk_plist['AllDisksAndPartitions'][0]['DeviceIdentifier']

I don't like it either, but thats the only way to get to the 'disk3' string you need to verify. That stuff is too easy to miss.

That is exactly what i had before!?! Also the return is not needed.

 

[NoSmokingBandit]

You did that that a few commits ago. Did I tell you to change that? Don't listen to me
Looking at the changes in github it looks like you were cleaning up your variable names and probably accidentally took that out. It makes a lot more sense the way you originally had it.

This thread must be sooooo boring for anyone but us.

If definitely throws an exception without returning something. It comes from line 52 in disk.py. If I put in the return statement it works perfectly.

Working for a platform you don't have access to is incredibly difficult. I'm impressed with your diskutil and plist skills.

 

[lukas_2511]

You did that that a few commits ago. Did I tell you to change that? Don't listen to me
Looking at the changes in github it looks like you were cleaning up your variable names and probably accidentally took that out. It makes a lot more sense the way you originally had it.

This thread must be sooooo boring for anyone but us.

If definitely throws an exception without returning something. It comes from line 52 in disk.py. If I put in the return statement it works perfectly.

Working for a platform you don't have access to is incredibly difficult. I'm impressed with your diskutil and plist skills.

Okay, I guess i have found the problem... should've removed the dummy-exception for the missing feature... The return statement made the function end before it would reach this part of the code... so now it should work.

 

[NoSmokingBandit]

One more thing.

                if sys.version_info.major == 3:
                    diskutil_plist = plistlib.loads(bytearray(diskutil_output, 'utf-8'))
                else:
                    diskutil_plist = plistlib.readPlistFromString(diskutil_output)
 
                disk_plist = plist['AllDisksAndPartitions'][0]

Can you spot the error?

I do that kind of stuff all the time. Brackets has a great plugin called WordHint that helps a lot with keeping things straight.

That said, here is my stuff. Should work well with the latest commit on linux, should work soon enough on osx.

https://www.dropbox.com/sh/dlen9rls1njvv9y/AAD4_rnigMGZaaB3-R0RzYqaa?dl=0

Just drop both of those in the folder containing "sky3ds.py". The python 3 version was converted with 2to3, so I don't know if it actually works at all. There were only minor changes.

For those who don't know what version they have just try gui.py first. Or run "python -v" in terminal and look for 2.X or 3.X in the lines that print out.

 

[lukas_2511]

Can you spot the error?

I do that kind of stuff all the time. Brackets has a great plugin called WordHint that helps a lot with keeping things straight.

That said, here is my stuff. Should work well with the latest commit on linux, should work soon enough on osx.

https://www.dropbox.com/sh/dlen9rls1njvv9y/AAD4_rnigMGZaaB3-R0RzYqaa?dl=0

Just drop both of those in the folder containing "sky3ds.py". The python 3 version was converted with 2to3, so I don't know if it actually works at all. There were only minor changes.

For those who don't know what version they have just try gui.py first. Or run "python -v" in terminal and look for 2.X or 3.X in the lines that print out.

Fixed it. It's problematic to write code without being able to really test it

I'll take a look at your code in a minute, but I guess i'll go sleep a bit before merging anything into the git repository.