sky3ds diskwriter python clone (for linux & osx)

Discussion in '3DS - ROM Hacking, Translations and Utilities' started by lukas_2511, Jan 4, 2015.

  1. lukas_2511
    OP

    lukas_2511 GBAtemp Regular

    Member
    125
    45
    Jan 4, 2015
    Gambia, The
    Hey there,

    I'm currently working on a Python version of DiskWriter to be able to use it on Linux and OSX machines.
    Not sure if anyone attempted this before, I wasn't able to find much information on how DiskWriter stores data to the sd-card, so I had to figure out most by just looking at hexdumps of my sd-card.

    Edit:

    For new users to this thread: The tool is actually already working. Under OSX it can't detect the disksize (yet), but I am working on that. If you know how to do things in python you can hardcode the size and this tool will work!

    Things I still want to improve: 3dz unique-id import (instead of using the data from template.txt), unique-id re-import on savegame import. Both should be relatively easy, I just didn't really need those features yet, so I didn't do that work.
     
    pattho, benseab23, Dax_Fame and 5 others like this.


  2. GothicIII

    GothicIII GBAtemp Fan

    Member
    488
    134
    Jan 4, 2015
    Gambia, The
    Hi,
    I registered just for this. I did a Hexdump research by myself.
    I also have an alternative method to make the 11th game work on red sky3ds without the button. It would be great if i can share my knowledge to help you with the tool.
    Ill post my findings in German. PM me if you want.

    Code:
    Aufbau der µSD:
     
    0x0000000 DiskWriter ROM-Informationen bis 0xF8.
    0x0000100 String "ROMs"
    0x0000108 Filler mit "FF" bis 0x1BD
    0x00001BE Anfang? Der FAT32-Partition.
    0x0100000 Card1 Spielstände im Abstand von 0x100000 bis 0x2000000. Max 31 Spielstände möglich.
    0x2000000 Beginn der ersten ROM. Dies ist Sektor Nr. 0x10000
     
    Cubic Ninja.3ds hat einen Null-byte-Bereich bei 0x57A8E00 ~87MB mit einer Länge von 0x28571F0 ~40MB (Auffüllunng von Bytes bis zu einer Größe von 2^x)
     
    ====================
    DiskWriter Startsektor und Größe in Sektoren:
    A B
    0x850000 0x800000
     
    Wert A ist der Sektor in hex., in dem die ROM anfängt.
    Um die Speicheradresse zu ermitteln diesen Wert mit Faktor 200 nehmen.
     
    Wert B ist die Größe der ROM in Sektoren. Um den Wert in Bytes zu ermitteln, Wert B nach Dez umrechnen und mit Faktor 512 nehmen.
     
    Abgelegt werden die Informationen auf der SD am Anfang der Image (0x0), aber Byte-Swapped.
    Aus:
    00008500 00008000
    wird also:
    00000085 0000 |0080
    Wert A |WertB
     
    Die ersten 6 unswapped Bytes ist die Sektornummer zur ROM; das letzte Byte bestimmt die ROM-Größe. Byte 7 ist ein Filler mit 00(?)
     
    ====================
    Manipulierter Header:
    An Stelle 0x15FE in der ROM, sind die letzten 2 Bytes der Schlüssel! Der Save muss mit identischen Bytes in der template.txt abschließen!
     
    FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ
    43 54 52 2D 50 2D 41 56 32 50 00 00 00 00 00 00 CTR-P-AV2P......
    .
    .
    .
    00 00 00 00 00 00 00 00 00 00 00 00 00 00 5D FB
     
    Man braucht nur die Headerinformationen mit der Länge 0x200 und einen Pointer auf diesen Header, um die Sky3DS zum Absturz zu bringen.
     
    ====================
     
    Pointer für die Headerinformationen anlegen:
    Unregistered ROMs funktionieren nicht bei µSD <4GB bzw. wenn zu wenig Speicherplatz vorhanden ist.(nicht verifiziert)
    Den Manipulierten Header an 0xMaxSektor-0x1600 kopieren und am Anfang der SD-Image die DiskWriter-Informationen schreiben:
     
    Schema:
    XXXX XXXX XXXX YYYY
     
    bei einer 4GB µSD ist folgender Eintrag zu tätigen:
    [Adresse: 0xE8BFEA00/0x200=0x745FF5]
    0000 7400 F55F 0200
    (ByteSwapped: 0000 0074 5FF5 0002)
     
    bei einer 8GB µSD:
    [Adresse: 0x1D23FEA00/0x200=0xE91FF5]
    0000 E900 F51F 0200
     
    bei einer 32GB µSD:
    [Adresse: 0x76E47EA00/0x200=0x3B723F5]
    0000 B703 F523 0200
     
    ====================
     
    Die card1-Saves liegen ab Adresse 0x100000 (1MB) im Abstand von 0x100000 (1MB) voneinander.
    D.h. es können maximal 31 card1 Saves abgelegt werden.
     
    Der erste Sektor mit ROM-Daten liegt immer bei Adresse 0x2000000.
    Die Reihenfolge der Spielstände entspricht der Reihenfolge im DiskWriter. Der Spielstand vom Spiel ganz oben in der Liste
    hat also die Adresse 0x100000. Ändert sich die Reihenfolge der Spiele, dann ändert sich auch die Reihenfolge der Spielstände.
     
    DiskWriter hat aber einige Bugs beim Sichern der Spielstände. Einige Gamesaves werden doppelt angelegt. (Muss näher untersucht werden)
     
    Jedenfalls lassen sich die Spielstände nur dann restaurieren, wenn der Header, mit dem das Spiel zuletzt gespeichert wurde, identisch ist.
    Leider ändert sich dies mit der Zeit bei Updates der template.txt. Aus dem Grund, sollte der injected Header mitgesichert werden.
     
    Der Header wird von DiskWriter direkt in die ROM geschrieben.
    Er beginnt in der ROM bei Adresse 0x1400, hat eine Länge von 0x200 (512 Bytes) und endet bei 0x15FF
     
    Sobald diese Bedingungen erfüllt sind, muss Sky3DS den Spielstand neu einlesen. Das geht, in dem man den roten Taster betätigt
    und die ROM nochmal lädt (nicht verifiziert) oder die µSD Karten 2x wechselt (verifiziert).
     
    ====================
    Encryptete Card2-Spielstände:
     
    Normalerweise wird der Save innerhalb der Image angelegt. Auf einer 4GB µSD hab
    ich folgendes Verhalten an der CubicNinja ROM festgestellt:
     
    Für Offset 0x2000000 mit der Länge 0x7FFFFFF (128MB) beginnen Daten außerhalb der ROM:
    bei 0x79280000 und enden bei 0x7928045B (Länge 0x45C) (beginnt bei 1938MB)
    bei 0x79281000 und enden bei 0x79281107 (Länge 0x108)
    bei 0x79282000 und enden bei 0x79286FFF (Länge 0x5000)
    bei 0x79301000 und enden bei 0x7930DFFF (Länge 0xD000)
    Ende
     
    Ein Vergleich mit 2 CubicNinja Saves zeigt Veränderungen:
    bei 0x79280000
     
    bei 0x79282000
    bei 0x79301000
    Greetings
     
    cearp likes this.
  3. lukas_2511
    OP

    lukas_2511 GBAtemp Regular

    Member
    125
    45
    Jan 4, 2015
    Gambia, The
    So uhm yea, I guess i figured most of the stuff out now. Thanks GothicIII for your notes, even though I didn't really need them in the end.

    I have put the source on https://github.com/lukas2511/sky3ds.py

    In the repo-wiki I documented the layout of the sd-card and savegame backups, and also some other stuff.
     
    cearp likes this.
  4. naxil

    naxil GBAtemp Advanced Fan

    Member
    740
    96
    Oct 26, 2011
    Italy
    works today? i have try with linux 12.04.. and continue to askme request module.. but i have install some python-requests* package....
    Sorry...
    Python3 module instead Python module need to be installed .....
     
  5. lukas_2511
    OP

    lukas_2511 GBAtemp Regular

    Member
    125
    45
    Jan 4, 2015
    Gambia, The

    Well, it's based on Python 3, and it depends on https://github.com/coagulant/progressbar-python3 and the appdirs package, but otherwise it should just work.
    I am actually using this (under ArchLinux), and the only problems I had were problems that have been reported with DiskWriter too, and are more a design problem with sky3ds itself rather than the disk writing utility.
     
  6. naxil

    naxil GBAtemp Advanced Fan

    Member
    740
    96
    Oct 26, 2011
    Italy
    ok-- now it works, iam ubuntu 12.04 user, i buy sky3ds in the next week, and i do some test ok? i tell u with pm or in that thread my test! thank u for think about NON WINDOWS user, idk why today oeople release simple program ONLYWIN....
     
  7. Nuxx20

    Nuxx20 GBAtemp Regular

    Member
    244
    86
    Jan 6, 2015
    United States
    Any chance someone could compile this for OS X?
     
  8. Bug_Checker_

    Bug_Checker_ GBAtemp Advanced Fan

    Member
    950
    444
    Jun 10, 2006
    United States
    cearp likes this.
  9. Nuxx20

    Nuxx20 GBAtemp Regular

    Member
    244
    86
    Jan 6, 2015
    United States
    So I got it to run but can't really define the SDCARD, doesn't matter what i type after "sky3ds.py -d", it just won't accept it. I tried "/dev/disk1" and "/dev/disk1s1" but it always gives the same error message: "please specify exactly one operation".

    I'm clueless about python, please help :)
     
  10. lukas_2511
    OP

    lukas_2511 GBAtemp Regular

    Member
    125
    45
    Jan 4, 2015
    Gambia, The

    Maybe try specifiying an operation? ;)
    I didn't set any default, so you have to actually tell it to do something (like "-l" to see if it can read your disk)

    EDIT: I would also suggest to use something like /dev/rdisk2, but /dev/disk2 (or whatever your disk is) should work too.
     
  11. Nuxx20

    Nuxx20 GBAtemp Regular

    Member
    244
    86
    Jan 6, 2015
    United States
    I thought I specified the command. I typed "sky3ds.py -d /dev/disk1" without success.
     
  12. lukas_2511
    OP

    lukas_2511 GBAtemp Regular

    Member
    125
    45
    Jan 4, 2015
    Gambia, The

    "-d /path/to/disk" doesn't specify a command, it just specifies the path to the disk. Try something like "sky3ds.py -d /path/to/disk -l".
     
  13. cearp

    cearp the ticket master

    Member
    7,398
    4,645
    May 26, 2008
    Tuvalu
    on mac I would just type /Volumes/SDCARDNAME , that doesn't work here?
     
  14. lukas_2511
    OP

    lukas_2511 GBAtemp Regular

    Member
    125
    45
    Jan 4, 2015
    Gambia, The

    "/Volumes/SDCARDNAME" is the mountpoint of the tiny FAT partition on the SD-Card. This tool needs access to the raw disk, like DiskWriter would on Windows.

    You can find out what disk your sdcard is by running "diskutil list", it should show you all disks in the system.
    You may also need to "diskutil unmount /Volumes/SDCARDNAME" before sky3ds.py can access the disk.

    Ideally you would look for an option to get osx to not mount the FAT partition every time (maybe make it invalid somehow? it's not needed anyway), because after modifications to the raw disk it seems to remount the card, which is problematic.
     
  15. Nuxx20

    Nuxx20 GBAtemp Regular

    Member
    244
    86
    Jan 6, 2015
    United States
    Gotcha. Tried this but it returns a "permission denied" error when the disk is mounted, and it "cannot find the disk" error if I unmount it.

    I'm on OSX Yosemite, using Python 3.4 with both dependencies installed
     
  16. lukas_2511
    OP

    lukas_2511 GBAtemp Regular

    Member
    125
    45
    Jan 4, 2015
    Gambia, The

    You have to make sure that you really unmount the disk, not eject it. After ejecting it is not available anymore until you physically remove and reinsert the card (which will mount it again).

    Also it looks that OSX tries to mount the disk every time the raw device gets closed...
    Since my code is still a bit dirty and reopens the disk several times I guess it wouldn't work at the moment without finding a way to stop OSX from opening the disk.

    I'm going to clean up the code before adding a few features I had in mind (like using Unique IDs inside of gateway game backups so people can continue using their card2 savegames and stuff).
    I'll change the code to only open the disk once, so it would at least run once without getting interrupted by OSX mounting the disk. May not be comfortable but at least usable.
    Also I'm going to have a look if I can somehow remove the FAT partition completely without sky3ds locking up. That would eliminate this problem, but will require a reformat.
     
  17. Bug_Checker_

    Bug_Checker_ GBAtemp Advanced Fan

    Member
    950
    444
    Jun 10, 2006
    United States

    Just trying hiding the FAT signature(1 bit changed) similar to http://gbatemp.net/threads/ustealth-wii-u-format-disk-nag-workaround.352786/
     
  18. pipoco

    pipoco Newbie

    Newcomer
    6
    2
    Dec 12, 2014
    Brazil
    Just a tip about what exactly Sky3DS DiskWriter does... it's build with .NET, you can simply decompile it with thousands of free tools and just look literal what it really do. Just google .net decompiler.

    You can just transcode it :3
     
  19. lukas_2511
    OP

    lukas_2511 GBAtemp Regular

    Member
    125
    45
    Jan 4, 2015
    Gambia, The
    Well uhmm... from my starting post on this thread:

    Also I have finished reverse-engineering the disk format some time ago, and this tool now works perfectly fine. Well, at least for me ;)
    There are some more things I want to clean up and features I want to add before actually marking this thing as "released".
     
  20. Cliemacfr

    Cliemacfr Member

    Newcomer
    25
    1
    Oct 3, 2006
    France
    Hello.
    I've got this in the terminal on OSX :

    Found updated template.txt. Converting...
    Traceback (most recent call last):
    File "/Users/*********/Downloads/sky3ds.py-master/sky3ds/disk.py", line 32, in __init__
    self.diskfp = open(disk_path, "r+b")
    PermissionError: [Errno 13] Permission denied: '/dev/disk2'

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
    File "sky3ds.py", line 51, in <module>
    disk = disk.Sky3DS_Disk(args.disk)
    File "/Users/********/Downloads/sky3ds.py-master/sky3ds/disk.py", line 34, in __init__
    raise Exception("Couldn't open disk, can't continue.")

    Exception: Couldn't open disk, can't continue.

    what am I doing wrong ? (I actually succeed in passing the "Permission Denied" once but not anymore.)

    UPDATE : if i use sudo to launch python, i don't have the Permission denied anymore but I have a 0 byte problem...

    thanks.
    Clié