ROM Hack sky3ds diskwriter python clone (for linux & osx)

lukas_2511

Well-Known Member
OP
Member
Joined
Jan 4, 2015
Messages
126
Trophies
0
Age
29
XP
389
Country
Gambia, The
Hey there,

I'm currently working on a Python version of DiskWriter to be able to use it on Linux and OSX machines.
Not sure if anyone attempted this before, I wasn't able to find much information on how DiskWriter stores data to the sd-card, so I had to figure out most by just looking at hexdumps of my sd-card.

Edit:

For new users to this thread: The tool is actually already working. Under OSX it can't detect the disksize (yet), but I am working on that. If you know how to do things in python you can hardcode the size and this tool will work!

Things I still want to improve: 3dz unique-id import (instead of using the data from template.txt), unique-id re-import on savegame import. Both should be relatively easy, I just didn't really need those features yet, so I didn't do that work.
 

GothicIII

Well-Known Member
Member
Joined
Jan 4, 2015
Messages
793
Trophies
0
Age
34
XP
1,615
Country
Gambia, The
Hi,
I registered just for this. I did a Hexdump research by myself.
I also have an alternative method to make the 11th game work on red sky3ds without the button. It would be great if i can share my knowledge to help you with the tool.
Ill post my findings in German. PM me if you want.

Code:
Aufbau der µSD:
 
0x0000000 DiskWriter ROM-Informationen bis 0xF8.
0x0000100 String "ROMs"
0x0000108 Filler mit "FF" bis 0x1BD
0x00001BE Anfang? Der FAT32-Partition.
0x0100000 Card1 Spielstände im Abstand von 0x100000 bis 0x2000000. Max 31 Spielstände möglich.
0x2000000 Beginn der ersten ROM. Dies ist Sektor Nr. 0x10000
 
Cubic Ninja.3ds hat einen Null-byte-Bereich bei 0x57A8E00 ~87MB mit einer Länge von 0x28571F0 ~40MB (Auffüllunng von Bytes bis zu einer Größe von 2^x)
 
====================
DiskWriter Startsektor und Größe in Sektoren:
A B
0x850000 0x800000
 
Wert A ist der Sektor in hex., in dem die ROM anfängt.
Um die Speicheradresse zu ermitteln diesen Wert mit Faktor 200 nehmen.
 
Wert B ist die Größe der ROM in Sektoren. Um den Wert in Bytes zu ermitteln, Wert B nach Dez umrechnen und mit Faktor 512 nehmen.
 
Abgelegt werden die Informationen auf der SD am Anfang der Image (0x0), aber Byte-Swapped.
Aus:
00008500 00008000
wird also:
00000085 0000 |0080
Wert A |WertB
 
Die ersten 6 unswapped Bytes ist die Sektornummer zur ROM; das letzte Byte bestimmt die ROM-Größe. Byte 7 ist ein Filler mit 00(?)
 
====================
Manipulierter Header:
An Stelle 0x15FE in der ROM, sind die letzten 2 Bytes der Schlüssel! Der Save muss mit identischen Bytes in der template.txt abschließen!
 
FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ
43 54 52 2D 50 2D 41 56 32 50 00 00 00 00 00 00 CTR-P-AV2P......
.
.
.
00 00 00 00 00 00 00 00 00 00 00 00 00 00 5D FB
 
Man braucht nur die Headerinformationen mit der Länge 0x200 und einen Pointer auf diesen Header, um die Sky3DS zum Absturz zu bringen.
 
====================
 
Pointer für die Headerinformationen anlegen:
Unregistered ROMs funktionieren nicht bei µSD <4GB bzw. wenn zu wenig Speicherplatz vorhanden ist.(nicht verifiziert)
Den Manipulierten Header an 0xMaxSektor-0x1600 kopieren und am Anfang der SD-Image die DiskWriter-Informationen schreiben:
 
Schema:
XXXX XXXX XXXX YYYY
 
bei einer 4GB µSD ist folgender Eintrag zu tätigen:
[Adresse: 0xE8BFEA00/0x200=0x745FF5]
0000 7400 F55F 0200
(ByteSwapped: 0000 0074 5FF5 0002)
 
bei einer 8GB µSD:
[Adresse: 0x1D23FEA00/0x200=0xE91FF5]
0000 E900 F51F 0200
 
bei einer 32GB µSD:
[Adresse: 0x76E47EA00/0x200=0x3B723F5]
0000 B703 F523 0200
 
====================
 
Die card1-Saves liegen ab Adresse 0x100000 (1MB) im Abstand von 0x100000 (1MB) voneinander.
D.h. es können maximal 31 card1 Saves abgelegt werden.
 
Der erste Sektor mit ROM-Daten liegt immer bei Adresse 0x2000000.
Die Reihenfolge der Spielstände entspricht der Reihenfolge im DiskWriter. Der Spielstand vom Spiel ganz oben in der Liste
hat also die Adresse 0x100000. Ändert sich die Reihenfolge der Spiele, dann ändert sich auch die Reihenfolge der Spielstände.
 
DiskWriter hat aber einige Bugs beim Sichern der Spielstände. Einige Gamesaves werden doppelt angelegt. (Muss näher untersucht werden)
 
Jedenfalls lassen sich die Spielstände nur dann restaurieren, wenn der Header, mit dem das Spiel zuletzt gespeichert wurde, identisch ist.
Leider ändert sich dies mit der Zeit bei Updates der template.txt. Aus dem Grund, sollte der injected Header mitgesichert werden.
 
Der Header wird von DiskWriter direkt in die ROM geschrieben.
Er beginnt in der ROM bei Adresse 0x1400, hat eine Länge von 0x200 (512 Bytes) und endet bei 0x15FF
 
Sobald diese Bedingungen erfüllt sind, muss Sky3DS den Spielstand neu einlesen. Das geht, in dem man den roten Taster betätigt
und die ROM nochmal lädt (nicht verifiziert) oder die µSD Karten 2x wechselt (verifiziert).
 
====================
Encryptete Card2-Spielstände:
 
Normalerweise wird der Save innerhalb der Image angelegt. Auf einer 4GB µSD hab
ich folgendes Verhalten an der CubicNinja ROM festgestellt:
 
Für Offset 0x2000000 mit der Länge 0x7FFFFFF (128MB) beginnen Daten außerhalb der ROM:
bei 0x79280000 und enden bei 0x7928045B (Länge 0x45C) (beginnt bei 1938MB)
bei 0x79281000 und enden bei 0x79281107 (Länge 0x108)
bei 0x79282000 und enden bei 0x79286FFF (Länge 0x5000)
bei 0x79301000 und enden bei 0x7930DFFF (Länge 0xD000)
Ende
 
Ein Vergleich mit 2 CubicNinja Saves zeigt Veränderungen:
bei 0x79280000
 
bei 0x79282000
bei 0x79301000

Greetings
 
  • Like
Reactions: cearp

lukas_2511

Well-Known Member
OP
Member
Joined
Jan 4, 2015
Messages
126
Trophies
0
Age
29
XP
389
Country
Gambia, The
So uhm yea, I guess i figured most of the stuff out now. Thanks GothicIII for your notes, even though I didn't really need them in the end.

I have put the source on https://github.com/lukas2511/sky3ds.py

In the repo-wiki I documented the layout of the sd-card and savegame backups, and also some other stuff.
 
  • Like
Reactions: cearp

naxil

Well-Known Member
Member
Joined
Oct 26, 2011
Messages
808
Trophies
0
XP
505
Country
Italy
works today? i have try with linux 12.04.. and continue to askme request module.. but i have install some python-requests* package....
Sorry...
Python3 module instead Python module need to be installed .....
 

lukas_2511

Well-Known Member
OP
Member
Joined
Jan 4, 2015
Messages
126
Trophies
0
Age
29
XP
389
Country
Gambia, The
works today? i have try with linux 12.04.. and continue to askme request module.. but i have install some python-requests* package....
Sorry...
Python3 module instead Python module need to be installed .....


Well, it's based on Python 3, and it depends on https://github.com/coagulant/progressbar-python3 and the appdirs package, but otherwise it should just work.
I am actually using this (under ArchLinux), and the only problems I had were problems that have been reported with DiskWriter too, and are more a design problem with sky3ds itself rather than the disk writing utility.
 

naxil

Well-Known Member
Member
Joined
Oct 26, 2011
Messages
808
Trophies
0
XP
505
Country
Italy
Well, it's based on Python 3, and it depends on https://github.com/coagulant/progressbar-python3 and the appdirs package, but otherwise it should just work.
I am actually using this (under ArchLinux), and the only problems I had were problems that have been reported with DiskWriter too, and are more a design problem with sky3ds itself rather than the disk writing utility.

ok-- now it works, iam ubuntu 12.04 user, i buy sky3ds in the next week, and i do some test ok? i tell u with pm or in that thread my test! thank u for think about NON WINDOWS user, idk why today oeople release simple program ONLYWIN....
 

Nuxx20

Well-Known Member
Member
Joined
Jan 6, 2015
Messages
244
Trophies
0
XP
221
Country
United States
So I got it to run but can't really define the SDCARD, doesn't matter what i type after "sky3ds.py -d", it just won't accept it. I tried "/dev/disk1" and "/dev/disk1s1" but it always gives the same error message: "please specify exactly one operation".

I'm clueless about python, please help :)
 

lukas_2511

Well-Known Member
OP
Member
Joined
Jan 4, 2015
Messages
126
Trophies
0
Age
29
XP
389
Country
Gambia, The
So I got it to run but can't really define the SDCARD, doesn't matter what i type after "sky3ds.py -d", it just won't accept it. I tried "/dev/disk1" and "/dev/disk1s1" but it always gives the same error message: "please specify exactly one operation".

I'm clueless about python, please help :)


Maybe try specifiying an operation? ;)
I didn't set any default, so you have to actually tell it to do something (like "-l" to see if it can read your disk)

EDIT: I would also suggest to use something like /dev/rdisk2, but /dev/disk2 (or whatever your disk is) should work too.
 

cearp

瓜老外
Developer
Joined
May 26, 2008
Messages
8,582
Trophies
1
XP
7,583
Country
Tuvalu
So I got it to run but can't really define the SDCARD, doesn't matter what i type after "sky3ds.py -d", it just won't accept it. I tried "/dev/disk1" and "/dev/disk1s1" but it always gives the same error message: "please specify exactly one operation".

I'm clueless about python, please help :)

on mac I would just type /Volumes/SDCARDNAME , that doesn't work here?
 

lukas_2511

Well-Known Member
OP
Member
Joined
Jan 4, 2015
Messages
126
Trophies
0
Age
29
XP
389
Country
Gambia, The
on mac I would just type /Volumes/SDCARDNAME , that doesn't work here?


"/Volumes/SDCARDNAME" is the mountpoint of the tiny FAT partition on the SD-Card. This tool needs access to the raw disk, like DiskWriter would on Windows.

You can find out what disk your sdcard is by running "diskutil list", it should show you all disks in the system.
You may also need to "diskutil unmount /Volumes/SDCARDNAME" before sky3ds.py can access the disk.

Ideally you would look for an option to get osx to not mount the FAT partition every time (maybe make it invalid somehow? it's not needed anyway), because after modifications to the raw disk it seems to remount the card, which is problematic.
 

Nuxx20

Well-Known Member
Member
Joined
Jan 6, 2015
Messages
244
Trophies
0
XP
221
Country
United States
"-d /path/to/disk" doesn't specify a command, it just specifies the path to the disk. Try something like "sky3ds.py -d /path/to/disk -l".

Gotcha. Tried this but it returns a "permission denied" error when the disk is mounted, and it "cannot find the disk" error if I unmount it.

I'm on OSX Yosemite, using Python 3.4 with both dependencies installed
 

lukas_2511

Well-Known Member
OP
Member
Joined
Jan 4, 2015
Messages
126
Trophies
0
Age
29
XP
389
Country
Gambia, The
Gotcha. Tried this but it returns a "permission denied" error when the disk is mounted, and it "cannot find the disk" error if I unmount it.

I'm on OSX Yosemite, using Python 3.4 with both dependencies installed


You have to make sure that you really unmount the disk, not eject it. After ejecting it is not available anymore until you physically remove and reinsert the card (which will mount it again).

Also it looks that OSX tries to mount the disk every time the raw device gets closed...
Since my code is still a bit dirty and reopens the disk several times I guess it wouldn't work at the moment without finding a way to stop OSX from opening the disk.

I'm going to clean up the code before adding a few features I had in mind (like using Unique IDs inside of gateway game backups so people can continue using their card2 savegames and stuff).
I'll change the code to only open the disk once, so it would at least run once without getting interrupted by OSX mounting the disk. May not be comfortable but at least usable.
Also I'm going to have a look if I can somehow remove the FAT partition completely without sky3ds locking up. That would eliminate this problem, but will require a reformat.
 

Bug_Checker_

Well-Known Member
Member
Joined
Jun 10, 2006
Messages
950
Trophies
0
XP
663
Country
United States
You have to make sure that you really unmount the disk, not eject it. After ejecting it is not available anymore until you physically remove and reinsert the card (which will mount it again).

Also it looks that OSX tries to mount the disk every time the raw device gets closed...
Since my code is still a bit dirty and reopens the disk several times I guess it wouldn't work at the moment without finding a way to stop OSX from opening the disk.

I'm going to clean up the code before adding a few features I had in mind (like using Unique IDs inside of gateway game backups so people can continue using their card2 savegames and stuff).
I'll change the code to only open the disk once, so it would at least run once without getting interrupted by OSX mounting the disk. May not be comfortable but at least usable.
Also I'm going to have a look if I can somehow remove the FAT partition completely without sky3ds locking up. That would eliminate this problem, but will require a reformat.


Just trying hiding the FAT signature(1 bit changed) similar to http://gbatemp.net/threads/ustealth-wii-u-format-disk-nag-workaround.352786/
 

pipoco

Member
Newcomer
Joined
Dec 12, 2014
Messages
6
Trophies
0
Age
34
XP
33
Country
Brazil
Just a tip about what exactly Sky3DS DiskWriter does... it's build with .NET, you can simply decompile it with thousands of free tools and just look literal what it really do. Just google .net decompiler.

You can just transcode it :3
 

lukas_2511

Well-Known Member
OP
Member
Joined
Jan 4, 2015
Messages
126
Trophies
0
Age
29
XP
389
Country
Gambia, The
Just a tip about what exactly Sky3DS DiskWriter does... it's build with .NET, you can simply decompile it with thousands of free tools and just look literal what it really do. Just google .net decompiler.

You can just transcode it :3

Well uhmm... from my starting post on this thread:

I decompiled DiskWriter, but since there are no comments and (decompiled) code is a bit messy in some parts

Also I have finished reverse-engineering the disk format some time ago, and this tool now works perfectly fine. Well, at least for me ;)
There are some more things I want to clean up and features I want to add before actually marking this thing as "released".
 

Cliemacfr

Active Member
Newcomer
Joined
Oct 3, 2006
Messages
25
Trophies
0
XP
202
Country
France
Hello.
I've got this in the terminal on OSX :

Found updated template.txt. Converting...
Traceback (most recent call last):
File "/Users/*********/Downloads/sky3ds.py-master/sky3ds/disk.py", line 32, in __init__
self.diskfp = open(disk_path, "r+b")
PermissionError: [Errno 13] Permission denied: '/dev/disk2'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "sky3ds.py", line 51, in <module>
disk = disk.Sky3DS_Disk(args.disk)
File "/Users/********/Downloads/sky3ds.py-master/sky3ds/disk.py", line 34, in __init__
raise Exception("Couldn't open disk, can't continue.")

Exception: Couldn't open disk, can't continue.

what am I doing wrong ? (I actually succeed in passing the "Permission Denied" once but not anymore.)

UPDATE : if i use sudo to launch python, i don't have the Permission denied anymore but I have a 0 byte problem...

thanks.
Clié
 
General chit-chat
Help Users
  • mthrnite @ mthrnite:
    praise be to gabe
    Gift
  • Tom Bombadildo @ Tom Bombadildo:
    It do be quite nice
    Gift
  • mthrnite @ mthrnite:
    i'll have to be content to play marioland on my BRAND NEW USEFUL FUNCTION MULTI GAME EMBODIED chinese trash
    Gift
  • Tom Bombadildo @ Tom Bombadildo:
    Great for emulation, and runs PC games better than my GPD Win 3 lol
    Gift
  • Tom Bombadildo @ Tom Bombadildo:
    Which I suppose I should get rid of at this point
    Gift
  • mthrnite @ mthrnite:
    i keep evertying till it's one big blue piece of battery crust
    Gift
  • Tom Bombadildo @ Tom Bombadildo:
    Yummy, that's the best part
    Gift
  • Gift
  • gudenau @ gudenau:
    I love how GBATemp randomly
    logs me out.
    Gift
  • Dark_Phoras @ Dark_Phoras:
    Take the hint, man
    Gift
  • KenniesNewName @ KenniesNewName:
    @Tom Bombadildo, upgrade the SSD init?
    Gift
  • KenniesNewName @ KenniesNewName:
    Apparently we have ancestors called Homo erectus
    Gift
  • Tom Bombadildo @ Tom Bombadildo:
    It was quite literally the first thing I did with it, never even turned it on with the stock 64GB lol
    Gift
  • Dark_Phoras @ Dark_Phoras:
    And several species of human existed, only sapiens survived
    Gift
  • Tom Bombadildo @ Tom Bombadildo:
    Nabbed 512GB for like $70
    Gift
  • KenniesNewName @ KenniesNewName:
    Ye I see them on eBay for about$60 but all are refurbished can't be that bad to get refurbished ones off eBay
    Gift
  • KenniesNewName @ KenniesNewName:
    Plan on getting a m.2 drive and 512gb SD card for roms
    Gift
  • Tom Bombadildo @ Tom Bombadildo:
    I've been looking into designing a small casing in the back of it actually, trying to integrate another 2230 M.2 slot along with as small a USB-C dock I can find to sort of make a plug and play M.2->USB expansion myself
    Gift
  • Tom Bombadildo @ Tom Bombadildo:
    Expand the storage with faster NVMe, and also have charging and maybe a micro HDMI port
    Gift
  • Tom Bombadildo @ Tom Bombadildo:
    But haven't had a whole lot of time to actually finish anything
    Gift
  • KenniesNewName @ KenniesNewName:
    Let me know if you managed anything I've thought about that being possible as well it would be nice to have that and maybe a built in mini HDMI port with 4 usbs
    Gift
  • KenniesNewName @ KenniesNewName:
    But freaking valve pushed my order to July even though I pre-ordered two days after release :angry:
    Gift
  • KenniesNewName @ KenniesNewName:
    I'll review it :tpi:
    Gift
  • KenniesNewName @ KenniesNewName:
    Won't be long before freaking china floods AliExpress with attachments
    Gift
  • smurgburglar @ smurgburglar:
    how do i get rid of or change the text underneath my username?
    Gift
    smurgburglar @ smurgburglar: how do i get rid of or change the text underneath my username?