ROM Hack Edit Smash Models?

TheConfuZzledDude · Oct 29, 2015

I'm gonna look at dtls.py and see if I can figure out what's going on myself. You'll probably beat me to it, but it would be nice to do something serious

NyaakoXD · Oct 29, 2015

I want to switch the low res models used during the fights with the higher res ones (the ones you see during victory screens and when the game is paused during battle). You can tell that there's a difference when you goto training, zoom in, and choosing the speed to go "Hold L for 1/4 speed". You can actually see the difference when you're pressing L repeatedly.

gudenau · Oct 29, 2015

LinkmstrYT said:
I want to switch the low res models used during the fights with the higher res ones (the ones you see during victory screens and when the game is paused during battle). You can tell that there's a difference when you goto training, zoom in, and choosing the speed to go "Hold L for 1/4 speed".

That might be because of lag problems though.

NyaakoXD · Oct 29, 2015

gudenaurock said:
That might be because of lag problems though.

But, with the New3DS, it may be possible with its extra power. >w>;;

Sammi Husky · Oct 29, 2015

I'm also writing a Repacker. It'll be released as both a .py file to go along with comex's scripts and i'll also be adding executable forms of the extractor and packer to my repo https://github.com/Sammi-Husky/Sm4sh-Tools

Deleted member 356526 · Oct 29, 2015

Sammi Husky said:
I'm also writing a Repacker. It'll be released as both a .py file to go along with comex's scripts and i'll also be adding executable forms of the extractor and packer to my repo https://github.com/Sammi-Husky/Sm4sh-Tools

thanks babe

darklordrs · Oct 29, 2015

gudenaurock said:
I just added that to my sig.

genuinely flattered lmaooo

edit:

Sammi Husky said:
I'm also writing a Repacker. It'll be released as both a .py file to go along with comex's scripts and i'll also be adding executable forms of the extractor and packer to my repo https://github.com/Sammi-Husky/Sm4sh-Tools

HAH, I KNEW IT. asking you if you would from every place I could did work!

gudenau · Oct 29, 2015

What the heck is this doing?

Code:

def invertify(msg):
    return ''.join(chr(~ord(msg[i]) & 0xff) for i in xrange(4)) + msg[4:]

TheConfuZzledDude · Oct 29, 2015

gudenaurock said:
What the heck is this doing?

Code:

def invertify(msg): return ''.join(chr(~ord(msg[i]) & 0xff) for i in xrange(4)) + msg[4:]

It inverts the binary e.g 10010010 becomes 01101101
Something to do with the CRC

gudenau · Oct 29, 2015

TheConfuZzledDude said:
It inverts the binary e.g 10010010 becomes 01101101
Something to do with the CRC

I know my binary operations, what I do not know is python.

--------------------- MERGED ---------------------------

Python CRC32 yields 642569344
Java CRC32 yields 264cd480

I must be doing something wrong.

Sammi Husky · Oct 30, 2015

gudenaurock said:
I know my binary operations, what I do not know is python.

--------------------- MERGED ---------------------------

Python CRC32 yields 642569344
Java CRC32 yields 264cd480

I must be doing something wrong.

Invertify(msg) is only inverting the first 4 characters of the string iirc, so if your inverting the whole string, that may be why. If not, then i can't really think of any reason the crc would be different

gudenau · Oct 30, 2015

Sammi Husky said:
Invertify(msg) is only inverting the first 4 characters of the string iirc, so if your inverting the whole string, that may be why. If not, then i can't really think of any reason the crc would be different

I used a static array of bytes for both.

Java:
new byte[]{ (byte)0x8d, (byte)0x9a, (byte)0x8c, (byte)0x90, (byte)'u', (byte)'r', (byte)'c', (byte)'e' }
Python:
'\x8d\x9a\x8c\x90urce'

RetroGamer732 · Oct 30, 2015

I have a Noob Question. How do i decompress/decrypt the files in dt with Python. I got the dtls.py file from Comex's smash-stuff, I just don't know what to do. Any help would be greatly appreciated.

gudenau · Oct 30, 2015

RetroGamer732 said:
I have a Noob Question. How do i decompress/decrypt the files in dt with Python. I got the dtls.py file from Comex's smash-stuff, I just don't know what to do. Any help would be greatly appreciated.

Use this one:

Code:

import struct, sys, zlib, os
from collections import OrderedDict
from cStringIO import StringIO

dtfn, lsfn, outdir, dtlstype = sys.argv[1:]
if not os.path.exists(outdir):
    os.makedirs(outdir)
tmpPath = os.getcwd()+'/tmp'
if not os.path.exists(tmpPath):
    os.makedirs(tmpPath)
dtfp = open(dtfn, 'rb')
lsfp = open(lsfn, 'rb')
lsfp.read(4)
count, = struct.unpack('<I', lsfp.read(4))
dt_offsets = OrderedDict()
dt_total_size = 0
for i in xrange(count):
    if dtlstype == 'u':
        crc, start, size, dt_index, unk = struct.unpack('<IIIHH', lsfp.read(16)) #TODO: Apparently there's indexed dt files (ie dt00, dt01)
    else:
       crc, start, size = struct.unpack('<III', lsfp.read(12))
    dt_offsets[crc] = (start, size)
    dt_total_size += size
#assert lsfp.read(1) == ''

def get_file((start, size)):
    dtfp.seek(start)
    compressed = dtfp.read(size)
    data = compressed
    if compressed.startswith('\xcc\xcc\xcc\xcc'):
        z = compressed.find('\x78\x9c')
        if z != -1 and z <= 0x300:
            decompressed = zlib.decompress(compressed[z:])
            return decompressed, True
    return data, False

def invertify(msg):
    return ''.join(chr(~ord(msg[i]) & 0xff) for i in xrange(4)) + msg[4:]

def stupidcrc(filename):
    return zlib.crc32(invertify(filename)) & 0xffffffff

resource = dt_offsets[stupidcrc('resource')]
resource_data, was_compressed = get_file(resource)
open(tmpPath+'/resource.dec', 'wb').write(resource_data)
assert resource_data.startswith('RF')
offset_to_compressed, = struct.unpack('<I', resource_data[4:8])
rf, hl1, _, hl2, \
_0x18_entries_len, timestamp, compressed_len, decompressed_len, \
start_of_strs_plus, len_strs \
    = struct.unpack('<10I', resource_data[:0x28])

resource_dec = zlib.decompress(resource_data[offset_to_compressed:])
rdfp = StringIO(resource_dec)
open(tmpPath+'/resource.bin', 'wb').write(resource_dec)

rdfp.seek(start_of_strs_plus - hl1)
num_segments, = struct.unpack('<I', rdfp.read(4))
segments = [rdfp.read(0x2000) for seg in xrange(num_segments)]

def get_from_offset(off, len):
    # this is actually just a linear mapping, so pretty useless - but at least
    # this documents what needs to happen for any repackers
    seg_off = off & 0x1fff
    return segments[off / 0x2000][seg_off:seg_off + len]

parts = []
offset_parts = []

known_crcs = set()
resource_total_size = 0

if 1:
    num_offsets, = struct.unpack('<I', rdfp.read(4))
    extension_offsets = struct.unpack('<%dI' % num_offsets, rdfp.read(4 * num_offsets))
    extensions = []
    for i, exto in enumerate(extension_offsets):
        ext = get_from_offset(exto, 64)
        ext = ext[:ext.find('\0')]
        extensions.append(ext)
        #print i, repr(ext)

    rdfp.seek(0)
    num_8sized, = struct.unpack('<I', rdfp.read(4))
    rdfp.read(num_8sized * 8)
    another_size, = struct.unpack('<I', rdfp.read(4))
    rdfp.read(another_size)
    while rdfp.tell() < _0x18_entries_len:
        off_in_chunk, name_offset_etc, cmp_size, dec_size, timestamp, derp1 = struct.unpack('<IIIIII', rdfp.read(0x18))
        ext = name_offset_etc >> 24
        name_offset = name_offset_etc & 0xfffff
        name = get_from_offset(name_offset, 128)
        if name_offset_etc & 0x00800000:
            reference, = struct.unpack('<H', name[:2])
            ref_len = (reference & 0x1f) + 4
            ref_reloff = (reference & 0xe0) >> 6 << 8 | (reference >> 8)
            name = get_from_offset(name_offset - ref_reloff, ref_len) + name[2:]
        if '\0' in name:
            name = name[:name.find('\0')]
        name += extensions[ext]

        nesting_level = derp1 & 0xff
        localized = bool(derp1 & 0x800)
        final = bool(derp1 & 0x400)
        compressed = bool(derp1 & 0x200)
        parts = parts[:nesting_level - 1] + [name]


        path = ''.join(parts)

        if final:
            start, size = None, None
            crc_path = 'data/' + path.rstrip('/') + ('/packed' if compressed else '')
            crc = stupidcrc(crc_path)
            if crc in dt_offsets:
                offset = dt_offsets[crc]
            else:
                offset = None
        else:
            offset = None
        offset_parts = offset_parts[:nesting_level - 1] + [offset]

        outfn = os.path.join(outdir, path)
        if path.endswith('/'):
            if not os.path.exists(outfn):
                os.mkdir(outfn)
        else:
            for part in offset_parts[::-1]:
                if part is not None:
                    chunk_start, chunk_size = part
                    break
            else:
                if dtlstype == 'u':
                    continue
                else:
                    raise Exception("%s: nothing to look at" % path)
            assert off_in_chunk + cmp_size <= chunk_size
            dtfp.seek(chunk_start + off_in_chunk)
            cmp_data = dtfp.read(cmp_size)
            # XXX why isn't 'compressed' right?
            if cmp_data.startswith('x\x9c'):
                file_data = zlib.decompress(cmp_data)
            else:
                file_data = cmp_data
            assert len(file_data) == dec_size
            open(outfn, 'wb').write(file_data)


        resource_total_size += cmp_size

if 0:
    for missing in set(dt_offsets.keys()) - known_crcs:
        print 'Missing', missing

And execute "python dtls.py dt ls output u" for an updated version, or "python dtls.py dt ls output o" for the original one on the catradge.

RetroGamer732 · Oct 30, 2015

gudenaurock said:

Use this one:

Code:

import struct, sys, zlib, os
from collections import OrderedDict
from cStringIO import StringIO

dtfn, lsfn, outdir, dtlstype = sys.argv[1:]
if not os.path.exists(outdir):
    os.makedirs(outdir)
tmpPath = os.getcwd()+'/tmp'
if not os.path.exists(tmpPath):
    os.makedirs(tmpPath)
dtfp = open(dtfn, 'rb')
lsfp = open(lsfn, 'rb')
lsfp.read(4)
count, = struct.unpack('<I', lsfp.read(4))
dt_offsets = OrderedDict()
dt_total_size = 0
for i in xrange(count):
    if dtlstype == 'u':
        crc, start, size, dt_index, unk = struct.unpack('<IIIHH', lsfp.read(16)) #TODO: Apparently there's indexed dt files (ie dt00, dt01)
    else:
       crc, start, size = struct.unpack('<III', lsfp.read(12))
    dt_offsets[crc] = (start, size)
    dt_total_size += size
#assert lsfp.read(1) == ''

def get_file((start, size)):
    dtfp.seek(start)
    compressed = dtfp.read(size)
    data = compressed
    if compressed.startswith('\xcc\xcc\xcc\xcc'):
        z = compressed.find('\x78\x9c')
        if z != -1 and z <= 0x300:
            decompressed = zlib.decompress(compressed[z:])
            return decompressed, True
    return data, False

def invertify(msg):
    return ''.join(chr(~ord(msg[i]) & 0xff) for i in xrange(4)) + msg[4:]

def stupidcrc(filename):
    return zlib.crc32(invertify(filename)) & 0xffffffff

resource = dt_offsets[stupidcrc('resource')]
resource_data, was_compressed = get_file(resource)
open(tmpPath+'/resource.dec', 'wb').write(resource_data)
assert resource_data.startswith('RF')
offset_to_compressed, = struct.unpack('<I', resource_data[4:8])
rf, hl1, _, hl2, \
_0x18_entries_len, timestamp, compressed_len, decompressed_len, \
start_of_strs_plus, len_strs \
    = struct.unpack('<10I', resource_data[:0x28])

resource_dec = zlib.decompress(resource_data[offset_to_compressed:])
rdfp = StringIO(resource_dec)
open(tmpPath+'/resource.bin', 'wb').write(resource_dec)

rdfp.seek(start_of_strs_plus - hl1)
num_segments, = struct.unpack('<I', rdfp.read(4))
segments = [rdfp.read(0x2000) for seg in xrange(num_segments)]

def get_from_offset(off, len):
    # this is actually just a linear mapping, so pretty useless - but at least
    # this documents what needs to happen for any repackers
    seg_off = off & 0x1fff
    return segments[off / 0x2000][seg_off:seg_off + len]

parts = []
offset_parts = []

known_crcs = set()
resource_total_size = 0

if 1:
    num_offsets, = struct.unpack('<I', rdfp.read(4))
    extension_offsets = struct.unpack('<%dI' % num_offsets, rdfp.read(4 * num_offsets))
    extensions = []
    for i, exto in enumerate(extension_offsets):
        ext = get_from_offset(exto, 64)
        ext = ext[:ext.find('\0')]
        extensions.append(ext)
        #print i, repr(ext)

    rdfp.seek(0)
    num_8sized, = struct.unpack('<I', rdfp.read(4))
    rdfp.read(num_8sized * 8)
    another_size, = struct.unpack('<I', rdfp.read(4))
    rdfp.read(another_size)
    while rdfp.tell() < _0x18_entries_len:
        off_in_chunk, name_offset_etc, cmp_size, dec_size, timestamp, derp1 = struct.unpack('<IIIIII', rdfp.read(0x18))
        ext = name_offset_etc >> 24
        name_offset = name_offset_etc & 0xfffff
        name = get_from_offset(name_offset, 128)
        if name_offset_etc & 0x00800000:
            reference, = struct.unpack('<H', name[:2])
            ref_len = (reference & 0x1f) + 4
            ref_reloff = (reference & 0xe0) >> 6 << 8 | (reference >> 8)
            name = get_from_offset(name_offset - ref_reloff, ref_len) + name[2:]
        if '\0' in name:
            name = name[:name.find('\0')]
        name += extensions[ext]

        nesting_level = derp1 & 0xff
        localized = bool(derp1 & 0x800)
        final = bool(derp1 & 0x400)
        compressed = bool(derp1 & 0x200)
        parts = parts[:nesting_level - 1] + [name]


        path = ''.join(parts)

        if final:
            start, size = None, None
            crc_path = 'data/' + path.rstrip('/') + ('/packed' if compressed else '')
            crc = stupidcrc(crc_path)
            if crc in dt_offsets:
                offset = dt_offsets[crc]
            else:
                offset = None
        else:
            offset = None
        offset_parts = offset_parts[:nesting_level - 1] + [offset]

        outfn = os.path.join(outdir, path)
        if path.endswith('/'):
            if not os.path.exists(outfn):
                os.mkdir(outfn)
        else:
            for part in offset_parts[::-1]:
                if part is not None:
                    chunk_start, chunk_size = part
                    break
            else:
                if dtlstype == 'u':
                    continue
                else:
                    raise Exception("%s: nothing to look at" % path)
            assert off_in_chunk + cmp_size <= chunk_size
            dtfp.seek(chunk_start + off_in_chunk)
            cmp_data = dtfp.read(cmp_size)
            # XXX why isn't 'compressed' right?
            if cmp_data.startswith('x\x9c'):
                file_data = zlib.decompress(cmp_data)
            else:
                file_data = cmp_data
            assert len(file_data) == dec_size
            open(outfn, 'wb').write(file_data)


        resource_total_size += cmp_size

if 0:
    for missing in set(dt_offsets.keys()) - known_crcs:
        print 'Missing', missing

And execute "python dtls.py dt ls output u" for an updated version, or "python dtls.py dt ls output o" for the original one on the catradge.

gudenaurock said:

Use this one:

Code:

import struct, sys, zlib, os
from collections import OrderedDict
from cStringIO import StringIO

dtfn, lsfn, outdir, dtlstype = sys.argv[1:]
if not os.path.exists(outdir):
    os.makedirs(outdir)
tmpPath = os.getcwd()+'/tmp'
if not os.path.exists(tmpPath):
    os.makedirs(tmpPath)
dtfp = open(dtfn, 'rb')
lsfp = open(lsfn, 'rb')
lsfp.read(4)
count, = struct.unpack('<I', lsfp.read(4))
dt_offsets = OrderedDict()
dt_total_size = 0
for i in xrange(count):
    if dtlstype == 'u':
        crc, start, size, dt_index, unk = struct.unpack('<IIIHH', lsfp.read(16)) #TODO: Apparently there's indexed dt files (ie dt00, dt01)
    else:
       crc, start, size = struct.unpack('<III', lsfp.read(12))
    dt_offsets[crc] = (start, size)
    dt_total_size += size
#assert lsfp.read(1) == ''

def get_file((start, size)):
    dtfp.seek(start)
    compressed = dtfp.read(size)
    data = compressed
    if compressed.startswith('\xcc\xcc\xcc\xcc'):
        z = compressed.find('\x78\x9c')
        if z != -1 and z <= 0x300:
            decompressed = zlib.decompress(compressed[z:])
            return decompressed, True
    return data, False

def invertify(msg):
    return ''.join(chr(~ord(msg[i]) & 0xff) for i in xrange(4)) + msg[4:]

def stupidcrc(filename):
    return zlib.crc32(invertify(filename)) & 0xffffffff

resource = dt_offsets[stupidcrc('resource')]
resource_data, was_compressed = get_file(resource)
open(tmpPath+'/resource.dec', 'wb').write(resource_data)
assert resource_data.startswith('RF')
offset_to_compressed, = struct.unpack('<I', resource_data[4:8])
rf, hl1, _, hl2, \
_0x18_entries_len, timestamp, compressed_len, decompressed_len, \
start_of_strs_plus, len_strs \
    = struct.unpack('<10I', resource_data[:0x28])

resource_dec = zlib.decompress(resource_data[offset_to_compressed:])
rdfp = StringIO(resource_dec)
open(tmpPath+'/resource.bin', 'wb').write(resource_dec)

rdfp.seek(start_of_strs_plus - hl1)
num_segments, = struct.unpack('<I', rdfp.read(4))
segments = [rdfp.read(0x2000) for seg in xrange(num_segments)]

def get_from_offset(off, len):
    # this is actually just a linear mapping, so pretty useless - but at least
    # this documents what needs to happen for any repackers
    seg_off = off & 0x1fff
    return segments[off / 0x2000][seg_off:seg_off + len]

parts = []
offset_parts = []

known_crcs = set()
resource_total_size = 0

if 1:
    num_offsets, = struct.unpack('<I', rdfp.read(4))
    extension_offsets = struct.unpack('<%dI' % num_offsets, rdfp.read(4 * num_offsets))
    extensions = []
    for i, exto in enumerate(extension_offsets):
        ext = get_from_offset(exto, 64)
        ext = ext[:ext.find('\0')]
        extensions.append(ext)
        #print i, repr(ext)

    rdfp.seek(0)
    num_8sized, = struct.unpack('<I', rdfp.read(4))
    rdfp.read(num_8sized * 8)
    another_size, = struct.unpack('<I', rdfp.read(4))
    rdfp.read(another_size)
    while rdfp.tell() < _0x18_entries_len:
        off_in_chunk, name_offset_etc, cmp_size, dec_size, timestamp, derp1 = struct.unpack('<IIIIII', rdfp.read(0x18))
        ext = name_offset_etc >> 24
        name_offset = name_offset_etc & 0xfffff
        name = get_from_offset(name_offset, 128)
        if name_offset_etc & 0x00800000:
            reference, = struct.unpack('<H', name[:2])
            ref_len = (reference & 0x1f) + 4
            ref_reloff = (reference & 0xe0) >> 6 << 8 | (reference >> 8)
            name = get_from_offset(name_offset - ref_reloff, ref_len) + name[2:]
        if '\0' in name:
            name = name[:name.find('\0')]
        name += extensions[ext]

        nesting_level = derp1 & 0xff
        localized = bool(derp1 & 0x800)
        final = bool(derp1 & 0x400)
        compressed = bool(derp1 & 0x200)
        parts = parts[:nesting_level - 1] + [name]


        path = ''.join(parts)

        if final:
            start, size = None, None
            crc_path = 'data/' + path.rstrip('/') + ('/packed' if compressed else '')
            crc = stupidcrc(crc_path)
            if crc in dt_offsets:
                offset = dt_offsets[crc]
            else:
                offset = None
        else:
            offset = None
        offset_parts = offset_parts[:nesting_level - 1] + [offset]

        outfn = os.path.join(outdir, path)
        if path.endswith('/'):
            if not os.path.exists(outfn):
                os.mkdir(outfn)
        else:
            for part in offset_parts[::-1]:
                if part is not None:
                    chunk_start, chunk_size = part
                    break
            else:
                if dtlstype == 'u':
                    continue
                else:
                    raise Exception("%s: nothing to look at" % path)
            assert off_in_chunk + cmp_size <= chunk_size
            dtfp.seek(chunk_start + off_in_chunk)
            cmp_data = dtfp.read(cmp_size)
            # XXX why isn't 'compressed' right?
            if cmp_data.startswith('x\x9c'):
                file_data = zlib.decompress(cmp_data)
            else:
                file_data = cmp_data
            assert len(file_data) == dec_size
            open(outfn, 'wb').write(file_data)


        resource_total_size += cmp_size

if 0:
    for missing in set(dt_offsets.keys()) - known_crcs:
        print 'Missing', missing

And execute "python dtls.py dt ls output u" for an updated version, or "python dtls.py dt ls output o" for the original one on the catradge.

Ok, Thank You

Sammi Husky · Oct 30, 2015

My version also printed the currently extracting file so that it's easier for others to know when it's done, but that's purely cosmetic really. If you want that, just add

Code:

print outfn

at the bottom after

Code:

open(outfn,'wb').write(file_data)

gudenau · Oct 30, 2015

Sammi Husky said:
My version also printed the currently extracting file so that it's easier for others to know when it's done, but that's purely cosmetic really. If you want that, just add

Code:

print outfn

at the bottom after

Code:

open(outfn,'wb').write(file_data)

Did you make yours accept the updated version?

Sammi Husky · Oct 30, 2015

I didn't update mine yet, no. The one you posted is the merged fixes from shinyQuagsire and mine. The only difference being that it doesn't print the filenames in the console. The fully updated version would be

Code:

import struct, sys, zlib, os
from collections import OrderedDict
from cStringIO import StringIO

dtfn, lsfn, outdir, dtlstype = sys.argv[1:]
if not os.path.exists(outdir):
    os.makedirs(outdir)
tmpPath = os.getcwd()+'/tmp'
if not os.path.exists(tmpPath):
    os.makedirs(tmpPath)
dtfp = open(dtfn, 'rb')
lsfp = open(lsfn, 'rb')
lsfp.read(4)
count, = struct.unpack('<I', lsfp.read(4))
dt_offsets = OrderedDict()
dt_total_size = 0
for i in xrange(count):
    if dtlstype == 'u':
        crc, start, size, dt_index, unk = struct.unpack('<IIIHH', lsfp.read(16)) #TODO: Apparently there's indexed dt files (ie dt00, dt01)
    else:
       crc, start, size = struct.unpack('<III', lsfp.read(12))
    dt_offsets[crc] = (start, size)
    dt_total_size += size
#assert lsfp.read(1) == ''

def get_file((start, size)):
    dtfp.seek(start)
    compressed = dtfp.read(size)
    data = compressed
    if compressed.startswith('\xcc\xcc\xcc\xcc'):
        z = compressed.find('\x78\x9c')
        if z != -1 and z <= 0x300:
            decompressed = zlib.decompress(compressed[z:])
            return decompressed, True
    return data, False

def invertify(msg):
    return ''.join(chr(~ord(msg[i]) & 0xff) for i in xrange(4)) + msg[4:]

def stupidcrc(filename):
    return zlib.crc32(invertify(filename)) & 0xffffffff

resource = dt_offsets[stupidcrc('resource')]
resource_data, was_compressed = get_file(resource)
open(tmpPath+'/resource.dec', 'wb').write(resource_data)
assert resource_data.startswith('RF')
offset_to_compressed, = struct.unpack('<I', resource_data[4:8])
rf, hl1, _, hl2, \
_0x18_entries_len, timestamp, compressed_len, decompressed_len, \
start_of_strs_plus, len_strs \
    = struct.unpack('<10I', resource_data[:0x28])

resource_dec = zlib.decompress(resource_data[offset_to_compressed:])
rdfp = StringIO(resource_dec)
open(tmpPath+'/resource.bin', 'wb').write(resource_dec)

rdfp.seek(start_of_strs_plus - hl1)
num_segments, = struct.unpack('<I', rdfp.read(4))
segments = [rdfp.read(0x2000) for seg in xrange(num_segments)]

def get_from_offset(off, len):
    # this is actually just a linear mapping, so pretty useless - but at least
    # this documents what needs to happen for any repackers
    seg_off = off & 0x1fff
    return segments[off / 0x2000][seg_off:seg_off + len]

parts = []
offset_parts = []

known_crcs = set()
resource_total_size = 0

if 1:
    num_offsets, = struct.unpack('<I', rdfp.read(4))
    extension_offsets = struct.unpack('<%dI' % num_offsets, rdfp.read(4 * num_offsets))
    extensions = []
    for i, exto in enumerate(extension_offsets):
        ext = get_from_offset(exto, 64)
        ext = ext[:ext.find('\0')]
        extensions.append(ext)
        #print i, repr(ext)

    rdfp.seek(0)
    num_8sized, = struct.unpack('<I', rdfp.read(4))
    rdfp.read(num_8sized * 8)
    another_size, = struct.unpack('<I', rdfp.read(4))
    rdfp.read(another_size)
    while rdfp.tell() < _0x18_entries_len:
        off_in_chunk, name_offset_etc, cmp_size, dec_size, timestamp, derp1 = struct.unpack('<IIIIII', rdfp.read(0x18))
        ext = name_offset_etc >> 24
        name_offset = name_offset_etc & 0xfffff
        name = get_from_offset(name_offset, 128)
        if name_offset_etc & 0x00800000:
            reference, = struct.unpack('<H', name[:2])
            ref_len = (reference & 0x1f) + 4
            ref_reloff = (reference & 0xe0) >> 6 << 8 | (reference >> 8)
            name = get_from_offset(name_offset - ref_reloff, ref_len) + name[2:]
        if '\0' in name:
            name = name[:name.find('\0')]
        name += extensions[ext]

        nesting_level = derp1 & 0xff
        localized = bool(derp1 & 0x800)
        final = bool(derp1 & 0x400)
        compressed = bool(derp1 & 0x200)
        parts = parts[:nesting_level - 1] + [name]


        path = ''.join(parts)

        if final:
            start, size = None, None
            crc_path = 'data/' + path.rstrip('/') + ('/packed' if compressed else '')
            crc = stupidcrc(crc_path)
            if crc in dt_offsets:
                offset = dt_offsets[crc]
            else:
                offset = None
        else:
            offset = None
        offset_parts = offset_parts[:nesting_level - 1] + [offset]

        outfn = os.path.join(outdir, path)
        if path.endswith('/'):
            if not os.path.exists(outfn):
                os.mkdir(outfn)
        else:
            for part in offset_parts[::-1]:
                if part is not None:
                    chunk_start, chunk_size = part
                    break
            else:
                if dtlstype == 'u':
                    continue
                else:
                    raise Exception("%s: nothing to look at" % path)
            assert off_in_chunk + cmp_size <= chunk_size
            dtfp.seek(chunk_start + off_in_chunk)
            cmp_data = dtfp.read(cmp_size)
            # XXX why isn't 'compressed' right?
            if cmp_data.startswith('x\x9c'):
                file_data = zlib.decompress(cmp_data)
            else:
                file_data = cmp_data
            assert len(file_data) == dec_size

            print outfn
            open(outfn, 'wb').write(file_data)

        resource_total_size += cmp_size

if 0:
    for missing in set(dt_offsets.keys()) - known_crcs:
        print 'Missing', missing

RetroGamer732 · Oct 30, 2015

gudenaurock said:

Use this one:

Code:

import struct, sys, zlib, os
from collections import OrderedDict
from cStringIO import StringIO

dtfn, lsfn, outdir, dtlstype = sys.argv[1:]
if not os.path.exists(outdir):
    os.makedirs(outdir)
tmpPath = os.getcwd()+'/tmp'
if not os.path.exists(tmpPath):
    os.makedirs(tmpPath)
dtfp = open(dtfn, 'rb')
lsfp = open(lsfn, 'rb')
lsfp.read(4)
count, = struct.unpack('<I', lsfp.read(4))
dt_offsets = OrderedDict()
dt_total_size = 0
for i in xrange(count):
    if dtlstype == 'u':
        crc, start, size, dt_index, unk = struct.unpack('<IIIHH', lsfp.read(16)) #TODO: Apparently there's indexed dt files (ie dt00, dt01)
    else:
       crc, start, size = struct.unpack('<III', lsfp.read(12))
    dt_offsets[crc] = (start, size)
    dt_total_size += size
#assert lsfp.read(1) == ''

def get_file((start, size)):
    dtfp.seek(start)
    compressed = dtfp.read(size)
    data = compressed
    if compressed.startswith('\xcc\xcc\xcc\xcc'):
        z = compressed.find('\x78\x9c')
        if z != -1 and z <= 0x300:
            decompressed = zlib.decompress(compressed[z:])
            return decompressed, True
    return data, False

def invertify(msg):
    return ''.join(chr(~ord(msg[i]) & 0xff) for i in xrange(4)) + msg[4:]

def stupidcrc(filename):
    return zlib.crc32(invertify(filename)) & 0xffffffff

resource = dt_offsets[stupidcrc('resource')]
resource_data, was_compressed = get_file(resource)
open(tmpPath+'/resource.dec', 'wb').write(resource_data)
assert resource_data.startswith('RF')
offset_to_compressed, = struct.unpack('<I', resource_data[4:8])
rf, hl1, _, hl2, \
_0x18_entries_len, timestamp, compressed_len, decompressed_len, \
start_of_strs_plus, len_strs \
    = struct.unpack('<10I', resource_data[:0x28])

resource_dec = zlib.decompress(resource_data[offset_to_compressed:])
rdfp = StringIO(resource_dec)
open(tmpPath+'/resource.bin', 'wb').write(resource_dec)

rdfp.seek(start_of_strs_plus - hl1)
num_segments, = struct.unpack('<I', rdfp.read(4))
segments = [rdfp.read(0x2000) for seg in xrange(num_segments)]

def get_from_offset(off, len):
    # this is actually just a linear mapping, so pretty useless - but at least
    # this documents what needs to happen for any repackers
    seg_off = off & 0x1fff
    return segments[off / 0x2000][seg_off:seg_off + len]

parts = []
offset_parts = []

known_crcs = set()
resource_total_size = 0

if 1:
    num_offsets, = struct.unpack('<I', rdfp.read(4))
    extension_offsets = struct.unpack('<%dI' % num_offsets, rdfp.read(4 * num_offsets))
    extensions = []
    for i, exto in enumerate(extension_offsets):
        ext = get_from_offset(exto, 64)
        ext = ext[:ext.find('\0')]
        extensions.append(ext)
        #print i, repr(ext)

    rdfp.seek(0)
    num_8sized, = struct.unpack('<I', rdfp.read(4))
    rdfp.read(num_8sized * 8)
    another_size, = struct.unpack('<I', rdfp.read(4))
    rdfp.read(another_size)
    while rdfp.tell() < _0x18_entries_len:
        off_in_chunk, name_offset_etc, cmp_size, dec_size, timestamp, derp1 = struct.unpack('<IIIIII', rdfp.read(0x18))
        ext = name_offset_etc >> 24
        name_offset = name_offset_etc & 0xfffff
        name = get_from_offset(name_offset, 128)
        if name_offset_etc & 0x00800000:
            reference, = struct.unpack('<H', name[:2])
            ref_len = (reference & 0x1f) + 4
            ref_reloff = (reference & 0xe0) >> 6 << 8 | (reference >> 8)
            name = get_from_offset(name_offset - ref_reloff, ref_len) + name[2:]
        if '\0' in name:
            name = name[:name.find('\0')]
        name += extensions[ext]

        nesting_level = derp1 & 0xff
        localized = bool(derp1 & 0x800)
        final = bool(derp1 & 0x400)
        compressed = bool(derp1 & 0x200)
        parts = parts[:nesting_level - 1] + [name]


        path = ''.join(parts)

        if final:
            start, size = None, None
            crc_path = 'data/' + path.rstrip('/') + ('/packed' if compressed else '')
            crc = stupidcrc(crc_path)
            if crc in dt_offsets:
                offset = dt_offsets[crc]
            else:
                offset = None
        else:
            offset = None
        offset_parts = offset_parts[:nesting_level - 1] + [offset]

        outfn = os.path.join(outdir, path)
        if path.endswith('/'):
            if not os.path.exists(outfn):
                os.mkdir(outfn)
        else:
            for part in offset_parts[::-1]:
                if part is not None:
                    chunk_start, chunk_size = part
                    break
            else:
                if dtlstype == 'u':
                    continue
                else:
                    raise Exception("%s: nothing to look at" % path)
            assert off_in_chunk + cmp_size <= chunk_size
            dtfp.seek(chunk_start + off_in_chunk)
            cmp_data = dtfp.read(cmp_size)
            # XXX why isn't 'compressed' right?
            if cmp_data.startswith('x\x9c'):
                file_data = zlib.decompress(cmp_data)
            else:
                file_data = cmp_data
            assert len(file_data) == dec_size
            open(outfn, 'wb').write(file_data)


        resource_total_size += cmp_size

if 0:
    for missing in set(dt_offsets.keys()) - known_crcs:
        print 'Missing', missing

And execute "python dtls.py dt ls output u" for an updated version, or "python dtls.py dt ls output o" for the original one on the catradge.

Ok, I'm really bad at this. I have a few questions. 1.) Where do I put "python dtls.py dt ls output o" in the code. 2.) Is there any way that I can just copy and paste this into Python. I'm on version 2.7 of Python if that info is necessary

gudenau · Oct 30, 2015

Sammi Husky said:

I didn't update mine yet, no. The one you posted is the merged fixes from shinyQuagsire and mine. The only difference being that it doesn't print the filenames in the console. The fully updated version would be

Code:

import struct, sys, zlib, os
from collections import OrderedDict
from cStringIO import StringIO

dtfn, lsfn, outdir, dtlstype = sys.argv[1:]
if not os.path.exists(outdir):
    os.makedirs(outdir)
tmpPath = os.getcwd()+'/tmp'
if not os.path.exists(tmpPath):
    os.makedirs(tmpPath)
dtfp = open(dtfn, 'rb')
lsfp = open(lsfn, 'rb')
lsfp.read(4)
count, = struct.unpack('<I', lsfp.read(4))
dt_offsets = OrderedDict()
dt_total_size = 0
for i in xrange(count):
    if dtlstype == 'u':
        crc, start, size, dt_index, unk = struct.unpack('<IIIHH', lsfp.read(16)) #TODO: Apparently there's indexed dt files (ie dt00, dt01)
    else:
       crc, start, size = struct.unpack('<III', lsfp.read(12))
    dt_offsets[crc] = (start, size)
    dt_total_size += size
#assert lsfp.read(1) == ''

def get_file((start, size)):
    dtfp.seek(start)
    compressed = dtfp.read(size)
    data = compressed
    if compressed.startswith('\xcc\xcc\xcc\xcc'):
        z = compressed.find('\x78\x9c')
        if z != -1 and z <= 0x300:
            decompressed = zlib.decompress(compressed[z:])
            return decompressed, True
    return data, False

def invertify(msg):
    return ''.join(chr(~ord(msg[i]) & 0xff) for i in xrange(4)) + msg[4:]

def stupidcrc(filename):
    return zlib.crc32(invertify(filename)) & 0xffffffff

resource = dt_offsets[stupidcrc('resource')]
resource_data, was_compressed = get_file(resource)
open(tmpPath+'/resource.dec', 'wb').write(resource_data)
assert resource_data.startswith('RF')
offset_to_compressed, = struct.unpack('<I', resource_data[4:8])
rf, hl1, _, hl2, \
_0x18_entries_len, timestamp, compressed_len, decompressed_len, \
start_of_strs_plus, len_strs \
    = struct.unpack('<10I', resource_data[:0x28])

resource_dec = zlib.decompress(resource_data[offset_to_compressed:])
rdfp = StringIO(resource_dec)
open(tmpPath+'/resource.bin', 'wb').write(resource_dec)

rdfp.seek(start_of_strs_plus - hl1)
num_segments, = struct.unpack('<I', rdfp.read(4))
segments = [rdfp.read(0x2000) for seg in xrange(num_segments)]

def get_from_offset(off, len):
    # this is actually just a linear mapping, so pretty useless - but at least
    # this documents what needs to happen for any repackers
    seg_off = off & 0x1fff
    return segments[off / 0x2000][seg_off:seg_off + len]

parts = []
offset_parts = []

known_crcs = set()
resource_total_size = 0

if 1:
    num_offsets, = struct.unpack('<I', rdfp.read(4))
    extension_offsets = struct.unpack('<%dI' % num_offsets, rdfp.read(4 * num_offsets))
    extensions = []
    for i, exto in enumerate(extension_offsets):
        ext = get_from_offset(exto, 64)
        ext = ext[:ext.find('\0')]
        extensions.append(ext)
        #print i, repr(ext)

    rdfp.seek(0)
    num_8sized, = struct.unpack('<I', rdfp.read(4))
    rdfp.read(num_8sized * 8)
    another_size, = struct.unpack('<I', rdfp.read(4))
    rdfp.read(another_size)
    while rdfp.tell() < _0x18_entries_len:
        off_in_chunk, name_offset_etc, cmp_size, dec_size, timestamp, derp1 = struct.unpack('<IIIIII', rdfp.read(0x18))
        ext = name_offset_etc >> 24
        name_offset = name_offset_etc & 0xfffff
        name = get_from_offset(name_offset, 128)
        if name_offset_etc & 0x00800000:
            reference, = struct.unpack('<H', name[:2])
            ref_len = (reference & 0x1f) + 4
            ref_reloff = (reference & 0xe0) >> 6 << 8 | (reference >> 8)
            name = get_from_offset(name_offset - ref_reloff, ref_len) + name[2:]
        if '\0' in name:
            name = name[:name.find('\0')]
        name += extensions[ext]

        nesting_level = derp1 & 0xff
        localized = bool(derp1 & 0x800)
        final = bool(derp1 & 0x400)
        compressed = bool(derp1 & 0x200)
        parts = parts[:nesting_level - 1] + [name]


        path = ''.join(parts)

        if final:
            start, size = None, None
            crc_path = 'data/' + path.rstrip('/') + ('/packed' if compressed else '')
            crc = stupidcrc(crc_path)
            if crc in dt_offsets:
                offset = dt_offsets[crc]
            else:
                offset = None
        else:
            offset = None
        offset_parts = offset_parts[:nesting_level - 1] + [offset]

        outfn = os.path.join(outdir, path)
        if path.endswith('/'):
            if not os.path.exists(outfn):
                os.mkdir(outfn)
        else:
            for part in offset_parts[::-1]:
                if part is not None:
                    chunk_start, chunk_size = part
                    break
            else:
                if dtlstype == 'u':
                    continue
                else:
                    raise Exception("%s: nothing to look at" % path)
            assert off_in_chunk + cmp_size <= chunk_size
            dtfp.seek(chunk_start + off_in_chunk)
            cmp_data = dtfp.read(cmp_size)
            # XXX why isn't 'compressed' right?
            if cmp_data.startswith('x\x9c'):
                file_data = zlib.decompress(cmp_data)
            else:
                file_data = cmp_data
            assert len(file_data) == dec_size

            print outfn
            open(outfn, 'wb').write(file_data)

        resource_total_size += cmp_size

if 0:
    for missing in set(dt_offsets.keys()) - known_crcs:
        print 'Missing', missing

Still do not understand why everyone loves Python so much, it is quite a odd language to me.

--------------------- MERGED ---------------------------

RetroGamer732 said:
Ok, I'm really bad at this. I have a few questions. 1.) Where do I put "python dtls.py dt ls output o" in the code. 2.) Is there any way that I can just copy and paste this into Python. I'm on version 2.7 of Python if that info is necessary

Put the code in a file called dtls.py next to dt and ls from the romfs, then in a command prompt/terminal run "python dtls.py dt ls output o" and wait. The output directory should contain the smash assets.

ROM Hack Edit Smash Models?

Active Member

( ͡° ͜ʖ ͡°)

Largely ignored

( ͡° ͜ʖ ͡°)

Well-Known Member

GBATemp's Local Octopus, Open 9am-2am. "Not Yet"

Well-Known Member

Largely ignored

Active Member

Largely ignored

Well-Known Member

Largely ignored

Well-Known Member

Largely ignored

Well-Known Member

Well-Known Member

Largely ignored

Well-Known Member

Well-Known Member

Largely ignored

Similar threads

Popular threads in this forum