ROM Hack 3DS ROM Tool: rom_tool

[tHciNc]

I havent had any issues with trimming

 

[bkifft]

The cause of that error is rom_tool thinks the ROM has an incorrect size. It's possible retrieved values used to determine if the size is correct, may not be obtained properly in linux, I had to jump some hoops to get it to work on Mac OS X and that might have broken something for linux. I'll look at it tomorrow morning.

EDIT: I retroactively change the license for this code fragment to WTFPLv2.

easy fix:

Spoiler

u64 GetFileSize_u64(char *filename)
{
        u64 size;
#ifdef _WIN32
        int fh;
        u64 n;
        fh = _open( filename, 0 );
        n = _lseeki64(fh, 0, SEEK_END);
        _close(fh);
        size = (n / sizeof(short))*2;
#else
#ifdef __linux__
        struct stat st;
        stat(filename, &st);
        size = st.st_size;
#else
        FILE *file = fopen(filename,"rb");
        fseeko(file, 0L, SEEK_END);
        size = (u64)ftello(file);
        fclose(file);
#endif //__linux__
#endif //_WIN32
        return size;
}

some cursory googling suggests that stat() also exists in Mac OS, might even be mandatory for POSIX systems (can't be arsed to look that up at 4am). seems also to exist on Windows (including the usual trailing _).

now to find out why your solution doesn't work (which it should by a first look)...

nevertheless: three thumbs up for the tool, highly appreciated.

edit: found it: _FILE_OFFSET_BITS 64 needs to be set before including the respective headers, thus another fix would be to move those three flags to the top of lib.h or use the -D flag in the compiler call.

you should use the stat() call anyway, as it should be way faster, as it querries the filesystem records and thus doesn't have to read the whole file byte by byte. (might be wrong there, but it's twelve past four and I'm tipsy from mulled wine, heck, lets do a small test).

edit 2: ok, after my next cup of wine I'll admit that there shouldn't be too much of a speed diffrence, as a good fseek() implementation shouldn't walk through the file on SEEK_END but jump there.

 

[Tomobobo]

Wow, yeah thanks this is awesome! Trimming is a breeze.

 

[Jiro2]

I am running this on Linux, having gotten 3.1 from Github and built it with the Makefile.

Every single time I try to trim something which is 2G or larger in size, I get something like this:

% rom_tool -t "Shin Megami Tensei - Devil Survivor Overclocked.3ds"
CCI_FILE_SIZE = 0xffffffffffffffff
MEDIA_SIZE = 0x80000000
CCI_IMAGE_SIZE = 0x48085c00
CCI_S_TRIM_SIZE = 0x47ba3a00
[!] CCI is malformed
[+] Trimming CCI
[*] Completed Successfully

CCI file size is always all F's (which is -1 in hex).

Is it safe to ignore these errors?

 

[bkifft]

I am running this on Linux, having gotten 3.1 from Github and built it with the Makefile.

Every single time I try to trim something which is 2G or larger in size, I get something like this:

% rom_tool -t "Shin Megami Tensei - Devil Survivor Overclocked.3ds"
CCI_FILE_SIZE = 0xffffffffffffffff
MEDIA_SIZE = 0x80000000
CCI_IMAGE_SIZE = 0x48085c00
CCI_S_TRIM_SIZE = 0x47ba3a00
[!] CCI is malformed
[+] Trimming CCI
[*] Completed Successfully

CCI file size is always all F's (which is -1 in hex).

Is it safe to ignore these errors?

erm... you might want to try one of the two solutions i posted 2 posts above yours. the easier one is to move the three #define statements from the bottom of lib.h to the top.

 

[3DSGuy]

I am running this on Linux, having gotten 3.1 from Github and built it with the Makefile.

Every single time I try to trim something which is 2G or larger in size, I get something like this:

% rom_tool -t "Shin Megami Tensei - Devil Survivor Overclocked.3ds"
CCI_FILE_SIZE = 0xffffffffffffffff
MEDIA_SIZE = 0x80000000
CCI_IMAGE_SIZE = 0x48085c00
CCI_S_TRIM_SIZE = 0x47ba3a00
[!] CCI is malformed
[+] Trimming CCI
[*] Completed Successfully

CCI file size is always all F's (which is -1 in hex).

Is it safe to ignore these errors?

No it's not, I'll have an update with the fix by bkifft pushed to Git soon.

(Yeah I know I'm late for fixing it, I've been snowboarding in another country. So rom_tool has little priority ATM)

Unfortunately rom_tool has reached the point in its development, where I am unhappy with 90% of the implementation. And my direction for the program cannot be addressed without a re-write. After the bug fix, it'll likely be superceded by another program.

 

[3DSGuy]

rom_tool 3.2 source is available. Fixes linux problems among other things. I'll release this properly with Win/Mac binaries when I have time.

 

[Jiro2]

The new version no longer gives me the errors but the files produced are the same size and identical to the files produced by the 3.1 version...

 

[3DSGuy]

The new version no longer gives me the errors but the files produced are the same size and identical to the files produced by the 3.1 version...

And is that bad? What does rom_tool say about the produced files?

 

[Jiro2]

And is that bad? What does rom_tool say about the produced files?

You said that it wasn't safe to ignore the errors in 3.1. But it's still producing the same trimmed ROM that I got when I ignored the errors in 3.1.

rom_tool used on the produced files doesn't say anything that's obviously unusual and for a trimmed ROM it shows the CCI data size the same as the CCI file.

 

[3DSGuy]

You said that it wasn't safe to ignore the errors in 3.1. But it's still producing the same trimmed ROM that I got when I ignored the errors in 3.1.

rom_tool should have stopped when it detected a 'malformed' rom. I assumed you weren't able to 'use' roms with a filesize of >=2GB with rom_tool 3.1.

Apparently I didn't comment out debug code in rom_tool 3.1, so despite not finding the correct file size it continued anyway.

rom_tool 3.1 produced the same files, because there was nothing wrong with the ROM. I said it was unsafe to ignore the errors because rom_tool may have behaved weirdly.

 

[Subtle Demise]

If anyone has trouble using the command line and wants to trim many 3DS roms at once, I created a batch file to make things 10000% easier. Extract the zip into the same folder as all of your .3ds files and double-click Trim.bat, then you're done.

 

[duke_srg]

It's weird, but seems that _FILE_OFFSET_BITS 64 not working for truncate on my system. Failed with restore for files over 2Gb and with truncate for 4Gb files, until recompiled with changed to truncate64.

 

[bkifft]

It's weird, but seems that _FILE_OFFSET_BITS 64 not working for truncate on my system. Failed with restore for files over 2Gb and with truncate for 4Gb files, until recompiled with changed to truncate64.

Would you mind telling what system (OS, compiler and their respective versions) that is?

 

[duke_srg]

It is a QNAP NAS embedded linux

# cat /proc/version
Linux version 3.4.6 (root@NasX86-9) (gcc version 4.1.3 20070929 (prerelease) (Ubuntu 4.1.2-16ubuntu2)) #1 SMP Fri Apr 26 03:31:24 CST 2013
# ipkg list_installed | grep gcc
gcc - 4.2.1-5 - The GNU Compiler Collection.
# gcc --version
gcc (GCC) 4.2.1
Copyright (C) 2007 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

 

[bkifft]

It is a QNAP NAS embedded linux

# cat /proc/version
Linux version 3.4.6 (root@NasX86-9) (gcc version 4.1.3 20070929 (prerelease) (Ubuntu 4.1.2-16ubuntu2)) #1 SMP Fri Apr 26 03:31:24 CST 2013
# ipkg list_installed | grep gcc
gcc - 4.2.1-5 - The GNU Compiler Collection.
# gcc --version
gcc (GCC) 4.2.1
Copyright (C) 2007 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Ok, that's weird indeed, as (e)glibc should wrap truncate() calls to truncate64() if it's a 64 bit system or _FILE_OFFSET_BITS == 64...

What version of libc are you using? (ldd --version)

 

[duke_srg]

# ldd --version
ldd (GNU libc) 2.6.1
Copyright (C) 2007 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Written by Roland McGrath and Ulrich Drepper.

Yes, it SHOULD wrap truncate to truncate64, but it did't. Checked unistd.h, seems to look fine. But still can see truncate in utils.o and at the same way fseeko64 %)

upd: Tried on another Ubuntu server, only truncate in utils.o exports.
upd2: Note a build warning. No warnings for fseek or any other, just for truncate.

# make
gcc --std=c99 -Wall -I.   -c -o utils.o utils.c
[COLOR=#ff0000]utils.c: In function 'TruncateFile_u64':[/COLOR]
[COLOR=#ff0000]utils.c:198: warning: implicit declaration of function 'truncate'[/COLOR]
gcc --std=c99 -Wall -I.   -c -o main.o main.c
gcc --std=c99 -Wall -I.   -c -o ncsd.o ncsd.c
ncsd.c: In function 'GetNCSDData':
ncsd.c:261: warning: implicit declaration of function 'PrintCCIPartitionData'
ncsd.c: At top level:
ncsd.c:374: warning: conflicting types for 'PrintCCIPartitionData'
ncsd.c:261: warning: previous implicit declaration of 'PrintCCIPartitionData' was here
g++ -o rom_tool -static-libgcc -static-libstdc++ -lm utils.o main.o ncsd.o
g++: unrecognized option '-static-libstdc++'

Changing with unistd.h gave nothing. Seems like it is build like libraries should imlement it right way but it's not.

 

[Jiro2]

I am having problems using this on a large file.

% ls -l *.3ds
-rw-r--r-- 1 xxx xxx 4294967296 Feb 8 20:40 bbb-p-btre.3ds
% rom_tool
[!] Must Specify Arguments
CTR_Toolkit - CCI Tool
Version 3.2 (C) 3DSGuy 2013
Usage: rom_tool [options] <cci filepath>
OPTIONS Possible Values Explanation
-h, --help Print this help.
-i, --info Display CCI info
-p, --partition_info Display CCI partitions
-r, --restore Restore unused bytes to CCI file.
-t, --trim Remove unused bytes from CCI file.
-u, --remove_update Remove update data
-x, --extract= Dir-out Extract CCI partitions to directory
% rom_tool -i *.3ds
[+] CCI Image Details
Media Type: CARD1
Media Size: 4 GB (32 Gbit)
CCI Data Size: 3342 MB (0xd0e91000 bytes)
CCI File:
> Size 4096 MB
> Status Full Size
Additional Device: EEPROM
Partition Count: 3
CUP Version: 6.2.0U
Save Crypto: 6.0.0 KeyY Method
[+] CXI Partition
Product Code: CTR-P-BTRE
Company Code: 00
Unique ID: 00fc5
Build Type: Release
SDK Version: 4.2.5 Release
Req. Kernel Version: 2.35-0
[+] CFA Partitions
E-Manual: Yes
DLP Child: No
Update Data: Yes
[*] Completed Successfully
% rom_tool -t *.3ds
[+] Trimming CCI
[!] Failed to trim CCI
[!] Failed

/usr/local/src/ctr_toolkit-master/rom_tool% head -3 lib.h
#define _FILE_OFFSET_BITS 64
#define _LARGEFILE_SOURCE
#define _LARGEFILE64_SOURCE
/usr/local/src/ctr_toolkit-master/rom_tool%

 

[Jiro2]

I found my problem, sort of. I also got a warning about implicit declaration of truncate(). From the man page:

Feature Test Macro Requirements for glibc (see feature_test_macros(7)):

truncate():
_BSD_SOURCE || _XOPEN_SOURCE >= 500 ||
_XOPEN_SOURCE && _XOPEN_SOURCE_EXTENDED
|| /* Since glibc 2.12: */ _POSIX_C_SOURCE >= 200809L

Adding a #define _BSD_SOURCE to lib.h fixed the problem as well as not giving a warning about truncate.

My system is Mageia 2.

 

[leeksausage]

Can anyone assist me with this guys?

Using a mac, steps so far:

Copied rom_tool into my directory with the roms in.
terminal - cd into the directory so it shows on the command line
try to run the room_tool code (i.e. rom_tool -t Pokemon X.3ds) comes back with: -bash: rom_tool: command not found

Tried a number of different approaches, no luck with them. Anyone got an "idiots guide" for me or something?