Hacking NDSTokyoManager

[BrianTokyo]

Got some ideas in mind to identify the roms which would be much faster than using those DAT files.

 

[Narin]

Got some ideas in mind to identify the roms which would be much faster than using those DAT files.

Oh, such as? I'm curious.

 

[kazumi213]

No-Intro Project offers DAT files for CMPro-compatible ROM managers which only contain data for dumps from commercial titles, as close as possible to the contents on the original media. All hacked variations are filtered out from the DATs.

In addition to these standard DATs, No-Intro started offering Parent-Clone DATs (MAME concept). Currently we offer P-C DATs for NDS, N64 (both from DAT-o-MATIC), GBA and SNES (available at No-Intro forum). All these P-C DATs are written in the new generic XML format developed by Logiqx (maintainer of MAME DAT) giving support to a couple of features we asked for. CMPro already supports this format and RomCenter will do in the near future.

These features are ROM merging support (in the specific way described below) and "one game, one ROM" (1G1R) filtering. Both features are based on region preferences specified by the user.

To show the ROM merging feature, here's an example for the GBA system:

WarioWare, Inc. - Minigame Mania (Europe) (En,Fr,De,Es,It)
ÂÂÂÂMade in Wario (Japan)
ÂÂÂÂWaliou Zhizao (China)
ÂÂÂÂWarioWare, Inc. - Mega Microgames! (USA)
ÂÂÂÂWarioWare, Inc. - Mega Microgames! (USA) (Beta)
Watashi no Makesalon (Japan)
Whac-A-Mole (USA)
Whistle! - Dai 37 Kai Tokyo-to Chuugakkou Sougou Taiiku Soccer Taikai (Japan)
Who Wants to Be a Millionaire (Europe)
ÂÂÂÂChi Vuol Essere Milionario (Italy)
ÂÂÂÂQui Veut Gagner des Millions (France)
ÂÂÂÂQuiere Ser Millonario (Spain)
ÂÂÂÂWeekend Miljonairs (Netherlands)
ÂÂÂÂWer wird Millionaer (Germany)
ÂÂÂÂWho Wants to Be a Millionaire (Australia)

Now, If user sets region preference to "Spain > USA > Europe > Japan..." listed ROMs in the example above will be merged into 5 archives named:

WarioWare, Inc. - Mega Microgames!.zip
Watashi no Makesalon (Japan).zip
Whac-A-Mole (USA).zip
Whistle! - Dai 37 Kai Tokyo-to Chuugakkou Sougou Taiiku Soccer Taikai (Japan).zip
Quiere Ser Millonario.zip

Note the "family" archives (with more than 1 ROM) are named using untagged title from prefered region. Standard archives (single ROM) are named using title with tags as usual. ROMs inside the archives will remain named as listed in the example. Leaving tags for archives with a single ROM is both accurate and allows to quickly identify family/single ROM archives just by browsing.

The 1G1R filtering feature allows the user to build a ROM collection which contains only 1 game per family, that which matches his region preference, ignoring all other region duplicates. So in the example above, with the same region preference settings, CMPro will only recognize the following ROMs:

WarioWare, Inc. - Mega Microgames! (USA)
Watashi no Makesalon (Japan)
Whac-A-Mole (USA)
Whistle! - Dai 37 Kai Tokyo-to Chuugakkou Sougou Taiiku Soccer Taikai (Japan)
Quiere Ser Millonario (Spain)

So you could move out from your collection 10 ROMs and you wouldn't be missing any game. Always the "best" ROM (retail and highest revision, most complete edition, etc) is picked.

For those who want to merge as described above but using 7z format, No-Intro also offers XMDB files for GoodMerge.

 

[BrianTokyo]

Nice, thanks for the info. I will look into this.

 

[lonewolf80]

Will it look more or less like DS-X Manager (with the icons, publisher, true game name, and your rating)? OR like an iTunes, where you can choose you "iPod" To ync it to every time you plug it in. And don't forget the ecchi skins!

 

[BrianTokyo]

Hmm hard to say atm, but not really like iTunes

 

[_mrshl_]

everything that comes to my mind right now:
1. multiple roms per archive / region duplicates in one archive
2. SHA and/or MD5 checks additional to CRC if possible
3. user-adjustable compression parameters (7z has quite a few)
4. support for more archive types (atleast for decompression - 7z.dll gives you this option)
5. maybe an option to use an external de-/compression program
6. would be kind to provide a way to let the user use newer/other versions of 7z.dll
7. of course custom rom naming schema via "%x"-parameters (with optional params - eg. for mutlilang roms an optional "(Mn)" )
8. user-adjustable language-codes (like in offlinelist)
9. region/duplicate filter
10. resizeable window
11. maybe an additional linux version (especially if not running via wine)
12. boxart/screenshot/nfo download
13. show/export logfile for batch processing
14. export miss/have lists
15. ?...

I assume this project will (sadly..) become closed source either, right?!? Which language you code in, btw?
If one would have acces to the source, a verbose debug mode would be a nice addition to track down errors so that _you_ won't have to recreate and track down every little problem

.

Regards
mrshl

 

[BrianTokyo]

Yeah it is closed source like my trimmer. I make in Delphi for win32 (no .net)

 

[kobykaan]

Ok ...let me get this straight then..... ALL the roms that have several regions/countrys/versions of that specific game will/would be packed into 1 archive ??

Wouldnt it be a massive PITA having several of the same files in 1 archive, when it comes to extracting and putting onto a flash card??

I mean to extract just the one you need...if theres 5 or 6 or more in the one archive then its going to more of a mess about trying to get that 1 file out is it not??

 

[_mrshl_]

Yeah it is closed source like my trimmer. I make in Delphi for win32 (no .net)

Meaning you will only support windows in future? Do you know how well your trimmer works under linux via wine? I am running winxp right know; just would like to know..
What do thing about my suggestions? Are there a few on your todo list?

Regards

 

[BrianTokyo]

My trimmer works very well so far in Wine. There were some versions which stopped working in Linux, but I fixed that last year. The only lacking feature in linux/wine is 7zip support.

This means the manager will also work in Linux/wine. (But no native version as the userbase is too small to justify the extra work)

Not sure about giving so much support for 7zip, it's a lot slower than RAR.

About using different 7zip DLLs.. regarding my trimmer, if you pick the "UAC compatible" version, you can use a different DLL there, as they are not packed in the exe like in the normal version. Many of the options you suggested are already on my mind.


About being open source or not... I think that would be a really bad idea. Just look at Linux, there are a million different distros and no standard at all, which makes it very difficult for hardware companies to give a proper support. It's become so diverse that (in my opinion) it's a hell..

I don't want my software to have 384783 different versions from unknown people, which bring new bugs from something I made. Besides that, my trimmer wouldn't be as famous nowadays if it had been open source, as anybody could have made their own parting from my routines.

In short, I don't support open source, at all.

 

[_mrshl_]

My trimmer works very well so far in Wine. There were some versions which stopped working in Linux, but I fixed that last year. The only lacking feature in linux/wine is 7zip support.

This means the manager will also work in Linux/wine. (But no native version as the userbase is too small to justify the extra work)

Not sure about giving so much support for 7zip, it's a lot slower than RAR.

About using different 7zip DLLs.. regarding my trimmer, if you pick the "UAC compatible" version, you can use a different DLL there, as they are not packed in the exe like in the normal version. Many of the options you suggested are already on my mind.


About being open source or not... I think that would be a really bad idea. Just look at Linux, there are a million different distros and no standard at all, which makes it very difficult for hardware companies to give a proper support. It's become so diverse that (in my opinion) it's a hell..

I don't want my software to have 384783 different versions from unknown people, which bring new bugs from something I made. Besides that, my trimmer wouldn't be as famous nowadays if it had been open source, as anybody could have made their own parting from my routines.

In short, I don't support open source, at all.

Nice to hear, that it is working quite well with wine.
But there are some things I disagree with you

:
1. 7zip vs RAR: Don't have WinRar currently installed on my machine, but from tests I ran some months back I can recall that 7zip can be even faster than rar, because the compression is extremely asymmetric, meaning compression may be very slow in "ultra"-mode but decompression is as fast as compressed in fast-mode; in rar the decompression gets slower too, the higher the compression is defined.
2. opensource: if projects are managed correctly and good, then there won't be such a mess (look at mame, snes9x --> there are other versions too, but the main versions are/were developed on and on, too).
But of course it is your choice wether you release your sources or not - if you do such a good jub with your trimmer I am fine with it being closed

.

Regards

 

[BrianTokyo]

*nods at mame: I never know where to look for the real original version

 

[BurnChao]

Are you still working on this? This thread is awfuly quite.

If you are, will it compress the roms it stores/manages? I seen that you were talking about rar/zip/etc support, but I couldn't tell if you meant that it just reads/decompresses, or that it will also compress.

 

[BrianTokyo]

Hi, dunno why I stop receiving reply emails from gbatemp often ;( Haven't worked on this for a while so I'm not sure if I will ever finish it; working on other stuff atm.

 

[villadelfia]

My major gripe with every rom manager is that they exclusively support CRC32. CRC32 Just isn't secure, and a file is easily changed to get a collision, possibly inserting malicious code in the process.

Since MD5 is also fatally cracked at this point, my request is an implementation of SHA1 or even better SHA256. Both are still secure for checking files.

 

[BurnChao]

I'm sorry to hear that you won't be developing this. Since no other manager is compatible with your TokyoTrimmer, I really wanted to use this. The only other option is to go with the built in trimmer of my manager.

Oh well. Thanks for all the work you have done, it's appreciated.

 

[gameguy95]

Hi there guys,

There's a bunch of Rom managers around already, so you'll wonder why another one. Well, I've tried a few of them and (with the utmost respect to the programmers for their amazing work and contribution) I don't feel like any of them suits my needs, so here I am starting a new one. Just liked happened with my trimmer when I didn't find any suitable trimmers around that would trim 30 GB of roms, keeping the wifi info, and not locking up the computer in the process, so NDSTokyoTrim was born and is a today a very successful and known application in this community with thousands of downloads every month

"Why anounce it before it's ready" you'll ask. I have a very good reason. I want to ask you guys what you would like to see in it. What features you want and others lack, tell me your needs and expectations, and if they make sense I will work on them. Just like happened with my trimmer, most of the current features are requests and ideas from its users which I took very seriously and make today's tokyo trimmer

Here are my initial ideas:

- Support both NDS and GBA roms, using the same methods as my trimmer.
- Support rar, zip and 7-zip without extra DLLs needed in the disk, just like NDSTokyoTrim does today, very portable!
- Compatible with Linux/wine (just like my trimmer)
- You can expect it to be extremelly fast. If you use my trimmer you know what I mean

- Trimming of both NDS and GBA roms in a batch. Expect the same features of my trimmer here (bricker detection, proper wifi detection, fix of wrong trims, sram paching etc)
- Various tags to mark roms you've already played, set a score, some comments. I think this is very important because most people downloads 2000 roms and plays only a few.
- Many sorting options (size, region, name, trimmed, wifi, homebrew etc)
- Absolutely NO NEED FOR .NET framework. You will never see an app from me which requires that crap.


So here we start, let's discuss this!

(btw, this is the trimmer I'm talking about if you don't know about it yet: http://gbatemp.net/index.php?showtopic=55162)

i am sad to see that neither this nor TokyoTrim have made any progress since 2008. i would love to at least see a beta version of this to know what i should expect of it when you finally finish it.

 

[BrianTokyo]

I had a new version of the trimmer since ages, just unfinished, it's in my mind to make a new release very soon

 

[gameguy95]

I had a new version of the trimmer since ages, just unfinished, it's in my mind to make a new release very soon

i hope so.

what about tokyomanager? 'cause if you scrapped the project, you could at least tell us. lease: