Hacking Nitro Zip

[BassAceGold]

So I've been playing around with zlib for a bit and finally managed to get it to output valid files with the DSTwo SDK. This got me thinking about creating a compressed rom type format. What I was thinking is that you'd have to just decompress the rom before playing. No on the fly decompression because that isn't really possible.

It would be fairly easy to incorporate such format into a menu like BAGplug (auto decompression and deletion of uncompressed files), if enough people were really into the idea.

Here are some (loosely accurate) tests of the format I have conducted on two opposite sides of the spectrum ( 16mb, 32mb, 64mb, 128mb, 256mb):
*R4DS Rom Trimmer 2.0 was used*
Tetris DS:
normal size - 16 mb
trimmed size - 13.1mb
compressed size - 6.4mb
compression time (on DS) - ~1 minute
decompression time - < 1 minute

Pokemon Soul Silver:
normal size - 128 mb
trimmed size - 120mb
compressed size - 58mb
compression time (on DS) - ~4minutes
decompression time - ~1 minute

Pokemon Black:
normal size - 256 mb
trimmed size - 256 mb
compressed size - 105 mb
compression time (on DS) - ~11 minutes
decompression time - ~1 and a half minutes

Dementium 2:
normal size - 64 mb
trimmed size - 54.2 mb
compressed size - 39.2 mb
compression time (on DS) - ~ 3 minutes
decompression time - ~1 minute

Brothers in Arms DS:
normal size - 32 mb
trimmed size - 30.9 mb
compressed size - 17.5 mb
compression time (on DS) - ~3 minutes
decompression time - < 1 minute

Club House Games:
normal size - 16 mb
trimmed size - 10 mb
compressed size - 8.3 mb
compression time (on DS) - < 1 minute
decompression time - < 30 seconds

Max compression level was used on the DSTwo, so faster compression times can be achieved using a lighter level, or if a pc application was made for compression.
The format itself is just a gzip file, with bits of the nds rom header appended to the end, so menus such as BAGplug or iMenu can still access rom information without having to unzip the rom itself.

So what my question is, is there really a need (or want) for this or are people just happy enough using trimmed roms? If there isn't much interest in this, then this idea won't go far.

Source code for the compression / decompression routines can be found here:
http://bag.nfnet.org/wp-content/uploads/2011/04/nzip-src.zip

Source was built using zlib 1.5.2

 

[Cuelhu]

this sounds brilliant even though I don't trim my ROMs.

 

[Rydian]

The people that would want to squeeze more games onto an SD are usually using 2GB ones, in which the space reserved for an uncompressed game could just fit more games in the first place. On something like a 4GB or 8GB card the savings add up a lot more, but people with those generally don't care about an extra game or two. I fall into the second category, I wouldn't even consider this.

I like the idea, but I don't think people would really use it, I don't want you to waste your time.

 

[BassAceGold]

The people that would want to squeeze more games onto an SD are usually using 2GB ones, in which the space reserved for an uncompressed game could just fit more games in the first place. On something like a 4GB or 8GB card the savings add up a lot more, but people with those generally don't care about an extra game or two. I fall into the second category, I wouldn't even consider this.

I like the idea, but I don't think people would really use it, I don't want you to waste your time.

It just depends on the combination of games on your card. You may be able to fit one or two extra games on smaller cards if all your games compress small enough while leaving enough space for your biggest uncompressed rom (you can also trim and compress), while others with larger cards may notice the space gain a lot more with more games being compressed.

 

[geoflcl]

What a novel idea!

Although something of this sort wouldn't be a concern for everybody (as Rydian said), I happen to be one of those lamers who use a 2GB card to cram all of their flashcart paraphernalia. Any saved space is quite beneficial, and I find the compression rates you've shown to be quite impressive!

Heck, even if you don't decide to go far with this idea, you've made me ecstatic just by this proof of concept. Wonderful work!

EDIT: I have just realized the problem of decompression free space for smaller cards. Oopse. Consider my above post exemplary of my stupidity.
(I really need to stop browsing the 'Temp while struggling to stay awake. :S )

 

[gumbyx84]

This is interesting. While I agree with Rydian that its useless on smaller cards, I can see this being useful to people like me with 8GB cards. Would love to help you with this (coding and testing) if you choose to continue, but sadly I don't have a DSTWO.

PS Would it be possible to use this in DSi mode with the iEvo?

 

[Boriar]

I want it so much because:

1. I hate trim my roms backups. I can't explain it better or way but I dislike it

2. I like carry on my mSD all the roms I want to play but only play one to three simultaneusly, so the other ones may be compressed to preserve space.

Until now I use DSCompress v1.2 Author: Carpman http://voidptr.org but use dstwo processor will be great!!!

Edit: I can test it or translate something if you need it

 

[Deleted_171835]

I would use this!

I have a 2GB microSD card in my DSTWO and I can't fit every game I want to play even after I trim the rom. This would be great!

 

[BassAceGold]

This is interesting. While I agree with Rydian that its useless on smaller cards, I can see this being useful to people like me with 8GB cards. Would love to help you with this (coding and testing) if you choose to continue, but sadly I don't have a DSTWO.

PS Would it be possible to use this in DSi mode with the iEvo?

The code itself is fairly portable. Anything that has a zlib port can decompress/compress it, just depends on the system itself if the process will be fast or not.

QUOTE(SoulSnatcher @ Apr 15 2011, 04:21 PM) I would use this!

I have a 2GB microSD card in my DSTWO and I can't fit every game I want to play even after I trim the rom. This would be great!

Keep in mind that you do need to leave enough free space on your card to hold the largest uncompressed rom you have.

 

[gumbyx84]

This is interesting. While I agree with Rydian that its useless on smaller cards, I can see this being useful to people like me with 8GB cards. Would love to help you with this (coding and testing) if you choose to continue, but sadly I don't have a DSTWO.

PS Would it be possible to use this in DSi mode with the iEvo?

The code itself is fairly portable. Anything that has a zlib port can decompress/compress it, just depends on the system itself if the process will be fast or not.

... This really makes me want to get a DSTWO and iEvo. I would be all over helping you code or even test stuff for ya. Been wanting to get back in development for awhile and this sounds like fun. :-D

 

[FAST6191]

A word on Tetris DS- if you picked the Japanese or US version the SDAT file has a meg or two of padding on it which when coupled with the regular trimming leads to that massive compression you saw.

Nice proof of concept though- if you are needing more code there is DSCompress as well.

 

[Schlupi]

I fully support this idea! I have an 8 GB, 2 GB, and 4 GB (hahaha) and would love to help test this out.

 

[BassAceGold]

A word on Tetris DS- if you picked the Japanese or US version the SDAT file has a meg or two of padding on it which when coupled with the regular trimming leads to that massive compression you saw.

Nice proof of concept though- if you are needing more code there is DSCompress as well.

I compressed untrimmed roms only. I might just test a bunch more roms to get a more accurate average rate of compression.

Updated first post with more results.

 

[BassAceGold]

Just posting a little bit based on the information in the first post. If you happened to have 512mb of free space, enough to hold all the games listed in the first post, and you compress them, you'd have 277.6 mb of free space left over. This space is enough for another compressed game (a small one) while still having enough room to decompress the largest rom (256mb), assuming you don't fill up the left over space with other things.

 

[gumbyx84]

Those compression ratios are nice, as are the decompression times. I'm guessing you would make a PC app that would allow you to compress ROMs. Compression time would be much faster if done on PC.

 

[BassAceGold]

Added source for the compression / decompression routines if anyone wants to make something (sorry for the sparse comments, hopefully the code is self explanitory) of it in the first post. If anyone makes any modifications to the code just post it here so we don't get multiple different formats created.

 

[Another World]

my only concern is the extra writing being done to the card each time a file is decompressed. if i play that file 10 times in a day its essentially written to the card 10 times. i'm thinking about the limited amount of read/writes a msd has.

this is a really cool idea for nds roms, tho i'm not sure how practical it would be.

-another world

 

[BassAceGold]

my only concern is the extra writing being done to the card each time a file is decompressed. if i play that file 10 times in a day its essentially written to the card 10 times. i'm thinking about the limited amount of read/writes a msd has.

this is a really cool idea for nds roms, tho i'm not sure how practical it would be.

-another world

That's dependent on the menu using it. If the menu was smart, it wouldn't delete the uncompressed file unless a new compressed file was selected to unzip. So basically once you unzip a file, you wouldn't need to keep unzipping it until you select a new game to unzip.

 

[twiztidsinz]

my only concern is the extra writing being done to the card each time a file is decompressed. if i play that file 10 times in a day its essentially written to the card 10 times. i'm thinking about the limited amount of read/writes a msd has.

this is a really cool idea for nds roms, tho i'm not sure how practical it would be.

-another world

http://en.wikipedia.org/wiki/Flash_memory#Memory_wear
They're rated to 100,000 writes, and using your 10 writes a day and doubling it for saves is 20 writes/day
100,000 / 20 = 5,000 / 365 = 13.6 years.

Add in something like 'wear leveling' that some cards use and you're going to get a longer life out of it.

 

[Boriar]

my only concern is the extra writing being done to the card each time a file is decompressed. if i play that file 10 times in a day its essentially written to the card 10 times. i'm thinking about the limited amount of read/writes a msd has.

this is a really cool idea for nds roms, tho i'm not sure how practical it would be.

-another world

That's dependent on the menu using it. If the menu was smart, it wouldn't delete the uncompressed file unless a new compressed file was selected to unzip. So basically once you unzip a file, you wouldn't need to keep unzipping it until you select a new game to unzip.

Also, you can let user to set on/off a deleting option. In that way with on actived anyone can select a game and if it's already decompressed play it directly (not delete game at all) and with off activated only keep the decompressed one until a different game will be selected, saving more time/space like you choose.