SNES Compression System

Discussion in 'Other Consoles & Oldies' started by N1warhead, Oct 8, 2017.

?
  1. Yes

    11 vote(s)
    64.7%
  2. Perhaps we'll see

    4 vote(s)
    23.5%
  3. No

    2 vote(s)
    11.8%
  1. N1warhead
    OP

    N1warhead Member

    Newcomer
    1
    Oct 8, 2017
    United States
    Hello, my name is David Watts.
    As the title states this is about SNES Classic Compression.

    I am in the works designing a (hopefully) to be implemented compression system for Hakchi2.
    I've been running tons of tests and and optimistic I can make a catered Snes compression system.

    As I'm sure a lot of you have realized, despite there being (7zip) compression with Hakchi2, you are still limited to what you can put on your Snes Classic in terms of amount of games, this is what I am trying to alleviate, every test I've done so far on my program I am writing is showing positive results in terms of patterns, (chunks, etc). Not just that but I've been looking through various SNES games with a Hex Editor and am seeing quite a lot of the same patterns in specific portions of their data. Which is stuff I can account for, optimize and thus give a better result that is catered specifically for this situation.

    But with this being said, I am not trying to be a Hakchi competitor, it's in my hopes that Cluster will implement the final touches into Hakchi once I have it completed this, because after all we all want the same thing - to enjoy our beloved games!

    I will post updates here as often as I can, and if you have any questions please feel free to ask! Also you can stay most informed on my Twitter (I just started my personal one) been using my business one for years. So you can find me on Twitter at @N1warhead

    Thanks guys, hope to make this trip of memory lane better for all of us!



    Faq Below

    Q: What format will this be?
    A: Strangely enough I didn't realize this until after I made the format that it will be .snesc (Snes Compression), it didn't click in my head that it meant as well "Snes Classic" haha.

    Q: How does this work?
    A: As the moment of writing this there are Three stages of optimizing the data, Step One - just get the data we want, Step Two - run the data through an algorithm that has pre-defined common patterns that I find so shortening down 1,000 [0's] to pat0x1000 or something like that. the Third Step is it will then again scan the remaining data for smaller but repeating byte sequences and combining them. As we know bytes don't EVER go above 255, so how can we possibly deal with lets say the following sequence [250, 250, 250] clearly there isn't a byte that can replace this, and that is true, there isn't. However, who's to say we don't just get them 3 bytes and output to 250x3 so when we go back to reconstruct this data we see '250x3' which means to then INPUT [250, 250, 250] back into the structure.

    Q: How long have you been working on this?
    A: To be truthful, literally just started it like 10 hours ago, non-stop coding, setting up UI Interfaces, Directory stuff, gathering data, exporting data, etc.
    I have a lot of experience with compression converting things such as 8K BMP Photos down to roughly 450 bytes. (Like actual photos, not a solid color), solid colors could make an entire image literally 10 bytes. I just didn't see a need to work on this compression system until I seen the results of the SNES Hardware Specs to know if it was even needed - even though I figured it would be, but who knows, there might be a 500 Gig HD in there somewhere hehe.

    Q: Will this be free?
    A: Absolutely, why charge, not a Scalper like some people :P

    Q: Do you have a release date?
    A:I want to be truthful with you, no I do not. It's still very early in development, once I get over this initial hurdle of learning the patterns, I will then be able to give you guys a better idea of a release date.

    Q: Will you release the source code and program?
    A:Absolutely, I will be using my GitHub page when it is ready.

    Q: What license will this be under?
    A: As of now I'm going with MIT so you can do whatever you want with it, don't care about credit, I just want to do a good thing.

    Q: But doesn't Hakchi already have compression, why would they implement your system?
    A: Yes, they most certainly do have Compression, but it's a general compression system that yes it does work, but it isn't catered to this type of data, it's a general purpose compressor. My system will be catered directly to the *.sfrom format and compressing it to my *.snesc format, I don't know if NES Classic uses that same format - I was never lucky enough to get that console. But if it does then you should benefit from this too. I'll try to get in contact with Hakchi creators to make this happen, if not, I suppose I'll make my own System to upload games and such. I just don't see a point in re-inventing the wheel when I can just give it some better tread and get more mileage out of it. But time will tell, we'll see what happens.


    EndOfFaq
     
    Michu, DarthDub, Thoradin and 2 others like this.
  2. BL4Z3D247

    BL4Z3D247 GBAtemp Stoner

    Member
    7
    Oct 22, 2008
    United States
    I'm so high, I don't even know!
    This sounds pretty sweet. I'm all for anything that would help us jam even more games into these mini consoles. I'm sure if your project works well Cluster will implement it into a release if you contacted him. Either way I hope this all works out for you, best of luck.
     
    Last edited by BL4Z3D247, Oct 8, 2017
  3. WeeFlats

    WeeFlats Newbie

    Newcomer
    1
    Oct 8, 2017
    United Kingdom
    Sounds cool, I'll gladly test it if you need help with that sort of thing.

    Any idea how much space you could potentially save beyond 7z/gzip?
     
  4. Jhynjhiruu

    Jhynjhiruu GBAtemp Advanced Fan

    Member
    6
    Dec 31, 2016
    Sounds awesome! Would it be at all possible to combine multiple ROMs into one file and simply have the SNESC load a different portion of the same file? This could theoretically save a little bit of space.
     
  5. azoreseuropa

    azoreseuropa GBAtemp Guru

    Member
    11
    Nov 6, 2002
    United States
    Azores/Portugal
    Yeah, I hate a Scalper like some people.. Now, you are a true person. Go for it. I am supporting you all the way.
     
  6. N1warhead
    OP

    N1warhead Member

    Newcomer
    1
    Oct 8, 2017
    United States
    Hey guys sorry for not answering I had went to bed (I am not a Day Walker) I walk at night :P

    Thanks man, I'm certainly going to attempt it at least. Appreciate your support.

    Sounds like a plan to me, I'll be releasing the GUI Application (Windows only) as it's C# and EXE, the testing that will need to be done on that is to see if there's any games that don't compress right, I'm in the middle of tweaking out some bugs that are doing just that right now, then perhaps have you upload that specific game (uncompressed) to make sure there wasn't any damages to the game... I hope I make sense...... > Compress > UnCompress it > Sync Newly UnCompressed to Console - see if it starts and plays right.
    Well I can't be for certain at this moment how much I can save, I have a lot of data gathering to get, then I have to line by line go over them (whether through a loop) or by hand, then I'll start by creating a pattern file, sorta how Anti-Virus works, it stores the patterns somewhere, the app reads the data and if it detects a common pattern within that Virus list, then do something.

    But give me a little while to get it all good, last thing I want is for it to quadruple brick your SNES haha. So I'm taking this one step at a time.
    I just figured instead of keeping quiet about it to at least let you guys know I'm having a go at it.

    Yeah same here, that's the reason I couldn't get an NES Classic last year, I wasn't about to spend 600+ dollars on it when I could just build my own or play it right on my PC haha. Thanks for your support mate.
     
    DarthDub and BL4Z3D247 like this.
  7. azoreseuropa

    azoreseuropa GBAtemp Guru

    Member
    11
    Nov 6, 2002
    United States
    Azores/Portugal
    No problem but you can try Amazon. About $210 dollars.
     
  8. N1warhead
    OP

    N1warhead Member

    Newcomer
    1
    Oct 8, 2017
    United States
    Thanks for the heads up, eh I'm gonna try and go for the official release when they sell them again coming soon, rather just get it brand new from a trusted source. I've had all kinds of bad experiences with Amazon, ordered a Vega 64 Graphics card and got a 30 dollar board graphics card in the package instead haha.
     
  9. azoreseuropa

    azoreseuropa GBAtemp Guru

    Member
    11
    Nov 6, 2002
    United States
    Azores/Portugal
    Ouch. Never had an experience like that before. Sorry about that. Try Ebay. By the way, I bought NES Classic Mini with 100 percent positive seller last summer. And SNES Classic Mini on the way. :)
     
  10. N1warhead
    OP

    N1warhead Member

    Newcomer
    1
    Oct 8, 2017
    United States
    Yeah I've only had positives with Newegg for electronic orders, I mean it wasn't the sellers faults, it was people thieving at the shipping centers.
    But I'll see what I can do, does the NES Classic need compression at all? Or do all the games fit on it's storage fine?
     
  11. azoreseuropa

    azoreseuropa GBAtemp Guru

    Member
    11
    Nov 6, 2002
    United States
    Azores/Portugal
    If you want to then you can! :)
     
    Last edited by azoreseuropa, Oct 9, 2017
    N1warhead likes this.
  12. N1warhead
    OP

    N1warhead Member

    Newcomer
    1
    Oct 8, 2017
    United States
    Heh, yeah..

    Wow I just did a major bug in the code for this, went from the normal 2,049 KB game to over a gig hahahaahah, oops. Y'all want that, that be epic, get 0.5% of a game haha.
     
  13. N1warhead
    OP

    N1warhead Member

    Newcomer
    1
    Oct 8, 2017
    United States
    Crap man sorry I thought I answered you back, apologize. Got in the coding lol.

    However, at this moment I'm sticking strictly to single files per game, perhaps once I get this working (and if it's efficient) I'll look into doing it as a single file. One step at a time mate, I won't leave any stones unturned. So it's not a matter of if, but when.. But for now, strictly file per game, not a bunch in one file. But later (perhaps in an update?)... Hope that helps.

    Take care,
    David.
     
    Last edited by N1warhead, Oct 9, 2017
  14. N1warhead
    OP

    N1warhead Member

    Newcomer
    1
    Oct 8, 2017
    United States
    Just letting you guys know I'm working on it..
    Right now I'm creating my own encoding format for it, not really on a bit level, but using short hand expressions.

    Like instead of saving 3 digit bytes, save them as two byte short hand expressions.(or Single bytes). I'm going to study more Games to find out the most common bytes and make them single digit bytes.
    That one byte can make a big different if you consider the size of games being 2 million plus bytes.
    Lets say every single byte in a game was 3 digits, (yeah I know not likely). But if you remove the extra 1 byte from every number,
    you've literally just removed a lot of data.

    That's just one part, the second part is combining the data, like instead of saving (h5,h5,h5,h5,h5,A1) I can just save it as (h5x4 A1) (well that's the readable version at least).

    But yeah, I'm getting progress guys. Not near finished yet, just wanted to keep you guys in the loop.

    EDIT Uploaded an Image:
    Example Image of the current lists (Might switch them to an Array later), more so just testing right now. And likewise, statics are for testing.
    StringInstructors.

    Pretty much how this will work is, there will be a loop somewhere, it will read the current set of bytes going through with the list up top, if it equals a set in the first List, it will then select the value and return from the bottom List. Then output the byte values for that return.
     
    Last edited by N1warhead, Oct 10, 2017
  15. Pleng

    Pleng Custom Title

    Member
    9
    Sep 14, 2011
    Thailand
    So basically you think can code a compression algorithm that is more efficient than 7zip in the specific case of SNES roms...?
     
  16. N1warhead
    OP

    N1warhead Member

    Newcomer
    1
    Oct 8, 2017
    United States
    Long story short: Yes.

    I mean yeah this system might could be used for other things, there's nothing inherently different with any other file, they are all just bytes of data.
    However, I am catering this towards games for the SNES specifically. This way it's a more controlled environment with the common data I anticipate.

    I'm certainly going to try. If for some reason I can't 100% figure it out, I'll still release the source code, perhaps someone else can knock it out. I just hate to not try, because I'm sure we all would like to fit more games on our Classics.

    I do have some experience creating my own Compression algorithms and such, so this isn't a first time thing for me.
    Mainly with image compression, however. It's not easy to compress a full fledged BMP file down to a few hundred bytes and re-store it with the same quality it had.
     
    DarthDub likes this.
  17. bailli

    bailli GBAtemp Regular

    Member
    6
    Oct 16, 2006
    Gambia, The
    Tinkering with stuff is always a good idea to learn something.
    But you should expect to get - in the very best scenario - only a marginal improvement over gzip...

    Also your opening post suggests you have no idea how most compression algorithms work - especially since you called the compression the SNES mini already uses 7zip instead of gzip.
     
  18. The Real Jdbye

    The Real Jdbye Always Remember 30/07/08

    Member
    19
    GBAtemp Patron
    The Real Jdbye is a Patron of GBAtemp and is helping us stay independent!

    Our Patreon
    Mar 17, 2010
    Norway
    Alola
    Good luck with this. Honestly it sounds a bit too good to be true. 7-zip went through a lot of work over a long period of time to get to where it is today and as it stands it has one of the best compression rates on the market, it might be hard to surpass it, even for a specific purpose such as this.
    The thing with general purpose compression is that it's designed to identify patterns in files and in that way tailor the compression to the specific files you're trying to compress, much in the same way as you're trying to do with .sfrom files.
    So I am kind of doubtful, but I'd love to be proven wrong.
    How much space does the SNES classic have anyway? The full USA SNES romset I have is 628 MB individually zipped, and with 7zip would likely be even smaller. There might be some stuff missing but it should be fairly complete. Would that not fit on the SNES classic?
     
    DarthDub, 0100100001001001 and Minox like this.
  19. Minox

    Minox Thanks for the fish

    Former Staff
    14
    Aug 27, 2007
    Japan
    Assuming you can truly achieve a better compression rate than 7zip I think this could be a great thing, but I think it's fair to say that there's still a lot of work that needs to be done before proving whether or not it can truly be more effective than 7zip.
     
  20. N1warhead
    OP

    N1warhead Member

    Newcomer
    1
    Oct 8, 2017
    United States
    Yeah I'm not going to lie and claim I know it all, anyone who does is a fool lol.
    I've been coding for quite a long time, and I'm always learning new things all the time.

    But about the 7zip stuff, that's just what I read that it used, I've been more a rar guy and haven't strayed to far from it.

    Oh man I totally get what you're saying.
    And what Hakchi told me, I have like 250 something megs of space. Which isn't much. (perhaps that was a bug?)
    But I'm going to give it a shot though.

    100% agreed. It will take time, I just wanted to let you guys know I was at least in the works on it. I'm just the kind of person that truly believes the future holds great things, and the problem with the future is, data is just getting bigger and bigger and without a new up to date compression systems, that will ultimately be holding us back. Yeah 7Zip and all that right now might be okay, but when downloads become petabytes there's going to need to be something better.

    ---------

    Now with this being said, just have a little faith in me guys, it's not the first time I've been down a dark dirty road at night and came out victorious :D
     
    DarthDub likes this.
Quick Reply
Draft saved Draft deleted
Loading...