What compression algorithm do you use commonly?
- Thread starter dAVID_
- Start date
- Views 3,345
- Replies 40
Something's wrong with the one you got, since I'm on 5.3 and still get RAR5 and have been ever since version 4.2...
Strange. By the way, I downloaded FreeArc, and used max compression on a folder containing Terraria save data. It beat LZMA2 on Ultra compression in 7z format. Wow!
You can install it with Cygwin.
Edit:
32bit: http://ring.u-toyama.ac.jp/archives/pc/gnu-win32/x86/release/lrzip/lrzip-0.621-1.tar.xz
64bit: http://ring.u-toyama.ac.jp/archives/pc/gnu-win32/x86_64/release/lrzip/lrzip-0.621-1.tar.xz
Last edited by Joom,
I usually don't compress stuff anymore. Most modern containers I work with are already optimized.
In my opinion "Compressing" really comes in handy just to distribute stuff and fool people.
In my opinion "Compressing" really comes in handy just to distribute stuff and fool people.
I just use RAR (not RAR5) because it's what I have and what I'm used to. I would use RAR5, but it's not supported by everything yet. The files I share usually aren't huge, so I'm not too worried about maximizing the compression ratio.
Your best bet is, if you're using Windows 10, to use the built-in Windows Compact Tool. Here are some results for popular games https://www.reddit.com/r/Windows10/..._the_windows_compact/?st=J8THCDU8&sh=b22efea6.Even already compressed files should be able to be compressed again to a certain extend. I mainly want to compress my game library (the setup files) so I can store them away on one drive rather than having 2 drives totaling 3TBsAnd, if possible, to compress some of my anime library (24TBs)...
As for your statement "Even already compressed files should be able to be compressed again to a certain extend." that only holds true if "to a certain extent" can mean "not at all by any means". Not all data can be compressed again, and depending on how you compressed it the first time, recompressing it can lead to an even larger file size.
Just because I said what I said doesn't mean I literally meant recompressing the compressed files, I meant compressing the originals using a better algorithm. An already compressed file can't lose more than 1% of it's size tops regardless of method, as it can't build it's dictionary properly for all addresses.
You don't have to bother with Bash. PATH variables exist amongst all operating systems. Your PATH defines where the system can execute binaries without defining the specific path to the file. However, if you don't feel like messing with all of that, just place the EXE on the root of your C drive, open a CMD, and type the name of the EXE (including the extension). It should run. Also, tab completion exists; something people tend to not know about. This means that you can type the first few characters of a command or file name, hit tab, and the prompt will automatically complete it for you. Just a protip to make your life easier.
Yeah, you're better off installing it with Cygwin. Windows isn't impervious to Dependency Hell.
What's the best method for storing files longterm reliably? I used to zip and compress everything with winrar until a lot of my games archives got corrupted
Sent from my Phone using Tapatalk
Sent from my Phone using Tapatalk
It depends upon the type of files, how big the files are, how long term is long term, what kind of access speeds you need*, what kind of security you need and how much you are willing to invest.
*stick a USB drive in your bank vault and you are good, at least until the need crops up for something on a Friday night and the bank does not open until Monday morning sort of thing.
A few bytes in some files is nothing -- so a couple of pixels in a picture or a few milliseconds of audio are not what they once were? Meh, especially if I can just download it again. Or it could be everything. Compression makes this worse as any corruption runs the risk of being magnified across the whole file.
A baseline for long term that most would use is how long you need to store your taxes for in places around the world. You are rarely going to get much more than a decade and almost never greater than 2 decades for that. Given I still play games from the 70s though...
A rule in data backup is data that does not exist in more than one (preferably offline*) location does not exist. Which is to say your USB drive next to your computer is fine but if your house floods, burns down or I break in I am certainly taking that nice couple of hundred gig USB drive with me too. Depending upon how extreme you want to get then this can get even worse (for personal things then it is usually not so bad, for a big business then where storing a backup USB at a house across town is more than enough for most if a big disaster hits and they lose everything it is going to be bad) but let us not go there.
*you might have the best RAID setup in the world, and copies in multiple folders but my piece of ransomware hits your system and encrypts the lot and you are hosed. Many found this out in the little ransomware fad a while back. Said ransomware can't leap out of a system and break your burned CD in half... though maybe for version 2.0 that could be an option.
Magnetic
Optical
Flash
Online
What about parity files?
What are your conditions? Hot and humid is worse for most things than dry and moderate. As my basement is dry and moderate all year round for free it is easy for me, however if I were in somewhere hot and wet (your location says CA and having been there myself I have seen much heat and humidity there which would mean some kind of sealing or air conditioning at times.
Back to the types
Magnetic.
Two main types being tape and hard drive. Tape is slow to read, annoying to store and somewhat cumbersome, however there is a reason lots of people/companies with serious data storage needs will still use it.
Hard drives are mechanical devices and subject to all sorts of things. I have 30 year old ones going strong, and I have had ones fail after months of light use. Unpowered storage is also tricky -- sometimes the bearing grease becomes more like glue after a few years, however dodgy bearings are replaced day in and day out by people without any real computer skills so it is possible but that starts getting expensive very fast.
Optical
A burned CD is just a piece of dye a laser burned. Their long term reliability is very questionable. While I have certainly still got working ones from the (late) 90s I equally have theoretically good ink that went on DVDs I burned some 5 years earlier.
Flash
The long term/data rot problem of flash is an interesting one and one I am never quite sure about -- much of it seems like fear spread by big spinning rust/big tape, others there is a physical basis for.
Online
Most things like dropbox will say while we make every effort to store your data we are not responsible for corruption. Hard drives do fail, power surges do happen and so on and so on. You want a big boy one that does it really truly properly with multiple redundant arrays of systems and you will pay handsomely for it.
At the same time some of those are a bit touchy about warez for whatever reason and you may have to encrypt things, which in many cases is like compression in that failures of the base file can have interesting effects.
It should also be noted most of these online storage companies last maybe 7-12 years on a good run. Maybe you will pick the winner, and maybe said winner won't become a bloated mess that gets taken out by some new hotness in a few years, or suddenly decide they are too big to be free.
Still for small files a few emails sent around will do well. By similar token it will suck but I have books that have been around for centuries, you can do such a thing too and type it all back out (or OCR it as the case may be) if you wanted. Wouldn't want to do that for a game archive in the terabytes, a few megabytes of data though is far better.
Parity files then.
You may have encountered them if you used usenet or certainly types of RAID array, or maybe you have not.
The maths behind it all is fascinating but the short version is you can make files that not only check the data is good but can recreate broken segments if it is not owing to the way files are just maths. More parity files, more damage is able to be fixed, more storage is needed for it all, you can go beyond 100% file recovery as well if you want to account for the parity files themselves being broken.
Good luck.
My files I intend to try to keep forever. Only needing to access them once every great while so read/write specs don't matter much to me. I have entire ROM collections, every major game release for all systems so that's a couple gigs each psp game for example, to 30gbs for ps3 games, to roughly 60 for ps4 dumps. And then theres homebrew collections for each console which range from a few mbs to a hundred mbs or so. So I have a range of file sizes to work with.
For my most precious files I have 2 8tb 2-drive raid enclosures. And to save money I started to just buy 2 8tb usb drives then just copy everything manually to both drives so every drive still has a backup.
I've experienced data loss before and it was devastating for me so I want to be as safe as possible. Due to the nature of my files I can't just re-download them whenever. Not easily at least. I spend hours and hours collecting ROMSets and such which won't be on the internet forever and in some cases they get nuked fairly quickly so a lot of my data is pretty rare like my undub and rom hacks collections which get taken down or the file hoster dies off for example.
I did some reading on parity files and discovered Parchive for windows to create par2 files for zip or 7zip archives. Is there an app that can batch create zip and par2 files say for each separate folder? If not I'd have to do that for over thousands of folders. And what about keeping the par2 files sorted with each archive for easy recovery in case of corruption. Can you combine the zip and par2 files? Preferably in batch mode as well? Something that's user friendly for a noob.
Thanks for all the knowledge btw
Sent from my Phone using Tapatalk
Similar threads
- Question
- No one is chatting at the moment.
-
-
-
-
@
Skelletonike:
Pondered about the switch, but had bought the pokemon one not long ago and I don't really need two switches (gave my previous one to a cousin). -
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
