What compression algorithm do you use commonly?

[dAVID_]

Mine is LZMA for the good compressed sizes, which were better compared to WinRAR.

 

[Joom]

RAR5 beats LZMA. It's unfortunate it's still not the standard, though. There are other algorithms with much higher ratios, though. I can't remember the main one warez uploaders use sometimes (it's unpopular since it can take days to compress a release), but I've seen 2+ GB files compressed down to just a couple megabytes.

Edit: FreeArc is one that can achieve this, but I don't know the algorithm used.

 

[ThoD]

RAR5 beats LZMA. It's unfortunate it's still not the standard, though. There are other algorithms with much higher ratios, though. I can't remember the main one warez uploaders use sometimes (it's unpopular since it can take days to compress a release), but I've seen 2+ GB files compressed down to just a couple megabytes.

Curious about that too, considering how I once found a site with PS2 ISOs that were 1.5-4GBs uncompressed but hardly 5MBs compressed... Would certainly make uploading a lot of stuff easier and would be great for storing backups!

As for what I use, I just go with the default RAR5 settings as I can't be bothered to take too long to compress most stuff so I don't change dictionary sizes and whatnot at all.

 

[Joom]

@ThoD, FreeArc is a tool that can achieve it. There's another out there that I'm trying to remember that uses a proprietary archival format, but I can't remember it (Monkey something or other). Keep in mind that only raw data can be compressed. Compressed media files and other existing archives, for example, can't be recompressed.

 

[ThoD]

@ThoD, FreeArc is a tool that can achieve it. There's another out there that I'm trying to remember that uses a proprietary archival format, but I can't remember it (Monkey something or other). Keep in mind that only raw data can be compressed. Compressed media files and other existing archives, for example, can't be recompressed.

Even already compressed files should be able to be compressed again to a certain extend. I mainly want to compress my game library (the setup files) so I can store them away on one drive rather than having 2 drives totaling 3TBs And, if possible, to compress some of my anime library (24TBs)...

 

[Joom]

Already compressed files technically can be recompressed, but it really won't be that much of a difference depending on the compression algorithm already used on them. Lossless compression especially really can't be compressed anymore than it already is (FLAC, PNG, etc. are examples of files that use lossless compression). Game ISOs can be compressed down to such small sizes because they tend to be filled with padding files, and raw data that isn't compressed.

 

[ThoD]

Already compressed files technically can be recompressed, but it really won't be that much of a difference depending on the compression algorithm already used on them. Lossless compression especially really can't be compressed anymore than it already is (FLAC, PNG, etc. are examples of files that use lossless compression). Game ISOs can be compressed down to such small sizes because they tend to be filled with padding files, and raw data that isn't compressed.

Most of my video files (anime, movies, etc.) are in either AVI or MP4 format, both of which aren't that much compressed and if I convert them to something like Matroska I could make them up to 12 times smaller with a tiny loss in quality, so compressing to at least half size with a good algorithm shouldn't be too hard to do, I just need to find the one that will do the job though. I used to go with gzip's algorithm, but it's gotten outdated at this point so I'm looking for best one out there.

 

[InsaneNutter]

Keep in mind with ISO's sometimes they are just padded out with garbage data to allow the console better read times to where the data actually is on the disc. That's why you can sometimes save multiple gigs compressing an ISO.

You would never achieve something like that compressing video, the best you can really do there is re-encode video in a more efficient format such as x265, however you would likely loose quality then anyway depending on your settings.

 

[ThoD]

Keep in mind with ISO's sometimes they are just padded out with garbage data to allow the console better read times to where the data actually is on the disc. That's why you can sometimes save multiple gigs compressing an ISO.

You would never achieve something like that compressing video, the best you can really do there is re-encode video in a more efficient format such as x265, however you would likely loose quality then anyway depending on your settings.

I already have the video files compressed to 74% of the original and that's with a bad algorithm, so with a good one it should be possible to hit like 60%. I don't even care if the archive ends up being normal or solid, as I will only be taking files out, not adding more in.

 

[StackMasher]

I use lrzip to backup the games on my SSD, it compressed saints row 4 (20.5GB) to 7.1GB

 

[Joom]

Most of my video files (anime, movies, etc.) are in either AVI or MP4 format, both of which aren't that much compressed and if I convert them to something like Matroska I could make them up to 12 times smaller with a tiny loss in quality, so compressing to at least half size with a good algorithm shouldn't be too hard to do, I just need to find the one that will do the job though. I used to go with gzip's algorithm, but it's gotten outdated at this point so I'm looking for best one out there.

Keep in mind that video codecs are just containers. They can be either raw, or compressed. Typically, MP4, AVI, and even Matroska are already compressed for release. The codecs themselves are just containers that house the actual video data. The data within is what is or isn't compressed. So if you were to convert your existing videos to MKV, you risk a major loss in quality.

--------------------- MERGED ---------------------------

You would never achieve something like that compressing video, the best you can really do there is re-encode video in a more efficient format such as x265, however you would likely loose quality then anyway depending on your settings.

Quality loss would happen regardless. In the same sense that you can't convert an MP3 to FLAC and expect an increase in quality, you can't re-encode x264 (for example) to x265 and expect better quality. There's actually a major loss in quality.

 

[ThoD]

Keep in mind that video codecs are just containers. They can be either raw, or compressed. Typically, MP4, AVI, and even Matroska are already compressed for release. The codecs themselves are just containers that house the actual video data. The data within is what is or isn't compressed.

--------------------- MERGED ---------------------------


Quality loss would happen regardless. In the same sense that you can't convert an MP3 to FLAC and expect an increase in quality, you can't re-encode x264 (for example) to x265 and expect better quality. There's actually a major loss in quality.

I know, but the algorithm I used was bad, but still complicated one that stripped the files of containers/extensions, compressed, then reassigned them.

 

[Joom]

I know, but the algorithm I used was bad, but still complicated one that stripped the files of containers/extensions, compressed, then reassigned them.

You'd still be much better off using the source data for conversion rather than the already compressed files. You essentially cut the quality in half each time you recompress an already compressed file. This is why scene groups release ISO dumps of DVDs and Blu-ray discs internally on top sites; it's all for the sake of creating "mainstream" releases.

 

[Jayro]

LZMA2, still don't have any better open-source contender in the ring. I just hate how painfully slow it is, and that it doesn't take advantage of I/O bandwidth, nor processor speed. My CPU could be pegged at 100% at 800MHz or at 3.4GHz, and it would finish 7zipping a 11GB ISO image in the same exact time.

 

[ThoD]

LZMA2, still don't have any better open-source contender in the ring. I just hate how painfully slow it is, and that it doesn't take advantage of I/O bandwidth, nor processor speed. My CPU could be pegged at 100% at 800MHz or at 3.4GHz, and it would finish 7zipping a 11GB ISO image in the same exact time.

LZMA2 is actually dependent on RAM and disk write/read speed, so CPU doesn't matter. Faster RAM helps cut down time by a lot and so does having some excess since it can spike for no reason.

 

[BvanBart]

That is a great compression ratio!

 

[Jayro]

LZMA2 is actually dependent on RAM and disk write/read speed, so CPU doesn't matter. Faster RAM helps cut down time by a lot and so does having some excess since it can spike for no reason.

I've run it from my HDD, my SSD, and my RAMdisk, all with marginal results of a few seconds apart. Nothing substantial enough to matter.

 

[ThoD]

I've run it from my HDD, my SSD, and my RAMdisk, all with marginal results of a few seconds apart. Nothing substantial enough to matter.

Then you just hit the limit of how fast you can do it with that method, too bad it's still slow af Fastest possible method with optimal compression would require something like CUDA C to use the cells of the GPU for acceleration along with the GPU and 2600+MHz RAM, but that would never go well as non-NVidia cards would never be able to process the files...

 

[FAST6191]

Most of my video files (anime, movies, etc.) are in either AVI or MP4 format, both of which aren't that much compressed and if I convert them to something like Matroska I could make them up to 12 times smaller with a tiny loss in quality

MP4, AVI and MKV are container standards. The only thing they have to do with compression is what compression standards they commonly support (AVI does very few when all is said and done, fewer still if you go official). Most people compress video in a lossy format as well which is not great for the lossless stuff I imagine this thread started with the intention of finding.

Back on topic.
Compression wise most things I do are a token effort and I normally care more about bundling files. Commonly said files are similar as well which makes cross file compression (like what you get in 7zip as opposed to regular zip). Nobody will thank for a 20 minute decompression of a few meg word document either -- bandwidth and storage is plentiful and cheap, time spent futzing with compression, much less when the recipients are not technically minded, is worse than having a download in the background.
If I am somehow working with a massive file I will be there in person or remote controlling the machine once the base file is sent.

 

[DinohScene]

I just zip shit together if needed.