What compression algorithm do you use commonly?

Joom

 ❤❤❤
Member
Joined
Jan 8, 2016
Messages
6,024
Trophies
1
Location
US
Website
mogbox.net
XP
5,940
Country
United States
RAR5 beats LZMA. It's unfortunate it's still not the standard, though. There are other algorithms with much higher ratios, though. I can't remember the main one warez uploaders use sometimes (it's unpopular since it can take days to compress a release), but I've seen 2+ GB files compressed down to just a couple megabytes.

Edit: FreeArc is one that can achieve this, but I don't know the algorithm used.
 
Last edited by Joom,

ThoD

GBATemp Addict (apparently), but more like "bored"
Member
Joined
Sep 8, 2017
Messages
3,631
Trophies
1
Age
26
XP
3,017
Country
Greece
RAR5 beats LZMA. It's unfortunate it's still not the standard, though. There are other algorithms with much higher ratios, though. I can't remember the main one warez uploaders use sometimes (it's unpopular since it can take days to compress a release), but I've seen 2+ GB files compressed down to just a couple megabytes.
Curious about that too, considering how I once found a site with PS2 ISOs that were 1.5-4GBs uncompressed but hardly 5MBs compressed... Would certainly make uploading a lot of stuff easier and would be great for storing backups!

As for what I use, I just go with the default RAR5 settings as I can't be bothered to take too long to compress most stuff so I don't change dictionary sizes and whatnot at all.
 

Joom

 ❤❤❤
Member
Joined
Jan 8, 2016
Messages
6,024
Trophies
1
Location
US
Website
mogbox.net
XP
5,940
Country
United States
@ThoD, FreeArc is a tool that can achieve it. There's another out there that I'm trying to remember that uses a proprietary archival format, but I can't remember it (Monkey something or other). Keep in mind that only raw data can be compressed. Compressed media files and other existing archives, for example, can't be recompressed.
 
  • Like
Reactions: ThoD

ThoD

GBATemp Addict (apparently), but more like "bored"
Member
Joined
Sep 8, 2017
Messages
3,631
Trophies
1
Age
26
XP
3,017
Country
Greece
@ThoD, FreeArc is a tool that can achieve it. There's another out there that I'm trying to remember that uses a proprietary archival format, but I can't remember it (Monkey something or other). Keep in mind that only raw data can be compressed. Compressed media files and other existing archives, for example, can't be recompressed.
Even already compressed files should be able to be compressed again to a certain extend. I mainly want to compress my game library (the setup files) so I can store them away on one drive rather than having 2 drives totaling 3TBs:P And, if possible, to compress some of my anime library (24TBs)...
 

Joom

 ❤❤❤
Member
Joined
Jan 8, 2016
Messages
6,024
Trophies
1
Location
US
Website
mogbox.net
XP
5,940
Country
United States
Already compressed files technically can be recompressed, but it really won't be that much of a difference depending on the compression algorithm already used on them. Lossless compression especially really can't be compressed anymore than it already is (FLAC, PNG, etc. are examples of files that use lossless compression). Game ISOs can be compressed down to such small sizes because they tend to be filled with padding files, and raw data that isn't compressed.
 
Last edited by Joom,

ThoD

GBATemp Addict (apparently), but more like "bored"
Member
Joined
Sep 8, 2017
Messages
3,631
Trophies
1
Age
26
XP
3,017
Country
Greece
Already compressed files technically can be recompressed, but it really won't be that much of a difference depending on the compression algorithm already used on them. Lossless compression especially really can't be compressed anymore than it already is (FLAC, PNG, etc. are examples of files that use lossless compression). Game ISOs can be compressed down to such small sizes because they tend to be filled with padding files, and raw data that isn't compressed.
Most of my video files (anime, movies, etc.) are in either AVI or MP4 format, both of which aren't that much compressed and if I convert them to something like Matroska I could make them up to 12 times smaller with a tiny loss in quality, so compressing to at least half size with a good algorithm shouldn't be too hard to do, I just need to find the one that will do the job though. I used to go with gzip's algorithm, but it's gotten outdated at this point so I'm looking for best one out there.
 

InsaneNutter

Well-Known Member
Member
Joined
Dec 26, 2007
Messages
958
Trophies
1
Age
36
Location
Yorkshire, UK
Website
digiex.net
XP
2,496
Country
Keep in mind with ISO's sometimes they are just padded out with garbage data to allow the console better read times to where the data actually is on the disc. That's why you can sometimes save multiple gigs compressing an ISO.

You would never achieve something like that compressing video, the best you can really do there is re-encode video in a more efficient format such as x265, however you would likely loose quality then anyway depending on your settings.
 

ThoD

GBATemp Addict (apparently), but more like "bored"
Member
Joined
Sep 8, 2017
Messages
3,631
Trophies
1
Age
26
XP
3,017
Country
Greece
Keep in mind with ISO's sometimes they are just padded out with garbage data to allow the console better read times to where the data actually is on the disc. That's why you can sometimes save multiple gigs compressing an ISO.

You would never achieve something like that compressing video, the best you can really do there is re-encode video in a more efficient format such as x265, however you would likely loose quality then anyway depending on your settings.
I already have the video files compressed to 74% of the original and that's with a bad algorithm, so with a good one it should be possible to hit like 60%. I don't even care if the archive ends up being normal or solid, as I will only be taking files out, not adding more in.
 

Joom

 ❤❤❤
Member
Joined
Jan 8, 2016
Messages
6,024
Trophies
1
Location
US
Website
mogbox.net
XP
5,940
Country
United States
Most of my video files (anime, movies, etc.) are in either AVI or MP4 format, both of which aren't that much compressed and if I convert them to something like Matroska I could make them up to 12 times smaller with a tiny loss in quality, so compressing to at least half size with a good algorithm shouldn't be too hard to do, I just need to find the one that will do the job though. I used to go with gzip's algorithm, but it's gotten outdated at this point so I'm looking for best one out there.
Keep in mind that video codecs are just containers. They can be either raw, or compressed. Typically, MP4, AVI, and even Matroska are already compressed for release. The codecs themselves are just containers that house the actual video data. The data within is what is or isn't compressed. So if you were to convert your existing videos to MKV, you risk a major loss in quality.

--------------------- MERGED ---------------------------

You would never achieve something like that compressing video, the best you can really do there is re-encode video in a more efficient format such as x265, however you would likely loose quality then anyway depending on your settings.
Quality loss would happen regardless. In the same sense that you can't convert an MP3 to FLAC and expect an increase in quality, you can't re-encode x264 (for example) to x265 and expect better quality. There's actually a major loss in quality.
 
Last edited by Joom,

ThoD

GBATemp Addict (apparently), but more like "bored"
Member
Joined
Sep 8, 2017
Messages
3,631
Trophies
1
Age
26
XP
3,017
Country
Greece
Keep in mind that video codecs are just containers. They can be either raw, or compressed. Typically, MP4, AVI, and even Matroska are already compressed for release. The codecs themselves are just containers that house the actual video data. The data within is what is or isn't compressed.

--------------------- MERGED ---------------------------


Quality loss would happen regardless. In the same sense that you can't convert an MP3 to FLAC and expect an increase in quality, you can't re-encode x264 (for example) to x265 and expect better quality. There's actually a major loss in quality.
I know, but the algorithm I used was bad, but still complicated one that stripped the files of containers/extensions, compressed, then reassigned them.
 

Joom

 ❤❤❤
Member
Joined
Jan 8, 2016
Messages
6,024
Trophies
1
Location
US
Website
mogbox.net
XP
5,940
Country
United States
I know, but the algorithm I used was bad, but still complicated one that stripped the files of containers/extensions, compressed, then reassigned them.
You'd still be much better off using the source data for conversion rather than the already compressed files. You essentially cut the quality in half each time you recompress an already compressed file. This is why scene groups release ISO dumps of DVDs and Blu-ray discs internally on top sites; it's all for the sake of creating "mainstream" releases.
 
Last edited by Joom,

Jayro

MediCat USB Dev
Developer
Joined
Jul 23, 2012
Messages
11,434
Trophies
4
Location
WA State
Website
ko-fi.com
XP
13,567
Country
United States
LZMA2, still don't have any better open-source contender in the ring. I just hate how painfully slow it is, and that it doesn't take advantage of I/O bandwidth, nor processor speed. My CPU could be pegged at 100% at 800MHz or at 3.4GHz, and it would finish 7zipping a 11GB ISO image in the same exact time.
 

ThoD

GBATemp Addict (apparently), but more like "bored"
Member
Joined
Sep 8, 2017
Messages
3,631
Trophies
1
Age
26
XP
3,017
Country
Greece
LZMA2, still don't have any better open-source contender in the ring. I just hate how painfully slow it is, and that it doesn't take advantage of I/O bandwidth, nor processor speed. My CPU could be pegged at 100% at 800MHz or at 3.4GHz, and it would finish 7zipping a 11GB ISO image in the same exact time.
LZMA2 is actually dependent on RAM and disk write/read speed, so CPU doesn't matter. Faster RAM helps cut down time by a lot and so does having some excess since it can spike for no reason.
 

Jayro

MediCat USB Dev
Developer
Joined
Jul 23, 2012
Messages
11,434
Trophies
4
Location
WA State
Website
ko-fi.com
XP
13,567
Country
United States
LZMA2 is actually dependent on RAM and disk write/read speed, so CPU doesn't matter. Faster RAM helps cut down time by a lot and so does having some excess since it can spike for no reason.
I've run it from my HDD, my SSD, and my RAMdisk, all with marginal results of a few seconds apart. Nothing substantial enough to matter.
 

ThoD

GBATemp Addict (apparently), but more like "bored"
Member
Joined
Sep 8, 2017
Messages
3,631
Trophies
1
Age
26
XP
3,017
Country
Greece
I've run it from my HDD, my SSD, and my RAMdisk, all with marginal results of a few seconds apart. Nothing substantial enough to matter.
Then you just hit the limit of how fast you can do it with that method, too bad it's still slow af:P Fastest possible method with optimal compression would require something like CUDA C to use the cells of the GPU for acceleration along with the GPU and 2600+MHz RAM, but that would never go well as non-NVidia cards would never be able to process the files...
 

FAST6191

Techromancer
Editorial Team
Joined
Nov 21, 2005
Messages
35,822
Trophies
3
Website
trastindustries.com
XP
26,155
Country
United Kingdom
Most of my video files (anime, movies, etc.) are in either AVI or MP4 format, both of which aren't that much compressed and if I convert them to something like Matroska I could make them up to 12 times smaller with a tiny loss in quality
MP4, AVI and MKV are container standards. The only thing they have to do with compression is what compression standards they commonly support (AVI does very few when all is said and done, fewer still if you go official). Most people compress video in a lossy format as well which is not great for the lossless stuff I imagine this thread started with the intention of finding.

Back on topic.
Compression wise most things I do are a token effort and I normally care more about bundling files. Commonly said files are similar as well which makes cross file compression (like what you get in 7zip as opposed to regular zip). Nobody will thank for a 20 minute decompression of a few meg word document either -- bandwidth and storage is plentiful and cheap, time spent futzing with compression, much less when the recipients are not technically minded, is worse than having a download in the background.
If I am somehow working with a massive file I will be there in person or remote controlling the machine once the base file is sent.
 

You may also like...

General chit-chat
Help Users
    The Real Jdbye @ The Real Jdbye: https://media.tenor.com/O3B3P07kGdoAAAAC/yamn-cdi_link.gif