[Request] I need a faster MD5 hash calculator (must beat 23 seconds for a 5GB file)

[Jayro]

I've currently switched from IgorWare Hasher to RapidCRC Unicode, and the speed difference is night and day; RapidCRC Unicode can calculate an MD5 hash in 23 seconds for a 5GB ISO file:

So far, I've found FastSum comes close in second place at 28 seconds for the same file:

But are there even faster MD5 hasher apps available? (Maybe some with GPU acceleration?)

All tests are done from an 8GB 2,200 MB/s DDR3 RAMdisk, so disk i/o shouldn't be a bottleneck.

 

[Whovian NineThreeSixNine]

I personally use http://code.kliu.org/hashcheck/ - I don't know if it would beat the 23 seconds, though, but it might.

 

[Monado_III]

I can get 15-17 seconds on a 6,780,092,416 byte file consistently using md5sum and my program below on linux, does it have to be a gui?

Spoiler

gcc -lssl -lcrypto -O3 -Wall -o file file.c

#include <stdio.h>
#include <openssl/md5.h>

int main(int argc, char **argv)
{
   unsigned char c[MD5_DIGEST_LENGTH];
   if(argc == 1)
       return 0;
   char *filename=argv[1];
   unsigned int i;
   FILE *inFile = fopen (filename, "rb");
   MD5_CTX mdContext;
   unsigned int bytes;
   unsigned char data[1024];
   if (inFile == NULL)
       return 0;
   MD5_Init (&mdContext);
   while ((bytes = fread (data, 1, 1024, inFile)) != 0)
      MD5_Update (&mdContext, data, bytes);

   MD5_Final (c,&mdContext);
   for(i = 0; i < MD5_DIGEST_LENGTH; i++)
      printf("%02x", c[i]);
   printf (" %s\n", filename);
   fclose (inFile);
   return 0;
}

I got it from here, I made some minor adjustments to make it faster (-O3) and let the user specify the file

$ time md5sum SX4E01.wbfs && time ./file_gcc SX4E01.wbfs && time ./file_clang SX4E01.wbfs
6c7939e99feba22f06d2f55d3f45d09d  SX4E01.wbfs

real    0m16.647s
user    0m13.597s
sys    0m1.956s
6c7939e99feba22f06d2f55d3f45d09d SX4E01.wbfs

real    0m15.956s
user    0m13.156s
sys    0m2.368s
6c7939e99feba22f06d2f55d3f45d09d SX4E01.wbfs

real    0m15.112s
user    0m12.617s
sys    0m2.150s

 

[Joom]

Why do people still use MD5? SHA-256 is much more efficient and accurate.

 

[Jayro]

Why do people still use MD5? SHA-256 is much more efficient and accurate.

I'm just benchmarking, otherwise I typically do SHA-256 also. I just need speed now.

 

[Joom]

I'm just benchmarking, otherwise I typically do SHA-256 also. I just need speed now.

Fair enough.