In my case it always load in about 8 seconds. Under WiiU it takes more time because it checks both USB ports (don't know why), even when I specifically tell it just check port0.
I'm only using it on a Wii, and it's only on a single Wii system too. Also I hardly ever change what is on my USB drive, maybe once a month at most, and yet it still seems to do both the "8 seconds" or the "1 minute".
Even weirder, after discovering that there's some fragmentation on my USB drive, I decided to defragment it. Once it was completely defragmented, USB Loader GX would always load in, as you say it, about 8 seconds. While I was at it, I decided to defragment my SD card as well, but once I did that USB Loader GX started doing the random 8 seconds or 1 minute load-time again...
And FYI, I know that defragmenting flash memory isn't a good idea. So instead of actually doing a defrag, I backed up all the files to my computer, reformatted the drive, and then copied the files back. Such a method results in considerably less write cycles and is much faster overall than defragmenting on slower flash memory.
Also a heads up, my USB drive is a 16GB flash drive. I know that defragging isn't useful on flash memory performance-wise, but I thought that maybe it'd make a difference for the loading of USB Loader GX (which it seemed to). But I defragged my SD card because custom BRSTM files MUST be defragged, otherwise you'll get horrible static.