Yeah, I just did a bunch of testing and the forwarder works just fine with the boot.dol on the USB. Not sure where I got the SD-only idea from...
Anyway, like I just said, I did a bunch of testing and I've greatly narrowed down the issue. I did discover some other unrelated issues in USB Loader GX, but I'd rather solve one problem before working on a different one.
First off, I just want to say that I timed the "1 minute" load time and it hangs at "Initializing USB device..." for exactly 62 seconds.
So rather than telling the whole process, I'll just say the result - the problem is caused by my 16GB Corsair Voyager flash drive. The SD card is fine as confirmed when I used my external HDD and/or a 1GB flash drive, and my 62-second load time still happened when I put the boot.dol on my 16GB flash drive and left my SD card ejected.
However, during my testing, I actually DID find a pattern to whether or not it loads in 10 seconds or in over a minute. If I load a Wii game from USB Loader GX, the next time I load USB Loader GX it'll take over a minute to load. If I load USB Loader GX but then just turn off the system, go to the homebrew channel, or go to the Wii menu without loading a game, the next time I try to load USB Loader GX it will take over a minute.
HOWEVER! In USB Loader GX if I load a GameCube game via Devolution, the next time I load USB Loader GX it will load in only 10 seconds.
So uh, any idea what the crap is going on here? For reference the 16GB flash drive is just single FAT32 partition formatted with a 64K allocation unit size.