USB Loader GX is slow to list the games if you have a lot (more than 1000) to display at the same time.
it's usually slow if you enable EmuNAND, it has to scan all your emuNAND folders to generate the game list.
One solution is to disable the game type you don't except to play soon (if you are in a mood to play gamecube, disable Wii and emuNAND display).
If you don't have that much games (100-300) then, maybe the issue is your HDD itself.
You also mentioned
nintendont is slow to load the game list, which is listing only gamecube games, so I think something is wrong with your HDD.
Either your HDD is slow
Either your FAT32 file table is having synchronization issue (USBLoaderGX has an option to fix that, called "sync FAT32"), but it was an issue only in case you used a very old homebrew to write a file on that drive, using a very old FAT library.
either your FAT32 partition is very big (4TB single partition?), and parsing the partition's file allocation table (the FAT actually) is taking a long time. It's also an issue on windows and the drive can take a moment before being available as a drive letter.