Yes, very normal. As I ripped more of my games to an attached hard drive or to an SD, the load times would increase for USB Loader GX to complete its scan after starting. I can tell you that I tried several different configurations and have witnessed others USB Loader GX configs in person and it never goes beyond 30-60 seconds even in extreme cases. It's a justified delay, since the loader has to (briefly) make sure that everything is still there that is supposed to be there and it has to load a chunk of the metadata for the titles and stuff you initially see all at once.
You'll still see roughly the same load time with any configuration, but having everything on a proper external drive is best. I tried various combinations to mix config files or metadata on the SD Card and all the titles on the hard drive and it doesn't help at all. Keeping everything (for vWii) on one external hard drive is the quickest. SD interface on the WiiU is extremely slow, it doesn't matter how fast of a card you throw at it.
Single partition, FAT32, 32kB Cluster Size, USB 2.0 or greater, up to 4TB works fine. 4kB sectors are best.
USB flash drive (thumb drive, key, whatever you prefer to call it) are much less ideal.
SD cards are least ideal.
For Seagate, you use Disc Wizard to set the drive to 4kB sectors.
For Western Digital you use WD Quick Formatter to set it to 4kB sectors.
If you can't find Disc Wizard (or an equivalent tool made by Seagate) for MacOS, run a Windows VM in Virtualbox, Parallels, Bootcamp, VMWare Player, etc.
See here:
https://gbatemp.net/threads/usb-loader-gx.149922/page-1107#post-6429740
Sorry about the pictures missing. Host must have deleted them and I can't find the originals.