some devices have compatibility issues, and flash drives are less compatible than Hard Drives.
it's possible that your USB3.0 device will never work, but there are things you can try.
what I understand, you are not using SD card at all, right? everything is on your USB?
Are you launching the loader from the homebrew channel, or from a USBLoaderGX channel you installed to your system menu?
using the homebrew channel to list homebrew on USB can usually force a proper drive init, making it work in the loader.
if you already tried launching from HBC on USB, then let's try something else.
launch the loader (with the USB2.0 so it doesn't freeze) and go to the settings > loader settings.
Set the "loader's IOS" to 58.
exit, and copy these two files to your bigger flash drive:
/apps/usbloader_gx/GXGlobal.cfg
/apps/usbloader_gx/meta.xml
try to launch it now. It should use IOS58 to boot the loader and use that "driver" to mount the USB.
or if you don't want to do the setting change on one drive and copy to the other, you can try to edit the meta.xml directly on the non working drive.
you need to edit the meta.xml in a text editor to delete the 2 lines saying "delete to use arguments", then replace --ios=250 with --ios=58