I have a sneaking suspicion that when it has a playlist with the core listed, or core info files that give it the path of the core, it goes "oh, I need to switch to a new core to play this!" even if the currently loaded core is perfectly capable.
Switching cores is a somewhat slow affair, since it unloads the current core+RetroArch and tells HBL to load the new core+RetroArch from scratch. The new RA has to load up the menu assets again, which is the truly sluggish wiiu thing.
I wrote a little hack for whatever the current git master is that logs out when files are opened and closed with timestamps -
check it out! there's two logs in there - the first one where all the messages are prefixed with [fs-cafe] is using the native Cafe OS filesystem drivers, like on a console with no CFW. [fs-lfat] uses iosuhax (included in Mocha, stripped-down version in CBHC) and libfat. Keep in mind when looking at the timestamps that the logging itself likely sucks up several milliseconds each line, since it's networking, and yeah. Cafe native is pushing a 20s boot!