So to put this issue to rest once and for all, I did 10 tests of 3 SD, 3 times each with different formats, here are the results.
All performed on a New 3DS XL, with the official copy of WarioWare, Inc. Mega Microgame$!. Cards formatted with Minitool Partition Wizard each time.
---
4GB Toshiba SDHC @ FAT32 16KB Cluster Size
0/10 times the issue was present.
4GB Toshiba SDHC @ FAT32 32KB Cluster Size
0/10 times the issue was present.
4GB Toshiba SDHC @ FAT32 64KB Cluster Size
1/10 times the issue was present.
32GB SanDisk SDHC @ FAT32 16KB Cluster Size
0/10 times the issue was present.
32GB SanDisk SDHC @ FAT32 32KB Cluster Size
0/10 times the issue was present.
32GB SanDisk SDHC @ FAT32 64KB Cluster Size
1/10 times the issue was present.
64GB Samsung EVO SDXC @ FAT32 16KB Cluster Size
0/10 times the issue was present.
64GB Samsung EVO SDXC @ FAT32 32KB Cluster Size
0/10 times the issue was present.
64GB Samsung EVO SDXC @ FAT32 64KB Cluster Size
2/10 times the issue was present.
---
Seems clusters above 32KB cause the issue. So riding the line by being on 32 seems to be the best option, since it seems going lower makes the console significantly slower, especially on the largest card sizes.
In addition, it seems it may be more common on larger card sizes, possibly related to larger cards being slower?
Edit: fixed a couple typos I made.