You just need to make backup and restore the emunand with emunand tool on the PC, that won't affect nothing and take several minutes.
Also filling the rest the card free space with, say, copies of emunand.bin will be also quick.
Both will eliminate possible emunand and FAT partition problematic sectors which could cause the delays and underruns.
As you said it is always happens in approximately the same place, so this is possible. With the lower speed it still could be read with no issues, when emuNAND booting and working those sectors could be rarely used, and when accessing from the card reader, all underruns are handled properly, so you don't see any issues.
I know, we must handle those too in the rxTools, but no solution for now.