The cause of the troubles does seem to be data not being read fast enough from the card.
I dropped some of the audio from the Japanese rom in an attempt to fit it on my old EZ2 and found out that it did not crash on my EZ4. You can do the same by pulling apart the rom with NDSTool and deleting a portion of the end of the soundfile (called sound_data.sdat) and then rebuilding, I have a list of offsets for the files you can use on my site:
http://www.4shared.com/file/7846194/59a6fbf/PORsound.html
Later I repointed the soundfile to dodge the massive wave files at the end of the soundfile (most of the other sounds were smaller than my cards cluster size) which made it far more stable again but did not sacrifice too much sound.
Interesting stuff. I didn't know you could disassemble roms like that. BTW, if you want to look at the C

OR file list + offsets, open the XML he linked to in Excel. It'll give you a nice row/col format.
Here's a sample:
.........................offset...size.....size(dec)
-------------------------------------------------
SEQ_MUSIC_DUMMY..........00046600.000001D4.#NAME?
SEQ_MUSIC_ST_CASTLE......000467E0.00002D30.#NAME?
SEQ_MUSIC_ST_JAIL........00049520.0000A510.#NAME?
SEQ_MUSIC_ST_TREASURE....00053A40.00004EF8.#NAME?
SEQ_MUSIC_ST_CLOCK.......00058940.00003B44.#NAME?
SEQ_MUSIC_ST_TOP.........0005C4A0.0000926C.#NAME?
SEQ_MUSIC_ST_DESERT......00065720.000057C0.#NAME?
SEQ_MUSIC_ST_MISTOWN.....0006AEE0.00003BBC.#NAME?
SEQ_MUSIC_ST_CIRCUS......0006EAA0.00004D68.#NAME?
SEQ_MUSIC_ST_FOREST......00073820.00003DBC.#NAME?
SEQ_MUSIC_ST_DESERT_RE_A.000775E0.0000156C.#NAME?
SEQ_MUSIC_ST_DESERT_RE_B.00078B60.00002304.#NAME?