I agree that it has something to do with the 1.54 firmware. When the game first came out I was playing it on the latest beta firmware at that time(1.54B I believe), and I had not one single freeze. I wasn't using a patched rom either, I was using the save fix cheat. Once I updated to stable 1.54, bam...the game would start freezing. And others were reporting the same problem on the TC forums after updating to 1.54.
I have played this game for hours on my DS-Linker and Supercard DSOne to test this, and not one single freeze.
So I defintely think it's a problem with 1.54 and not the memory cards.