I'm not sure if the read/access times for different types of cards carry over but I am using an A-data 50x sd card and don't notice any slowdown with ds games and the only GBA game with slowdown that I've played is Kingdom Hearts (and only when there are a bunch of those green health restore/moogle point sphere's bouncing around, none during gameplay).
So there have been some posts regarding read/access times mattering for the supercard lites, but I don't recall hearing anything about other media.
In general I think after a memory format has been established (CF/SD) there really are no speed problems as the majority of cards on the market are rather fast. For newer media (MiniSD [back when it was new I mean, I haven't heard about speed issues regarding miniSD for a while], and MicroSD) you might want to look for a card with better speed.