That's plenty fast enough. Something is wrong your Wiis internet connection. It may be something like your router being far away and it keeps losing connection or the Wifi card in your Wii may be dying.
If you have a broken sd card reader then I would make sure the connection between your Wii and your router is solid (not to far away, etc...) or if that doesn't work I would look into getting one of the USB<->Ethernet adapters.
I've honestly got no clue what's wrong. I guess my Wii is just dead and or dying. It is a 2006 Wii after all.
You can see in the picture my living room setup. Wii's right near the router (Green light, bottom left corner.) Oh well.
I take it there's much I can do other than try again? Will my system break if I restart it?