The reason why I got an SD card this size is because I want to get CFW (Custom Firmware) so then I can have space for all the games I want to download.
Well while is not my business why you use larger cards than intended, Nintendo surly didn't expect anyone to just download every single game available or even back up ones that aren't on eshop servers, but again, is your decision. 32GB is what nintendo considered their standard.
To be honest i think you may have a defective (Fake) Card. Apperently they exist somehow. Generic looking cards, sell for very low prices and has high storages spaces, have very poor write speeds. Mostly you should avoid any unknown brands or ones that look cheep and generic for example have a sticker on it, without basic information.
Is better to stick to trusted brands like sandisk, kingston, luxor or whatever the ones listed by nintendo. Find ones that look like this, it should least have the "MICRO SDXC" Logo on it and The writing speed class that shows "U1" Or "C10" Something similar. Also Expect to pay more than $60 for it, mostly people would buy a very cheap card, especially having high storage.... easy to know is supicious.
You can also
CLICK HERE For list of tested and working sd cards by nintendo for nintendo 3ds and new 3ds systems.

It also tells you that is recommended to use the sd formatter tool for best results, and that nintendo does not guarantee performance for Micro sd cards.