Like you said before, it depends on the person. Myself, I would find it insanely annoying, due to me using my 3DS for my portable music, so I tend to pull the SD card out quite a bit to update my library on there. Of course, I could probably find a way to rig it so it would stay closed without the screw, I'm generally fine altering housings and whatnot. It's a minor annoyance, yes, but it's an annoyance that wasn't present in the previous iterations.
Most eople that use the SD card just for games will probably be fine with it, and since most people have smartphones or some other device for handling their media, that will probably be most people. I just don't get why they decided to use a screw, were people having troubles with their SD card falling out of the previous models? I know my 3DS holds it just fine, but I don't know about the XL as I don't own one.
I'm not too concerned about it myself, though, since I don't plan to get one. I don't have anything major against the N3DS, but I can't see a reason to spend another $160 for a mildly better version of something I already have and works just fine. If it gets exclusives I want to play, not just ports, I might rethink that.