For your purposes (running 3DS games off of the SD card as .cias), downgrading seems to be a good idea. Though I do encourage you to check out what you can do besides that. There are some really cool homebrew apps out there that aren't dependent on your system having kernel access, like
Picroxx,
Draw Attack!, or if you're into emulation, the
Retroarch cores that have been ported seem pretty popular. And then, if tinkering with game assets is something you fancy, you can dump games most games with
braindump to take a look at their innards. With the right tools, you can create ROM hacks for some games
like Pokemon X/Y and OR/AS, or
replace the music in Smash 4. There's even some translations for games that are only in Japan right now,
like Labyrinth no Kanata. And yet, there's more so much more.
The only thing I have to add here about downgrading is if you choose to downgrade, it's wiser to use julian20's version of sysupdater to downgrade, which can be found in here:
https://gbatemp.net/threads/simple-guide-to-downgrading-your-3ds.407888/
safesysupdater is based off of the sysupdater version that was used near the beginning of memchunkhax2 implementation; it either uninstalls all titles first and then installs them, or it installs NATIVE_FIRM last, I don't remember. Either way, I advocate using julian20's version because it updates NATIVE_FIRM and the safe-mode titles first, and then does the rest. safesysupdater is only called 'safe' because it checks that you're using the right files for your system version and that the hashes check out so you don't have any .cias that are corrupted in the slightest. You can check the MD5 checksums and all that yourself, and you can also verify that you have the right update pack, meaning you have .cias for the Old 3DS and NOT the New 3DS (I recommend going to that iso site to search for a thread labeled "[MEGA] [CIA/BIN] MSET 4.X & 6.X backups, Browser backups, Update packs (All Regions)". If you choose to go there for that, you want "9.2.0-20U(Full).zip". I can't stress this enough because, as dumb as it sounds, people have bricked their 3DS using the wrong pack, either the O3DS files for the N3DS, or vice versa. It happens.
Just note that if something goes wrong with downgrading, you may end up with a brick - it's never completely safe. The best thing you can do is educate yourself (which it looks like you've been doing so). Last thing to note is if you softbrick and are able to enter into recovery mode, you'll have to update to 10.5, thus not being able to downgrade anymore.