I think you're misunderstanding. You dont have to use the SD card ONLY for Android. But you must format your card in a certain way.
Once it's set up, you dont need to keep moving stuff back and forth. You can use emummc, android, linux, and ofw all together. You then use Hekate to choose what you want to boot into.
https://wiki.switchroot.org/en/Android/Setup-11
You will have to format your card during this process, and you will lose however much you partition for Android. So if you have a 256gb card, you can partition 32gb of that for Android, and you will still have ~224gb left for Switch games and such.
Unless you mean you only have a 32gb sd card. In that case, I would look at upgrading cards first before you install Android.