I think the only way to run unofficial .3ds roms is using gateway software.It won't work if it never got a physical release, therefore your suggestion indeed won't work. I don't want to argue this anymore.
I just tried this and it didn't work. I had a .3ds that I tried to decrypt like you said, and GM9 said it wasn't encrypted. So then I encrypted it, and it was marked as a .bad by the cart again. I don't know what I'm doing wrong.