Not at all! Android is actually a very efficient operating system. You might not think so because of the bloated crap that ships on modern devices. If you take AOSP and remove all the background Google stuff, you have a solid linux-based operating system.
But to port an Android game to the 3DS, you wouldn't necessarily need to port the entire operating system. Android apps run on the Dalvik virtual machine, which is open source. If you port that, along with all the APIs used by a particular game/app, you could get it to run. Games in particular tend not to use too many system APIs, as they rely on the game engine to abstract that stuff away. This process can be made easier because there are davlik bytecode decompilers out there, so you could actually see exactly which system APIs are used, and possibly modify the code to not use them.
Although, ABI compatability for native code is probably going to be a lost cause. I don't think the ancient ARM chips used in the 3DS can run armeabi-v7a instructions, which is the oldest architecture officially supported by Android nowadays (not necessarily supported by all games/apps tho). Emulation or virtualization could potentially be used, but it likely wouldn't be fast enough for any games. Maybe decompilation could work?
Like I said before, none of this would be easy, but it is possible.