Creating a game engine from scratch requires knowledge beyond what most developers in the scene can handle, and the ones that could would not have the time or resources. It's a massive project.
You will need to implement sound, collisions, create a physics engine, polygon renderer, lighting, cull invisible surfaces, handle animation and 3D models, draw skinned meshes, get a particle system setup etc. I could go on for a long time, its just not really worth it.
Best bet for 3D homebrew games is Unity with all its Switch SDK dependencies (NX addon for Unity, Clang for compiling, NVN API etc.) , but that is obviously illegal, and completely unrelated to OpenGL in LibNX.