I remember seeing a post about Lua LovePotion being able to do 3D.
Even if you don't know OpenGL you're going to need at least first-year uni Linear Algebra knowledge (probably 2nd year for American universities). And if you want to build a physics engine you'll need equivalent level calculus.