1) Cause it is not the native executable format. You need to trick the Wii U into loading a different binary format.
2) Yes, you need to change the code: .elf is coded against a completely different environment than .rpx (dynlibs vs WUT and stuff like that).
3) Cause the homebrew scene wanted to start but the .rpx format / how to load native, unsigned binaries wasn't known back then. So this hacky way of loading .elf just got found before the way of loading .rpx. Also .rpx needs an advanced SDK which didn't exist back then, instead each homebrew basically used its own SDK (dynlibs and stuff).
4) Hard to say but more and more stuff gets ported from .elf to .rpx lately as next-gen CFWs (Aroma) can't run .elf anymore.
4) As you said: Answer is no.