Well, there are hints that the private battles run as dedicated servers on a host wii, such as that the hosting player (at least from experience) never has lag on any of the objectives and that certain hacks like colorhax sometimes cause joining players to get the same ink-colors...
The tricky parts would be both locking out nintendo (more dns-blacklisting maybe?) to some extend and find an alternate way to join a lobby without a working friendlist...
But i am just thinking out loud, these are mere theories of ways to tackle.
Also you talk little cheating. when it worked for brawl, mario kart, sunshine and galaxy, why shouldnt we be able to figure out splatoon maps and make our own ones? Imagine Splatoon on Bob Omb Battlefield or UT3 Deck or something similar! How frigging awesome would that be?
And all that is exactly holding this back is nintendo banning people for once. Hacking doesn't need to be unfair little shitcheats for some scrubs who can't play the weapons and configurate their gear normally...
What most people don't see is that mods have an unmatched potential to keep multiplayer games alive, just look at how much ctgpr and project m are still being played EVERYWHERE and for a lot of highly customizable multiplayergames on pc (unreal tournament and trackmania just to name 2 wellsuited examples) this has held up for even longer.