There are no reports of booting via RCM causing a ban, and there are no reports of failed payloads causing bans.
I don't believe Nintendo could detect if you tried to send a payload and failed, but they can detect if you boot from RCM.
As others pointed out though we can't give a answer, as we don't know what Nintendo considers ban worthy, I do however find it extremely unlikely that RCM activity in general would prompt a ban.