What counts here varies, and may vary again in the future if Nintendo figures out another tell they can use or implement a new check that the hackers do not catch (very possible with fairly basic efforts on Nintendo's part).
For the purposes of this discussion we are mostly concerned with bans for modding (dodgy credit cards and being a bastard online are different matters).
The idea is that before being banned there will be some kind of tell the adversary (this being Nintendo and their agents) can use to detect that some modding had previously happened here, even if none is presently ongoing, by means of reading the NAND (permanent storage). If all you then have is a copy of this after detectable incidents have happened (possible with some methods of hacking, or just that you did not care to get a backup before you went and made it dirty) then some consider this a bad thing, and your chances of being banned rise dramatically if you do decide to try taking things online to the official servers. Whether you can clean a NAND image (or maybe Nintendo might have already got word that it was and are just biding their time before a ban) is up in the air, though the earlier methods of wiping logs all the time might in and of itself be a tell (if I, I being Nintendo, know lots of games had been played prior to this and all of a sudden at next check in I see this is apparently a never been touched device then something is up).
To that end anything that touches the NAND and appears as homebrew/hacked will be considered as making it dirty. Fortunately we have some means of intercepting the system before it fully boots (you often want that in hacking anyway as you get more access to more of the system). At this point you can then take a backup to maybe restore later, or make a virtual NAND image to boot to in turn and keep everything unhacked in that whilst having an (offline) part of the NAND (or another virtual image) to do all your nefarious pirate/cheaty/hackery/homebrew deeds upon.
What Nintendo checks for at any given point is mostly unknown and could be considered to be anything (they have full source code to things, nice debuggers, can quite happily browse sites like this to see what the kids are doing, what they think will protect them and develop counters). They still have physics to deal with (if your NAND and any identifying info are taken out of the equation and all done offline then not like they magically can tell) and beyond that they don't care that much* (adding checks and tests takes time in coding and time in booting/running the device) so mostly limit it to a few types of quick and easy checks**, though again they could get some motivated coder do something in the future and you have to rely on the developers of your custom firmwares and hack methods to be on top of things. This difference between what is theoretically able to be checked for and what is practically checked for then forms the difference in opinions the first reply goes into.
*you are never going to get all the hackers and doing so will just end up pouring resources into a losing battle (as a hacker I have all the time in the world and playing attacker-defender is great fun, Nintendo then have to pay their coders and suffer their users saying all these checks and protections and updates every 5 minutes are annoying). If however you can make the lives of the guide followers on forums, worry the parents of the kids buying the devices and otherwise make people believe online is a valuable thing that it would be unpleasant to lose then you can do that more cheaply.
**they could theoretically have local network play send the local partner a packet when playing saying next time you are online then this device with this ID you were playing against was hacked. Please tell the mothership. Possibly could get a repeat situation going on as well in some kind of offline distributed network. It is not even a hard technical feat. Most however view this as very unlikely to be implemented.