To answer your question, the recent bans handed out by Nintendo was due to detecting unauthorised software being used while connected to their server. As such, using software like Vgmstream3DS while the 3DS is connected to the internet.
It has nothing to do with Custom Firmware or anything outside the scope of Homebrew software.
Best way to avoid bans, is by using .3dsx versions of Homebrew software and use Luma3DS's Input Redirection to open Homebrew Launcher. And to be offline while using Homebrew CIAs.