Mods and homebrew is quite a different activity with minimal overlap.
For homebrew you will probably be pointed at
https://devkitpro.org/ as it has Switch options. Most likely to be coding in C and maybe some C++ for this one. There should be plenty of open source projects to look at in addition to the examples that provides.
https://switchbrew.org/w/index.php?title=Main_Page has some info on hardware as well.
There are not really any high level scripting languages I would want to point someone at right now outside of whatever exists within the various Linux and Android ports (on the older consoles we tend to see some nice high level languages and they get their own little community).
Mods, or ROM hacking to most when it is not on PC, is much the same whether you are dealing with something loaded from tape or the latest PC game. The Switch has a file system so you will have that to play with (anything on a cartridge older than the DS will probably be a big blob of code, no nice file names/sizes/extensions/directories).
Otherwise pull it apart, figure out the formats for the aspect you want to edit (will tend to be closer to the hardware, some abstract known format, something Nintendo provided or something more game focused than pure abstract), edit and go from there. If you want to edit what is taken care of with code then you get to learn a more complicated form of coding called assembly (which differs between CPU types, and the surrounding hardware for what it speaks to).
Switch emulation is not particularly good at this point so debugging might well have to be/be better served by hardware based methods (easy enough to add things in there and speak to it by network to your PC
https://gbatemp.net/threads/atmospheres-gdb.602435/ is more for cheats but memory reading, breakpoints and such is debugging, learning to make cheats
https://web.archive.org/web/20080309104350/http://etk.scener.org/?op=tutorial though Switch stuff has some quirks with memory locations, both helps in hacking by giving you a jumping off point and is a great way to learn things)
https://wiki.multimedia.cx/index.php?title=Category:Game_Formats http://wiki.xentax.com/index.php/Game_File_Format_Central http://www.amnoid.de/gc/ https://gbatemp.net/threads/gbatemp-rom-hacking-documentation-project-new-2016-edition-out.73394/ is more for the GBA and DS but the principles are the same really. By similar token you might also like
https://www.romhacking.net/start/
Not sure what we are pointing people at for fiddling with 3d formats -- you occasionally see plugins for Blender or other 3d modelling programs.