As I only managed to get it to work on linux (windows has python instead of python3 and I was too lazy to rename it everywhere)
I can only try to explain how to setup a linux build environment. I won't explain though how to setup a linux operating system itself
but its not too hard (I used VMWare Player 12 and Ubuntu Gnome 15 64bit). I guess you'll find many tutorials on that one on Youtube etc.
Anyways, once you got linux booted up, you can do the following steps to setup your development environment:
1. Open a terminal, and paste the whole block of commands you can find
here and press return.
2. After that, enter the following command (without quotation marks) and press return: "sudo ln -s /devkitARM /opt/devkitpro/devkitARM" (You have to enter your password there)
3. Go to the desktop via "cd Desktop" and make a folder with "mkdir 3DS" (or whatever you wanna call it)
4. Switch into that folder via "cd 3DS"
5. Download a copy of homemenuhax via "git clone
https://github.com/yellows8/3ds_homemenuhax.git"
6. Switch into the folder via "cd 3ds_homemenuhax" (should be the name of the new folder)
7. Issue the command found on the homemenuhax github website (and change it to your liking).
The default is "make clean && time make LOADSDPAYLOAD=1 USE_PADCHECK=0x200 ENABLE_LOADROPBIN=1 ENABLE_HBLAUNCHER=1 LOADSDCFG_PADCHECK=1 MENUROP_PATH=menurop_prebuilt"
8. After pressing return, it should compile. When it's done, you'll see that <username>@ubuntu:<directory>$ prompt again. Should take around a minute with the default settings.
If its finished very quickly, you might wanna check if there is an error in the output and post that.
9. If everythings alright, you might need to compile the 3dsx file itself, which you can do by switching into the themehax-installer subfolder via "cd themehax_installer" and issue the "make" command there as well. Just type "make" and press return.
10. Now you should have the themehax_installer.3dsx and smdh files in that subfolder and all the themefiles in the themepayload subfolder. That's pretty much all you need.
Note: if you want to embed a custom theme, you have to append a "THEMEDATA_PATH=<relative path to an uncompressed body_LZ.bin file> to the command at point 7. You will notice that it takes a WHOLE lot longer (took me an hour on a fast PC) because it embeds the theme into all the files in the themepayload folder, which are something like 64 different files..
Hope I didn't forget something
//Note: To prevent any problems, look at the file size of the uncompressed body_LZ.bin file (if you plan to use a custom theme). If that number is dividable by 4, you shouldn't have any problems. If not,
read here to solve that.