I'm having difficulties getting Oot3dHax to work as well. I'm on 11.0.0.33U with a New 3DS XL, and I had a similar issue where I couldn't boot into the freakyhax fix as well. When I run Oot3dhax, the game crashes and I get the "An error has occurred forcing the software to close. The system will now restart." error. I used Powersaves to transfer the newest 6-26-16 release to my cartridge and it looks fine. I've also tried various combos of the starter files and boot.3dsx files to no avail (http://i.imgur.com/rbi3q1k.jpg).
I went on #3dsdev to get some help but I'm not sure if there is something that screwed up when I initially ran freakyhax a week ago. I was able to get it to the boot sector screen with the ropbin text but then my 3DS froze on that screen and never finished booting into the Homebrew menu (I think the bottom screen may have been very dark greenish?). I rebooted my 3DS and afterward I could never get freakyhax to launch that boot screen (someone said something about "getting the boot sector going") and tried multiple different file combos, save resets, etc.
I thought Oot3dhax would work in my case, but I've gotten the same crashing error. I used my Powersaves to copy over the C2 image of US N3DS_U_25600_usa_9221 and it looks good from what I can tell and runs all the way up until I pull out the ocarina and freezes. I'm wondering if due to that initial time that I tried to boot/install Homebrew channel and my 3DS froze, that I'm running into all these problems. Any help would be great.
If you are using freaky forms deluxe to do freakyhax use this page
http://plutooo.github.io/freakyhax/
If you are using oothax with powersaves, use this page.
https://github.com/yellows8/oot3dhax