First, from the readme
"Make sure to download the otherapp payload for your firmware and save it in the steelhax folder as payload.bin"
If you didn't do that, try it. If not then steelhax installer still cannot be run under 11.2. Try the workaround below:
"Yes I tested this, I'm able to launch steelhax on 11.2 both on the A9LH 3DS and stock 3DS. It just takes some tries before it finally works. It's VERY unstable but it'll eventually load without crashing the system.
1. Install game
2. Make a save by entering the game and setting a mii
3. Go into JK Save Manager (3dsx or cia versions are fine)
4. export the save
5. Copy and replace the save file located in JKSM/Saves/Steel diver sub wars/
6. Import the save with JKSM
7. add the steeldiver folder with the code.bin in it
8. Download the 11.1.0-34U otherapp bin file and rename it to payload. Place it in the steelhax folder
9. Launch the game many many times until it eventually works
You'll need the steelhax folder with payload and code.bin on 3DS #2 to launch steelhax on 3DS #2 after the system transfer. It's launched the same way, trying over and over until it works."