Update-11-5-and-homebrew says nothing about whether the old payload works on 11.5. If it really doesn't work, you need to update the payload (which is written in the save data of oot3d). I don't know if the latest release of oot3dhax installer(in the homebrew starter kit on
smealum.github.io/3ds/) can write the payload into save data correctly on 11.5 ofw. If not, you can ask for a exploitable savefile for 11.5fw here(or somewhere else) and write savefile with powersave, otherwise you may need to have a 3ds on cfw to do this.
BTW, I think you have made a wrong decision to update the system. You can still use ctrhttpwn to access online sevice on 11.2 while the 11.4 update has patched it.
I used to having the same thought like you: hey, oot3dhax is unpatchable, why do i need to install cfw to get access to hbl? However, by sending many system updates nintendo really did a lot to prevent installing oot3dhax and this really anoyyed me. Finally, I chose to install cfw on my 3ds and now I can update with no risk of losing access to hbl(and it's much easier than before since the luma 8.1 released). I highly recommend you to do this too, this will make your life easier, really.