whenever I did the reboot to bootloader/update.bin it would lead me in to sys cfw
Weird, so someone (or you) deliberately swapped the update.bin with the fusee.bin or something? Since it boots into sysNAND CFW? And not into Hekate?
so what I did after clicking on it I pressed volume down & it led me back into hakate
Yep, I definitely don't know what's going on your Switch.

As I said, I have zero experience with hardmodded Switches.

Different people are following different tutorials available online, leading to even more different setups. I doubt that there are two nearly identical setups.
so can you guide me on the format thing
I've already posted
here, just make sure to adjust where necessary.
I tried haku33 but it didn't work, should I just go with the normal format
Only if you've already made a at least one NAND backup.
I did remove all the apps that were installed on it deleted the old user & created a new one, should I just do the regular format that is in the settings?
I would make a NAND backup right now and attempt the usual factory reset and, if it works, great. If not, restore the old backup and do not attempt to factory reset.
& Also what's with the keydump error thing when it load into tegraexplorer? Did I do something wrong with the console that it gives me that message or is it just a regular message that pops up on that menu?
I guess you need to update your
Lockpick_RCM and dump the up-to-date ones?
& Why is it booting into that menu instead of directly booting into hekate? Is it because something got messed up on the sd card?
Can't say for sure, it could be that you've swapped some of the payloads around. For example, I use the Reboot to Payload app on my Switch to reboot into hekate's payload instead of usual Atmosphere payload i.e. I just renamed hekate*.bin to reboot_payload.bin and removed the old one. Something similar must have been done to your Switch as well.
If you remove your SD card and you can boot into OFW while holding both volume buttons without an issue and if you get to No SD screen if you try to boot normally, I'd say that you're good to go to backup your SD card, then format it and set it up again from scratch by following NH Switch Guide.