LayeredFS overrides files within the romfs with whatever is present in the folder or adds on. SpecialK is sort of similar in that it can override, but it works at the other end -- replacing textures after loading. LayeredFS is orders of magnitude more powerful in that it can replace anything (whereas SoecialK is mostly just videocard stuff like textures) but you do have to repack (which means decrypting usually -- assuming they kept the Switch version encrypted as some games drop that due to already being encrypted since that would hurt performance.)
For all that they are somewhat similar in principle, the mechanisms are completely opposite and stuff for one won't work on the other. Also, SpecialK relies on hooking into Direct3D or etc to work, so is impossible in any other environment.
Honestly, I always felt like SpecialK was far too limited and that it was very disappointing that so many modders stop trying to figure out how to extract and repack game files (admittedly not easy) when it functions because it limits what can be done to very little outside if simple texture replacements. Perhaps with this latest console release some may try again, but sadly probably not.
