PS1 on PS3 is software, while PS2 on PS3 is 100% hardware on the NTSC 20/60 GB models (PS2 CPU+GPU), 50% on the NTSC 80 GB and PAL 60 GB models (PS2 GPU+emulated CPU), and fully software on all later models (PS2 Classics emulator, physical discs cannot be used without hacks). I'm not sure how the PS1/PS2 Classics emulation works on the PS4, since it renders PS2 games at a higher resolution, but it doesn't have the real PS1/2 hardware for sure (maybe it uses the PS4's GPU like PCSX2 uses the graphics card of the PC it runs on?).
Either way, yes, technically there is always an emulator, even when running on real CPU/GPU. Nothing on these Sony consoles runs in a compatibility layer, not even PS1 on PS2 (since the GPU is emulated on the Emotion Engine CPU). The closest thing to a compatibility layer for backwards-compatibility I know is Nintendont on the Wii and Wii U.