It's hardware. With CFW you can load an emulated PSP CFW. AFAIK the only compatibility issues are with PS1 games, and then there's only about 5 that have issues. The most popular one being Digimon World 3.
I believe it's software-based. There are no "real" issues with compatibility for PSP games. The Vita will play literally any PSP game you throw at it, however, Sony has blacklisted some games for various reasons, so they take either Adrenaline or some finagling to install. A lot of PSP games and PSOne Classics that cant be downloaded directly to Vita can also be installed via PS3.