It really comes down to how much functionality you want.
Do you want to have basically a plug-and-play experience? Get a PSP. Anything that could be modded with it has been made, so there's really no barriers. Emulators are pretty good, though you won't get full-speed in some systems like SNES or GBA- but you'll also have the chance to try out N64 emulation, which is pretty cool.
However, keep in mind that the PSP-1000 has half the ram that later models have, so you'll see even less homebrew performance from that. If you're not too worried about that and just really want those PS1 games, Grab yourself an SD card adapter for storage, slap a ton of games into it, and you're off to the races (Following a guide for CFW, of course). Get used to having only one stick for those DualShock PS1 titles, though.
Now, if you want to maximize your experience of PSP homebrew... you might actually consider getting a Vita. Not sure if you're aware, but the PS Vita actually comes with native PSP backwards compatibility- to the point that, through modding, you can even get PSP homebrew running on it, and it runs BETTER than on a regular PSP. Plus, you can play PSP titles with cool upscaling filters, if you're into that. This functionality is only unlocked with CFW, however, and Sony's still releasing updates that may break the hacks, so be sure you're dilligent NOT to update your Vita, or else you'll have to wait for those hacks to be updated. That, and you'd have to source a Vita, which might be difficult nowadays.
All-in-all, if you just want to get playing now, go for the PSP, and if you think it could be better, down the line try and grab yourself a Vita or a PSTV. Either way, you're set.