Hacking a PSP or hacking PSP games? For the former most of it should amount to "means to extract zip/7z/rar files and the the ability to copy, delete and move files to the PSP memory card or whatever the go appears as", for the latter then read on
Best. Get bootcamp or a virtual machine and run Windows on it. I enjoy a good "ah, apple thinks it is computers" joke as much as anybody but in all seriousness as far as ROM hacking is concerned it really is a Windows only game (preferably some combo of 7 and XP) for just about everything*.
*a few Linux people did get stuff done on the DS (which is about the same sorts of problems faces), they only got there by writing everything themselves or porting it around, and still Linux is in a similar position to OSX in terms of ease of ROM hacking.
Still
The eboot files are the main executable of the game, though they can include any of the various assets games use to do what they do. Primarily though they are "supposed" to be for actual code, and the eboots themselves are takes on the ELF executable format like most other Sony systems. Depending upon the age of the game and the source you are using it may also be encrypted, however we did used to use the actual PSP to decrypt so there is that. Afraid I don't know what happened with this decryption lark since we got all the keys for things.
If you want to get stuck right in then I guess there is a version of IDA for osx
https://www.hex-rays.com/products/ida/support/download.shtml but that is rather expensive and probably of limited use.
The very most basic tool of a ROM hacker is a hex editor, I don't know what we are suggesting for osx these days but
http://ridiculousfish.com/hexfiend/ appears to be free and tick many of the boxes I look for when picking a basic hex editor, how useful it will be for anything slightly off the beaten path that we do in ROM hacking remains to be seen. While you can technically do anything within a hex editor (you have the whole file visible right there and can change it to whatever you like) it is really pointless to do anything.
I don't think there is much in the way of a tile editor worth linking up here, the PSP did use some common formats but if it is in the eboot then chances are any graphics are going to be closer to hardware than high level formats. There is not even a Java one you can try for anything more than NES style graphics formats.
