Don't know about iso but it does fine with bin/cue. Make a new folder in ux0:/app/RETROVITA/ and put them there so it doesn't freeze when you sleep the system.
Link to the .ISO via a .CUE file and try that. Or, you could convert from .ISO to .BIN+.CUE.
Also make sure syscard3.pce is located in (retroarch folder)/system, and you've explicitly set the "System" folder in Settings->Directories to that directory.
Guys... We could open a bounty for hardware acceleration.
Or...
Is there any ETA for this?
You mean hardware shader support? That would be nice.
If they crack it it's gonna sort all the problems out.
N64 maybe even Saturn dare I say it.....
Sorry to burst your bubble, but it's not exactly that easy.
For starters, the Vita has (ATM) absolutely no support for open graphics rendering standards (I.E. Vulkan, OpenGL[ES], DirectX, etc.), which means that the project would require one of two things:
1) The goddess Hakurei descends from upon Heaven/Makai and provides a Vita2OGL/D3D/Vulkan interpreter that gives the 2011-era SoC modern graphics rendering capabilities.
2) Either the already-existing Libretro team or a Vita-specific developer can cleanly convert all the aspects of the already developed Vita2D code (I.E. Software Rendering) to whatever early 3D renderer Rinnegatemate or whomever else has developed. It may not be the most efficient, but it'll be functional; perhaps a short-term boost.
Following from that (Yay for professional segues), the hardware-accelerated renderers we have
now aren't the most well-developed or, more accurately, ubiquitous and well-understood. Outside of some basic demos, it's not enough to convert something that was more-or-less made for the most ubiquitous modern rendering technologies to something that (to us with our homebrewed SDKs) barely understands what "OpenGL" is.
TL;DR, It's not
impossible, but will take more effort than is likely worth in the end. The 3DS is undergoing a similar mid-life crisis as well regarding rendering (To my understanding).