I was thinking a shift-button that you would hold down in order to modify the behavior of other buttons.
The Vita has 8 buttons aside from the directional pad. If you need more buttons than this, you could assign one of the buttons such that while you hold it down, other buttons can have an alternate mapping.
For example, the Stella emulator (Atari 2600) needs six buttons to emulate setting the positions of the important, but seldom changed, Color/BW and both difficulty switches alone. On top of this, to get to the Retroarch menu you have to mash a combo like [select + start + L1 + R1], throwing all those switches out of whack and/or resetting your game, something you definitely don't want to do on that system. With a modifier key, you could map all those switches off onto an alternate mapping that's only active while you hold down a particular button.
It can allow for cleaner, uncluttered mappings. The difference between this and Hotkey Enable is that it would apply to in-game buttons too, not just hotkeys.
I thought this was a feature available in most of the other implementations of Retroarch, but missing in the Vita version - but on review I found that isn't true. Since I wrote that, I discovered that it's not part of Retroarch in general; my memory was playing tricks on me. It would have to be new feature for RA and therefore not something I could add myself to the main branch. Off the top of my head, I can't think of any way to implement it without reworking RA's interface in a major way, which is not going to happen. It *can* be done via an alternate joypad driver without touching RA itself.