Piezo clicker is only a hypothetical solution, I consider a battery backed dongle far more sane. I mentioned it only because it is theoretically possible, not because I think it is amazing.
As for how to protect the circuitry from the clicker, I would (try to) use a small ferrite ring that has been converted into a small wire-wound transformer. The piezo crystal has two spikes generated, a positive voltage, and a negative voltage-- because it works by pushing electrons out of their orbitals in the crystal lattice mechanically. So, when you push down on it, electrons are pushed out, and when you let go, they rush back in. This effectively makes it an AC power source, if a very inefficient one, so a transformer would work to convert its high voltage--low amperage into a higher amperage--lower voltage wave. Then you could use a rectifier of appropriate amperage, and all that fun stuff. Naturally, you would have to measure the wave properties of the clicker and do some math to determine how many winds on each part of the bead you need to get the voltage/amperage ranges you want. You would need to click it fairly quickly to get good results, but it should work.That would let you jab electricity into the capacitor reasonably safely. After that, it is just discharge control hardware, similar to the previously linked UPS. It should all easily fit on something the size of a USB stick. I am thinking-- Ferrite ring transformer lays flat, and surrounds the piezo clicker on one side of the PCB, along with a rectifier and the cap. A load resistor on a flipflop prevents the micro (and low-power status LED, indicating the micro is turned on) from turning on until enough charge has been injected into the cap to overcome it, which then triggers the flipflop, bypasses the resistor and turns on the discharge control hardware and the micro, which would be on the other side.
Or something kinda like that.
I really do not want to design such a circuit.
Please dont make me. Electrical engineering is not my thing.
Again, from a sanity perspective, battery is just way smarter. This would just not require a battery at all, might be smaller/lighter, and depending on quality of parts, might last longer.