I have not looked into the vita hardware and means it uses to achieve brightness control, though I would be surprised if it is not some kind of LED brightness controlled with PWM. If it is a separate hardware control then chances are slim without busting out a soldering iron, though there might be some undocumented stuff/headroom. If the OS/background processes generate the pulse then you have scope for something. Be aware that if you go too low with PWM you start to see flicker, indeed if you have a camera that can record fairly quickly then get a blank screen, drop the brightness and then you might see it there. It is the same for brightness increase, though a lot of things will operate at a 100% duty cycle for the PWM and going over that will see you overvolt (if that is even possible) or something else that probably will involve a soldering iron. With OLED stuff I am not sure what type of OLED they use, big boy OLED works by having the pixels generate their own light and that might alter the scope for things, if it is just LCD with an OLED backlight then it should be like above.
The forest for the trees method is find youself a bit of sticky back plastic (or tape it if you are playing the cowboy) but have it slightly shaded/tinted. The type of plastic you choose might trouble the touchscreen but I reckon that stuff people use to put on their windows should do and does readily come in tinted form.
Depending upon what causes the outside issue you might have some scope for something here, though it might involve playing with polarising filters. Quick test if you have a set of those 3d glasses for 3d cinemas then they have circularly polarised lenses (would have preferred straight polarised for a test but those glasses are an easy thing for people to find). Stick them on and rotate your head a bit.