Should be doable yeah (obviously assuming you can direct audio to those outputs when running with HDMI, that is not usually a problem though and comes up more if you are running from VGA). I have been known to tap off internal speakers at points like this but that is just me.
Personally I think you would be a lot more happy in the long run if you get a basic altoids amp type affair going on (if your TV has a USB port you can power things from they are quite good for this) and fed a standard RCA/phono to 3.5mm into that.

 

My apologies I made that more complex than it possibly should have been. Most TV manufacturers, especially now we have all gone flat panel, recognise their speakers are awful and will tend to output things over the phono/rca ouput leads at all points in time (probably will not even mute the main TV speakers)*. HDMI can support other combos of speakers (2.1,3.0....5.1.....) than straight stereo but as you just wanted stereo headphones that should not be a problem at all. Now if you want to try using a VGA input and directing audio around the place then it gets tricky with some TVs I have had the misfortune of setting up, I should have made that clearer.

No USB... I am impressed. It does not have to be an active USB you can watch videos from and just some little service port or something (they will tend to be powered still and more than enough to power an amp rather than remember to have a 9v battery or something).

*bonus 2 is it will delay the audio by however much the TV delays output for upscaling and whatever else as well.