You're missing out on a lot...
I like to see him playing rhythm games with the sound off

.
While I think the audio is an important part of the gaming experience it depends on the game and situation.
I like the extra noises you can hear with headphones, but I also find a bother having the headphones on my head all the time(mostly if I need to constantly put them in and out to go somewhere around my house) and of course I'm not gonna use headphones to play games like smash if I'm playing with others or simply talking with someone in the room.
For the most part I just use the TV speakers as I don't have space on my room to fit a 5.1 that my father retired for a bluray hometheater. I only use headphones on home consoles at night to not bother anyone(somewhat debatable with the sounds of the controller buttons).
On portables I use headphones more often but I still find it bothersome for the same reason as above.
I like to play rhythm games and as those are played better with good audio I use headphones often, but the audio quality of the Vita and PSPgo, for example, is good enough that I don't need headphones if the ambient is quiet.
Rhythm games on Nintendo DSi, I feel the audio quality is better from the speakers as the audio quality of said games is not as good as the PSP/Vita rhythm games(I'm looking at you Osu and Utacchi).
If I'm out I crank up the volume and hope I can hear anything if the ambient is noisy, mute the sound if quiet or use headphones if I need the audio(again rhythm games).
TL;DR I use headphones if I feel like it, if I need to hear the game audio(rhythm games) or to not bother other people.