I haven't studied the matter extensively, but I feel the main problem with the Vita's sales, especially compared to the 3DS, isn't necessarily that it's "too powerful" in general, but that its too close in power to its console counterpart. Let's look at them individually.
The 3DS. It was released early 2011, when the Wii was largely considered a dead or near-dead system by many. The Wii had its trademark motion controls, but the 3DS provided a more "traditional" handheld experience, albeit with the dual screens and touchscreen of the DS, plus 3D. A high price point and lack of games made for a poor launch, but this was rectified with the release of triple A first party games and a price drop towards the end of the year. By the time the Vita came out about a year after the 3DS, the 3DS already had a rather established library and great sales to convince developers to make games for it, as well as the fact that it was probably just a more promising investment than the Wii to most developers.
Now the Vita released a year behind the 3DS. In addition to having a lackluster library, it was released in wake of a still-going-strong companion console, the PS3 (unlike the Wii, which was lacking at this point). Aside from that, the handheld felt like a mini version of the PS3 in some ways. In addition to being afraid of taking a risk developing for the Vita, devs felt called to the 3DS, which already had proven itself with sales and a bigger library. And even more so, the PS3 was competing against the Vita, in a way. The Vita could output near-console graphics, but being a much-less proven system, what motivation do developers have to make a "handheld console game" for the Vita, when it could just go ahead and make a higher-definition version of the same game on the PS3, which has a much more proven fanbase?
So in conclusion, I think it boils down to mainly the fact that A) the 3DS had a year head start and was successful, and B) the Vita was too much, both too soon and too late. Had the 3DS not been released a year before and become successful, and had the Vita either been less of a powerhouse, or perhaps been released AFTER the next generation of consoles came out, it might have done better.
Anyway, sorry if that's jumbled and poorly written and makes no sense. I wrote it off and on over the course on an hour while juggling other things, and I'm really tired. But that's just my personal guess of what might have happened.
