Depends what you're into, I guess. If you're not really into Nintendo first party stuff, it's gonna cut down the vastness of the library significantly. But it still enjoyed pretty strong third party support, so I'd say yes. I'd recommend getting a New 3DSXL. Color doesn't really matter; I like the shell of my Samus edition, but prefer the multicolored face buttons of the standard units). But I recommend the New 3DS XL simply because I actually really love the stereoscopic 3D (and I prefer the larger screens). It's a unique, defining feature of the console. I really don't think you should go without it unless you're one of those cretins who's just completely against it. Certain games utilize it very well, to the point where it really adds to the experience. The New 3DS has the "super stable" 3D, so you can actually move the thing a little bit without it going all wonky if you don't hold it completely still as the original model did, so it's pretty much a must. Plus, it has a little more horsepower than the Old 3DS, which gives you access to a couple more official games, and gives you better performance with certain homebrew, mods, and emulation. Also, it comes with a second analog nub out of the box, which makes a lot of the 3DS games much more playable than they would be without. I also recommend performing a simple hardware mod by simply ripping out the rubber nub for the second analog and replacing it with a PSP cap. Very cheap, easy mod that doesn't even require you to take apart your 3DS. Makes it much more comfortable to use, and slightly more intuitive.
Not only do you get the 3DS' amazing library, but you also get the DS' amazing library. Lots of unique experiences you can't really get anywhere else, due to the unique nature of the hardware. On top of that, if you jailbreak it, you have access to the entire GBA library -- which is also amazing -- with perfect emulation. Same goes for GB & GBC. If you're into the 16 bits, you can add near-perfect Genesis, SNES, & TurboGrafx emulation to the list. As far as 5th gen consoles go, emulation is hit or miss. There's been a couple of amazing, optimized 3D ports, such as Tomb Raider & Mario 64. But mileage definitely varies. N64 emulation is mostly crap, and as I understand it PSX emulation has improved, but still very hit or miss. I just stick to the Vita for that.
I have a Vita as well. And you can do some awesome stuff with it that you can't do with the 3DS as far as some of the games, emulation, and ports, but the 3DS simply has a better library (in my opinion), and a more vibrant modding and homebrew community. I'm finding new stuff to do on it all the time.
If money is tight... after you make that initial investment in the handheld, and a high volume SD card (which are pretty cheap these days), you could potentially play pretty much everything for free via download.