Probably making the ultimate version of Pokemon thats practically all the pokemon in one is too ambitious of a project. By your logic, you might as well ask why can't every new mario game just be that year's version of a new Mario All Stars.
Also games shouldn't be too long. The statistics of games are that only 20% or less on average number of gamers actually mangage to see the end/beat the game these days. Making games that extend past 20-30 hours of original content is more or less wasted effort on the development cycle. You're talking about a project that would take several times longer to develop, and overall be under appretiated by a smaller percentage of audiences.
Also looking at a sales point of view, Pokemon games need no help making their single-adventure games gross tens of millions of sold copies and making the game longer than the standard will hardly increase sales, and cost more in production and time. Time is money, and its in Nintendo's/Gamefreak's best interest to be spilling out as many Pokemon games as fast as possible is the most effecient way of cutting production costs and getting a return on investment as soon as possible. Taking too long making a game is too costly, and rarely produces good results. I mean there are still dummies that thing Final Fantasy Versus/XV is going to be the best game ever? lol.
The fact is that even the best, highest rated, most beloved games in this world have a hard time pushing past 10-15 million copies sold. Pokemon as it is, almost consistently breaks those milestones, so why would anyone spend more time and money making a perfect formula last longer if its not going to yield better results. I have a hard time seeing a Pokemon with all the badges, towns, maps, pokemon, badges and etc selling more than 25 million copies in it's life, it would seem they'd have to sell 40 million+ in a 6 year life span to justify the development cost and schedule, otherwise they've worked harder on a game thats not as profitable.
BUT Time and Money is NOT a reason, its just a practical limitation. The likely reason why Pokemon doesn't include the entire catalog of Pokemon content is often the same reason that OTHER games don't follow that same suit. You're thinking of a game as a game and not as a piece of media or art. 1 game = 1 experience just as 1 album doesn't = 2 albums, 1 movie isn't 2 movies, 1 book isn't two books. Theres compilation titles that are created as bargains, but rarely does any content producer ever include all the experience of one thing into the same experience of another. Not every game gets to be Super Mario All Stars + Mario World.
Saturday dude. 
JM 
