When Dr Robotnik met Knuckles and sent him to destroy Sonic, he probably didn't realize just quite how well his plan would work.
When Sonic The Hedgehog landed on the Sega Megadrive in 1991 he instantly became Sega's mascot. He was fast. He was blue. He was spiky. Which apparently, and generally unknown before his debut, was every school boy's dream! Sonic 2 came around the following year, and again was an instant success. It featured a second character, Tails. He was everything Sonic wasn't; furry, chirpy and generally a bit boring. All in all a perfect sidekick - who wasn't going to give the main star any grief.
The final two games to appear on the Mega Drive, Sonic 3 and Sonic & Knuckles featured a new character called (unsurprisingly) Knuckles. In Sonic 3, Knuckles was a new nemesis fighting against Sonic, as Dr Robotnik had duped him into believing Sonic was the bad guy. In Sonic & Knuckles, however, Knuckles was a playable character. And here in lay the problem. Knuckles could fly (well glide), you see. And he could climb walls. Really, How cool was that? Why would you want to play as Sonic when playing as the pink dude meant you could float around until you hit a wall, and then climb up said wall? Sonic had very little going for him. He was forced to rely on his name to make people choose him to play with. I mean, Sega kitted him out with a flash kick, he was a bit faster, and he could make better use of some of the shields that were available but, come on, none of that is as cool as flying and climbing!
It's possible that Sega were actually being a bit clever. It's not unreasonable to hypothesize that Sega had seen the Sonic games were running out of possibilities. I mean, how many more games could you make sticking to the same run and jump formula? Power-ups like capes and firepower, as found in the Mario series, wouldn't really work in this style of game. So maybe, like the creators of The Leauge of Gentlemen and The (British) Office, the owners of the IP decided that there were only 4 games worth of content available in the series, and that he'd need to be retired, in 2D at least, after Sonic and Knuckles. Knuckles was perhaps to be the new hero for Sega.
It would have been a very noble decision to make, but with two great flaws. Firstly, spin-offs are rarely as popular as the series from which they originated and secondly, the 32X and Saturn were just around the corner.
While Sega were busy destroying the Sonic franchise by all but killing off the hedgehog in the main series, and creating various quirky titles like 3D Blast and Knuckles Chaotix, they should have been bang on making a beautiful 2D Sonic platformer for launch with their Saturn console. The improved graphic capabilities of the system would have meant there were at least another 2 games in the series before people started to notice the gameplay was getting repetitive. In which time they could have begun lifting Knuckles, or some other character's profile in their own series. It would have been seen as a risky move at the time; after all, the entire world was getting gooey over the Super Mario 64 demos and people were convinced that 3D platforming was the way to go. It was only proved that there was still a decent market for 2D platform games when, a decade later (ironically) Nintendo released the New Super Mario Bros on their DS system.
A 2D Sonic game on the Sega Saturn would have been pretty easy to make (the system was originally only really designed with 2D in mind, anyway; until Sony started demonstrating their PlayStation at which point Sega started bolting all kinds of additional hardware to help it compete), would have looked spectacular compared to the Mega Drive games, and would have given them the opportunity to delay some other launch games (notable Daytona USA) to give them time to fix some of their flaws. The updated graphics would have meant that Sega could have brought pretty much nothing else new to the table for the first next-gen Sonic title; they wouldn't have even had to think up new Zones; who, at the time, wouldn't have wanted to see new acts based in familiar but graphically up-to-date Marble Zone, Casino Night Zone, and Flying Battery?
A Sonic game would have made the Saturn sell far more than Daytona and Virtua Fighter combined, and as I have eluded to already, those games when released without flaws would have strengthened the system, rather than weakened it.
So it's my view that Knuckles enforced on Sega a dilemma that they didn't need to deal with for at least another 3 or 4 years. So convinced were that that Sonic was running out of steam, that they held him back and doomed their next gen consoles to failure.