How can I put this into words...
It's hard to say if it's SquareEnix that lost his touch, or instead just wasn't able to cope with the expectations and evolution of games in general. It's hard to explain in a few words.
If you look at the evolution of gaming, you'll notice alot of things changed :
-The worldmap was somewhat required as a medium to travel, has given place to more beautifully detailed areas to move through the areas and stories. It's a matter of taste in that aspect I suppose, but how many games can you count since the PS2 era that had an actual overworld map? I know of two offhandly that did fine with it too (Dragon Quest 8 and Suikoden 5), while most of the games went with a more 'scaled' exploration (FFX and FFXII among others).
It's hard to say which one is better, but they are used differently. DQ8 had a beautiful, colored overworld to explore, but at the same time, due to the size of it, it also felt smaller to explore in a certain way. Yet I won't say that it wasn't amazing either. Suikoden 5 instead scaled down the models to the worldmap, giving it a very retro look, which was different than what they had attempted before in S3 and S4, that didn't have a worldmap proper (well it was an ocean in S4's case).
-Probably due to lack of programming space back in the NES/SNES days, but nowadays all games have built in tutorials, you can essentially ignore the instruction manual, as everything is given when you start the game.
-Somewhat related to the worldmap, the graphics have been increasingly detailed and lifelike. Where as before you had to do your best with a 40x20 pixel sprite, today we have fully-scaled 3D models.
-Battle systems have vastly evolved from the turn-based system the NES had. While you could say that Dragon Quest's battle system has not changed all that much since the first games, it still showed an evolution, wether it's in graphics or in gameplay, such as the skill system or more tactical uses of tension. All games try to make themselves unique, including FF games, instead of just trying to be a reiteration of the previous game's ATB system with tweaks to how you use spells, limit breaks or job classes.
Basically, what I mean is that it's very hard to properly gauge what people will want to see in a game. Do you want to see twelve times the same battle system with different stories, or be tossed some new elements in every game, changing your strategies or devising different manners of maximizing your team's power? That's mostly up to you.
As for FF13, I can note several 'good' points, but also bad points. I enjoyed the game for what it was, what it had to offer. I wouldn't call it terrible by any stretch, but it's greatest fault is being a FF : the high expectations that people have for it makes the fault look worse than they would be for another game.
Same thing happened with FF8. Alot of gamers had started their console- or RP-life with PS1 and FF7. The greatest downfall of FF8 for them was that it was NOT FF7-2. But if you take it for what it is, it had one of the most customizable battle system of any game. It tried new things like no-gil from monsters, or no treasure chest, and weapons that aren't bought but modified/crafted instead. It was a gamble honestly, but it wasn't the first time they did it either.
If you look further back, FF2's battle system was a very long shot from the original. No jobs/classes distinctions, essentially 4 characters that are the same and you can 'mold' into a role that you want, you could make 4 times the same character. More flexibility, yes, but also less uniqueness depending on your playstyle. Also no experience points, a stat growing system that was later recycled in the SaGa series.
Beside that, all of the FF games had a similar battle system, wether it was turn based or ATB, even FFX was a more 'tactical' take on the ATB, in which you could work with the results of your actions clearly seen as making your (or opponent's) waiting time longer or shorter. The rest is what changes, wether it's the job system (FF3-5-Tactics), or lack thereof (FF6-7-8), characters stuck in their jobs (FF4-FF9), the spell/skill system (Drawing/GF in FF8, Materias in FF7, equipment in FF9-TAdv), level advancement (experience, Sphere grid in FFX), and many more changes.
So FF13's battle system did not come as a surprise, and I did not find it horrible by any stretch. It was very active, very complex in a way. It had a 'controlled chaos' uniqueness, and it's in that way that I see that SquareEnix did a great job. The way they paced the battle system to learn every nuance of it, and the fact that you get to control all of the characters in a given role in turn, I felt it was a great idea. Being tossed in the chapter 10-13's battle system direction would end up in a large amount of frustration and game overs no doubt.
The difficulty is what I expected as well. Another evolution of games is in having the main story being relatively easy, for the more casual gamer that only wants to sit through the story, and then a large amount of extra that is more challenging for those that love it. But here comes one of FF13's cave-ins.
The biggest lack was in the sidequests. There was essentially nothing hidden, no side-tracking involved that would allow you to find a few precious items or gain extra levels by straying off path. There is essentially no back-tracking, no exploration, nothing left hidden by the time you went through an area. This is what they called linear.
But at the same time, I understand that if they left you stray off the path too much, it could have thrown off the pacing of learning the battle system. Also the way the story was made, you don't get to pick your team until chapter 10 or so. So for the story to remain coherent, it was probably necessary to avoid all of that extra skipping. But when the team gets together, it could have used a bit more of those extras.
So I wouldn't say they lost their touch, I'd rather say that while trying new things, they weren't able to properly read what people wanted out of the FF games. It seems they have listened and are making a very interesting looking FF13-2 and VS. I'm looking forward to both titles.