Reviews in the Media - Flawed and Inaccurate?

[kuwanger]

Reviews in the media are flawed and inaccurate, sure. That's, well, a fundamental thing about reviews of an entertainment medium, isn't it, since it inherently is subjective. I wouldn't say reviews "are on the decline" in any meaningful way, though. By that, I mean, even if there's a short term problem with reviews now, I see it being "fixed" over time as people start to weight reviews less or certain reviewers/sources higher than others. That's just the cyclical nature of things. I mean, how many "gaming" review magazines have spawned and died over the last 30 years?

As for a complaint about the scoring system, well, it's the same with used game prices. I mean, one would think demand would have enough of an effect that the price of a game would be a good indicator of the fun/entertainment/whatever of a game in a genre--as different genres should have different buy/sell patterns--, but for those who have bought even a few used games, there doesn't seem to be much rhyme or reason to it there, either. That doesn't make the price meaningless. It just means it's silly to judge a game based on its price/cover.

In any case, if I had to make a wild guess about the "decline of reviews"--a wild guess as I haven't begun to read enough reviews really every to make a judgment on the general quality of game reviews--, I'd take it as more the 'inexperienced" part being a factor as so many games enter the market, there's a glut of shovelware and a glut of new reviewers trying to push a lot of reviews of it through (to filter out the "good" from the "bad").

Oh, and a small guess, but the whole RPG grinding being a complaint? Well, if you're a person trying to review several games, having to grind is a great way to make a game (a) feel monotonous--Pokemon, Animal Crossing, etc all do this too, but they at least try to color some of it with items or new monsters or whatever to it feels you're accomplishing something real instead of merely making stat X go up 15 points so you'll do 15*y points more damage per hit against the area boss--and (b) feel like it's holding you back from enjoying all those other games you've set out to review. I mean, one can sort of presume that a reviewer by definition is in it for the experience of playing a lot of different games--sampling them, really--and not devoting 80+ hours per game.

Just think of reviewers like doctors. You can always go for a second opinion. And in the end, it's our time so your choice on whether Pokemon really *is* a 10.0 or not. Me? I haven't tried out Pokemon and don't have much interest to, but that's because I'm too busy with a whole other set of games I want to play. More power to people who love or hate Pokemon or the games I love or whatever. And I'm sure I disagree strongly with a lot of people who love the games I hate or vice versa.

*tries to forget the Metroid: Other M plot*

 

[Ryukouki]

Reviews in the media are flawed and inaccurate, sure. That's, well, a fundamental thing about reviews of an entertainment medium, isn't it, since it inherently is subjective. I wouldn't say reviews "are on the decline" in any meaningful way, though. By that, I mean, even if there's a short term problem with reviews now, I see it being "fixed" over time as people start to weight reviews less or certain reviewers/sources higher than others. That's just the cyclical nature of things. I mean, how many "gaming" review magazines have spawned and died over the last 30 years?

As for a complaint about the scoring system, well, it's the same with used game prices. I mean, one would think demand would have enough of an effect that the price of a game would be a good indicator of the fun/entertainment/whatever of a game in a genre--as different genres should have different buy/sell patterns--, but for those who have bought even a few used games, there doesn't seem to be much rhyme or reason to it there, either. That doesn't make the price meaningless. It just means it's silly to judge a game based on its price/cover.

In any case, if I had to make a wild guess about the "decline of reviews"--a wild guess as I haven't begun to read enough reviews really every to make a judgment on the general quality of game reviews--, I'd take it as more the 'inexperienced" part being a factor as so many games enter the market, there's a glut of shovelware and a glut of new reviewers trying to push a lot of reviews of it through (to filter out the "good" from the "bad").

Oh, and a small guess, but the whole RPG grinding being a complaint? Well, if you're a person trying to review several games, having to grind is a great way to make a game (a) feel monotonous--Pokemon, Animal Crossing, etc all do this too, but they at least try to color some of it with items or new monsters or whatever to it feels you're accomplishing something real instead of merely making stat X go up 15 points so you'll do 15*y points more damage per hit against the area boss--and (b) feel like it's holding you back from enjoying all those other games you've set out to review. I mean, one can sort of presume that a reviewer by definition is in it for the experience of playing a lot of different games--sampling them, really--and not devoting 80+ hours per game.

Just think of reviewers like doctors. You can always go for a second opinion. And in the end, it's our time so your choice on whether Pokemon really *is* a 10.0 or not. Me? I haven't tried out Pokemon and don't have much interest to, but that's because I'm too busy with a whole other set of games I want to play. More power to people who love or hate Pokemon or the games I love or whatever. And I'm sure I disagree strongly with a lot of people who love the games I hate or vice versa.

*tries to forget the Metroid: Other M plot*

Oh ho, you caught on! Used game prices was probably going to be the next article I was planning to discuss!

 

[Transdude1996]

Oh ho, you caught on! Used game prices was probably going to be the next article I was planning to discuss!

In my town, there's a local game store that sells used games for all consoles, even the Atari 2600 and the Virtual Boy. When I asked them about how they determine their prices, they told me that the price is determined by the rarity of the game.

If that's really true, then that means there's a lot of copies for F-Zero GX still floating around because I was able to get it for under $5 from them.

 

[Ryukouki]

In my town, there's a local game store that sells used games for all consoles, even the Atari 2600 and the Virtual Boy. When I asked them about how they determine their prices, they told me that the price is determined by the rarity of the game.

If that's really true, then that means there's a lot of copies for F-Zero GX still floating around because I was able to get it for under $5 from them.

LOL, my game store will take brand new games after release and then open them and sell as used. How do I know? The disk I get in return is spotless, the instruction manual has not a single crease or wrinkle, and it has that new game smell to it. .-.

 

[pwsincd]

Dinoh is right , you personally are the reviwer and you should ignore alltogether the dumb scores from media biased reviewers. You tube videos , gamer comments , forum threads are my source for game reviews and from there ill make my mind up on wether ill be investing , unlike movies , ill use the imdb rating as a guide as to wether ill bother watching something .

 

[aiat_gamer]

That's a good question. His review was factually proven to be flawed at some point if I recall correctly. When I played it I thought the controls were a huge step up from the waggle era of Twilight Princess. If you play the game the way it is meant to be played (again, this ideal is dependent on a number of factors) it plays fantastic. If I had to give an on the spot score I would award it an 8.5, subjective to a number of factors that I account in an as-general-as-possible approach. This score is in comparison to other games in the Zelda franchise. It also is subjective to improvements over mechanisms that needed improvement from titles past. Some major strengths could include the stronger story, the unique puzzles, the strategy that is involved in fighting monsters. Some of the point deductions could be subject to the linearity that some gamers may have faced. Other issues could include the forced inclusion of Wii Motion Plus. I can go into the nitty gritty all day but I might as well write a full review on it.

I will be honest, I have not played Skyward Sword, but I did follow the reviews on it closely. Based on what I heard and read, seems like in retrospect, this game is a clear example of people being so excited that they just overlooked major flaws within the game. I read many instances that said the game suffered from poor pacing, getting repetitive and some control problems and also the fact that it holds hands to the point of tedium.
This is what I find very problematic in reviews, the main example I can think of that I also have played is Bioshock: Infinite. Looking at the review, I can not believe how everyone just chose to overlook some serious problems with the game. I mean basically the gameplay in the game is deeply flawed, so much that it became a chore to actually finish the game! As I expected, people who look at the game objectively after the excitement wore off actually were able to see through the initial impressions and acknowledge the problems within the game. I have burned many time like this and that is the main reason I usually never buy games when they come out in the first month or so.

 

[Cyan]

Ahh, so many long comments, I didn't read them all yet but I'll start posting what I've read and answered so far.

Hell, how about writing multiple short reviews for a game (with different approaches), and asking people for feedback? That could be fun.

That's the idea I had when Costello asked what could be done to improved the score table of GBAtemp's review section. I thought it would be a good idea to allow multiple official reviewers score the same game, and each official reviewer having a list of their own games/genre score to let people know that they like the same genre as them. For example, if someone who never played jRPG before and are not a fan of this genre but liked the game he/she reviewed, then the reader who also don't like RPG may think it's interesting as a "starting game in the jRPG genre". And other official reviewers who are old school fan of jRPG may find that game dull and readers will know, based on their own preferences, which reviewer they want to follow.
That system was never made available to the review center, because it's based on a single reviewer system, and multiple readers comments. reviewers could make their own review in the comments, but it doesn't have the same impact as if it was fully integrated to the site (like tabs for each reviewer, and a reviewer's profile with game/console/preferences/backlog list, etc.)

I also had to make it clear in the beginning that I was not a fan of random encounters or grinding which were often staples of the genre. Some may say that because of that, someone like me would have no business reviewing JRPGs with my limited experience and my disdain for certain tropes that the genre has seen a lot of. But maybe instead, my outsider view will be more accessible to other greenhorns of the genre and I can help give them a place to start if they want to branch out.

I think this is what should be done on all reviews, it let's the reader know if you are a fervent fan of the genre, of the series, if you played all past games or not, etc.
Telling your backlog gaming is a good start in knowing if your review is biased or not, and to know if you based it on the past released games or if you are new to the genre. It gives more strength to your opinion, and readers should take that in account.
I'm currently trying new genres of video games, I was a fan of RPG and Adventure games only. But these games tend to be very long to complete and I miss time now that I'm working. So now I prefer short games, easy to complete, I'm open to new game genres and I try to read other gamer's opinion of the games (instead of official reviews) to know if they liked a game or dislike it, and why. I feel the gamers I know have a more impact and a proper opinion of a game, because I'm used to know which game they like.

That being said, I don't think someone like me should be reviewing those types of games on a big review site because scores actually matter there, and the scores seen by that many people should really be given by someone who represents the majority of the audience who is likely to want to play the game rather than an outsider.

On the contrary. I think a website which want to cover more people as possible should take in account that there are "newbies" in each genre. Some people like to test new games. There are people beginning their Play-life, we are not all 30 years old with 20 years of gaming background !
To cover more people visiting the website, they should know if you wrote your reviews based on your heavy gaming past, or if you are also a newbie in that genre.

For example, and a very easy one :
You have to review a game you never expected to play : one of the "imagine" series, let's say "My Secret World by Imagine".
You probably won't be a fan of that game due to your own age, your own opinion on shovelware, etc.
But there are A LOT of "imagine" games in the series, which means they are a popular series out there, and a LOT of kids like theses games. If you review it, you will probably think the game is bad and give it a really bad note, but you didn't based your note on the real audience, which are the kids who are used to play these games and actually like what you disliked.
If someone who like these type of games review one of them, the score he/she will give will be more accurate and the reader who want to know if the game is good or bad will have more interest into reading that reviewer's opinion instead of yours. That's why I thought having multiple official reviewer's notes/opinions within a single game's review is better (for small or bigger website alike). Readers just have to know which reviewer has the most chance to provide a trusting, less biased, review for themselves.

A single reviewer system will always be biased because it won't cover every reader's opinions.
It's not biased for you, it's your own opinion, but it will be for readers with a different opinion than yours. you can't please everyone, even if you properly note the game without taking in account money or free games you might earn by lying on the score.


hmm, now I'll go read the thread again, so many replies I skipped!

Edit:
I think a review should also tell if the reviewer completed the game, or just played the first stage/mission/etc.
I would trust more a review if the reviewer completed the game and know what he is talking about.

About the notation, I don't take numbers for granted and I always test the games myself to make my own idea.
I would prefer a system based on comparison instead of a notation based on ... nothing?
When a reviewer gives a note, it's based on what? on other games he liked? on other same gen games? on his self enjoyment? A notation system needs to be based on a concrete value (what is 0, and is 10, etc.)
I think not giving a note at all, is better. it also force the readers to actually read the review and not only read the notes.

 

[Ziko]

Sorry I didn't read through your opening post TC, I just saw a wall of text and my TL;DR function just turned on. So, sorry if I address some things you already stated.

IMO, it seems like reviews have become very skewed during the 7th Gen consoles and it looks like it may get even worse. Take a look at Jimquisition's Hate Out Of Ten and Dragon's Frown videos, and look at the backlash the Escapist and GameSpot recieved after not giving GTA V a 10/10. That's how bad things have gotten. And with the fact that a 5 year old can now access the internet, things are only going to get worse.

One of the modern problems I've seen happening these days is that nearly every reveiwer treats the rating scale like their still in school (anything below 60% is a fail, see Rooster Teeth's Game On PSA, and I know it's a parody). The problem with this is that, if a game receives a score lower than about 70%, it's considered a failure (E.G. Sonic games delisted if they have an average Metacritic score). The problem is that with the increase in this mindset is that people pass-over a lot of good games just because of their score. And the are a lot of good games out there that receive a 5 or a 4 as a rating. Personally, my solution to the problem was to create my own rating scale (10-8: the game is worth owning, 7-5: the game is worth looking into, 4-2: You can find better games but at least give it a try sometime, 1-0: Don't even bother).

As for the solution to gaming bias, I just try to find reviewers who I consider to be unbiased, or, at least, answer questions I want to ask about the game. Just saying, but I think that the most unbiased reviewer on the internet of CGR because all he really does is answer the simple question, "Is this game fun?". Even though I trust him a lot, I'm also subscribed to some other reviewers as well because, not only do I want a second opinion, but I also want to hear what are some problems with the game.

I use the old X-Play 5.5 score system because to me, it's the more easier and informative system. That's how I determine whether or not it's worth playing to me. Also, I look to the guys who've been in the business long enough and they tell me or some folks I know whether that game is worth buy or not. Guys like Adam Sessler and Tommy Tallarico who are movers and shakers in the industry. Then we get guys like AngryJoe who tend to ride on the coattails of people like them and they either botch the review or often give us misinformation about the product and well, that leads to problems later. I'm not trying to slander everybody but that's just how I determine things.

I get the feeling that they don't play the whole game in a lot of cases. They write their review based on only a couple hours of playing. Like a review of their preview.

At any rate, I never read just ONE review for any game, I read many. I like to read the lowest ratings first, it helps temper me against the glowing "gamegasm" reviews. You start to see through the holes in the reviews as you go from low to high ratings. Some of the things mentioned in low reviews aren't mentioned in high reviews, and vice versa. By using the low to high review method, I am rarely disappointed with a game I actually buy. The diversity of the reviews gave me a much better idea of what I'm buying into than going from high to low.

But that's commercial reviews. If I really want to know if a game is my type of game or not is to read GAMER reviews, go to forums dedicated to that game and read-around the site, and don't forget to carefully read the "support" section.

That's very true. Most of the major guys only play maybe 3-4 hours and then determine whether the game is garbage or not.

It does, look how reviewers were treated when GTA V was reviewed and the reviewers didn't give it a 10. Bribed reviews are way more common in Japan than the US which is why I don't like Famitsu reviews.

Famitsu is the king of bribery in Japan. In fact, they gave Medarot DS which is one of my all time favorite DS games, a freaking score of 7 compared to other games like Pokemon Black and White at that time. The excuse was not enough content to maintain a long playthrough of the game as well as a somewhat weak plot. Maybe this explains why 7 was so much better on the 3DS and in fact the events of DS don't even occur in the 3DS game. Nothing but an excuse to tell Natsume to get their shit together and boy they did.

 

[Dunny]

Oh yikes, I never even read about the Kane and Lynch incident. I think I heard of it in an off hand conversation, but DAMN. That's interesting to see.

EDIT: Wait, what someone got FIRED from a poor review score? Wow, that's interesting and surprising, to say the least. I hope to God that I never end up in a space like that. That's awful.

It's been going on for as long as game reviews have existed. I'm sure we all recall the tongue-in-cheek Crash! Parody "Unclear User" which was published when it was proven that a competitor was taking money for positive reviews - Crash! proved it, published their knowledge only to be taken to court and have it all covered up. Every single issue of Crash! still for sale was recalled and destroyed, simply because a software house had more money than the reviewer.

It's sad, but you can't trust any reviews at all, and never have been able to.

D.

 

[Ryukouki]

It's been going on for as long as game reviews have existed. I'm sure we all recall the tongue-in-cheek Crash! Parody "Unclear User" which was published when it was proven that a competitor was taking money for positive reviews - Crash! proved it, published their knowledge only to be taken to court and have it all covered up. Every single issue of Crash! still for sale was recalled and destroyed, simply because a software house had more money than the reviewer.

It's sad, but you can't trust any reviews at all, and never have been able to.

D.

Never had a PS3 so I never go through their reviews. But yeah, its a bit sad that the media can play such influences into scoring.

 

[slingblade1170]

I think major gaming sites review in a very biased way, some are probably paid to review a high score.

 

[Dunny]

Never had a PS3 so I never go through their reviews. But yeah, its a bit sad that the media can play such influences into scoring.

The incident I was talking about occurred in 1985, so it's been around for a very long time - pretty much as long as there's been video game review magazines, in fact.

Back then, of course, being a rather impressionable teenager I trusted mag reviews - Crash! was pretty reliable but even so the odd game turned up that was overrated. What really used to swing me though was standing in a computer shop browsing the boxes and looking at the artwork - if a game was well presented (and looked exciting with decent graphics etc) then I usually bought it. With the most expensive games being £5.99, there was an enormous amount available to me for pocket money.

Nowadays with the increasing lack of brick-and-mortar shops, it's getting to be digital downloads. There's videos and screengrabs to sway my opinion, and with word-of-mouth being so accessible in today's online societies it's easier to choose something I'll like.

So gaming review sites, for me at least, are getting less and less relevant.

But I still miss those magazines and I really miss the shops

D.

 

[KingdomBlade]

I heavily prefer the given system of Film Criticism and Music Criticism when it comes to numbered scores. I think that a score range of 0-10 with decimals is very inaccurate and random. Film criticism and music criticism often places scores of 0 to 5, or 0 to 4; a narrow spectrum of possible scores allow you to weigh better and is also easier to interpret for readers. A score of 4 could be paired with excellent, 3.5 would be great, 3, would be good, 2 would be lackluster, etc. The majority of film reviews display this kind of tone that often coincides with their rating. It's much harder to understand what an "8.2" is supposed to mean when paired with the review. What exactly is the ".2" supposed to be? Why not an 8? How much better is an "8.2" than an "8" or a "7.8?" It just seems really convoluted; it's not as if reviewers can actually use a specific and mathematical way of measuring.

However, the idea that there needs to be a 0-10 scale with decimals is already ingrained in video game reviewers; regardless of it being a very inaccurate way of measuring anything.

While there are plenty of problems with the reviewers themselves, I also find a big problem with the fans. This isn't just true with video games, this is true for all. For example: suppose a reviewer gives a low score to a generally acclaimed game. You can expect fans to ravage that reviewer. As long as the reviewers make valid points, I don't think fans should jump at them like that. I mean, fuck, chill out. You don't have to say the reviewer is a stupid asshole who has no idea what he's doing for having a differing opinion. The worst recent case I found was the Dragon's Crown review. I think that a criticism of the art style as being too exaggerated and slightly trivializes women is perfectly valid. If it detracts from her experience of the game, then there's no problem with that. It's just aggravating.