Metacritic's game scores

crea

Well-Known Member
OP
Member
Joined
Mar 20, 2013
Messages
167
Trophies
1
XP
1,019
Country
Lesotho
I would like to discuss this topic that has bugged me for some time now. The site Metacritic pretends to be a fair plattform for game scores, as they calculate an average score of independent reviews. But in my opinion there exist plenty problems with how Metacritic gathers its scores.

1) The professional scores are sometimes translated incorrectly.
Many sites do not use the 0 to 100 point system, with 100 highest, but a 0 to 5 point grade system, with 5 highest. While it is assumed that decimal points are used to discern finer game scores, this is not the case many of the times. To my experience, normally a very rough score of say 1, 2, 3, 4, then smaller steps of say 4.0 to 5.0 is used. Sometimes not even the 4.0 to 5.0 is diveded, leaving the scores with a crude 1, 2, 3, 4 or 5 scale.
Now, when translating such a grade system into the point system, taking into account the less detailed scores of the grade system, 5 does in no way always mean 100 points, as 4 does not always mean 80 points and so on. But that is exactly what metacritic does with its scores, just convert them in any way they think is fit. Which is wrong from the beginning.
A good example for this is the game "Bastion" for PS4.

2) The scores are added to lists when only single or a handful of reviews exist, thus rendering the "average" aspect of the score system moot.
This problem exists for both professional and amateur reviews.
Again, a good example for this is the game "Bastion" for PS4. It was not even released yet, but the few overwhelmingly positive scores it received from professional critics make it an instant ps4 super hit. This adds even further to the issue of bought reviews, as it becomes a stong tool to ramp up game sales (more that one critic at once cannot be wrong, can they?).

3) For professional scores, even amateur reviews are taken into account.
There are plenty of gaming review sites which let their users write reviews. Which is perfectly fine. But those reviews should not be added to the professional scores. But it happens a lot of times.

4) There is no quality control over which and how many professional reviews are added.
Which in my opinion makes the score system rather dubious and the motives behind the review add system questionable to the point it could be assumed financial motives play a role.
Some games only have very few reviews added, while many more are available, possibly distorting the overall score if only the more positive or more negative reviews are added.
 

Foxi4

Endless Trash
Global Moderator
Joined
Sep 13, 2009
Messages
30,818
Trophies
3
Location
Gaming Grotto
XP
29,788
Country
Poland
Metacritic isn't automatically fair just because it shows aggregate scores - if you have a collection of crooked scores then averaging them gives you a crooked average. By extension, Metascores are just as useless as any other score simply because the basis of Metascores is invalid or at least questionable from the get-go. Numerical scores in general are not a good expression of quality, opinion-based reviews from trustes sources are.
 
  • Like
Reactions: Wellington2k

RevPokemon

GBATemp's 3rd Favorite Transgirl
Member
Joined
Aug 24, 2014
Messages
4,839
Trophies
0
Age
27
Location
Fort Gay, West Virginia
XP
2,300
Country
United States
Honestly for me metacritic is worthless as when I look to buy a game I don't care about the average score rather I care more about what certain people think (my fav reviewer, youtuber I like, friend)
 

FAST6191

Techromancer
Editorial Team
Joined
Nov 21, 2005
Messages
36,798
Trophies
3
XP
28,284
Country
United Kingdom
Personally I do not care for review scores in general -- I find it hard to believe a simple number represents my several hundred words and dozen or so hours I probably just sunk into the game to review it.

Equally there is the whole numbers being weighted thing, or apparently in this universe 7 is the middle number in 0-10. Review aggregators do seem to occasionally account for this (taking reviews from a given publication and seeing where they fall relative to everybody else) which is always amusing to see every argument surrounding it.


"which let their users write reviews. Which is perfectly fine. But those reviews should not be added to the professional scores."
Explain.
Sure "OMG it is pokemon gold, I have pokemon silver and it is worth having both versions. 10/10" is not all that useful. On the other hand "professional games journalist" is a term that amuses me endlessly.

All this said I am probably not going to add anything new to the conversation that was not the better part of a decade ago in http://www.gamerevolution.com/features/mind_over_meta
 

crea

Well-Known Member
OP
Member
Joined
Mar 20, 2013
Messages
167
Trophies
1
XP
1,019
Country
Lesotho
"which let their users write reviews. Which is perfectly fine. But those reviews should not be added to the professional scores."
Explain.
I stumbled over some reviews that, after looking into them, were just user reviews but appeared in the "Critic reviews" side. However I did not remember which games had that and was not able to find one again to give an example. If I do, I'll add it.

Sure "OMG it is pokemon gold, I have pokemon silver and it is worth having both versions. 10/10" is not all that useful. On the other hand "professional games journalist" is a term that amuses me endlessly.
Me too! But that's how it is. The gaming industry is bigger than the movie industry now I heard. If this is true or not, it has become big business, so there are also professional journalists who spend their entire time on games. There are millions of additional dollars to make with games receiving good reviews, and journalists on any kind of topic should be knowledgeable and unbiased. In an utopian world.


All this said I am probably not going to add anything new to the conversation that was not the better part of a decade ago in http://www.gamerevolution.com/features/mind_over_meta
Thanks that was a great read and reflects my mixed feelings a lot and in much more detail.
 

RevPokemon

GBATemp's 3rd Favorite Transgirl
Member
Joined
Aug 24, 2014
Messages
4,839
Trophies
0
Age
27
Location
Fort Gay, West Virginia
XP
2,300
Country
United States
I stumbled over some reviews that, after looking into them, were just user reviews but appeared in the "Critic reviews" side. However I did not remember which games had that and was not able to find one again to give an example. If I do, I'll add it.


Me too! But that's how it is. The gaming industry is bigger than the movie industry now I heard. If this is true or not, it has become big business, so there are also professional journalists who spend their entire time on games. There are millions of additional dollars to make with games receiving good reviews, and journalists on any kind of topic should be knowledgeable and unbiased. In an utopian world.

The truth is the game review industry like other forms of reporting all are biased in a sense the make articles that are made partially to benefit said company and getting kick backs to praise ideas or products is quite common sadly
 

crea

Well-Known Member
OP
Member
Joined
Mar 20, 2013
Messages
167
Trophies
1
XP
1,019
Country
Lesotho
I think the 'unbiased' and 'same taste in games' parts as some of you say are much more important than anything else.

Now I just have to find some gamers online with exactly the same tastes as me :)
 

RevPokemon

GBATemp's 3rd Favorite Transgirl
Member
Joined
Aug 24, 2014
Messages
4,839
Trophies
0
Age
27
Location
Fort Gay, West Virginia
XP
2,300
Country
United States
I think the 'unbiased' and 'same taste in games' parts as some of you say are much more important than anything else.

Now I just have to find some gamers online with exactly the same tastes as me and a knack for expressing their opinions :)
Best way to do that is see if they rated or like the games you like and if they do then he will be a good influence (like me and the completionist to a sense)
 

Taleweaver

Storywriter
Member
Joined
Dec 23, 2009
Messages
8,685
Trophies
2
Age
43
Location
Belgium
XP
8,066
Country
Belgium
I'm not really sure how to respond to all this. Truth is, crea, I kind of read your post as if you make metacritic into something it is not. To me, it is a gathering place or a portal of different opinions. Of course these opinions are going to differ, and it would be a huge fallacy to think anything else. I might be wrong, but I don't think metacritic themselves don't assume they can truly compare an RTS on a PC with a platformer on wiiu (to name just one clear incomparable thing). If you see it as an attempt to boil down all reviews into one final percentage score, you'd be wrong. One could argue that exactly because it sets all the different opinions together, the end result should be unbiassed...but that doesn't make the process less wrong.

I would like to discuss this topic that has bugged me for some time now. The site Metacritic pretends to be a fair plattform for game scores, as they calculate an average score of independent reviews. But in my opinion there exist plenty problems with how Metacritic gathers its scores.

1) The professional scores are sometimes translated incorrectly.
Many sites do not use the 0 to 100 point system, with 100 highest, but a 0 to 5 point grade system, with 5 highest. While it is assumed that decimal points are used to discern finer game scores, this is not the case many of the times. To my experience, normally a very rough score of say 1, 2, 3, 4, then smaller steps of say 4.0 to 5.0 is used. Sometimes not even the 4.0 to 5.0 is diveded, leaving the scores with a crude 1, 2, 3, 4 or 5 scale.
Now, when translating such a grade system into the point system, taking into account the less detailed scores of the grade system, 5 does in no way always mean 100 points, as 4 does not always mean 80 points and so on. But that is exactly what metacritic does with its scores, just convert them in any way they think is fit. Which is wrong from the beginning.
A good example for this is the game "Bastion" for PS4.
What do you mean 'sometimes'? You've read reviews, have you not? The score a review gives to it is a sum of factors they weigh independently, based on the preferences of the reviewer and/or the genre and/or the franchise (I may be missing other factors). The whole reason sites use a system is because their readers demand them to (because those often just want a quick glance rather than read an entire article).

2) The scores are added to lists when only single or a handful of reviews exist, thus rendering the "average" aspect of the score system moot.
This problem exists for both professional and amateur reviews.
Again, a good example for this is the game "Bastion" for PS4. It was not even released yet, but the few overwhelmingly positive scores it received from professional critics make it an instant ps4 super hit. This adds even further to the issue of bought reviews, as it becomes a stong tool to ramp up game sales (more that one critic at once cannot be wrong, can they?).
At first I wanted to point out the difference between 'average' and 'median' (which is a valid remark and an important difference in statistics). But with your example, I think your point is more in the sense that once one reviewer has set the tone ("this game is awesome!!!!! :D :D "), other reviewers are more likely to influence their scores based on that review rather than staying completely unbiased. That could be a problem. But again: you should take the reviews as the guidelines, not the end score.

3) For professional scores, even amateur reviews are taken into account.
There are plenty of gaming review sites which let their users write reviews. Which is perfectly fine. But those reviews should not be added to the professional scores. But it happens a lot of times.
Hmmm...haven't noticed this, but that would be an issue, indeed. As with the previous points, it can be debunked quickly by checking the actual reviews rather than just follow the scores.

4) There is no quality control over which and how many professional reviews are added.
Which in my opinion makes the score system rather dubious and the motives behind the review add system questionable to the point it could be assumed financial motives play a role.
Some games only have very few reviews added, while many more are available, possibly distorting the overall score if only the more positive or more negative reviews are added.

Each review and each score can be bought, bribed or even be written by the makers themselves. And yes, smaller titles have fewer reviews (there are simply too many games coming out to have every site review every game...let alone it having to be quality as well). Though in truth, I think metacritic is less suspectible to bribery, exactly because reviews come from everywhere. I'll have to pay a huge amount of money on quite some reviewers to actually improve the professional rating, and that very act could backfire if it comes out. It could cause a massive flood of negative user reviews, claiming that the reviewers didn't do their job properly. And I don't know how you are, but if I see a large gap between the user and professional end scores, I refrain from buying until I know more.
 

crea

Well-Known Member
OP
Member
Joined
Mar 20, 2013
Messages
167
Trophies
1
XP
1,019
Country
Lesotho
I mean if you collect review scores, do it correctly or don't do it at all. While I tried to point out it is not possible to translate one score system directly into another, the article FAST6191 linked argues there isn't even a linear transition used between two different rating systems, but a made-up transition. I assume this was done by metacritic for exactly the reason that the rating systems are not compatible, but it is arbitrary no matter how you do it (linear would at leas be somewhat fair, then if you sensibly set the minimum and maximum values, it could at least be functional; for example an A+ can be 100 points, but an F cannot be 0, as one end of the scale needs to take a score range into account).

To point 4), it would seem less likely to get bribed if your site offers only a statistic or average score of other reviewers, but as the linked article also concluded, the weighting of the different sources is not published. This negates any credibility and somehow opens the doors again for manipulation (or bribing). Again, the undelying thoughts are at least understandable, not every little online review site or even hobbyist site can be thought equal to the ones established over many years. But in this case for gods sake, publish the weighting.
 

RevPokemon

GBATemp's 3rd Favorite Transgirl
Member
Joined
Aug 24, 2014
Messages
4,839
Trophies
0
Age
27
Location
Fort Gay, West Virginia
XP
2,300
Country
United States
I mean if you collect review scores, do it correctly or don't do it at all. While I tried to point out it is not possible to translate one score system directly into another, the article FAST6191 linked argues there isn't even a linear transition used between two different rating systems, but a made-up transition. I assume this was done by metacritic for exactly the reason that the rating systems are not compatible, but it is arbitrary no matter how you do it (linear would at leas be somewhat fair, then if you sensibly set the minimum and maximum values, it could at least be functional; for example an A+ can be 100 points, but an F cannot be 0, as one end of the scale needs to take a score range into account).

To point 4), it would seem less likely to get bribed if your site offers only a statistic or average score of other reviewers, but as the linked article also concluded, the weighting of the different sources is not published. This negates any credibility and somehow opens the doors again for manipulation (or bribing). Again, the undelying thoughts are at least understandable, not every little online review site or even hobbyist site can be thought equal to the ones established over many years. But in this case for gods sake, publish the weighting.
Online issue is if they published the weighting then rivals would have the formula and rip them off
 

vayanui8

Well-Known Member
Member
Joined
Nov 11, 2013
Messages
1,086
Trophies
0
XP
907
Country
United States
But metacritic is flawless. Reviews never show any bias and people never give bloated or nerfed scores to troll. Anyone who doubts the 100% legitimacy of every metacritic score need only see this to be proven wrong
fd84dce9d32dd8d5fe22c498bc8b432b-metacritic-says-cory-in-the-house-ds-is-almost-a-perfect-game.jpg

In all seriousness though, of course there will always be duds on sites where anyone can post a review, but if you aren't too picky on the numbers it can at least give you a pretty general idea of a games quality or if it was what other people were expecting.
 
  • Like
Reactions: Purple_Shyguy

Vengenceonu

Revenge is beneath me but accidents do happen.
Member
Joined
Jun 20, 2013
Messages
2,211
Trophies
2
Location
The C Standard Library
XP
2,326
Country
United States
Tbh, all reviews of all types are bullshit. No matter how good a game is, there will always be a group of people who think it's overrated or hate it. Hence people say "[insert game here] is overrated because [insert over used trope here], I can't believe this got [insert accolade here]". This is the reason there's at least 1 dislike on every youtube video after a certain point (like 5000 or so).

I will say this though, The most accurate reviews are when a game is universally panned. Games that are so abhorrently disgusting that it's only reason to exist is to comment on how disgusting it is... those reviews are glorious.
 

RevPokemon

GBATemp's 3rd Favorite Transgirl
Member
Joined
Aug 24, 2014
Messages
4,839
Trophies
0
Age
27
Location
Fort Gay, West Virginia
XP
2,300
Country
United States
But metacritic is flawless. Reviews never show any bias and people never give bloated or nerfed scores to troll. Anyone who doubts the 100% legitimacy of every metacritic score need only see this to be proven wrong
fd84dce9d32dd8d5fe22c498bc8b432b-metacritic-says-cory-in-the-house-ds-is-almost-a-perfect-game.jpg

In all seriousness though, of course there will always be duds on sites where anyone can post a review, but if you aren't too picky on the numbers it can at least give you a pretty general idea of a games quality or if it was what other people were expecting.
I remember when sonic boom came out and users gave it 10s just to raise the score
 

vayanui8

Well-Known Member
Member
Joined
Nov 11, 2013
Messages
1,086
Trophies
0
XP
907
Country
United States
I remember when sonic boom came out and users gave it 10s just to raise the score
thats the general issue. Too many people give 10s or zeroes to really get a precise score. However, unless significant trolling has occurred it seems that most games will get a relatively high score if they are well received and poor scores if they aren't. There are always exceptions though. Anyone who relies on 1 source exclusively to determine their purchases should be prepared to be disappointed though
 

TecXero

Technovert
Member
Joined
Apr 13, 2014
Messages
2,810
Trophies
0
Location
Mainframe
XP
1,040
Country
United States
Scores seem kind of pointless to me. I don't mind a general system of 1 to 5 (essentially breaking down to "Strongly Not Recommended, Not Recommended, Neutral, Recommended, and Strongly Recommended), as it just gives you a general idea of recommended or not, but trying to boil down an opinion to an accurate score just doesn't work. Games are far too subjective. Yeah, there are games that are universally bad, but when you get above 50%, it becomes a grey area.

Metacritic might work for user ratings, if the users weren't so black and white about things. The review scores are a joke. If you want a site that will give you a clearer idea of what to expect from a game, I'd recommend something like Giant Bomb. They have both reviews and first impressions. As for metacritic, it would require an entire overhaul and be more user driven while promoting users to look at the games more logically, before I take it seriously.
 

RevPokemon

GBATemp's 3rd Favorite Transgirl
Member
Joined
Aug 24, 2014
Messages
4,839
Trophies
0
Age
27
Location
Fort Gay, West Virginia
XP
2,300
Country
United States
thats the general issue. Too many people give 10s or zeroes to really get a precise score. However, unless significant trolling has occurred it seems that most games will get a relatively high score if they are well received and poor scores if they aren't. There are always exceptions though. Anyone who relies on 1 source exclusively to determine their purchases should be prepared to be disappointed though
Well it can hurt (or help) smaller indie he's who don't got butt load of reviews to balance it
 

Site & Scene News

Popular threads in this forum

General chit-chat
Help Users
  • S @ salazarcosplay:
    @BigOnYa there were relevant issues before south park was made, that's why i think a south park prequel/spinoff would be great. Randy and his friends in their child hood
    +1
  • BigOnYa @ BigOnYa:
    Yea, like them running in high school together, getting into stuff, and how they got hitched and had kids. And how the town of South Park was back then compared to now. That would be cool to see.
  • BakerMan @ BakerMan:
    yeah
  • The Real Jdbye @ The Real Jdbye:
    @salazarcosplay if they made a prequel, it would still be about current issues, cause it doesn't make sense to make it about stuff that happened 30 years ago that nobody cares about anymore
  • The Real Jdbye @ The Real Jdbye:
    it's too late
  • The Real Jdbye @ The Real Jdbye:
    the older south park episodes about particular issues usually age poorly since the topic is no longer relevant
  • The Real Jdbye @ The Real Jdbye:
    an exception is giant douche vs turd sandwich, that's always relevant :P
    +1
  • K3Nv2 @ K3Nv2:
    I was gone for like an hour and none of you thought to write or call pos
  • BigOnYa @ BigOnYa:
    We knew you were going to Sonic to get lunch.
  • K3Nv2 @ K3Nv2:
    Sonics fast I would've been home in 10 mins
  • BigOnYa @ BigOnYa:
    Meet and greet with AncientBoi then?
  • K3Nv2 @ K3Nv2:
    That would've gone slow he's old
    +1
  • ZeroT21 @ ZeroT21:
    sadly the person in question feels too young for his own good
  • K3Nv2 @ K3Nv2:
    We don't question people
  • ZeroT21 @ ZeroT21:
    me neither, i just bash them
  • K3Nv2 @ K3Nv2:
    We just question @AncientBoi
  • ZeroT21 @ ZeroT21:
    it wasn't a question, it was fact
  • BigOnYa @ BigOnYa:
    He said he had 3 different doctors apt this week, so he prob there. Something about gerbal extraction, I don't know.
    +1
  • ZeroT21 @ ZeroT21:
    bored, guess i'll spread more democracy
  • LeoTCK @ LeoTCK:
    @K3Nv2 one more time you say such bs to @BakerMan and I'll smack you across the whole planet
  • K3Nv2 @ K3Nv2:
    Make sure you smack my booty daddy
    +1
  • LeoTCK @ LeoTCK:
    telling him that my partner is luke...does he look like someone with such big ne
    eds?
  • LeoTCK @ LeoTCK:
    do you really think I could stand living with someone like luke?
  • LeoTCK @ LeoTCK:
    I suppose luke has "special needs" but he's not my partner, did you just say that to piss me off again?
    LeoTCK @ LeoTCK: I suppose luke has "special needs" but he's not my partner, did you just say that to piss me off...