Metacritic's game scores

Discussion in 'General Gaming Discussion' started by crea, Apr 8, 2015.

  1. crea
    OP

    crea GBAtemp Regular

    Member
    100
    39
    Mar 20, 2013
    Lesotho
    I would like to discuss this topic that has bugged me for some time now. The site Metacritic pretends to be a fair plattform for game scores, as they calculate an average score of independent reviews. But in my opinion there exist plenty problems with how Metacritic gathers its scores.

    1) The professional scores are sometimes translated incorrectly.
    Many sites do not use the 0 to 100 point system, with 100 highest, but a 0 to 5 point grade system, with 5 highest. While it is assumed that decimal points are used to discern finer game scores, this is not the case many of the times. To my experience, normally a very rough score of say 1, 2, 3, 4, then smaller steps of say 4.0 to 5.0 is used. Sometimes not even the 4.0 to 5.0 is diveded, leaving the scores with a crude 1, 2, 3, 4 or 5 scale.
    Now, when translating such a grade system into the point system, taking into account the less detailed scores of the grade system, 5 does in no way always mean 100 points, as 4 does not always mean 80 points and so on. But that is exactly what metacritic does with its scores, just convert them in any way they think is fit. Which is wrong from the beginning.
    A good example for this is the game "Bastion" for PS4.

    2) The scores are added to lists when only single or a handful of reviews exist, thus rendering the "average" aspect of the score system moot.
    This problem exists for both professional and amateur reviews.
    Again, a good example for this is the game "Bastion" for PS4. It was not even released yet, but the few overwhelmingly positive scores it received from professional critics make it an instant ps4 super hit. This adds even further to the issue of bought reviews, as it becomes a stong tool to ramp up game sales (more that one critic at once cannot be wrong, can they?).

    3) For professional scores, even amateur reviews are taken into account.
    There are plenty of gaming review sites which let their users write reviews. Which is perfectly fine. But those reviews should not be added to the professional scores. But it happens a lot of times.

    4) There is no quality control over which and how many professional reviews are added.
    Which in my opinion makes the score system rather dubious and the motives behind the review add system questionable to the point it could be assumed financial motives play a role.
    Some games only have very few reviews added, while many more are available, possibly distorting the overall score if only the more positive or more negative reviews are added.
     
  2. Foxi4

    Foxi4 On the hunt...

    pip Reporter
    23,671
    21,712
    Sep 13, 2009
    Poland
    Gaming Grotto
    Metacritic isn't automatically fair just because it shows aggregate scores - if you have a collection of crooked scores then averaging them gives you a crooked average. By extension, Metascores are just as useless as any other score simply because the basis of Metascores is invalid or at least questionable from the get-go. Numerical scores in general are not a good expression of quality, opinion-based reviews from trustes sources are.
     
    Wellington2k likes this.
  3. RevPokemon

    RevPokemon GBATemp's 3rd Favorite Transgirl

    Member
    4,845
    2,430
    Aug 24, 2014
    United States
    Fort Gay, West Virginia
    Honestly for me metacritic is worthless as when I look to buy a game I don't care about the average score rather I care more about what certain people think (my fav reviewer, youtuber I like, friend)
     
  4. FAST6191

    FAST6191 Techromancer

    pip Reporter
    23,868
    9,780
    Nov 21, 2005
    United Kingdom
    Personally I do not care for review scores in general -- I find it hard to believe a simple number represents my several hundred words and dozen or so hours I probably just sunk into the game to review it.

    Equally there is the whole numbers being weighted thing, or apparently in this universe 7 is the middle number in 0-10. Review aggregators do seem to occasionally account for this (taking reviews from a given publication and seeing where they fall relative to everybody else) which is always amusing to see every argument surrounding it.


    "which let their users write reviews. Which is perfectly fine. But those reviews should not be added to the professional scores."
    Explain.
    Sure "OMG it is pokemon gold, I have pokemon silver and it is worth having both versions. 10/10" is not all that useful. On the other hand "professional games journalist" is a term that amuses me endlessly.

    All this said I am probably not going to add anything new to the conversation that was not the better part of a decade ago in http://www.gamerevolution.com/features/mind_over_meta
     
  5. crea
    OP

    crea GBAtemp Regular

    Member
    100
    39
    Mar 20, 2013
    Lesotho
    I stumbled over some reviews that, after looking into them, were just user reviews but appeared in the "Critic reviews" side. However I did not remember which games had that and was not able to find one again to give an example. If I do, I'll add it.

    Me too! But that's how it is. The gaming industry is bigger than the movie industry now I heard. If this is true or not, it has become big business, so there are also professional journalists who spend their entire time on games. There are millions of additional dollars to make with games receiving good reviews, and journalists on any kind of topic should be knowledgeable and unbiased. In an utopian world.


    Thanks that was a great read and reflects my mixed feelings a lot and in much more detail.
     
  6. RevPokemon

    RevPokemon GBATemp's 3rd Favorite Transgirl

    Member
    4,845
    2,430
    Aug 24, 2014
    United States
    Fort Gay, West Virginia
    The truth is the game review industry like other forms of reporting all are biased in a sense the make articles that are made partially to benefit said company and getting kick backs to praise ideas or products is quite common sadly
     
  7. crea
    OP

    crea GBAtemp Regular

    Member
    100
    39
    Mar 20, 2013
    Lesotho
    I think the 'unbiased' and 'same taste in games' parts as some of you say are much more important than anything else.

    Now I just have to find some gamers online with exactly the same tastes as me :)
     
  8. RevPokemon

    RevPokemon GBATemp's 3rd Favorite Transgirl

    Member
    4,845
    2,430
    Aug 24, 2014
    United States
    Fort Gay, West Virginia
    Best way to do that is see if they rated or like the games you like and if they do then he will be a good influence (like me and the completionist to a sense)
     
  9. Taleweaver

    Taleweaver Storywriter

    Member
    5,694
    1,696
    Dec 23, 2009
    Belgium
    Belgium
    I'm not really sure how to respond to all this. Truth is, crea, I kind of read your post as if you make metacritic into something it is not. To me, it is a gathering place or a portal of different opinions. Of course these opinions are going to differ, and it would be a huge fallacy to think anything else. I might be wrong, but I don't think metacritic themselves don't assume they can truly compare an RTS on a PC with a platformer on wiiu (to name just one clear incomparable thing). If you see it as an attempt to boil down all reviews into one final percentage score, you'd be wrong. One could argue that exactly because it sets all the different opinions together, the end result should be unbiassed...but that doesn't make the process less wrong.

    What do you mean 'sometimes'? You've read reviews, have you not? The score a review gives to it is a sum of factors they weigh independently, based on the preferences of the reviewer and/or the genre and/or the franchise (I may be missing other factors). The whole reason sites use a system is because their readers demand them to (because those often just want a quick glance rather than read an entire article).

    At first I wanted to point out the difference between 'average' and 'median' (which is a valid remark and an important difference in statistics). But with your example, I think your point is more in the sense that once one reviewer has set the tone ("this game is awesome!!!!! :D :D "), other reviewers are more likely to influence their scores based on that review rather than staying completely unbiased. That could be a problem. But again: you should take the reviews as the guidelines, not the end score.

    Hmmm...haven't noticed this, but that would be an issue, indeed. As with the previous points, it can be debunked quickly by checking the actual reviews rather than just follow the scores.

    Each review and each score can be bought, bribed or even be written by the makers themselves. And yes, smaller titles have fewer reviews (there are simply too many games coming out to have every site review every game...let alone it having to be quality as well). Though in truth, I think metacritic is less suspectible to bribery, exactly because reviews come from everywhere. I'll have to pay a huge amount of money on quite some reviewers to actually improve the professional rating, and that very act could backfire if it comes out. It could cause a massive flood of negative user reviews, claiming that the reviewers didn't do their job properly. And I don't know how you are, but if I see a large gap between the user and professional end scores, I refrain from buying until I know more.
     
  10. crea
    OP

    crea GBAtemp Regular

    Member
    100
    39
    Mar 20, 2013
    Lesotho
    I mean if you collect review scores, do it correctly or don't do it at all. While I tried to point out it is not possible to translate one score system directly into another, the article FAST6191 linked argues there isn't even a linear transition used between two different rating systems, but a made-up transition. I assume this was done by metacritic for exactly the reason that the rating systems are not compatible, but it is arbitrary no matter how you do it (linear would at leas be somewhat fair, then if you sensibly set the minimum and maximum values, it could at least be functional; for example an A+ can be 100 points, but an F cannot be 0, as one end of the scale needs to take a score range into account).

    To point 4), it would seem less likely to get bribed if your site offers only a statistic or average score of other reviewers, but as the linked article also concluded, the weighting of the different sources is not published. This negates any credibility and somehow opens the doors again for manipulation (or bribing). Again, the undelying thoughts are at least understandable, not every little online review site or even hobbyist site can be thought equal to the ones established over many years. But in this case for gods sake, publish the weighting.
     
  11. RevPokemon

    RevPokemon GBATemp's 3rd Favorite Transgirl

    Member
    4,845
    2,430
    Aug 24, 2014
    United States
    Fort Gay, West Virginia
    Online issue is if they published the weighting then rivals would have the formula and rip them off
     
  12. vayanui8

    vayanui8 GBAtemp Maniac

    Member
    1,086
    936
    Nov 11, 2013
    United States
    But metacritic is flawless. Reviews never show any bias and people never give bloated or nerfed scores to troll. Anyone who doubts the 100% legitimacy of every metacritic score need only see this to be proven wrong [​IMG]
    In all seriousness though, of course there will always be duds on sites where anyone can post a review, but if you aren't too picky on the numbers it can at least give you a pretty general idea of a games quality or if it was what other people were expecting.
     
    Purple_Shyguy likes this.
  13. Vengenceonu

    Vengenceonu Revenge is beneath me but accidents do happen.

    Member
    2,205
    1,938
    Jun 20, 2013
    United States
    The C Standard Library
    Tbh, all reviews of all types are bullshit. No matter how good a game is, there will always be a group of people who think it's overrated or hate it. Hence people say "[insert game here] is overrated because [insert over used trope here], I can't believe this got [insert accolade here]". This is the reason there's at least 1 dislike on every youtube video after a certain point (like 5000 or so).

    I will say this though, The most accurate reviews are when a game is universally panned. Games that are so abhorrently disgusting that it's only reason to exist is to comment on how disgusting it is... those reviews are glorious.
     
  14. RevPokemon

    RevPokemon GBATemp's 3rd Favorite Transgirl

    Member
    4,845
    2,430
    Aug 24, 2014
    United States
    Fort Gay, West Virginia
    I remember when sonic boom came out and users gave it 10s just to raise the score
     
  15. vayanui8

    vayanui8 GBAtemp Maniac

    Member
    1,086
    936
    Nov 11, 2013
    United States
    thats the general issue. Too many people give 10s or zeroes to really get a precise score. However, unless significant trolling has occurred it seems that most games will get a relatively high score if they are well received and poor scores if they aren't. There are always exceptions though. Anyone who relies on 1 source exclusively to determine their purchases should be prepared to be disappointed though
     
  16. TecXero

    TecXero Technovert

    Member
    2,814
    909
    Apr 13, 2014
    United States
    Mainframe
    Scores seem kind of pointless to me. I don't mind a general system of 1 to 5 (essentially breaking down to "Strongly Not Recommended, Not Recommended, Neutral, Recommended, and Strongly Recommended), as it just gives you a general idea of recommended or not, but trying to boil down an opinion to an accurate score just doesn't work. Games are far too subjective. Yeah, there are games that are universally bad, but when you get above 50%, it becomes a grey area.

    Metacritic might work for user ratings, if the users weren't so black and white about things. The review scores are a joke. If you want a site that will give you a clearer idea of what to expect from a game, I'd recommend something like Giant Bomb. They have both reviews and first impressions. As for metacritic, it would require an entire overhaul and be more user driven while promoting users to look at the games more logically, before I take it seriously.
     
  17. IpsoFact0

    IpsoFact0 GBAtemp Fan

    Member
    428
    85
    Mar 30, 2015
    Canada
    The Six
    Why would you pay attention to the user score anyway? critic score is all that matters
     
  18. RevPokemon

    RevPokemon GBATemp's 3rd Favorite Transgirl

    Member
    4,845
    2,430
    Aug 24, 2014
    United States
    Fort Gay, West Virginia
    Well it can hurt (or help) smaller indie he's who don't got butt load of reviews to balance it
     
  19. Hyro-Sama

    Hyro-Sama I'm from the fucking future.

    Member
    4,260
    3,015
    Oct 25, 2009
    After Earth
    Don't mess with Cory in the House. Cory is a busta.
     
    vayanui8 likes this.
  20. vayanui8

    vayanui8 GBAtemp Maniac

    Member
    1,086
    936
    Nov 11, 2013
    United States
    I suppose, but like I said, you shouldn't rely on it entirely but it can be a solid resource at times