Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

The Problem With Metacritic

Soulskill posted about 2 years ago | from the saturday-morning-game-developers dept.

Games 131

Metacritic has risen to a position of prominence in the gaming community — but is it given more credit than it's due? This article delves into some of the problems with using Metacritic as a measure of quality or success. Quoting: "The scores used to calculate the Metascore have issues before they are even averaged. Metacritic operates on a 0-100 scale. While it's simple to convert some scores into this scale (if it's necessary at all), others are not so easy. 1UP, for example, uses letter grades. The manner in which these scores should be converted into Metacritic scores is a matter of some debate; Metacritic says a B- is equal to a 67 because the grades A+ through F- have to be mapped to the full range of its scale, when in reality most people would view a B- as being more positive than a 67. This also doesn't account for the different interpretation of scores that outlets have -- some treat 7 as an average score, which I see as a problem in an of itself, while others see 5 as average. Trying to compensate for these variations is a nigh-impossible task and, lest we forget, Metacritic will assign scores to reviews that do not provide them. ... The act of simplifying reviews into a single Metascore also feeds into a misconception some hold about reviews. If you browse into the comments of a review anywhere on the web (particularly those of especially big games), you're likely to come across those criticizing the reviewer for his or her take on a game. People seem to mistaken reviews as something which should be 'objective.' 'Stop giving your opinion and tell us about the game' is a notion you'll see expressed from time to time, as if it is the job of a reviewer to go down a list of items that need to be addressed — objectively! — and nothing else."

cancel ×
This is a preview of your comment

No Comment Title Entered

Anonymous Coward 1 minute ago

No Comment Entered


But it's all subjective anyway. (3, Interesting)

TheoGB (786170) | about 2 years ago | (#40683097)

Personally I think anything less than 7 out of 10 isn't worth my while bothering with. That's me and about time I have. Friends of mine, however, would give a film a 5 out of 10 and say it's still decent enough to stick on one night when you want something to watch. Even if Metacritic was exactly showing a score that we agreed was 'accurate' it wouldn't really matter. Aggregation of this sort is as good as doing it by eye yourself, surely?

Re:But it's all subjective anyway. (1)

Anonymous Coward | about 2 years ago | (#40683161)

It really doesn't matter what scale you use or how you weight them. If you add enough similarly-weighted ratings together, the central limit theorem essentially guarantees that the result will have a normal distribution. So in the end, the mean is irrelevant; it's really only the (score-mean)/stdev that matters.

Re:But it's all subjective anyway. (4, Funny)

Razgorov Prikazka (1699498) | about 2 years ago | (#40683175)

Obligatory XKCD reference: http://xkcd.com/937/

Re:But it's all subjective anyway. (1)

Anonymous Coward | about 2 years ago | (#40683313)

For the love of Baal, please at least use <a href [xkcd.com], m'kay?

Re:But it's all subjective anyway. (1)

SpooForBrains (771537) | about 2 years ago | (#40683729)

Does anyone else play the "Predict which XKCD this will be" game when Oblig XKCD links are posted?

(Those that don't know the strip numbers off by heart of course)

Re:But it's all subjective anyway. (3, Funny)

Razgorov Prikazka (1699498) | about 2 years ago | (#40683921)

Yeah, I surely hope that Randall Munroe makes a cartoon on the RaspberryPi or Bitcoins, that would make the prediction a whole lot easier on a lot of /. stories :-)

Re:But it's all subjective anyway. (1)

mooingyak (720677) | about 2 years ago | (#40685165)

Does anyone else play the "Predict which XKCD this will be" game when Oblig XKCD links are posted?

(Those that don't know the strip numbers off by heart of course)

Every time. I don't have the numbers memorized, but I do recognize 'recent' vs 'not recent' and that helps sometimes.

Re:But it's all subjective anyway. (1)

Anonymous Coward | about 2 years ago | (#40683469)

Nobody rates lower than 0 or higher than 100 so it can't be a normal distribution because the tail would be cut off. This would primarily be a problem for large standard deviations or extreme scores.

Re:But it's all subjective anyway. (1)

jellomizer (103300) | about 2 years ago | (#40684931)

The problem is a lack of agreement on what the median should be.

A normal distribution works when we know what the average value is (The peak of the distribution bubble) However we need a normal distribution on what the people think the normal is first to come up with the best median for the distribution of grades for a product.

Re:But it's all subjective anyway. (2)

mooingyak (720677) | about 2 years ago | (#40685183)

Nobody rates lower than 0 or higher than 100 so it can't be a normal distribution because the tail would be cut off. This would primarily be a problem for large standard deviations or extreme scores.

Not just that, but there are numbers in the low range that just aren't going to get used. Who rates something as a 3 out of 100? What does that mean? It was absolutely terrible except for one relatively minor part that was done right?

Re:But it's all subjective anyway. (0)

Anonymous Coward | about 2 years ago | (#40684007)

I don't think video game ratings are independent variables. I'm not sure about identically distributed either, but the independent one is a biggie.

Is that so? (2)

Warma (1220342) | about 2 years ago | (#40683271)

This line of thought seems faulty, but I have to admit that I feel the same way for most scores reviewing sites and magazines deal out. A score of less than 7 out of 10 is reserved for seriously failed works, and typically these works never merit a recommendation. However, if anything below 7 is not worth experiencing, you essentially only have five possible scores: bad, 7, 8, 9 and 10. The rest of the scale is simply wasted.

I always wondered if this is caused by school grades in youth influencing what people think as acceptable ratings. For example in Finland, grade school grades go from 4 (failed) to 10 (best), where 7 ends up being an average score and anything below it is considered poor.

Re:Is that so? (1)

TheoGB (786170) | about 2 years ago | (#40683499)

Certainly, I'm not suggesting it's logical for me to feel that way. But it's just how I am. With music I am more open than I was 10-15 years ago and I think it's all about how much time I have: I can stream music for free (legally) all day at work so I have a lot more time to check it out for myself, but I only get a few hours a day to spend watching films and reading. This means I don't really have time to risk watching a film rated 5 (unless clearly there is a personal rec, etc.) when I could be sifting through those 7-10s for something I might really love.

Re:Is that so? (5, Interesting)

SpooForBrains (771537) | about 2 years ago | (#40683753)

I work for a review platform. We have decided that you only really need four ratings, Bad, Poor, Good, Excellent. We don't have a neutral option because really neutral tends to mean bad.

Of course, quite a lot of our users (and our marketing department) seem to prefer stars. Because an arbitrary scale is so much more useful that simply saying what you think of something. Apparently.

Re:Is that so? (1)

Warma (1220342) | about 2 years ago | (#40683819)

Might I ask what you review and for a link to the site (if there is one)? It would be nice to check out reviews using this philosophy.

Re:Is that so? (0)

Anonymous Coward | about 2 years ago | (#40685059)

I blog music reviews. I use a binary rating system : BUY or DON'T BUY
It's all that matters...

Re:But it's all subjective anyway. (0)

Anonymous Coward | about 2 years ago | (#40683349)

Scores are useless when deciding what to watch, play, or listen. At least if you don't know the tastes and preferences of the person scoring it. If you know your tastes match that of the critic, the his recommendations have some value, in some other cases you just know because some critic hates the movie or game you are most likely going to enjoy it.

Re:But it's all subjective anyway. (-1)

Anonymous Coward | about 2 years ago | (#40684627)

Congratulations on being a weak-minded sheep.

Re:But it's all subjective anyway. (1, Offtopic)

jellomizer (103300) | about 2 years ago | (#40685009)

We often think of using the school level of grading.
  60 F (0.0)
60-69 D (1.0)
70-79 C (2.0)
80-89 B (3.0)
90-100 A (4.0).

Of course school grading has gotten corrupted to just mean. Less then C you don't go to college. C-B means you can go to a normal college, A you go to a well known college.

However it should be the following...
F is a Failure where they should take the class over again.
D They can advance to the next class... However they are at high risk of failing the next class, you should have the option to retake the class over again
C They passed they should advance to the next class.
B They Mastered the content in the class, and show a strong understanding in it.
A The class was too easy for them they should be advanced to a harder system.

I have found classes I got a C in are the ones I learned the most in.
the classes I got A's in I already knew most of the information and I learned very little.


Anonymous Coward | about 2 years ago | (#40683109)

We got gallons and inches and feet and yards !!

Solve it with Machine Learning (2)

LittleImp (1020687) | about 2 years ago | (#40683115)

This would be perfect for machine learning. Just analyze _all_ the scores from a certain source and calculate a most probable score with a standard deviation. Then assign a score from 0-100 accordingly. I don't know that much about machine learning so I'm sure an expert could find a way better algorithm for that.

"Trying to compensate for these variations is a nigh-impossible task" - definitely not.

Re:Solve it with Machine Learning (4, Interesting)

bWareiWare.co.uk (660144) | about 2 years ago | (#40683495)

Your description is not so much machine learning as basic math. If you just want each scoring system to have equal weight on the results then compensating for the variation is trivial.

Where machine learning would come in is to find underlying pattens in the data.

This could be used to weed out reviewers to lazily copy scores, or are subject to influence.

It would also allow them to test your scores for some games against the population of reviewers to find reviewers with similar tastes.

You could also use clustering algorithms to find niche games which got a few really strong scores but whose average was really pulled down because they don't have wide appeal.

Just use a bell curve (5, Insightful)

wjh31 (1372867) | about 2 years ago | (#40683151)

sounds in principle like a fairly simple solution. Put together a separate histogram of the scores by each reviewer. From this you can estimate what an average score really is and how many standard deviations an individual score is above or below. The meta-score then becomes the average number of sigma's the game is above or below the various averages. If necessary this score can be sanitised to something easier to read for those less familiar with Gaussian statistics

Re:Just use a bell curve (3, Insightful)

fph il quozientatore (971015) | about 2 years ago | (#40683283)

My thoughts exactly. This is not rocket science; it is sad to read that no one in their whole company seems to have a clue about statistics.

Re:Just use a bell curve (2)

Anonymous Coward | about 2 years ago | (#40683305)

Well it is extremely likely that Metacritic has a system like this implemented, but the article is just ignorant.

Re:Just use a bell curve (0)

Anonymous Coward | about 2 years ago | (#40683301)

No. This could be a bad idea. Suppose you only bothered to rate movies you liked. A bell-curve would be a very bad fit to their actual experience.

Bell curve doesn't work that well either... (5, Insightful)

F69631 (2421974) | about 2 years ago | (#40683329)

One reviewer might only rate highly hyped games which he expects to be good (nearly all fall to 60-100 range) and other reviewer tries out pretty much everything he encounters to find out those lone gems among less well-known indie games, etc. (let's say ranging from 20 to 95). We can't just take a bell curve of each and say "Game A is slightly above average on first reviewer's scale and Game B is slightly above average on the second reviewer's scale... so they're probably about equally good!". Sure, with large number of reviewers, you can still see which games do well and which won't but you have lost at least as much precision as you would have if you hadn't taken the bell curve in the first place.

That said, I don't know if reviews are that relevant anymore. I am active gamer but don't remember when was the last time I read a full review... There have been two times recently when I bought newer games from series I had played years ago (Cossacks and Anno 1602). I just wanted to take a quick peek on whether the games were considered about equally good, better or worse than the ones I had liked and whether they were very similar with just better graphics etc. or if some major concept had changed. That consisted mostly of looking the games up on Wikipedia and quickly glancing the first reviews I found using Google. I think I also checked the metascore, but it was more among the lines of "I'll buy it unless it turns out to have metascore under 60 or something". I didn't use that as exact metric.

Most games I buy are ones recommended to me by my friends, those recommended by blogs I follow (e.g., the Penny Arcade guys' news feed... you could consider those reviews, but they don't mention the games they hated, don't give scores, etc., just mention "Hey, that was pretty good. Try it out.") or those that just seem fun and don't cost much (When I noticed Orcs Must Die on Steam for under 5 euros, I didn't start doing extensive research on the critical acclaim of the game.)

Re:Bell curve doesn't work that well either... (1)

Deltaspectre (796409) | about 2 years ago | (#40684615)

I was thinking about this yesterday when I was looking through CmdrTaco's reddit AMA. If we wanted a truly fair system, couldn't we just do relational ratings? (It would probably be too much work server side, but it sure sounds good in my mind.) Have each reviewer rate a game in comparison to another game. Portal 1 better than 18 Wheels of Steel. Team Fortress 2 better than Data Jammers: FastForward. With enough people, everything would end up somewhere on the scale and it wouldn't all be squashed up towards the top half that it is now for half decent games.

Re:Just use a bell curve (0)

Anonymous Coward | about 2 years ago | (#40683365)

Right. I'm sure no one at a company whose business model is statistics remembers this stuff from grade 12 and that college course they got a "C" in.

Re:Just use a bell curve (0)

Anonymous Coward | about 2 years ago | (#40683385)

How can you be sure that reviewer doesn't review only good movies?

Re:Just use a bell curve (1)

Relic of the Future (118669) | about 2 years ago | (#40683451)

Go ahead and do all the math and I'll bet that 95% of scores move less than 5%. There's really very little wrong with a simple average.

Scope of a decision. (0)

Anonymous Coward | about 2 years ago | (#40683173)

Except most people use more than just the numeric score to make a gaming decision. There's also both the professional reviewers and the regular gaming public and their reviews.

FTA (0)

Anonymous Coward | about 2 years ago | (#40683227)

This might all sound unimportant in terms of how it affects the industry, but as noted above, it's not just gamers who look at Metacritic. Publishers do, too, and in some cases they rely on these scores too heavily, as evidenced by well-publicized stories about bonuses being tied to Metacritic. Most famously, Obsidian's Chris Avellone revealed on Twitter earlier this year that the developer missed out on receiving a bonus for its work on Fallout: New Vegas, which it was not entitled to royalties on, because it failed to reach the required Metascore. 85+ was what was required to receive the bonus; the game ended up at 84.

Metacritic needs to die for the simple fact that the giant steaming pile of monkey shit that is The Elder Scrolls: Oblivion should get a 94. Stories like the above quote just add more reasons for it to disappear.

Re:FTA (1)

anne on E. mouse cow (867445) | about 2 years ago | (#40683839)

Or perhaps you are the 1 in a hundred person who hates Oblivion when everyone else, myself included thinks the game is brilliant, I think it was more fun than Skyrim (fights too balanced and orchestrated - you can always win).

No, of course you're right, the rest of the world is delusional and millions of people just imagined having great fun playing Oblivion.

Why do you hate it anyway?

Re:FTA (0)

Anonymous Coward | about 2 years ago | (#40684077)

Oblivion gets too boring too quickly. Honestly, after opening the second gate is just rinse and repeat ad nauseum. I usually try to win every game I own, but in the case of Oblivion didn't even felt bad for not doing so. If boring, repetitive quests and lack of challenge is your thing, sure you'll have a great fun playing Oblivion, for me it was just a boring game with nice graphics.

And FYI, I'm not the guy above, I'm just giving a second opinion.

Re:FTA (1)

anne on E. mouse cow (867445) | about 2 years ago | (#40684417)

If boring, repetitive quests and lack of challenge is your thing, sure you'll have a great fun playing Oblivion.

So you didn't do the Mages Guild quests then, or the divines quests, a lot of uniqueness there. If you rushed to complete the game's main storyline then you're doing it wrong.

So if Oblivion is not interesting then what single player game is?

Re:FTA (1)

Lord_Naikon (1837226) | about 2 years ago | (#40684703)

I too found Oblivion to be boring. Once I set foot outside the starting dungeon I felt lost, without sense of purpose or direction. I liked skyrim a whole lot more due to its faster pacing and sense of urgency. The improved graphic fidelity also helped a lot. Other open world games that worked for me were the new Fallout series. Another example of an interesting single player game is obviously (the golden standard) Baldur's Gate 2, where none of the encounters are scaled and everything is unique.

Re:FTA (0)

Anonymous Coward | about 2 years ago | (#40685499)

Personally, I liked Oblivion, did almost all of the side quests, but I never finished the main storyline since it was less interesting than any of the side quests.

Re:FTA (0)

Anonymous Coward | about 2 years ago | (#40685963)

This is. [steampowered.com] Until Oblivion is modded with tracks, it will never be worth a 94.

Re:FTA (0)

Anonymous Coward | about 2 years ago | (#40686051)

I did some of them and it where the same thing really. As for other titles, Mass Effect \ ME2, SWKOTOR, the Bard's Tale or even the good ol' Nox were WAY better. Some of those titles were based on the same rinse and repeat philosophy, but the story line, narrative or character development saved them, something that Oblivion missed completely.

I see a tendency in newer RPGs to try to emulate the mechanic of successful online games like WoW, missing completely the fact that the success of those comes from the social elements that single player lacks. Personally I'd love to see a remake of the Ultima 4: Quest of Avatar, Entomorph: Plague of the Darkfall or similar innovative old school games using the Oblivion engine than the boring titles we've been getting of late.

Re:FTA (1)

Beyond_GoodandEvil (769135) | about 2 years ago | (#40684743)

Metacritic needs to die for the simple fact that the giant steaming pile of monkey shit that is The Elder Scrolls: Oblivion should get a 94. Stories like the above quote just add more reasons for it to disappear.
While I never played Oblivion, I recall the buggy mess that was its predecessor Daggerfall, so I understand the feeling. However, metacritic did help illustrate in stark contrast what a racket those gaming publications ratings were vs what the gen. public thought(I'm looking at you steamy pile of shit that was Dragon Age II). As for Chris Avellone, life lesson don't tie your bonus to shit you can't control, as a person who has worked in R&D, you can get burned if your bonus is based on other people's efforts. Btw, loved Fallout New Vegas and the DLCs(but that damn gamebryo engine is a buggy mess), but I am surprised FO:NV only got 84. Oh, shit DA:II has an 82 wtf? Yeah, FO:NV is much better than 84.

Hmmm? (1)

Lando (9348) | about 2 years ago | (#40683231)

Metacritic a scoring system that works for a lot of people and seems to work for them isn't perfect? News at 11. Really? So someone writes an opinion piece on it backed with opinion and this is interesting? The methods Metacritic use seem to be fair and work, so who cares that someone doesn't think they are perfect?

I'll use the numbers as a guideline, but not as fact. Just like wikipedia, it's a place to start. Now if the article was aimed at pointing out that the publishers put too much emphasis on the metacritic score, then there should have been more documentation from that side of things. As it is, it's just someone standing back and saying this system sucks and suggesting that it needs to be scraped without offering any constructive advice on how to fix it. Just like people that complain about the government, but does that mean we should have no government and live in anarchy? Or perhaps wouldn't it serve customers better to try and figure out how to fix metacritics problems rather than just complaining that metacritic has too much power because people find it useful?

Re:Hmmm? (1)

fluffythedestroyer (2586259) | about 2 years ago | (#40684289)

The problem is not metacritic, it's the people who read it are. They seem to take this whole critic thing very personal. but like you said, it's not perfect but people don't seem to get that. On top of that, the metacritic score can be "hacked" in some way to my knowledge just like any other scores you use average in. Just think about it, it's simple math, give a game a super bad review and another one a good review and you get a decent score. I don't think this is good if you ask me.

I don't get the point of it anyway (3, Insightful)

Riceballsan (816702) | about 2 years ago | (#40683241)

For me anyway, when it comes to finding good reviews of things, I've always found a mass average entirely useless. Just because 10,000 out of 15,000 like something, it has no bearing over if I will like it and quite often leads to the oposite. Instead what I do is I check reviews of the games movies shows etc... that I already have seen, I find the reviewers that are the closest match to how I felt about things in the past. Then I check their reviews of what I haven't seen. It isn't a perfect system, but it works overall, and tends to be more accurate to my tastes than other methods that I have tried. In addition of course actually reading detailed reviews with explanations of why they felt that way. If you are one who is looking for a game for a deep story and the review is 9/10 saying "Great explosions, incredible action at every turn, the graphics were spectacular, the story was a little weak but that is made up for by the incredible pace of the combat", odds are it isn't a good game for someone looking for a deep plot.

It can be used as an executive summary (4, Insightful)

Sycraft-fu (314770) | about 2 years ago | (#40683375)

I find it useful for that. If there's something you have little knowledge or information about it can give you a quick breakdown of what you might expect. For example if a game has a 90 metascore, you know it is something that you should probably look in to further, that is uncommonly high. If something has a 40 metascore, you can pretty much give it a miss, that is uncommonly low.

What it'll then get you for things you do want to look in to further is a list of reviews. So you can see what sites have reviewed it, and then go and read the specifics if you wish. Along those lines it is a quick way to find good and bad reviews. When I'm on the fence about something I like to see what people thought was good and bad. I can then weigh for myself how much those matter to me.

Average ratings really can be of some use to filter. I just don't give enough of a shit about every game to go and read multiple full reviews on it and research it. So if it isn't a game I was already interested in, I want a sort of executive summary to decide if I should give it any more time. Metacritic helps with that.

Two recent examples:

1) Endless Space. I had never heard of this game, an indy 4X space game apparently, though rather well developed. Ok well ambitious indy titles can be all over the map. Metascore is 78. That tells me it is worth looking at, it is on my list and I'll look at it more in depth when I feel like playing such a game.

2) Fray, a turn-based strategy sci-fi game. Again, something I hadn't heard of, however a kind of game I like so maybe I'd be interested. Metascore of 32. So no, not wasting time on that.

Other games I won't bother on the Metascore, just use it to find reviews. Like Orcs Must Die 2. Looking forward to that one, so I'll spend time researching it to see if I want to buy it. I liked the original enough it'll be worth looking at reviews, no matter what the score, so see if I think I'll like the next one.

Re:I don't get the point of it anyway (3, Funny)

thegarbz (1787294) | about 2 years ago | (#40683509)

Just because 10,000 out of 15,000 like something, it has no bearing over if I will like it and quite often leads to the oposite.

So you're not interested in my remake of Twilight staring Justin Bieber?

Well DUR? (1)

djsmiley (752149) | about 2 years ago | (#40683273)

Of course it can be wrong, but its a good indicator of what others think of the game.

Nothing on earth can tell you if YOU will like the game unless you play it.

I personally sometimes enjoy playing terrible games, (or games with terrible reviews) and find them quite charming.

Re:Well DUR? (0)

Anonymous Coward | about 2 years ago | (#40683551)

Of course it can be wrong, but its a good indicator of what others think of the game.

Nothing on earth can tell you if YOU will like the game unless you play it.

Well, a selection of, say, 5 people with similar tastes to yours can come pretty damn close.

The trouble is trying to come up with a one-size-fits-all score -- if you take the same raw data, and use a weighting matrix specific to you, it can be remarkably good. That matrix, of course, can best be built by you rating every game you play, and finding correlations -- and armed with those ratings from every user (instead of just from critics), you get a larger pool to find people with similar tastes, and the system just keeps getting more accurate.

Who Cares? (1)

Anonymous Coward | about 2 years ago | (#40683285)

They shouldn't worry, Slashdot beats Metacritic hands down for subjectively erroneous (mod) scoring.
Mod me up for the hell of it.

Picking some nits, are we? (0)

Anonymous Coward | about 2 years ago | (#40683307)

Who cares, it's sunny outside!

stupid article (1)

bloodhawk (813939) | about 2 years ago | (#40683317)

This article is a classic example of why game and movie rating are so terrible nowadays. since when is a 67 a terrible score, in proper scoring system it should be at least a pass. It seems sites tend to rate even trash with a range of 60-100. A B- or a 67 is NOT a terrible score, if scoring is done correctly this should be above average but not great. What is the point of having a 0-100 scale if you are not using the range.

Depends on what you mean by using the range (3, Interesting)

Sycraft-fu (314770) | about 2 years ago | (#40683433)

In most US schools, the scale is:

A: 100-90
B: 89-80
C: 79-70
D: 69-60
F (or sometimes E): 59-0

So while you can percentage wise score anywhere from 0-100 on an assignment and on the final grade, 59% or below is failing. In terms of the grades an A means (or is supposed to mean) an excellent grasp of the material, a B a good grasp, a C an acceptable grasp, a D a below average grasp but still enough, and an F an unsatisfactory grasp.

So translate that to reviews and you get the same system. Also it can be useful to have a range of bad. Anything under 60% is bad in grade terms but looking at the percentages can tell you how bad. A 55% means you failed, but were close to passing. A 10% means you probably didn't even try.

So games could be looked at the same way. The ratings do seem to get used that way too. When you see sites hand out ratings in the 60s (or 6/10) they usually are giving it a marginal rating, like "We wouldn't really recommend this, but it isn't horrible so maybe if you really like this kind of game." A rating in the 50s is pretty much a no recommendation but for a game that is just bad not truly horrible. When a real piece of shit comes along, it will get things in the 30s or 20s (maybe lower).

A "grade style" rating system does make some sense, also in particular since we are not rating in terms of averages. I don't think anyone gives a shit if the game is "average" or not, they care if it is good. The "average" game could be good or bad, that really isn't relevant. What is relevant is do you want to play a specific game.

Re:Depends on what you mean by using the range (0)

Anonymous Coward | about 2 years ago | (#40683475)

That is moronic, just because the school system uses an idiotic grading system doesn't mean the rest of the world should.if they are going to to that then the score system may as well be from 0-40, where a 0 is fail and everything in between is the real rating.

Re:Depends on what you mean by using the range (0)

Anonymous Coward | about 2 years ago | (#40683595)

It's not moronic, because that was the original source for a lot of the grading systems on american review sites. It was a metric we were all familiar with. A to F is not a linear value.

Re:Depends on what you mean by using the range (0)

Anonymous Coward | about 2 years ago | (#40683633)

"All familiar with"???, As someone that went to school in Australia and England I am certainly NOT familiar with it, every school I went to used a 0-100 rating.

Re:Depends on what you mean by using the range (0)

Anonymous Coward | about 2 years ago | (#40684209)

oh boo hoo... I'm special.

Re:Depends on what you mean by using the range (0)

Anonymous Coward | about 2 years ago | (#40683665)

Sure it does. Almost everyone that uses MetaCritic, goes through this school system and understands this grading system. The numbers have meaning based off of this. Similar to how if I say it's 100 degrees outside, the vast majority of the US instantly understands that it's really hot. I'm not going to go out and say well I like Celsius better, so lets make a US focused website that only shows the temperature in Celsius.

I don't really care enough to worry about the downsides of this scoring system. (Oh no we don't use a bunch of numbers very often... Just subtract 67 from every rating if you really care.) The important thing is that people understand what the numbers are supposed to symbolize.

Re:Depends on what you mean by using the range (0)

Anonymous Coward | about 2 years ago | (#40684045)

The system I'm familiar with (which is Canadian) has A go from 100-80, and everything else you said goes down 10 further. Which makes 67 a C+, not a B- but pretty close. Something in the 60 ranges should then (by the system I'm familiar with) be solidly average. 50s would be poor but not outright failure (a string of all 50s would be failure though). 70s would be above average, 80s good, and 90s exceptional.

Re:Depends on what you mean by using the range (1)

bruce_the_loon (856617) | about 2 years ago | (#40684057)

South Africa uses a different set of numbers, so the same grade letter means something different for us, and probably something else completely for other countries too.

  • A - 80% to 100%
  • B - 70% to 79%
  • C - 60% to 69%
  • D - 50% to 59%
  • E - 40% to 49%
  • F - 30% to 39%
  • G - 20% to 29%
  • H - 0% to 19%

Re:stupid article (1)

Zephyn (415698) | about 2 years ago | (#40685915)

It's called the Four Point Scale [tvtropes.org] for precisely this reason. Professional reviewers are very limited on how severely they can criticize the faults of a game when it comes from a big publisher.

When the critic reviews of a AAA game are all 80+, but the user reviews are in sub 4.0 red text, tread carefully.

A score without a review is useless (0)

RogueyWon (735973) | about 2 years ago | (#40683321)

I'd never, ever let a metacritic score determine whether or not I buy a game. That's not to say I don't let reviews (and review scores) influence a purchase, but I find metacritic useless.

When I check reviews of a game to work out whether I want to buy it, I'll look at the score, but it's only one small factor. What I'm actually looking for are certain factors that might be picked up in a review that will be highly likely to influence whether or not I like a game.

For example, I hate - and I do mean absolutely hate - being forced to replay long sections of a game after a death. If an overall positive review criticises a game for poor checkpointing, then I know that the game is highly likely to annoy me and I'm correspondingly less likely to buy it.

Then at other times, there are factors that drive a review score down, but don't bother me at all. A good recent example here is Lolipop Chainsaw. This one had a real spread of review scores, from 9/10 right down to about 4/10. It's clear that some of the reviewers didn't buy into the theme of the game. Others were disappointed that it wasn't (contrary to appearances) a button mashing hack-and-slash. I'd already worked out that the combat was basically a more Arkham Asylum-style methodical brawler and was quite happy with that. And the plot and setting struck me as hilarious. So I bought the game and loved it.

There's also the fact that not all sites mark to the same scale. IGN tends to be fairly "soft" in its marking - review scores tend to cluster around the 7.5+ range. But that's fine, because the review text still tends to pick up the major issues. Eurogamer tend to mark hard on most games and I generally trust their reviews - but their reviewers do feel like fanboys for a couple of companies (Blizzard and Nintendo in particular), so I know to discount them in those cases.

Anyway, tl;dr version - the factors that affect whether an individual will like a game will vary considerably depending on the individual. Trying to capture that in a single meta-score is never going to be workable.

B- = 67? (0)

Anonymous Coward | about 2 years ago | (#40683359)

Given the range A+ to F-, how do they get that B- equals 67? If each letter has 3 options (A+, A and A-), shouldn't B- be 72 (100/18*13)?

Re:B- = 67? (0)

Anonymous Coward | about 2 years ago | (#40685451)

If they're using the traditional American A-F scale, then there's no E, so the set is really A,B,C,D,F. 5x3 = 15.

We need a meta-meta-critic (2)

TechnoCore (806385) | about 2 years ago | (#40683377)

Even though meta critic has become the standard for measuring the quality of a game, they sadly do not check the quality or sincerity of the reviewers they pick. I myself work at a smaller indie game studio. Our last project got reviews between ranging from between 2 to 10. How that even is possible is due to several factors, though the main one being that some reviewers didn't really review the game at all. They just scraped at the surface of it, and Metacritic then used that score. Our game wasn't perfect, neither was it crap. It is fun, addictive, beautiful, with a few bugs. But was it a 2 or a 10? Never.

I know that the larger companies in the business keep track of every journalist and blog that has been lucky enough to have been taken up at Metacritic. If the reviewer is known for giving constantly low or bad reviews they will never receive a copy for reviewing. That doesn't hinder people from buying the game at release and then reviewing it anyway, though it might stop those important first reviews from being bad I guess. Guess we have to do the same at our little studio.

What is really needed is a meta-meta-critic. A site where journalists and reviewers themselves are rated based on their seriousness. Something like the system for rating comments here at Slashdot.

Re:We need a meta-meta-critic (1)

cheekyjohnson (1873388) | about 2 years ago | (#40683505)

Our game wasn't perfect, neither was it crap. It is fun, addictive, beautiful

That's completely subjective.

But was it a 2 or a 10?

Apparently to them, it was.

Re:We need a meta-meta-critic (1)

metacell (523607) | about 2 years ago | (#40684427)

Well then, you could argue that their opinion isn't very relevant, if they only scraped the surface of the game. Reviews are for those who want to shell out their hard-earned cash on a game, and they probably want to go a little deeper.

Re:We need a meta-meta-critic (1)

TechnoCore (806385) | about 2 years ago | (#40686091)

But was it a 2 or a 10?

Apparently to them, it was.

Sure it was, but should they be at Metacritic? Anyone can rate anything without looking if they like to, but it will be of little or no use to anybody else.

Re:We need a meta-meta-critic (0)

Anonymous Coward | about 2 years ago | (#40684119)

I agree with a site to rate reviewers, but Metacritic even with all the trolling going around serves a purpose: to balance the shameful game production sponsored "professional" critics. Titles like Mass Effect 3 showed that there is a HUGE divide between "professional" critics with averages over 90 and actual gamers in a game with glaring shortcomings (inexistent quest management, broken history line, crappiest ending ever, etc).

Re:We need a meta-meta-critic (0)

Anonymous Coward | about 2 years ago | (#40684847)

Actually, Metacritic has 1 major flaw the moment: Big Publishers gets a free pass, just like with roughly the rest of gaming journalism. So basically a AA game that deserves 30/100, will never get less than 60/100. Some might even be bribed up to 90/100, as with the rest of gaming journalism.
Think of it like British top gear, except with no criticism and just promotion. Would be a really boring show, and its not informative.

Give later reviews more weight.... (1)

antifoidulus (807088) | about 2 years ago | (#40683453)

They should give you an option to give more weight the later a review came out. I just find myself generally distrustful of 0-day reviews because they usually mean:
a)The reviewer didnt actually spend enough time with the game to give it a meaningful review and/or
b) the reviewer had access to the game early, which of course raises questions about objectivity.....
The best reviews IMO are those that come out at least a week after the games release....

Re:Give later reviews more weight.... (1)

flimflammer (956759) | about 2 years ago | (#40683625)

Well, it's not as though reviewers get the game the same day we do and then put them out the same day. They get them often weeks in advance, and save for really open ended games like Skyrim, most games can be thoroughly played in the window the reviewers are already given.

Re:Give later reviews more weight.... (1)

flimflammer (956759) | about 2 years ago | (#40683647)

I should also point out that just because reviewers are given games early doesn't raise many questions about objectivity. It's pretty much how the entire industry functions. The reviews you need to take with a grain of salt are the pre-release reviews that come out some time before the game actually launches. Publishers often impose restrictions on what you can say before release, but they have no control over what is said after release. As long as the reviews come out the day the game launches, there should be minimal question on objectivity. At that point it just depends on how much you trust that particular reviewer.

The average is never to be taken at face value (1)

homb (82455) | about 2 years ago | (#40683621)

Diablo 3 on Metacritic is the 2nd highest rated current game.
Don't take averages for truth, they're just averages. Use Metacritic as a source of reviews, find the reviewers (people) who you have the most affinity with over time, and then focus on what their own scores are.

Re:The average is never to be taken at face value (1)

nschubach (922175) | about 2 years ago | (#40684303)

Sure, but if you look at the user scores, it's also in the lowest 5-7.

http://www.metacritic.com/browse/games/score/metascore/year/pc?sort=desc&year_selected=2012 [metacritic.com]

Now, if there were only 10-15 reviews I can understand that this number may not be accurate... but generally the fans don't like the game and that tells me it's probably not worth playing, especially Blizzard fans who are usually not critical to the mothership. Currently, the majority of user reviews are all negative, by almost 2:1.

Re:The average is never to be taken at face value (1)

homb (82455) | about 2 years ago | (#40684349)

User scores are not reviewer reviews.
Reviewers have experience and thoughtful analyses but suffer from small sample size and conflicts of interest.
Users have the strength of numbers but suffer from groupthink and emotional coloring.

The way I use MC is that I get a good feel for the game based on user score averages, then look at the reviewers that I like and analyze their pros and cons. Then I make a decision, based on whether the cons are bad enough for me or not.

Re:The average is never to be taken at face value (1)

nschubach (922175) | about 2 years ago | (#40684475)

User scores are not reviewer reviews.

Not sure where I said they were...

Reviewers have experience and thoughtful analyses but suffer from small sample size and conflicts of interest.
Users have the strength of numbers but suffer from groupthink and emotional coloring.

In an entertainment medium I find emotional coloring an important metric, especially for companies that have such a huge following as Blizzard with such a rabid fan base. There are a few fans, but a majority are not happy with the game.
Professional reviewers have ad revenues to look into. If they continually rate EA games poorly, EA will pull it's advertising dollars from their income. Of course they are bias and there have been multiple... multiple stories and articles that state the same.

You can use it whatever way you like. I like to look at the user scores, compare that to the reviewer scores, compare that to my internal score based on what I read about the game (which of course is weighted highly), and then further decide if I want to purchase it. If the users hate it, there's a reason. I rarely trust "professional" reviewer scores.

It's good for avoiding rotten apples (1)

pijokela (462279) | about 2 years ago | (#40683683)

Metacritic is good for avoiding games that are complete crap. Other than that, you really have to read some of the reviews to decide which game you will like more.

I just played The Last Story (metascore 82) after buying it instead of Xenoblade Chronicles (92). After reading some reviews I was sure that I would prefer The Last Story even though Xenoblade has a greater score and the games are in the same genre. Xenoblade is usually praised e.g. for having lots of stuff to do, but I really wanted a game that I can end someday.

So a number does not replace really making an informed decision, duh.

Use it for Good, and not Lazy (1)

Bieeanda (961632) | about 2 years ago | (#40683725)

I like Metacritic. I consult it when I'm considering a new game on Steam, and I don't have gameplay footage or word of mouth to convince me otherwise. However, I don't use the aggregated score as anything other than an average.

There are links to all of the collated reviews. They're there so that you, the consumer, can perform your due diligence more easily. Personally, I like to scan the middling and bad reviews to get an idea of the kind of warts I should expect if I buy it. One reviewer's selling point may be a no-sale for me-- and I wouldn't know about it if I hadn't taken the opportunity to read it.

The metascore itself is handy at a glance, but using it as a final arbiter of worth and taste is a terrible idea. Just ask Obsidian.

1UP moans that everyone is is doing it wrong. (1)

Rogerborg (306625) | about 2 years ago | (#40683829)

Fortunately, all smart, discerning, handsome, virile, gamers can still get their properly graded reviews from 1UP. Phew!

Rottentomatoes (4, Interesting)

cheesecake23 (1110663) | about 2 years ago | (#40683863)

By that logic, Rottentomatoes (which averages reviews using only a binary fresh/rotten scale) should be utterly useless. Except it isn't. It's IMHO the most dependable rating site on the net.

It seems the magic lies not in the rating resolution, but in the quality and size of the reviewer pool (100+ for Rottentomatoes). In other words, make the law of averages work for you.

Re:Rottentomatoes (3, Interesting)

Dorkmaster Flek (1013045) | about 2 years ago | (#40685145)

Rotten Tomatoes uses a different system though. In fact, I really like their system. They look at a review and decide ultimately whether the critic enjoyed the movie enough to recommend it or not. It's like Siskel & Ebert's thumbs up or down system; fresh or rotten. The only factor is whether the enjoyed the movie or not. There's none of this trying to take a letter grade and turn it into a number from 1-100 bullshit. The Rotten Tomatoes rating is simply a percentage of the number of critics who liked the film enough to recommend it out of the total number of reviews, which I find much more useful. It's still no substitute for the most reliable method, which somebody else above mentioned: find a reviewer whose taste agrees with you on past films/games/whatever and see what they say about new ones. Rotten Tomatoes takes less time though.

Re:Rottentomatoes (2)

Zaphod The 42nd (1205578) | about 2 years ago | (#40685387)

Exactly, came here to say this.

Rottentomatoes rating is not a rating of how good a movie is. Rather, it is how likely you are to enjoy it. It is a probability!!

A movie with 10% on rottentomatoes doesn't mean its a movie worth a 10 grade, it means that only a niche audience enjoyed. So you're less likely to be part of that 10%, but its absolutely possible you still love the film, for very specific reasons. Similarly, a movie with a 98% rating isn't necessarily the best movie or a very high quality film, it just means that a large portion of the population will find it enjoyable overall.

If you treat metacritic the same way, there's nothing wrong at all.

Scores are unnecessary (1)

Hentes (2461350) | about 2 years ago | (#40683873)

Anyone who reviews videogames -- or any form of entertainment, really -- will tell you the score is but one part of the puzzle; in some cases, it's looked upon as a necessary evil, as certain outlets' experiments with ditching scores altogether have been deemed failures.

Yeah, that's why Rockpapershotgun have failed...oh wait.

That doesn't count their gamma rating, of course. (1)

Impy the Impiuos Imp (442658) | about 2 years ago | (#40683913)

Hmmmm...I hadn't realized this issue with Metacritic before. I now rate Metacritic's quality as 4.7 kumquats, down from 2-8 exahogsheads per quadriliter :(

As a game reviewer (1)

Anonymous Coward | about 2 years ago | (#40684089)

As a ten+ year game reviewer (shameless plug: game-over.net), I see the problem from the other side. Even on a single review board, there are variations in how "hard" individual reviewers score. Over the years we have tried to implement a scoring system, giving XX/20 for graphics, XX/20 for story, that kind of thing, but found disagreements among reviewers as to the weightings. Is good graphics equally as important as good plot? What about good music and sound effects? What one reviewer sees as retro, another may call dated. Artsy, bland, exciting, dull - all terribly non-quantifiable adjectives. Aggregation is fine, but over the long haul you're better off finding a single reviewer or just a few, and gauging their opinions for yourself.

truly (1)

PopeRatzo (965947) | about 2 years ago | (#40684433)

The problem with metacritic is that they don't take into account the Law of Truly Large Numbers.

(I don't know what that means, but I've decided I'm going to say this in all discussions involving statistics).

Works when people are sane (1)

TheSkepticalOptimist (898384) | about 2 years ago | (#40685055)

If your going to use some crazy scheme to rate games, then perhaps those websites should provide some kind of a translation value so that Metacritic can correctly identify the intention of the crazy review schemes.

But ultimately if you are getting reviews from 100's of websites, the aggregate value should be fairly accurate. I think a game that is rated 50% is bad compared to a game rated 90%, but I don't think people really care about the perceived quality between two games if their ratings are like 85% and 88%.

Of course, "professional" reviewers are among the most useless professions on the planet due to the sheer amount of online public opinion that come with every game or movie release.

Another problem (1)

ddtstudio (61065) | about 2 years ago | (#40685363)

Larger/louder/more voices drown out smaller/quieter/fewer voices -- regardless of the authority or quality of comment. (Unlike on /., which has moderation and meta-moderation based on content.)

True story: My sister and brother-in-law left their kids with their grandmother and escaped to see a movie and relative peace for a few hours. My sister came back, really angry with her husband. "But sweetie," he said, "_2012_ got a good score on Metacritic!"

Really. Happened.

Good thing there not reviewing people (0)

Anonymous Coward | about 2 years ago | (#40685367)

Oh wait...... they are (directors/actors/game reviewers were the beta, I think they stopped when they realized they were directly effecting peoples ability to do their job)

I can't wait until I walk into an interview and get told my metacritic score isn't high enough to be chosen for a job.

Metacritic isnt the problem, at all. (0)

Anonymous Coward | about 2 years ago | (#40685471)

Watch this http://www.escapistmagazine.com/videos/view/jimquisition/3607-Metacritic-Isnt-the-Problem

Value in variation of reviews, not the "score" (2)

swb (14022) | about 2 years ago | (#40685555)

I think the value in metacritic isn't the "score" but the variation across all reviews. You could have two titles with identical "80" scores, which would otherwise indicate both titles are equally well liked.

That being said, one title could have all of its reviews be between 70 and 90, while the other could have a lot of low scores and a lot of high scores. The high variation in scores tells you that there's something about that title that's amiss.

It would be interesting to see statistics compiled for reviewers, too. Do some reviewers always deviate above the average? Below? I would think a reviewer with a higher variability of ratings would be more trustworthy than one who was consistent with their reviews.

This is not the real problem (0)

Anonymous Coward | about 2 years ago | (#40685685)

The problem isn't how you average the scores, it's that they average the scores at all. If you actually read a bunch of different reviews from different reviewers, you'll learn that they all have their own proclivities and eccentricities, different things that they like or dislike, some are more likely to try to score the game against its target audience regardless of their personal preferences and some will not, things like that. All of those things are super subjective and can't really be accounted for in a mathematical model.

Shifting focus (0)

Anonymous Coward | about 2 years ago | (#40685827)

from sales figures to quality. I think it's a good thing even if the method isn't at all scientific.


tl;dr - Project managers only get their bonus if they get a metascore over X.

The problem is a completely different one (1)

Opportunist (166417) | about 2 years ago | (#40686127)

It's not one about 100 to 0 or A to F. It's one where people are hardly objective or, rather, how people rate games.

Take a look around. Games will get a rating of 90 to 100 if they're really good, 80 to 90 if they're halfway decent and 70 to 80 if they're kinda lukewarm. Then there's a big nothing until the 0-10 bracket for the stinkers. WTF?

This doesn't make any sense at all. But that's how things run today. Every game maker presses to get a 90+ review. Even if the game doesn't really deserve it. What is a 90+, actually? Or rather, what SHOULD it represent? Now, if game reviewing was anything like grades gotten in a class, a 90+ would mean something akin to "showing performance that surpasses the ordinary or shows understanding beyond what has been taught". By that analogy, a 90+ would only be available to games that set a new standard in some way, be it gameplay, graphics or a whole new genre.

Now, how many 90+ games of 2012 can you name that actually come even close to doing that?

Instead, every halfway good game gets a 90+. So what do we do with the really stellar one-a-year hits that redefine something? They get 95+ ratings, of course. So the rating system is now kinda like 95-98 (because 99 or 100 are just simply out, because, well, once a game got that there's no way to get over it) for the games that SHOULD be 90-100, 90-95 for games that SHOULD get a 70-90 and 80-90 for games that should have been "well received but nothing to write home about", which would probably be better suited in a 50-70 rating.

But 50-70 is already viewed as a failure. Despite representing what should be deemed above average (that's what it actually is), any game that got a 54 is a dud. Not because it got a 54, but because we learned that games that get a rating below 80 are actually games that we should stay away from.

That's what's wrong here. Not the grading system itself, but the fact that the system is out of proportion.

Load More Comments
Slashdot Account

Need an Account?

Forgot your password?

Don't worry, we never post anything without your permission.

Submission Text Formatting Tips

We support a small subset of HTML, namely these tags:

  • b
  • i
  • p
  • br
  • a
  • ol
  • ul
  • li
  • dl
  • dt
  • dd
  • em
  • strong
  • tt
  • blockquote
  • div
  • quote
  • ecode

"ecode" can be used for code snippets, for example:

<ecode>    while(1) { do_something(); } </ecode>
Sign up for Slashdot Newsletters
Create a Slashdot Account