Beta
×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

Chess Ratings — Move Over Elo

timothy posted more than 4 years ago | from the checkmate-and-perhaps-match dept.

Classic Games (Games) 133

databuff writes "Less than 24 hours ago, Jeff Sonas, the creator of the Chessmetrics rating system, launched a competition to find a chess rating algorithm that performs better than the official Elo rating system. The competition requires entrants to build their rating systems based on the results of more than 65,000 historical chess games. Entrants then test their algorithms by predicting the results of another 7,809 games. Already three teams have managed create systems that make more accurate predictions than the official Elo approach. It's not a surprise that Elo has been outdone — after all, the system was invented half a century ago before we could easily crunch large amounts of historical data. However, it is a big surprise that Elo has been bettered so quickly!"

cancel ×

133 comments

Sorry! There are no comments related to the filter you selected.

Indeed (5, Funny)

mooingyak (720677) | more than 4 years ago | (#33143658)

However, it is a big surprise that Elo has been bettered done so quickly!

Absolutely. I can almost guarantee no one thought that Elo would have been bettered done so quickly.

Re:Indeed (0)

Peach Rings (1782482) | more than 4 years ago | (#33143686)

Does Timothy even glance at the stories he approves or is it pure pin the tail on the donkey?

Re:Indeed (5, Funny)

Lord Byron II (671689) | more than 4 years ago | (#33143744)

Timothy is the bettered done editor of Slashdot!

Re:Indeed (0, Offtopic)

Hognoxious (631665) | more than 4 years ago | (#33144076)

# He's a bettered done kid

[bettered done baby]

Battered dome kid

[battered dome baby]

ooh ooooh ooh ooooh oo oo ooh ohh a hooway hooway hoowah hoowah/#

Fuck me, I'd forgotten what a pile of shite Deacon Park South Texas were. Thanks a bastarding bunch for reminding me, you heiferflap.

Surreal Gone Kid (1)

Dogtanian (588974) | more than 4 years ago | (#33145096)

Fuck me, I'd forgotten what a pile of shite Deacon Park South Texas were. Thanks a bastarding bunch for reminding me, you heiferflap.

WTF? Is this what happens when some late-1980s Scottish bands get mixed up in a transporter with a popular animation series?

If something that tenuous links to "Real Gone Kid" in your head, you must have some major trauma :-/

Let Him Know How You Feel (0)

Anonymous Coward | more than 4 years ago | (#33143962)

Does Timothy even glance at the stories he approves or is it pure pin the tail on the donkey?

Timothy's e-mail address is timothy@monkey.org according to his home page [monkey.org] . Tired of the half-assed submissions where he couldn't bother to read it over before submitting it for millions to read? Send him an e-mail.

Re:Indeed (-1, Offtopic)

Anonymous Coward | more than 4 years ago | (#33144590)

You actually think any of the editors actually read or *gasp* edit *gasp* the things they post? You must be new here.

Re:Indeed (0, Offtopic)

mike260 (224212) | more than 4 years ago | (#33144950)

Hopefully next time he will bettered posted checked done more carefully.

Re:Indeed (3, Funny)

Braintrust (449843) | more than 4 years ago | (#33143740)

Indubitably. It filled with hope the one that no one thought Elo would have been bettered done so quickly.

Re:Indeed (5, Funny)

camperdave (969942) | more than 4 years ago | (#33143954)

ELO hasn't done all that well since the big hair rock days of the late 1970s/early 1980s, pretty much since the drummer left to join Black Sabbath. I'm surprised at the band's connection to chess.

Re:Indeed (2, Insightful)

Hognoxious (631665) | more than 4 years ago | (#33144228)

The first time I Heard Bev Bevan had joined Sabbath I kind of went "WTF?". But they're all Brummies, along with a lot of heavy metal bands around that time. Priest, Magnum ... they probably all played in pubs together wwhen they were 15.

Similarly you couldn't be a serious goth in the 80s unless you were from Leeds, or a flare-wearing floppy-mopped tossbag in the 90s if you weren't a Manc.

The consequences... (0)

Anonymous Coward | more than 4 years ago | (#33144084)

Timothy bettered done goofed

Re:Indeed (0)

Anonymous Coward | more than 4 years ago | (#33144208)

Has anyone ever really been so far as to have had been bettered done so quickly?

Re:Indeed (0)

Anonymous Coward | more than 4 years ago | (#33144486)

Feeew. As a non-English native I was quite disturbed by yet another language construct which I don't understand.

Mx-doctor (2, Funny)

martin-boundary (547041) | more than 4 years ago | (#33145596)

However, it is a big surprise that Elo has been bettered done so quickly!

Absolutely. I can almost guarantee no one thought that Elo would have been bettered done so quickly.

Is it because elo would have been bettered done so quickly that you came to me?

Re:Indeed (0)

Anonymous Coward | more than 4 years ago | (#33145698)

I'm not surprised at all. Anyone with a good grasp of mathematics and probability can see that it just comes down to how complicated you make your formula to be. It's extremely like finding a function given a list of coordinates.

Bettered Done So Quickly (0)

Anonymous Coward | more than 4 years ago | (#33143662)

However, it is a big surprise that Elo has been bettered done so quickly!"

*facepalm*

Re:Bettered Done So Quickly (1)

Shikaku (1129753) | more than 4 years ago | (#33143716)

bettered done --> bested

Re:Bettered Done So Quickly (2, Funny)

Anonymous Coward | more than 4 years ago | (#33143974)

Battered done --> basted

Re:Bettered Done So Quickly (1)

HeckRuler (1369601) | more than 4 years ago | (#33143786)

Yeah, "bettered done"?
I was JUST ranting about how we shouldn't care about trivial things like spacing after periods, but this is just a sad excuse for journalism.

"beaten"
"surpassed"
"out done"
"blown out of the water"
And while it seems a little old-fashioned, "bested" would work.

Come on people, read it twice before submitting for millions to read.
Git 'er bettered done'ed!

First? (-1, Offtopic)

Anonymous Coward | more than 4 years ago | (#33143678)

I remember when it was near impossible to get first. :/

Re:First? (-1, Offtopic)

Anonymous Coward | more than 4 years ago | (#33145288)

So, you're a goldfish?

First Chess then the BCS! (1)

Shadow Wrought (586631) | more than 4 years ago | (#33143680)

Not that they'd use it, but it certainly couldn't hurt.

Re:First Chess then the BCS! (1)

PRMan (959735) | more than 4 years ago | (#33145042)

Technically, they are already using ELO-CHESS in the BCS, because Jeff Sagarin uses it in his rating system. So all that has to happen is for Jeff Sagarin to change his method.

Re:First Chess then the BCS! (1)

maxume (22995) | more than 4 years ago | (#33146128)

You really think they are ever going to take special status away from Notre Dame and implement a playoff?

been bettered done THAT quickly??? (5, Funny)

boneclinkz (1284458) | more than 4 years ago | (#33143694)

Elo-L

umm (4, Informative)

buddyglass (925859) | more than 4 years ago | (#33143698)

However, it is a big surprise that Elo has been bettered done so quickly!

Not really. Jeff Sagarin has had two systems of rating sports teams for a while now. One, ELO_CHESS, is based purely on win-loss, while the other, PURE POINTS, takes into account margin of victory. According to him, the latter is better at predicting future results. From his analysis:

In ELO CHESS, only winning and losing matters; the score margin is of no consequence, which makes it very "politically correct". However it is less accurate in its predictions for upcoming games than is the PURE POINTS, in which the score margin is the only thing that matters. PURE POINTS is also known as PREDICTOR, BALLANTINE, RHEINGOLD, WHITE OWL and is the best single PREDICTOR of future games.

how are victory margins relevant to chess? (4, Insightful)

l2718 (514756) | more than 4 years ago | (#33143920)

Indeed, Sagarin has shown that applying Elo in sports where the winner is based on points scored is not optimal, since the average margin of victory is a better predictor of strength than won-loss record. But this has nothing to do with applying the Elo method to its original setting of chess, where the outcome of the game is only "win/draw/loss" and there is no margin of victory.

Re:how are victory margins relevant to chess? (1)

buddyglass (925859) | more than 4 years ago | (#33144000)

It's not inconceivable that one might apply an artificial means of gauging "margin of victory" to the domain of chess. Some sort of differential in the "value" of the pieces remaining for each contestant when the game ends. For the three teams that beat ELO, do their ratings systems only take "win/loss" as input, or do they also get the board's configuration at the point when the game ended?

Re:how are victory margins relevant to chess? (5, Insightful)

thousandinone (918319) | more than 4 years ago | (#33144118)

This is pretty ridiculous. Margin of victory? Is there a committee overseeing ethical treatment of chess pieces now? If I sacrifice everything but my King and a Bishop to checkmate you, why is that intrinsically a better strategy than sparing some of my pieces?

There are definite merits to a sacrificial strategy- it's all about board control. Long as theres more than one or two legal moves available to your opponent, you can't really predict where he'll send his pieces. A queen in the middle of the board can cover a lot of distance and do some impressive maneuvers, but any given piece only occupies one spot. Control where your opponent moves, control the game. Not to mention that less pieces on the board gives you more options for where to move with your remaining pieces, and by allowing your pieces to be taken, you have a measure of control over where the free space on the board is.

Indeed, given the rules of the game, I would say a strategy that goes to great lengths to preserve as many of ones own pieces as possible is flawed...

Re:how are victory margins relevant to chess? (1)

Marcx77 (1193559) | more than 4 years ago | (#33144344)

Sorry, but... You can't checkmate with only a king and a bishop.

Re:how are victory margins relevant to chess? (3, Insightful)

SomeJoel (1061138) | more than 4 years ago | (#33144430)

Sorry, but... You can't checkmate with only a king and a bishop.

The hell you can't. It turns out, your opponent has pieces too! Have you ever even played chess?

Re:how are victory margins relevant to chess? (1)

magarity (164372) | more than 4 years ago | (#33146740)

When chess nerds talk about end game strategies it is implied that "a king and a __ " ending is one where the other player has just a king.

Re:how are victory margins relevant to chess? (3, Informative)

phantomfive (622387) | more than 4 years ago | (#33144540)

You know, you're really asking for it when you take a small point that isn't even relevant to his main point and attack it. Sorry, YOU'RE WRONG!!!!! [gameknot.com] .

If you ever find yourself in a game where you can sacrifice all your pieces to get to that position, DO IT!

Re:how are victory margins relevant to chess? (2, Insightful)

friedo (112163) | more than 4 years ago | (#33144372)

If some metric X is a statistically reliable method of predicting future success, then X can be defined as a margin of victory. Whether X is a function of the "values" of remaining pieces, or their positions on the board, or the number of moves, or whatever, is immaterial.

Re:how are victory margins relevant to chess? (1)

microbox (704317) | more than 4 years ago | (#33146338)

Except that if such a metric were used in the future - it would punish the most entertaining and trilling form of play.

In chess, you win or lose. If players started "grinding" just to raise their ratings - ick.

Re:how are victory margins relevant to chess? (1, Interesting)

buddyglass (925859) | more than 4 years ago | (#33144932)

If I sacrifice everything but my King and a Bishop to checkmate you, why is that intrinsically a better strategy than sparing some of my pieces?

Winning with only a king and a bishop remaining is no "better" than winning with all your pieces remaining. A win is a win. That said, winning a game while having many more pieces remaining than one's opponent may imply that the difference between your skill and your opponent's is greater than if you won with only a kind and bishop left. There may be some merit to working that into an algorithm if the goal is to predict the outcome of future matches.

Another data point that might be valuable is simply "how many moves did the game take before checkmate"? Without any other knowledge, the guy who beats me in 10 moves is likely to be a better player than the guy who takes 50 moves to beat me.

Re:how are victory margins relevant to chess? (2, Informative)

databuff (1789500) | more than 4 years ago | (#33144194)

Data only shows results - so there's no scope for gauging the margin of victory.

Re:how are victory margins relevant to chess? (1)

buddyglass (925859) | more than 4 years ago | (#33144944)

I take back what I said, then. It is moderately surprising that there have already been three solutions that outperform ELO based solely on win/loss.

Re:how are victory margins relevant to chess? (1)

DavidShor (928926) | more than 4 years ago | (#33146824)

I thought so too, but looking at the site, it seems relatively trivial to set up a Bayesian structural equation model that models evolution of individual player's ability. That will produce a ton of parameters, but hierarchal magic can take care of that. In fact, they even mention that on the official hint. It's clear to see why that would outperform ELO.

Re:how are victory margins relevant to chess? (1)

TubeSteak (669689) | more than 4 years ago | (#33144924)

But this has nothing to do with applying the Elo method to its original setting of chess, where the outcome of the game is only "win/draw/loss" and there is no margin of victory.

You can easily keep track of a "margin" by assigning point values for the pieces that have been taken.
http://en.wikipedia.org/wiki/Chess_piece_relative_value [wikipedia.org]

That metric loses some relevance since someone behind on points can easily have a strategic victory,
but there may still be some information of value gained from crunching the numbers.

Submission error (2, Informative)

TubeSteak (669689) | more than 4 years ago | (#33143706)

Already three teams [kaggle.com] have managed create systems that make more accurate predictions than the official Elo approach.

1 EdR* 0.729125
2 whiteknight* 0.731656
3 Elo Benchmark* 0.738107 {-- The "official Elo approach"

Maybe we're counting from zero and they forgot to put it on the leaderboard?

More like commenter error (3, Informative)

Anonymous Coward | more than 4 years ago | (#33143836)

That number is "Root Mean Square Error", so lower is better

Re:More like commenter error (3, Insightful)

digitig (1056110) | more than 4 years ago | (#33144696)

Yes, and count how many of them are better than the ELO approach.

Re:More like commenter error (0)

Anonymous Coward | more than 4 years ago | (#33145966)

42?

Re:More like commenter error (1, Informative)

Anonymous Coward | more than 4 years ago | (#33146714)

The leaderboard changes over time, and also consider this:

Update: The team Elo Benchmark (see the leaderboard), uses the Elo rating system. Note, the method for creating seed ratings for Elo Benchmark is being refined, so don't be surprised if the benchmark improves a little in the competition's first week.

Re:Submission error (0)

Anonymous Coward | more than 4 years ago | (#33143862)

No, the posted leaderboard is just predictions of which prediction approach would be more accurate... unfortunately, the meta-prediction algorithm isn't bettered done yet, hence the inaccurate results.

Re:Submission error (4, Informative)

databuff (1789500) | more than 4 years ago | (#33144232)

The Elo Benchmark was submitted a second time. I wrote to Sonas about this. Apparently the rating system has to be seeded. He tried a different approach to calculating seed ratings and this performed better - pushing him one place higher in the rankings.

Re:Submission error (1)

socsoc (1116769) | more than 4 years ago | (#33145974)

It changes and isn't done... Elo is 1st at the moment.

Re:Submission error (2, Informative)

Martian_Kyo (1161137) | more than 4 years ago | (#33147304)

1 Elo BenchmarkOpen 0.723834
2 EdROpen 0.729125
3 whiteknightOpen 0.731656
so at this moment elo is back on top.

Could it be that people have been done some quickly jumpening to conclusions?

I guess george [nanc.com] is working at /. now.

When I see the word Elo (0, Offtopic)

Ukab the Great (87152) | more than 4 years ago | (#33143714)

I can't think of anything other than 70's cheese and largest white afro up until the release of Bobobo-bo Bo-bobo.

Less than 24 hours ago (5, Funny)

LearnToSpell (694184) | more than 4 years ago | (#33143750)

Less than 24 hours ago, the readers of Slashdot launched a competition to find an editing algorithm that performs better than the official "editors" of the site. The competition requires entrants to build their comment systems based on the results of over 9,000 historical submissions. Entrants then test their algorithms by predicting the results of the next 7,809 dup^H^H^Hstories. Already three teams have managed to create systems that make more accurate predictions than the official /. approach. It's not a surprise that Timothy has been outdone -- after all, he was invented half a century ago before English had been standardized. However, it is no big surprise that Slashdot has been bettered done so quickly! The winner: Texas Instruments! [speaknspell.co.uk]

Re:Less than 24 hours ago (0)

Anonymous Coward | more than 4 years ago | (#33143940)

"Glicko Scoring" was suggested some years ago, but I don't know if anyone uses it.

Re:Less than 24 hours ago (1)

roman_mir (125474) | more than 4 years ago | (#33144140)

so what you are saying is that /. editing algorithm has been bettered done quickly?

Re:Less than 24 hours ago (-1, Troll)

Anonymous Coward | more than 4 years ago | (#33145612)

"English had been standardized"

until i die i will fight for standardised.

In other news... (2, Funny)

Last_Available_Usern (756093) | more than 4 years ago | (#33143760)

Organized crime members linked to gambling rackets have been endicted for kidnapping a busload of nerds after they refused to program similar algorithms in exchange for Warcraft game time and photoshopped Natalie Portman porn.

We all know that's not true though. They totally would have done it.

Re:In other news... (2, Funny)

easterberry (1826250) | more than 4 years ago | (#33144052)

They bettered have done it!

differences are minute (4, Interesting)

l2718 (514756) | more than 4 years ago | (#33143834)

Looking at the table, the differences in predictive power are small enough that it's not obvious they aren't due to chance alone; there needs to be some calculation that shows that the differences are meaningful validating the claim that the alternative methods actually extract more information than Elo does. Perhaps there is enough inherent randomness in Chess that even simple predictive models can extract most of the systematics so that what remains after Elo is mostly noise?

Re:differences are minute (1)

Monkeedude1212 (1560403) | more than 4 years ago | (#33144618)

Perhaps there is enough inherent randomness in Chess that even simple predictive models can extract most of the systematics so that what remains after Elo is mostly noise?

No. Chess has no random elements to it. You play against an opponent, with a very strict set of rules.

Now sometimes the rules differ from game to game (such as timing, whether they use something like 3/5 fischer or 20 moves an hour sort of thing), which can have drastic changes to the outcome. For example if you do something like 20 moves an hour, sometimes Chess players will be running short on time, and they'll deliberately try to speed up their 18th 19th and 20th move to get that extra hour of time.

The only other thing that could be considered random is who plays black and who plays white (some players are stronger at one than the other). But in most tournaments, it's round robin with even playing both sides anyways.

Aside from that - it's not random at all. You play against an opponent, with the same setup every game, and the only things left to chance are your strategies.

Re:differences are minute (2, Insightful)

l2718 (514756) | more than 4 years ago | (#33145226)

No. Chess has no random elements to it. You play against an opponent, with a very strict set of rules.

I don't think you understand what the discussion in this post is about. The game of chess has no element of randomness -- but the players do, and it's the players we are trying to model. Just because, on average, player A is better than player B, doesn't mean that player A will win every game. The fact is that the same player will play at different levels of ability on different days, and that is the randomness that is relevant to models trying to predict outcomes of chess games.

Basically all rating systems are based on the assumption that players' ability for a given game fluctuates around an "average ability level" according to some distribution, and the goal of the rating system is to discover the average (and perhaps spread) of this indvidual distribution. So even under best conditions the most the system can do is predict the outcome with an error coming from the distribution of abilities. Now assume the distributions are relatively wide -- then there will be a large statistical error even for the best system.

Returning to the main point, the discussion of the last paragraph has nothing to do with the fact that chess is deterministic. In fact, the fact that there is no randomness in chess makes things easier.

Re:differences are minute (2, Informative)

shimage (954282) | more than 4 years ago | (#33145254)

Bullshit. Mistakes are roughly stochastic, ergo, there are random elements in chess players' performance. This is why chess matches involve more than just two games.

Re:differences are minute (1)

hoytak (1148181) | more than 4 years ago | (#33145258)

... and the only things left to chance are your strategies.

and whether you had too much coffee that morning, failed to see that move 10 steps ahead, etc. In high level chess, it seems that these kind of things have enormous effects on the outcome of the game and are not things that can be easily modeled except as random effects. Thus there is definitely a random element in the outcome of the game; Kasparov vs. Deep Blue was a mix of wins and losses; definitely not a deterministic outcome.

Re:differences are minute (3, Interesting)

phantomfive (622387) | more than 4 years ago | (#33147024)

Mikhail Tal, one of the best players ever, would differ; because it's impossible to see deeply enough to know what the outcome of a move will be. He makes the point here [wikipedia.org] , and I'll quote a small piece:

Tal: - "Yes. For example, I will never forget my game with GM Vasiukov on a USSR Championship. We reached a very complicated position where I was intending to sacrifice a knight. The sacrifice was not obvious; there was a large number of possible variations; but when I began to study hard and work through them, I found to my horror that nothing would come of it. Ideas piled up one after another. I would transport a subtle reply by my opponent, which worked in one case, to another situation where it would naturally prove to be quite useless. As a result my head became filled with a completely chaotic pile of all sorts of moves, and the infamous "tree of variations", from which the chess trainers recommend that you cut off the small branches, in this case spread with unbelievable rapidity.

Now I somehow realized that it was not possible to calculate all the variations, and that the knight sacrifice was, by its very nature, purely intuitive. And since it promised an interesting game, I could not refrain from making it."

Journalist: - "And the following day, it was with pleasure that I read in the paper how Mikhail Tal, after carefully thinking over the position for 40 minutes, made an accurately-calculated piece sacrifice".

You will find that lots of chess players have reported making similarly intuitive moves.

Re:differences are minute (1)

Shadow Wrought (586631) | more than 4 years ago | (#33144656)

I realize they are "predicting" games that have already taken place, but how would this affect a realtime match? How much would it change your moves knowing you've been predicted to lose? Or to win?

Re:differences are minute (1)

dakameleon (1126377) | more than 4 years ago | (#33146270)

Welcome to the world of probability theory. In particular, get started with Bayes [wikipedia.org] and work your way from there.

Well, everyone knows (1)

Duradin (1261418) | more than 4 years ago | (#33143864)

Well, everyone knows that arena is serious business.

So they've got better... (4, Interesting)

frank_adrian314159 (469671) | more than 4 years ago | (#33143924)

Are the better entries as transparent? ELO's a pretty simple way do do this - add or subtract a few points from the rating based on a win or a loss based on the relative difference of the ratings. Would anyone understand (other than "It's a neural net") the ratings produced by these competitors? Would anything human be able to calculate them?

Also, are the new models' improvements in prediction statistically relevant? Or are they just fitting the noise? Both the training dataset and the test dataset seem rather small to me.

Finally, and most importantly, how stable are the ratings? If I'm drunk and lose to a "patzer", do I go down to his level? Fairness of tournaments having small numbers of games has a lot to do with rating stability (unless we're assuming a population periodically beset by huge random shifts in ability).

All-in-all, there's a lot of problems coming up with a good rating system. Opening the dataset to the world, saying "Have at it!", and looking at a single scorecard based solely on predictability is nowhere near sufficient.

Re:So they've got better... (2, Interesting)

greg1104 (461138) | more than 4 years ago | (#33144450)

Development of stock trading systems, which are also trying to rank things based on historical data, have this persistent problem there's been waaay more research into than chess rankings. If you train them on a bunch of historical data, you will discover the best system is invariably one that essentially does a giant curve fitting job on that exact data. One thing trading system developers do to address this are use techniques like walk forward testing [automated-...system.com] , where the system gets trained on one set of data but is only evaluated on a second set.

Luckily, this chess rating competition is using that sort of technique: "Competitors train their rating systems using a training dataset of over 65,000 recent results for 8,631 top players. Participants then use their method to predict the outcome of a further 7,809 games." In fact, the current leaderboard reflects results on only 1/10 of the training set. So long as real ranking is ultimately based on the unseen data set, not the training one, there's little risk of them fitting the noise in the training set and still winning.

Re:So they've got better... (1)

DavidShor (928926) | more than 4 years ago | (#33146962)

Walk-forward testing(a special type of something more general called cross-validation), is a bit over-rated. It's very intuitive, which is why it's used so much in the technical analysis crowd. But statistically, it's really roughly equivalent to multiplying the standard error by a constant factor unless you have severe model mis-specification(In which case you're doing something very wrong!)

In general, it's good to parametrize a range of plausible models, test the assumptions of the model, and conservatively build up if it makes theoretical sense, and don't be afraid to move beyond OLS. That does a lot more to protect against over-fitting then a couple of choice statistics.This can't always be done, Machine-Learning is a necessary evil in some fields, but for something low-dimensional like financial time series or Chess? Should be doable...

Re:So they've got better... (0)

Anonymous Coward | more than 4 years ago | (#33146836)

The weird part is the complexity found in the Elo system (specifically, the non-linear expected win% stuff) are not beneficial. Sonas himself (the guy who made this contest) actually got better results by simplifying Elo.

http://www.chessbase.com/newsdetail.asp?newsid=562

How stable the ratings are is irrelevant -- it's measuring the accuracy of predictions based on the rating system. If a rating system dropped you to patzer because you totally flubbed one game, then the predictions would not be accurate, and the system would get a poor mean squared error, etc. In general though, I think it's recognized that Elo was too static. I think FIDE changed the rating rules not that long ago, using a larger K factor (bigger rating changes per game) for players below a certain level.

I agree this is nowhere near sufficient, but it's not like all the chess federations are going to just drop Elo and go with the winner. The results may offer some innovative approaches though.

Apples and oranges? (1)

DerekLyons (302214) | more than 4 years ago | (#33143928)

Since the Elo system is not designed to predict future performance (it's designed to capture current relative rankings), then is it really surprising that programs designed to predict future performance are better at it?

Re:Apples and oranges? (2, Informative)

mooingyak (720677) | more than 4 years ago | (#33143988)

Since the Elo system is not designed to predict future performance (it's designed to capture current relative rankings), then is it really surprising that programs designed to predict future performance are better at it?

And if my current relative rank is higher than yours, doesn't that imply that if we play each other I should win? If not, what purpose does the rank serve?

Re:Apples and oranges? (2, Funny)

vlm (69642) | more than 4 years ago | (#33144146)

And if my current relative rank is higher than yours, doesn't that imply that if we play each other I should win? If not, what purpose does the rank serve?

Historical achievement, the glory of the grind. Much as my lower UID implies this comment should be more valuable than your high UID comment.

Re:Apples and oranges? (2, Interesting)

mooingyak (720677) | more than 4 years ago | (#33144410)

Much as my lower UID implies this comment should be more valuable than your high UID comment.

I used to think of myself as having a particularly high UID... until I realized that mine is actually lower than a majority of the total UIDs. Weirded me out a little. There are UIDs that are farther from the 1,000,000 mark than I am from Taco.

Re:Apples and oranges? (1)

Orion (3967) | more than 4 years ago | (#33145080)

Historical achievement, the glory of the grind. Much as my lower UID implies this comment should be more valuable than your high UID comment.

Wow... 69,642 is a low UID?

I *am* a god!

Re:Apples and oranges? (1)

DavidShor (928926) | more than 4 years ago | (#33146978)

Sometimes I suspect low UID users have a crawler that looks for people referencing low UIDs...

Re:Apples and oranges? (1)

DerekLyons (302214) | more than 4 years ago | (#33145142)

Since the Elo system is not designed to predict future performance (it's designed to capture current relative rankings), then is it really surprising that programs designed to predict future performance are better at it?

And if my current relative rank is higher than yours, doesn't that imply that if we play each other I should win?

That depends on the relative difference between the ranks. A narrow difference implies you might win, a wider difference implies you will win - and between the two lies a spectrum of gradual shifts from may to will. It's not an absolute quantitative measurement.

Re:Apples and oranges? (1)

databuff (1789500) | more than 4 years ago | (#33144282)

how do you test current relative rankings without using them to make predictions?

Re:Apples and oranges? (1)

DerekLyons (302214) | more than 4 years ago | (#33145156)

That's a damm good question, and one I don't know the answer to.

For a $50 voucher? (1)

pongo000 (97357) | more than 4 years ago | (#33143934)

I don't think so. The time I'd spend on this project is worth a bit more than $50...

Re:For a $50 voucher? (1)

Scatterplot (1031778) | more than 4 years ago | (#33144110)

$50 is bettered than nothin!

I can see why it's such a surprise... (1)

Anubis IV (1279820) | more than 4 years ago | (#33144070)

After all, it's not like other ideas [microsoft.com] haven't already been created in the meantime to address Elo's perceived shortcomings, right?

Elo in non-chess games (4, Insightful)

LambdaWolf (1561517) | more than 4 years ago | (#33144134)

Ah man, no matter how inadequate the Elo system may be for chess, it's much worse seeing it applied to other games where it doesn't belong, which happens regrettably often. The trouble is that the Elo system depends on the premise that nothing affects the outcome of a game other than the skill of each player (and who gets the white pieces).

In chess, that assumption is a pretty good approximation to reality, since every tournament game in run the same way. But many games do have variations in rules or format across different events, such as different maps or races in a real-time strategy game, or different card pools in Magic: The Gathering. Then Elo ratings are biased by how often a player has the chance to play to his strong areas. Players in turn are compelled to game the system: "I should avoid this event because they're using Format X and my rating will stay stronger if I stick to Format Y." The Elo system is meant precisely to obviate that kind of gamesmanship: chess players should need to think only about the strengths of their opponents, which (in principle) will be weighted fairly when calculating rating adjustments. But if there are other competitive factors, which is true for most any popular game invented in the last 30 years, Elo ratings become that much less meaningful.

Re:Elo in non-chess games (1)

selven (1556643) | more than 4 years ago | (#33145326)

Yes, linear ranking systems fail hard at anything as, let alone more, complex than rock-paper-scissors.

Re:Elo in non-chess games (0)

Anonymous Coward | more than 4 years ago | (#33146530)

Yes, linear ranking systems fail hard at anything as, let alone more, complex than rock-paper-scissors.

Elo is non-linear, so your point, although it may be valid, is irrelevant to the current discussion.

Re:Elo in non-chess games (1)

jhol13 (1087781) | more than 4 years ago | (#33146744)

The Elo system does not depend on the premise "nothing affects the outcome of a game other than the skill of each player".

Sure it is modelled according to that, but in practise it is very untrue even for chess. There are a lot of examples where player A has won player B N out of M times although according to rating difference very different outcome should have happened.

The chess events are not similar, I have played a few and they do vary considerably (number of games per day, travel, lighting, temperature, players, mood, ...).

Elo rating is much more meaningful than ratings used e.g. in snooker or tennis. No, I am not saying they should change their rating, it would not make much sense to change the culture of the game, after all the rating is just a bloody number.

Allow me to clarify (3, Funny)

jamrock (863246) | more than 4 years ago | (#33144264)

Three teams done bettered Elo with betterer done algorithms, and the submitter is surprised that it was bettered done so quickly. I'm done. Was that better?

He sounds like Lady Macbeth on crack.

Cheese! (0)

future assassin (639396) | more than 4 years ago | (#33144400)

Man I was like WTF? Cheese ratings? Got confused with seeing the packman icon.

Microsoft's TrueSkill beat Elo before this comp (1)

blaisethom (1563331) | more than 4 years ago | (#33144412)

I believe the algorithm used by Microsoft to match players for X-Box games was already beating Elo before this competition. They have a description of their algorithm at http://research.microsoft.com/en-us/projects/trueskill/ [microsoft.com]

Re:Microsoft's TrueSkill beat Elo before this comp (3, Informative)

Maarx (1794262) | more than 4 years ago | (#33144976)

Not to belittle what Microsoft did, but in the interest if giving credit where credit is due:

Here’s the problem with Battle.net 2.0: 2002s Warcraft III: Reign of Chaos is one of the most underrated video games ever created. And that’s before you learn its online apparatus is the foundation for modern matchmaking, where Blizzard Entertainment should get royalties every time you brag about your X-Box Live Trueskill rating. (Then again, I shouldn’t be giving Blizzard ideas right now.)

Here’s how Warcraft III matchmaking worked: Everyone starts at level one. The maximum level is fifty. You play players within six levels of your own. Win five games, gain a level. Lose five games, lose a level. The penalty for losing is reduced during levels one to nine. Thus, players who win half their games will become level ten.

It was simple and transparent. That was the hook, and people choked on it. It turned Warcraft III ladder play into what ICCUP serves for Starcraft players, a stomping ground so competitive that climbing the food chain gave you a shot at the guys who played for a living. That’s what a good online gaming system does.

The quote comes from Battle.net 2.0: The Antithesis of Consumer Confidence [the-ghetto.org] . I would encourage you to read the entire thing, but for reasons completely unrelated to this thread.

Shouldn't it be "Roll over ELO" (0)

Anonymous Coward | more than 4 years ago | (#33144518)

by Beethoven

Little early yet (0)

Anonymous Coward | more than 4 years ago | (#33144752)

It looks to me like the data-set is rather small and so are the differences in the results. I don't see a clear winner yet by any means.

Steven, 2,156 Elo at my best.

Glicko (better than Elo) has been around for years (1)

whatteaux (112343) | more than 4 years ago | (#33145580)

The Glicko chess rating system and its successor Glicko2 (creative, huh?) are better than Elo and have been around for years. Various online chess sites use it, as does the Australian Chess Federation.

What? (1)

clint999 (1277046) | more than 4 years ago | (#33145690)

Bull****. Mistakes are roughly stochastic, ergo, there are random elements in chess players' performance. This is why chess matches involve more than just two games.

validity and reliability of criterion (1)

Hartmut (458656) | more than 4 years ago | (#33145766)

The problem is not just to find another _method_ to predict game results, but to construct and evaluate a better workable scientific model of chess ability. That's hard, because the criterion 'game result' itself possibly is not a valid indicator of the quality of game play, and the stability of playing strength over time, which is reliability. To estimate these criteria, it is necessary, to design the data collection, as scientists do e.g. in experimental design.
In addition, the available tests of logistic models, like ELO, are not sufficient.

Elo Benchmark is #1 at this moment (1)

schneidafunk (795759) | more than 4 years ago | (#33145876)

1 Elo Benchmark 0.723834 3 6:03pm, Wednesday 4 August 2010
2 EdROpen 0.729125 2 11:47pm, Tuesday 3 August 2010
3 whiteknightOpen 0.731656 4 2:29am, Wednesday 4 August 2010

Re:Elo Benchmark is #1 at this moment (2, Funny)

daveime (1253762) | more than 4 years ago | (#33146944)

Pleased to say I jumped straight into the money at #7 with my first submission :-)

Where AM I going to spend a whole 50 Euros ? Maybe I'll donate it to Greece, seems like they need it.

Re:Elo Benchmark is #1 at this moment (1)

daveime (1253762) | more than 4 years ago | (#33147014)

Damnit ... $50 USD ... that's only 38.50 Euros.

Elo Anecdote (4, Informative)

afabbro (33948) | more than 4 years ago | (#33146324)

Not relevant specifically to this story, but I always laugh at the story of how a prisoner manpiulated the Elo system via closed pool ratings inflation [wikipedia.org] .

Short summary: said prisoner only played against other prisoners, who he'd trained. Due to careful scheduling of the games, he rose from his true strength (probably sub-master) to being the second-highest rated played in the U.S. in 1996.

System Feedback (1)

physicsphairy (720718) | more than 4 years ago | (#33146870)

The problem with this kind of modeling is that many "good fitting" algorithms would, if implemented, change the system itself. There's more to competition chess than just the rules on how to move pieces. For example, while a game in isolation would almost always be played to win, there are many times that because of information from ratings (or due to the method of the tournament) you would start the game being equally happy to draw, which will affect how you play.

Now, even if the difference in the number of pieces remaining (e.g.) is a much better predictor of who will win than the ELO system, if you were ever to actually implement it you would no longer be playing the game the ELO system was trying to track--suddenly you have made players more conservative, not as willing to sacrifice pieces for a better mating position. Possibly some would say you had ruined the game.

Load More Comments
Slashdot Login

Need an Account?

Forgot your password?

Submission Text Formatting Tips

We support a small subset of HTML, namely these tags:

  • b
  • i
  • p
  • br
  • a
  • ol
  • ul
  • li
  • dl
  • dt
  • dd
  • em
  • strong
  • tt
  • blockquote
  • div
  • quote
  • ecode

"ecode" can be used for code snippets, for example:

<ecode>    while(1) { do_something(); } </ecode>