Beta
×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

Crowd Wisdom Better At Predictions Than Top CIA Analysts

timothy posted about 5 months ago | from the random-walk-down-global-politics-street dept.

Stats 136

First time accepted submitter tkalfigo (1448133) writes "The Good Judgment Project is an experiment put together by three well-known psychologists and some people inside the intelligence community. What they aim to prove is that average, ordinary people in large groups and access just to Google search can predict far more accurately events of geopolitical importance than smart intelligence analysts with access to actual classified information. In fact there is a clearly identified top 1 percent of the 3000 predictors group, who have been identified as super-forecasters: people whose predictions are reportedly 30 percent better than intelligence officers."

cancel ×

136 comments

Sorry! There are no comments related to the filter you selected.

Luck resets every time you guess. (3, Interesting)

rmdingler (1955220) | about 5 months ago | (#46730705)

People ahead in guessing games such as these are probably more likely to regress to the mean than to continue defying probability.

Re:Luck resets every time you guess. (3, Insightful)

Arker (91948) | about 5 months ago | (#46730805)

Except that this has nothing to do with luck. It has to do with independent observers having less pressure on them to, consciously or subconsciously, produce rhetoric ostensibly concerning foreign policy but whose content is determined by domestic political needs.

Re:Luck resets every time you guess. (2)

fustakrakich (1673220) | about 5 months ago | (#46730919)

Why are you being modded down? For the font? And yes, they produce the results they are told to produce. That's how they play their game.

Re:Luck resets every time you guess. (-1)

Arker (91948) | about 5 months ago | (#46731161)

How would I know why I am being modded down? I appear to have some fans who come through periodically and give me a bunch of overrated mods, that might be it.

Font? That would be idiotic since I do not control the font on your screen, not even indirectly. For the benefit of anyone who does not understand, the user agent (web browser) has the final decision as to what font to use in displaying a webpage (if the concept of 'font' actually has meaning in the environment it runs in, which is not a given.) The web page (in this case, Slashdot) can give your browser suggestions, and often it has a long list of such suggestions which your browser can and often should be told to ignore.

I am three levels back, behind the web page. I am allowed to give it hints as well, but only by picking from a very short list. Every option on it sucks, the one I am using now sucks the least as default because it mucks with my text the least. If you do not like the font you are seeing on your screen then LEARN how to use your web browser and activate an over-ride. This is the web not television FFS.

Re:Luck resets every time you guess. (-1, Offtopic)

artor3 (1344997) | about 5 months ago | (#46731249)

Fuck off. That font is hideous and unreadable, and I shouldn't need to go digging in any settings to fix the mess you created.

You don't even seem to understand why a fixed-width font would be useful. It "mucks with your text the least"? You realize that you're writing in paragraphs, right? And that not everyone in the world is going to have their browser window at the same width?

You're just trying to be a special snowflake and show everyone how smart and techy you are.

Re:Luck resets every time you guess. (-1)

Arker (91948) | about 5 months ago | (#46731387)

"That font is hideous and unreadable"

Then perhaps you should uninstall it?

"I shouldn't need to go digging in any settings to fix the mess you created."

Are you really such a technological illiterate you think I control which fonts are installed and used on your computer?

Re:Luck resets every time you guess. (0)

Anonymous Coward | about 5 months ago | (#46732275)

You're not fooling anyone. Well, at least anyone who has used the site for more than a day. Fuck, your UID is low enough, if you can't comprehend the tag and font system here, you are truly retarded.

We know the <code> tag looks like

We know what it looks like when you switch "Plain Old Text" to "Code". Fixed width fonts have their purposes, and general conversation isn't one of them.

Reminding you that you're just looking like a twit won't help fix your behavior. Hopefully everyone will mod you troll, so people won't have to read your nonsense.

Re:Luck resets every time you guess. (-1)

Anonymous Coward | about 5 months ago | (#46731395)

Browser window width has ZERO impact on how fixed-width-font text is presented. There's no no-breaking-spaces in his text, only the occasional forced-line-break between paragraphs same as any other comment here on the site.

So please, if you're going to bitch something, don't make an elementary-school-level mistake in your claims for why something is bad.

- WolfWings, who's been too lazy to login to Slashdot for years now.

Re:Luck resets every time you guess. (0)

Anonymous Coward | about 5 months ago | (#46731619)

> That font is hideous and unreadable,

I happen to think it's much more readable. The font wasn't messed with, he simply wrapped his post in tt tags, which ... unsurprisingly, is converted to whatever font your browser has configured to handle Teletype Text. I hope you can understand why it is considered more readable by many people, who happen to use computers daily.

You can probably change your font handling via

Chrome - https://support.google.com/chr... [google.com] (look for fixed width font in your font settings)
Mozilla - https://support.mozilla.org/en... [mozilla.org]
(or use a plugin, or other guide, etc)

None of this has to do with his post and a great deal with your unfamiliarity with anything other than (probably) facebook. If you don't like it, as he suggested, change it.

Re:Luck resets every time you guess. (0)

Anonymous Coward | about 5 months ago | (#46731699)

Actually I am not using any tags at all. I am entering nothing but plain text with the occasional CR.

I am using the 'code' drop down which testing showed to be the only way to keep slashdot from adding lots of other extraneous and undesired tags. Yes, if you select plain text it still inserts tags.

And the funniest thing is that my posts show on my screen in exactly the same font as all the other posts. Or, well, that was true until just a minute ago at least. I played with my browser font settings and now my comments are even more readable. One more change and ALL the comments will look better.

It's not magic folks, it's world wide web 101.

Re:Luck resets every time you guess. (2)

Cenan (1892902) | about 5 months ago | (#46732081)

If you're using Firefox, you can download the Stylish plugin and make sure everybody conforms to your wishes by adding a rule for the "commentBody" class. Might also want to add one for the "tt" tag.

.commentBody {
              font-family: monospace;
              font-size: 14px;
        }

This makes everybody's posts appear to use code tags. I find it readable, you might not, but you do have a say in it.

Re:Luck resets every time you guess. (0)

fustakrakich (1673220) | about 5 months ago | (#46731281)

Deary me! Aren't we sensitive tonight! And check your prefs!

Re:Luck resets every time you guess. (-1)

Anonymous Coward | about 5 months ago | (#46731777)

Yes. I think his pussy hurts. "Cascading Style Sheets" do a pretty good job on word break/wrap. Squeeze the sides of this page. Damn kids are always whining about stupid shit.

Re:Luck resets every time you guess. (3, Insightful)

Flentil (765056) | about 5 months ago | (#46731397)

I think the majority here use Slashdot's default font and never messed with it, but you did mess with yours, making your posts stand out as odd. Blaming everyone else, making them responsible to fix the 'bug' you created on their screen, isn't very helpful. It's a lot easier to just mod you down than to delve into browser font settings and possibly mess up how we view all other websites, just because you like your posts on Slashdot to look like they were typed on an old-timey typewriter. Why don't you just fix your own browser font settings and not put the burden on everyone else?

Re:Luck resets every time you guess. (-1)

Anonymous Coward | about 5 months ago | (#46731439)

FFS you idiots are hilarious. If you have a hideous unreadable font on your computer get rid of it! It looks fine from here.

It's a lot easier to just mod you down than to delve into browser font settings and possibly mess up how we view all other websites

Well if it's a hideous and unreadable font on this website it will be a hideous and unreadable font on any other websites you go to as well, donchathink?

Re:Luck resets every time you guess. (1)

Anonymous Coward | about 5 months ago | (#46732427)

YOU DON'T LIKE ALL CAPS? just instruct your browser to use a font where caps appear as lowercase letters.

Re:Luck resets every time you guess. (2)

Ardyvee (2447206) | about 5 months ago | (#46732671)

No. Courier New is a pretty nice font for some things. Just not the /. comment section.

Re:Luck resets every time you guess. (1)

Attila Dimedici (1036002) | about 5 months ago | (#46732729)

I'm sorry, but if you are not doing something so that your posts appear in a different font than everyone else, why is your font different than everyone else? Oh, and your signature is in the same font as everyone else's post, not the font which your post is in. So, it seems probable that you have decided to post in a font other than the slashdot default.

Re:Luck resets every time you guess. (1)

grep -v '.*' * (780312) | about 5 months ago | (#46731301)

independent observers ,,, concerning foreign policy but whose content is determined by domestic political needs

Damn! That's a usage case I hadn't heard of before. In politics it was always someone else's fault when it didn't work out. Now it's going to be EVERYone else's fault.

Cloud-blaming! (tm)

Re:Luck resets every time you guess. (1)

ceoyoyo (59147) | about 5 months ago | (#46731637)

Yes, that's what the article says. In a scientific paper that would be in the discussion, possibly in the conclusions. Experienced scientists know that the discussion, and depressingly frequently the conclusions, are BS the authors made up that's not really supported by the data, one way or the other.

Re:Luck resets every time you guess. (2)

Sique (173459) | about 5 months ago | (#46732235)

Not only that, but people with some distance to the situation and only superficial knowledge are not blinded by facts and details. We have the same phenomenon with predicting sport results. There, people who are not absolute fans or professionals in the sport usually fare better at predicting results as they don't give too much weight on some details, or their own preferences, which in the long run prove to have much less influence than expected. Instead they basicly tend to put teams or athletes they often heard about in front and less famous ones more to the back.

So you have three group of people: Those who don't know anything, and who indeed are the worst predictors, then those who have some basic understanding of the situation, who are the best predictors, and then the specialists, who know intricate details and have very profound knowledge, who aren't very good at predicting because they give too much weight to single aspects and have some very strong opinions or emotions about the situation which blinds their judgement.

Re:Luck resets every time you guess. (0)

Anonymous Coward | about 5 months ago | (#46732453)

If you want good predictions. Look at betting sites.
Large amount of people putting their own money on the line for what they think.
When it comes to sports, the weather and the eurovision song contest betting sites will predict the result way better than any expert panel.

CIA uses Dirty Tricks Project (2, Interesting)

Anonymous Coward | about 5 months ago | (#46730715)

CIA cannot believe a wisdom based output, they have to believe that their actions will change the outcome.

Re:CIA uses Dirty Tricks Project (0)

Anonymous Coward | about 5 months ago | (#46731867)

They only have to release all that classified information to the public for the new, more accurate predictions to pour in. Result: CIA utilized wisdom and justifies its own existence at the same time.

But why would the CIA release their best results? (0)

Anonymous Coward | about 5 months ago | (#46730717)

Wouldn't the smart thing be that we think they're barely competent?

Re:But why would the CIA release their best result (2)

hey! (33014) | about 5 months ago | (#46731461)

While I am sure there are occasional situations where it might be advantageous to be thought foolish and incompetent, in general this is likely a bad thing.

It's like being thought *weak* in military terms. There in tactical situations you'd like the enemy to underestimate your strength, strategically it's better to be thought stronger than you actually are. If a hostile country is considering violating some treaty they have with us, we'd want them to think our intelligence agencies will catch them red-handed. Once they actually go down that road, we'd want them to think our agencies are completely incompetent.

Re:Common Knowlege (1)

hoboroadie (1726896) | about 5 months ago | (#46732457)

I'm not particularly smart, but doesn't EVERYBODY already know those CIA fools are incompetent?

Well... (2)

ComputersKai (3499237) | about 5 months ago | (#46730719)

There was a project affiliated with Google that aimed to predict disease outbreaks using the search engine, but that didn't turn out that well. In fact, it barely succeeded in any of its predictions.

Re:Well... (0)

Anonymous Coward | about 5 months ago | (#46730925)

There was a project affiliated with Google that aimed to predict disease outbreaks using the search engine, but that didn't turn out that well. In fact, it barely succeeded in any of its predictions.

So just like the people responsible for predicting what three flu strains to protect against during immunization season. In some years flu shots have a zero percent success rate not because they don't work, but because the predictions on what three strains to fight against were just so wrong that none of the three strains even infected one percent of the population. If the google flu project can do better, even if it is still a very poor effort, it is still better than professional people with the magic letters D and R in front of their names.

Re:Well... (3, Interesting)

Cenan (1892902) | about 5 months ago | (#46732109)

The problem with Google's prediction algorithm is that it consistently overshoots. The story was on /. about a month ago [slashdot.org] , as far as I can tell they're not only not predicting cases correctly, they aren't even attempting to distinguish between strains (how could they, they're predicting from search activity - flu victims rarely know their strain).

Re:Well... (0)

Anonymous Coward | about 5 months ago | (#46731169)

Wisdom of Crowds is almost the exact opposite of creating algorithms to mine big data.

Well yeah (4, Insightful)

MichaelSmith (789609) | about 5 months ago | (#46730739)

This is why its better to have elections than let the CIA select the government. AFAIK, anyway.

Re:Well yeah (3, Funny)

BradMajors (995624) | about 5 months ago | (#46731117)

The CIA disagrees.

Re:Well yeah (1)

cold fjord (826450) | about 5 months ago | (#46731199)

So is there some reason that is on your mind at the moment? Or did I miss that this was a "Best of Slashdot" from November 1968?

Re:Well yeah (1)

blahplusplus (757119) | about 5 months ago | (#46731215)

"This is why its better to have elections"

http://www.ted.com/talks/lawre... [ted.com]

Re:Well yeah (1)

Ceriel Nosforit (682174) | about 5 months ago | (#46732295)

Not TEDx. Nice.

Re:Well yeah (0)

Anonymous Coward | about 5 months ago | (#46731727)

The presidential candidate I voted for has lost every time. 0-4. :-(

Re:Well yeah (1)

kruach aum (1934852) | about 5 months ago | (#46732555)

The elections have nothing to do with informed decision making. Your news media has made sure of that.

Seems fishy (4, Insightful)

ceoyoyo (59147) | about 5 months ago | (#46730767)

I wonder if they properly controlled for luck. Take three thousand people and get them to make predictions and some of them are going to appear unusually accurate than others even if all of them are just making completely random guesses. You'd be surprised how many people don't correctly account for that. Every paper proposing clinical diagnostic criteria I've ever read, for example.

Re:Seems fishy (3, Interesting)

mikael (484) | about 5 months ago | (#46730909)

Why do you think it is purely luck? When you have these wild discussion parties - things like "is a bright blob of pixels on a Mars Rover image a cosmic ray, a high-voltage dust-devil, light contamination of a camera box, a gas geyser", you will have an incredible combination of experts - everyone from geologists, ranchers, hill-hikers, photographers, astronomers. Geologists will tell you want can and can't come from the ground, ranchers and hill-hikers will tell you things they have seen and never seen, photographers will tell you what visual artifacts can appear on a camera, and astronomers tell you what can fall from the sky and can't, and what those falling things look like.

It's like solving a giant logic problem where everyone can cross off or tick what what they know. Eventually the set of possible answers reduces down to one or two.

Re:Seems fishy (3, Interesting)

hibiki_r (649814) | about 5 months ago | (#46731597)

Except this isn't how it works at all.

The wisdom of crowds works doesn't have anything to do with having experts. After all, the experts have no way of influencing the crowd. It is a well defined phenomenon that works when people's biases are pretty random, so mistakes cancel each other out. It's a lower quality estimation mechanism than a market, where people that are sure of their answer can be 'louder' than those that don't know said answer, and it lacks the feedback mechanisms of a market, but still, it is helpful to predict things based on widely available information. Ask the crowd information few of them have any idea about, and their result will suck.

So what does the average beating CIA personnel? That the CIA's biases are large enough to need quite a bit of quality control.

Now, having a 1% of the respondents be far better than the CIA experts probably means nothing. If I invite 3000 people over to guess how 10 coin flips will turn out, chances are one or two of them will guess all of them correctly, but that would not make them seers capable of seeing the future. how many people were worse than 30% worse than those same CIA experts?

Re:Seems fishy (1)

ceoyoyo (59147) | about 5 months ago | (#46731655)

In statistics it's called an unbiased estimator. Most people know it as an average. It doesn't have any particular link to crowds and the behaviour is very well defined. It does, however, require that the individual estimates be wrong in a random way.

You managed to pick exactly the same example I did.

Re:Seems fishy (0)

Anonymous Coward | about 5 months ago | (#46732373)

I agree with almost all of your comment, except for this bit:

It's a lower quality estimation mechanism than a market, where people that are sure of their answer can be 'louder' than those that don't know said answer

It seems to me that allowing people to weight their votes the way the market does would make it less reliable than a conventional "wisdom of the crowds" measure. This is of course, just my opinion, and I would be very interested if you have any solid evidence to back up that particular assertion.

Re:Seems fishy (2)

Attila Dimedici (1036002) | about 5 months ago | (#46732753)

The wisdom of crowds works doesn't have anything to do with having experts.

You are right that the wisdom of crowds does not come from having experts. The wisdom of crowds comes from having a lot of people who all have a little bit of knowledge relevant to the subject. Some of that knowledge might be something that you would not necessarily think was relevant, but when applied as a filter on the other knowledge present produces a result much more accurate than an expert on the subject would ever produce.

The results of this study are not new. Back in the lat 70s, early 80s, there was a study which showed that a group of people with no particular expertise on the subject will reach a better decision than an individual expert on the subject as long as certain criteria are met in the group discussion. The most important criteria that needs to be met is that the groups deliberations must be such that the individual charisma of each person must not be allowed to influence the group discussion. My understanding is that they accomplished this by having all of the discussion occur in anonymous text (such as if all of the comments on slashdot were from Anonymous Coward).

Re:Seems fishy (1)

ceoyoyo (59147) | about 5 months ago | (#46731609)

Because that's not what's going on here. The example in the article is a pharmacist who somehow manages to be better than everyone else at predicting geopolitical events. Not a party with a bunch of experts in various fields hashing things out, just a pharmacist in her kitchen in her spare time.

Flip a coin ten times and there's only a tenth of one percent of a chance of it coming up heads every time. Flip a thousand coins ten times and there's only a small chance one won't come up heads ten times in a row. If you were doing this study and one of your subjects made ten correct calls such as "Russia will invade the Crimea before June" in a row, might you get a little bit excited?

Teela Brown (0)

Anonymous Coward | about 5 months ago | (#46730913)

Let's start breeding lucky people!

Re:Teela Brown (1)

K. S. Kyosuke (729550) | about 5 months ago | (#46731069)

We're already doing that. ;-) (Unless you believe this world is so awful that it's better to never get conceived in the first place!)

Re:Teela Brown (1)

TubeSteak (669689) | about 5 months ago | (#46731451)

Maybe you missed the reference.
Teela Brown [wikipedia.org] is a character from Larry Niven's Ringworld series.
Her defining characteristic is that she's a 6th generation of Birthright Lottery winners and thus, uniquely lucky.

Re:Teela Brown (1)

K. S. Kyosuke (729550) | about 5 months ago | (#46731471)

I did indeed, but my point is that all of us are winners of one such lottery. :-)

Re:Teela Brown (1)

CrimsonAvenger (580665) | about 5 months ago | (#46732607)

but my point is that all of us are winners of one such lottery. :-)

Alas, no.

The Birthright Lotteries in question were a literal lottery held annually to allow someone to have a child.

Everyone in the world was authorized to have one child.

Then, special people were authorized more than one (geniuses, that sort of thing).

And rich people were allowed to buy extra birthrights (the theory being that getting rich was a skill - note that taxation in place pretty much prevented inherited wealth).

And finally, if there were not enough children being born to replace losses in any particular year, there was the Birthright Lottery - winners got a license to have a child.

The odds of winning were on the orders of millions to one against. All of Teela's ancestors for the last six generations were winners of the Birthright Lottery. And she was lucky. Alas for the people who chose to send her to the Ringoworld, her luck did NOT extend to the people near her - odd things happened that were good for HER - if they were good for nearby people, cool, if not, tough....

Read the article. (0)

Anonymous Coward | about 5 months ago | (#46731065)

You should read the article, as the summery is confusing.

The project asks questions like, "What is the chance that North Korea will launch a multi-stage missile before June 2015?"

People enter a guessed % of probability. They get 3000 random people to respond. People's guesses are wildly all over the place. However. . .

When you average out all those responses, the resulting number is spooky accurate. So-called, "Wisdom of the crowd."

Luck has both nothing, and everything to do with it.

Re:Read the article. (1)

pepty (1976012) | about 5 months ago | (#46731589)

"What is the chance that North Korea will launch a multi-stage missile before June 2015?" People enter a guessed % of probability. They get 3000 random people to respond. People's guesses are wildly all over the place. However. . .

When you average out all those responses, the resulting number is spooky accurate. So-called, "Wisdom of the crowd."

Luck has both nothing, and everything to do with it.

How can a probability be spooky accurate when it is in reference to a singular event that can't be repeated over and over again?

Re:Read the article. (0)

Anonymous Coward | about 5 months ago | (#46731893)

How can a probability be spooky accurate when it is in reference to a singular event that can't be repeated over and over again?

Of course, you're right. I'd not thought of it correctly.

I am going to presume that the study authors managed to work out a metric upon which to base their claims of accuracy versus inaccuracy which makes sense.

Re:Read the article. (1)

ceoyoyo (59147) | about 5 months ago | (#46731675)

You should read the article. There's a little section about "wisdom of crowds" and then the balance of it is about particular people they've selected as being super accurate, such as the pharmacist they use as an example. If you take enough people and ask them to guess randomly, some of their guesses will line up very nicely with the answers to any questions. Purely by luck. If you cherry pick these randomly lucky guessers and don't properly allow for your cherry picking in your calculation of expected performance, you will be misled badly.

A different version of the same phenomenon confuses people who try to write classifiers. I have a friend who was trying to classify patients who did or did not have a disease. He put a bunch of measurements into the classifier, trained it, and look, it was 100% accurate! He was suspicious, so he put in a bunch of randomly generated numbers, trained it, and look, it was 100% accurate! Of course, neither version did any better than chance on data it hadn't been trained on.

Re:Read the article. (0)

Anonymous Coward | about 5 months ago | (#46731989)

They get 3000 random people to respond. People's guesses are wildly all over the place. However. . .

When you average out all those responses, the resulting number is spooky accurate. So-called, "Wisdom of the crowd."

Or, you know, it can turn out like when Reddit "helped" to identify suspects for the Boston marathon bombing.

Oh, wait...

Re:Read the article. (0)

Anonymous Coward | about 5 months ago | (#46732391)

For "wisdom of the crowds" to work, everyone has to make their guesses in isolation from each other*, that isn't what happened on Reddit.

*It shouldn't actually make much difference with small sub-groups of the crowd having some influence, but it will break down when large sub-groups or the whole crowd is interacting with each other on the question in hand as some members of the groups will have disproportionate influence on each other without those members being more likely to be correct.

Re:Read the article. (0)

Anonymous Coward | about 5 months ago | (#46732399)

I think it's telling that, as far as I heard, *no one* on Reddit suggested even one of the actual perpetrators as a suspect.

So, this is one of those scenarios where the wisdom of the crowd was so incorrect they had apparently 0% chance of being correct at all. Yes, the echo chamber effect is a problem, but that's not the only critical failure of this "crowd wisdom".

Re:Seems fishy (1)

TubeSteak (669689) | about 5 months ago | (#46731485)

I wonder if they properly controlled for luck.

You obviously didn't RTFA.

But she signed up, got a little training in how to estimate probabilities from the people running the program, and then was given access to a website that listed dozens of carefully worded questions on events of interest to the intelligence community, along with a place for her to enter her numerical estimate of their likelihood.

"Usually I just do a Google search," she said.

In fact, she's so good she's been put on a special team with other superforecasters whose predictions are reportedly 30 percent better than intelligence officers with access to actual classified information.

It's not luck they've selected for, it's the ability to make educated guesses.

Re:Seems fishy (1)

ceoyoyo (59147) | about 5 months ago | (#46731625)

I did read the article. Nothing in what you quoted is at all relevant. Perhaps you didn't understand my post?

Re:Seems fishy (0)

Anonymous Coward | about 5 months ago | (#46731769)

Take three thousand people and get them to make predictions and some of them are going to appear unusually accurate than others even if all of them are just making completely random guesses.

Meanwhile, take three hundred CIA analysts with tons of classified material and even if they're horrible inaccurate, not a one will get fired. Hell, in that circumstance, we're literally better off throwing dice and closing down the CIA entirely. Dice don't tend to cause blowback on their own.

Re:Seems fishy (1)

Solandri (704621) | about 5 months ago | (#46732055)

That was my thought too. For example, if you grab n people off the street and ask them to make a total guess at 10 coin flips, just by pure chance alone 1.1% of them are going to get 8, 9, or all 10 correct (80% or better "correct" rate). A cumulative binomial distribution of 8 correct of 10 trials with 50% success rate is 98.9%. Likewise the bottom 1.1% will guess correctly 20% or worse. This is the the usual "cause" of some research investigating psychic phenomenon. If the researchers aren't very well versed in statistics, they end up thinking that a percentage of the population is psychic (guesses better than average), and a percentage is anti-psychic (guesses worse than average). When in reality it's just luck.

The deviation shrinks as you increase the number of coin flips (i.e. the "correct" rate of the luckiest 1% of the population gets closer to 50%). Unfortunately the article lacks any info to really judge if this is what's going on. I wasn't able to find in TFA how many events they've had their "crowd" predict. From the project's blog, they've been at it for 3 years so I have to think they've had more than 10 predictions. But you also need to know the long-term rate of correct guesses, as that will skew the correct rate up as well. (It's also unclear what's meant by "30% better" - does that mean the experts were correct 60% and their top 1% were correct 90%? Or does it mean the experts were correct 60% and their top 1% were correct 78%?)

Reminds me of the Policy Analysis Market (4, Interesting)

khallow (566160) | about 5 months ago | (#46730793)

Back in 2003, there was a similar system called the Policy Analysis Market (PAM) that was close to being implemented. It got deep-sixed by some world-class idiots from Congress (see my opinion [kuro5hin.org] then). It's too bad that we have to go to a somewhat contrived surveying/polling system rather than use something that we know works.

For example, I think a PAM system would have given us (and I mean everyone not just US policy makers) insight into how the events of the Arab Spring revolutions would evolve even if it couldn't have predicted the original flash point.

Re:Reminds me of the Policy Analysis Market (4, Interesting)

TapeCutter (624760) | about 5 months ago | (#46731401)

even if it couldn't have predicted the original flash point.

Funny you should say that, the diplomatic cable leaks showed [wikileaks.org] that high level western diplomats in Syria were concerned about a civil war erupting due to the severe "fertile crescent" drought fuelling internal migration from rural areas to the cities (10% of Syria's total population simply abandoned their farms due to lack of water). The drought caused food prices to rise sharply and food riots became a regular occurrence in cities across the middle east and North Africa.

"flash point" - Have a look at why that protester set fire to himself in the public square and why it resonated so strongly across the Arab world, it wasn't because they all logged on to FB and suddenly realised their governments were tyrannical. Predicting this sort of social unrest is like predicting an earthquake in LA, you can be pretty confident that your prediction will come to pass but have no idea when.

Re:Reminds me of the Policy Analysis Market (2)

nut (19435) | about 5 months ago | (#46731679)

Back in 2003, there was a similar system called the Policy Analysis Market (PAM) that was close to being implemented. It got deep-sixed by some world-class idiots from Congress ...

Maybe they weren't idiots. Maybe the were protecting a lucrative after-Congress job [techdirt.com] market...

Re:Reminds me of the Policy Analysis Market (2)

gringer (252588) | about 5 months ago | (#46731981)

New Zealand has iPredict, where you can make money off correct guesses about the future:

https://www.ipredict.co.nz/ [ipredict.co.nz]

All of the superforecasters are cynics (0)

Anonymous Coward | about 5 months ago | (#46730821)

Cynics don't see the world the way they think it ought to be. Cynics see the world the way it is, and are hence capable forecasters of events.

Tail wags dog (0)

Anonymous Coward | about 5 months ago | (#46730823)

People are herd animals, and the stuff they start thinking about in groups can become a self-fulfilling prophecy.

Question of scale (3, Insightful)

Livius (318358) | about 5 months ago | (#46730827)

With enough people, there will be someone with insightful information, and probably a balance of opinions. Searching for bugs in open source works a little like that.

But in theory if a professional intelligence service had hard evidence that, for example, a politician is bluffing about something, then a policy can be adopted even if it goes against some conventional wisdom.

For example, the information that Saddam Hussein's WMD programme was a hoax prevented a rash invasion...., um, never mind.

Re:Question of scale (4, Funny)

mikael (484) | about 5 months ago | (#46730927)

Both the British and Americans used the same government contact for their information, but they didn't tell each other who that contact was. In fact, they had different codenames for the person. When they cross-referenced each others information, they got two confirmations.

Re:Question of scale (0)

Anonymous Coward | about 5 months ago | (#46731239)

But they knew they didn't assign a single codename a person....

Re:Question of scale (1)

Livius (318358) | about 5 months ago | (#46732741)

Professional intelligent agents were not fooled. People who only heard what they wanted to hear do not count as professionals.

Re:Question of scale (0)

Anonymous Coward | about 5 months ago | (#46731563)

The sad thing is there was nothing "rash" about it. They planned their bullshit, stacked it high. The armor was unnecessary for their objectives.

Downsizing (0)

Anonymous Coward | about 5 months ago | (#46730847)

I would say the CIA agents between ~1947-2007/2008/2009 are smarter then 2010-2014 CIA agents since the ones pink sliped work in the private sector...

Old news (3, Informative)

Crashmarik (635988) | about 5 months ago | (#46730883)

http://en.wikipedia.org/wiki/D... [wikipedia.org]

This has been known since the 60s. Only reason it keeps cropping up is the ego of the people involved in analysis, and the organizational inertia of the agencies involved.

Re:Old news (2)

Jmc23 (2353706) | about 5 months ago | (#46731797)

Did you read your link? Delphi method is basically what they are comparing against, a structured group of experts.

Re:Old news (1)

Crashmarik (635988) | about 5 months ago | (#46731935)

Not sure if you are trolling or not, but the articles make no mention of the comparison.

delphi method (0)

Anonymous Coward | about 5 months ago | (#46730897)

This sounds like the Delphi Method, developed by RAND in the 1950s.

Re:delphi method (1)

Jmc23 (2353706) | about 5 months ago | (#46731789)

Sounds like, but isn't.

Summary is confusing (0)

Anonymous Coward | about 5 months ago | (#46730921)

What they set out to prove has nothing to do with Google vs. classified stuff, but lone analyst vs. crowd.

According to one report, the predictions made by the Good Judgment Project are often better even than intelligence analysts with access to classified information

That is much different than the summary's claim.

The project is about testing a method of making predictions by using a large pool of people, wisdom of the crowd. That pool of people could very well be entirely composed of CIA analysts with access to classified information.

Duh (0)

Anonymous Coward | about 5 months ago | (#46731003)

Anyone who specializes in a given field will become myopic. Having worked in the intelligence community for decades, I've seen it first hand hundreds of times. Great analysts become worse at their jobs over time as they become more convinced of their 'rightness' and less apt to all doubt or introspection to impact their judgement. Human nature. Next story.

Fearfully (0)

Anonymous Coward | about 5 months ago | (#46731061)

Alex Jones is probably somewhere very near the top of that group. Woe is we! :)

My drunken pet vole makes better predictions... (1)

rumpledoll (716472) | about 5 months ago | (#46731165)

My drunken pet vole makes better predictions than those idiots at the CIA. News at 11, Captain Obvious.

Bell Curve (1)

aaronb1138 (2035478) | about 5 months ago | (#46731175)

I doubt there is any field where one percent of laymen aren't vastly superior to the majority of professionals. The same applies to art, engineering, science, and any other field of study. This is statistically normal.

Re:Bell Curve (1)

grep -v '.*' * (780312) | about 5 months ago | (#46731375)

I doubt there is any field where one percent of laymen aren't vastly superior to the majority of professionals. ... This is statistically normal.

Fine, but just like the quatrains of Nostradamus [nostradamu...ctions.org] : can you identify them correctly beforehand? Counting the perfect hits after the fact isn't fair. (But then again I guess it worked for Miss Cleo [weht.net] for a while [consumeraffairs.com] )

BTW: 16th century Mr. N. is an idiot. But he's better than the current sales-people paying attention to him with 5 centuries more experience. Oh, and multiple Blood Moons [latimes.com] are soon arriving -- buy your Tarot cards and ticket to safety [imdb.com] now, before it's too late!

Re:Bell Curve (3, Insightful)

TapeCutter (624760) | about 5 months ago | (#46731425)

Interesting, so why do "laymen" in the US keep electing zealots, crackpots, and "entrepreneurs" who are clearly lying to their face for fun and profit? Do you guys enjoy being treated with contempt?

Re:Bell Curve (0)

Anonymous Coward | about 5 months ago | (#46731503)

We need public figures to vilify, so we can feel better about our own shitty decisions.

Re:Bell Curve (0)

Anonymous Coward | about 5 months ago | (#46731525)

I believe many Americans enjoy *other* guys, guys they don't like, being treated with contempt.

Re:Bell Curve (0)

Anonymous Coward | about 5 months ago | (#46731951)

I don't trust the results of crowd wisdom in this case. First of all, the crowd is represents a cross-section of people with diverse biases and diverse suspicions. With large enough scale and enough diversity in the conspiracy theories the crowd dreams up, there's a great chance that some prediction will be right. Where the CIA differs from the crowd is that they have to be skillful at sifting through noise and finding information that is useful. The CIA's predictions have to be tempered with reason and likelihood and evidence that Google will not be privy too. So, why does it seem like the CIA gets everything wrong? Well, we don't exactly know all the things the CIA got correct because the CIA doesn't talk openly about things. Second, there's a lot of randomness in political events. The CIA can predict that a specific move by a foreign government will incite its citizens to protest. The CIA can predict the likelihood that a protest will get bloody and will lead to a military crackdown. The CIA cannot predict a knee-jerk reaction by a third-party foreign leader to make a bold political move such as annexing bits of Ukraine. Studies such as this one assume far to much equality between the information access available to the intelligence community and that available via Google search.

Here are my predictions:

1) Russia will attempt to annex lands in another former Soviet satellite or republic.
2) A new domestic policy scandal will lead the opposition party to demand the firing of top officials in the ruling government

re: TapeCutter (0)

Anonymous Coward | about 5 months ago | (#46732309)

Your question implies 'laymen' have another option. They don't. The politcal process eliminates the very people it needs. It's a shortcoming of all political systems.

Re:Bell Curve (1)

idji (984038) | about 5 months ago | (#46732379)

Laymen elect zealots, crackpots and "entrepreneurs" who are clearly lying to their fact for the Bread and Circuses [wikipedia.org]

Re:Bell Curve (1)

hibiki_r (649814) | about 5 months ago | (#46731651)

That doesn't make any sense for things where training is needed though. 1% of laymen being better than civil engineers at building extremely large bridges? 1% of laymen being better at fixing cars than a mechanic? How about 1% of laymen being better at basketball than NBA players? It makes absolutely no sense, because we are talking about things where the training time is extremely valuable, and guessing at random will not help you, because there are too many possible answers.

Even in yes/no questions, if 1% beats the professionals, it's because the questions are so hard, that the results might as well be random.

The most you could say is that we are bad at putting the most talented people at a certain field in the right position to use their advantage. For instance, I doubt that the Americans that have the best potential to be soccer stars happen to pick soccer, if just because it's not a very popular sport here compared to most of the world. However, in something like Basketball, where it's very easy to identify talent, as being very tall is a major advantage, it's very likely that we are pretty close to the best there is in the population: For instance, 17% of people in the US that are 7 feet play, or have played, in the NBA!

So yeah, that bell curve... go read again.

Professionalism and bullshit (0)

Anonymous Coward | about 5 months ago | (#46731193)

Seriously who could even argument against the fact that a democracy composed of fucking idiots wont like to vote for you ???

Also... (0)

Anonymous Coward | about 5 months ago | (#46731457)

It has been shown that judges make just as good of decisions in family court than hired gun psychologists. The "psychology" industry as a whole wastes a lot of taxpayer money.

How is this different? (0)

Anonymous Coward | about 5 months ago | (#46731731)

http://en.wikipedia.org/wiki/Ouija

Have they not read Bruner? (1)

mrgren (630298) | about 5 months ago | (#46731849)

Two words: Delphi Pool. [wikipedia.org] This is an idea from 1975's The Shockwave Rider.

Re:Have they not read Bruner? (0)

Anonymous Coward | about 5 months ago | (#46732527)

Hate to break it to you but no one reads crappy Sci fi from the 70s.

Can I Hear it for the.... â(TM) (1)

Rooked_One (591287) | about 5 months ago | (#46732213)

EYYYEEEEE ENNN TEEEE Ps?

That's where it's at the...

EYYYEEEEE ENNN TEEEE Pee... eeeeeees

That is because CIA analysis is politicised (0)

Anonymous Coward | about 5 months ago | (#46732215)

The CIA pursues a self serving agenda, and their analysis is coloured by that. Although the CIA has a history of involvement in some of the worst crimes against humanity ever committed, and subverting democracy across the world, I do actually have a respect for some of their analysts. Listen to what they say. They are generally not stupid.

Is it the group or its best? (1)

ponos (122721) | about 5 months ago | (#46732283)

I don't exactly get it. Is it the group as a whole that predicts accurately or its "best predictors"? Because clearly the first hypothesis favors direct democracy as a decision-making process. My intuitive guess is that when you pick a large enough group, some people within that group are clearly going to do better than specialists, because, in a certain way, they are themselves specialists.

Who are the Super Forecasters? (0)

LF11 (18760) | about 5 months ago | (#46732703)

I am really curious as to who makes up the "super forecasters" of these geopolitical problems. I suspect that they are merely the Libertarian contingent. 30 people out of 3000? Sounds about right.

Or, alternatively, are they spiritual people? People who partaken in psychedelic experiences? What defines this group?
Load More Comments
Slashdot Login

Need an Account?

Forgot your password?

Submission Text Formatting Tips

We support a small subset of HTML, namely these tags:

  • b
  • i
  • p
  • br
  • a
  • ol
  • ul
  • li
  • dl
  • dt
  • dd
  • em
  • strong
  • tt
  • blockquote
  • div
  • quote
  • ecode

"ecode" can be used for code snippets, for example:

<ecode>    while(1) { do_something(); } </ecode>