Beta

×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

Why Most Published Research Findings Are False

kdawson posted more than 5 years ago | from the peers-can-be-wrong-too dept.

Medicine 259

Hugh Pickens writes "Researchers have found that the winner's curse may apply to the publication of scientific papers and that incorrect findings are more likely to end up in print than correct findings. Dr John Ioannidis bases his argument about incorrect research partly on a study of 49 papers on the effectiveness of medical interventions published in leading journals that had been cited by more than 1,000 other scientists, and his finding that, within only a few years, almost a third of the papers had been refuted by other studies. Ioannidis argues that scientific research is so difficult — the sample sizes must be big and the analysis rigorous — that most research may end up being wrong, and the 'hotter' the field, the greater the competition is, and the more likely that published research in top journals could be wrong. Another study earlier this year found that among the studies submitted to the FDA about the effectiveness of antidepressants, almost all of those with positive results were published, whereas very few of those with negative results saw print, although negative results are potentially just as informative as positive (if less exciting)."

cancel ×

259 comments

Sorry! There are no comments related to the filter you selected.

Peer review helps (4, Interesting)

CRCulver (715279) | more than 5 years ago | (#25432787)

Peer review no doubt helps to limit people who intentionally want to cause problems. Sokal's bullshit paper on quantum gravity (see The Sokal Hoax [amazon.com] ) made it into print only through a non-peer-reviewed journal. While it is disturbing to think much published scholarship is unreliable, at least it isn't necessarily malicious.

Re:Peer review helps (5, Insightful)

symes (835608) | more than 5 years ago | (#25433133)

Peer review can both help and hinder - there's the reputation effect of guest authorship where having a well-known, senior, academic's name on the paper helps it through no matter how absurd the findings.

Then there are reviewers who review papers they do not have the expertise to review. And to be frank I've seen some pretty bloody ludicrous comments from supposedly expert reviewers - the sort of stuff 1st year students wouldn't make.

But I do think that the majority of researchers are dilligent and beleive in what they submit. And lets face it - if it is an emerging area and you have a neat result that either refutes someone else's grand theory or is just really novel you're going to want to see that in print. It is because we seek to replicate research that findings are later falsified. This isn't evidence that the system is broke it is pricesly how it should work. It is the work that can't be falsified that stands the test of time and contributes to our knowledge.

If there are people who think that falsifying published research is somehow a bad thing - that is shows there's a problem in research standards - the they really really need to go back to school and read some Karl Popper.

Re:Peer review helps (1)

pjt33 (739471) | more than 5 years ago | (#25434065)

If there are people who think that falsifying published research is somehow a bad thing - that is shows there's a problem in research standards - the they really really need to go back to school and read some Karl Popper.

I think this could be phrased more carefully to make explicit the difference between theoretical and experimental research. Theoretical research being falsified is the scientific method at work. Experimental research being falsified is less cut and dried. Sometimes it's due to previously unknown effects and the result is an increase in knowledge: sometimes it's due to poor analysis of the results - failure to account for systematic errors, or statistical incompetence, and it would be better that it not be published in the first place than that someone have to expend time and money on falsifying it.

Re:Peer review helps (3, Informative)

phantomfive (622387) | more than 5 years ago | (#25433347)

It isn't so much a problem of peer review. Peer review has limitations of course, to thoroughly review an article, one would have to repeat the experiment, which most reviewers (for good reasons) do not do. Peer review is good for what it does, give feedback to the authors of the paper, and as you said, it does have a filtering effect.

The other issue is, a lot of papers aren't really worth much at all. Nature might get their share of interesting articles, but in smaller journals, a lot of research ends up being something like, "I had this idea, and I did a small little experiment to see if it was worth anything. Maybe it is." But of course, with a small little experiment, your chances of being wrong are greater: it's just an entry-point for someone else to maybe continue research in an interesting direction. And I have done peer review, FWIW (and if you trust random guys you meet on the internet).

Re:Peer review helps (1, Offtopic)

ShieldW0lf (601553) | more than 5 years ago | (#25433845)

The problem is money. If we lived in a world where you gained power through the trust and esteem of your fellow man, no one would publish things they weren't confident in. But we don't. We live in a world where you gain power through the accumulation of leverage that allows you to dominate your fellow man despite their wishes and opinions. In this world, who gives a shit if you were wrong or not? Who gives a shit if anyone knows? It doesn't matter, as long as you got paid before they find out. Then you can trample over them either way.

You want to fix it, you move the power basis from economics to politics. Then, people will actually start caring about their reputation. Without addressing this fundamental problem, nothing will change in the slightest.

Need a revolution to do that though... the incumbents will without a doubt use any and all forces that their disposal to prevent a fair and equitable system from ever coming into existence.

Re:Peer review helps (1)

arktemplar (1060050) | more than 5 years ago | (#25434063)

One thing that I noticed was that this is may not apply to the engineering fields and to the more mathematical of sciences. I'm not sure about exactly whether the above can be construed as being valid or not ? should the headline be as general and should it be called 'science' if it's just medicine ?

Illin in the panicillin? (-1, Troll)

Anonymous Coward | more than 5 years ago | (#25432789)

Is she illin with the panicillin?
Is she reelin in the panicillin?
Is it feelin with the panicillin?
Are you steelin in the panacillin?

Panka Panka

Is she liable no suitifiable no not on trial but so suitifiable
Is she viable no suitifiable pliable style is so suitifiable
so reliable no suitifiable shes not on file but so suitifiable
im on the dial its so suitifiable its like im liable but more suitifiable

Re:Illin in the panicillin? (-1, Offtopic)

Anonymous Coward | more than 5 years ago | (#25433453)

You suck many, many miles of dick.

Obvious question ... (5, Funny)

2muchcoffeeman (573484) | more than 5 years ago | (#25432791)

How long until some researcher releases a study showing that Dr. Ioannidis' research findings are themselves wrong?

Re:Obvious question ... (5, Funny)

CaptainPatent (1087643) | more than 5 years ago | (#25432913)

Actually, someone already has. [slashdot.org]

Re:Obvious question ... (1)

PCMX (1029966) | more than 5 years ago | (#25433227)

Hehe. First we find that 89.2% of all statistics are made up and now research shows that most researches are false. What the hell are we to believe in now?!

Re:Obvious question ... (0)

Anonymous Coward | more than 5 years ago | (#25433831)

Belive in yourself, nobody else does.

Who needs a study: science != medicine/biology (5, Insightful)

Roger W Moore (538166) | more than 5 years ago | (#25433353)

How long until some researcher releases a study showing that Dr. Ioannidis' research findings are themselves wrong?

Who needs a study? Simply reading the article shows that he has fallen precisely into the trap that he is complaining about i.e. overstating his results. He forgets one very simple point: not all science is medicine/biology.

As a particle physicist I would strongly disagree with his conclusions, at least as applied to experimental particle physics. It is certainly true that some papers turn out to be wrong but this is rare and usually ends up as a 'big thing' in the field. Outside my field I'd be very surprised if the majority of physics or even chemistry papers turn out to be wrong (but I certainly not a chemist so this is just my impression).

As for medicine I can certainly see that they have a problem. Afterall how many times have we been told "don't eat X/do Y it is bad for you" only later to find out that actually it isn't half as bad as they thought and may even have benefits? Just because a lot of medical research is often flawed does not mean that all of science has the problem on the same scale.

So, Dr. Ioannidis either show us some data from chemistry, maths and physics or stop complaining that all of science has a problem on this scale. From where I stand your evidence points to a problem with bioscience/medical research only.

Re:Who needs a study: science != medicine/biology (-1)

metageek (466836) | more than 5 years ago | (#25433853)

oh yeah?

What about Cold Fusion? What about the fabricated nanotechnology data of Schon? What about the memory of water?

The issue is the same with physics, chemistry, and all the others. A large part of the problem is that the top journals *want* papers that can make the news on Thursday; and will select papers that may have not been fully vetted, and also have a bias towards "big shots" (who have much easier time publishing any kind of trash than do young researchers).

Re:Who needs a study: science != medicine/biology (4, Interesting)

Colonel Korn (1258968) | more than 5 years ago | (#25434289)

oh yeah?

What about Cold Fusion? What about the fabricated nanotechnology data of Schon? What about the memory of water?

The issue is the same with physics, chemistry, and all the others. A large part of the problem is that the top journals *want* papers that can make the news on Thursday; and will select papers that may have not been fully vetted, and also have a bias towards "big shots" (who have much easier time publishing any kind of trash than do young researchers).

Exceptionally rare outliers that were discovered very quickly, and these examples don't jive with the type of problem described in the article, which the GP nails when he points out how it is very concentrated in medical science.

Also, top journals don't "want" papers in the sense that they get the ones they want. Peer reviewers decide what's worth publishing, and I have yet to meet one who feels that an article should be published because it will make the evening news. Big shots do get a big advantage, but in most cases it's because they have a history of good research. Things DO slip through the cracks, but in Chemistry and Physics, those things are within error bars.

The GP's post is so damn good.

Re:Who needs a study: science != medicine/biology (3, Insightful)

khallow (566160) | more than 5 years ago | (#25434341)

What about Cold Fusion? What about the fabricated nanotechnology data of Schon? What about the memory of water?

What about them? That's only three (and all but the Schon research were strongly questioned at the time they came out). A few anecdotal cases don't disprove the grandparent's claims.

Re:Who needs a study: science != medicine/biology (1)

Bishop Rook (1281208) | more than 5 years ago | (#25434369)

From GP:

It is certainly true that some papers turn out to be wrong but this is rare and usually ends up as a 'big thing' in the field.

Cold fusion being debunked and Schon's fabricated data were both "big things" in the field. "Memory of water" has never been scientific in the first place, it's nothing but homeopathic quackery.

Re:Who needs a study: science != medicine/biology (0)

Anonymous Coward | more than 5 years ago | (#25433887)

string theory, if correct, will upset the entire field of physics - as well as almost every other theory in science. wait- most, if not all, of science is THEORY!

Re:Who needs a study: science != medicine/biology (5, Insightful)

Bishop Rook (1281208) | more than 5 years ago | (#25434333)

Definitely not most. All. The process of science is using theory to predict a result, carrying out an experiment to test whether that result occurs or not, and revising the theory if necessary.

We cannot ever prove that the current theory is, in fact, "correct." For all we know, there is some rule encoded into the stuff of reality that gravitation will reverse itself next Tuesday, and we can neither disprove this nor predict it. All science can offer is the minimally-complex theory that fits all currently known data.

Re:Who needs a study: science != medicine/biology (4, Insightful)

Btarlinian (922732) | more than 5 years ago | (#25434459)

I would argue that this problem is not only pretty much non-existent in chemistry and physics, but that even biology, at least cell and molecular biology do not have this issue either. Typically when a biologist publishes a protein structure or sequences an organism's DNA no one shows up later and says it is wrong. In fact, it's rather large news when it does.

For example, there was a bit of a controversy over protein crystallographers recently. A person had published a paper on a protein structure that seemed to contradict all previous though functions for the protein. It turned out that they had used the wrong parameters in their phasing program. However, this doesn't happen in most to most papers, and certainly not a majority of them.

I would say that this problem is mostly specific to medical research. By its very nature, medical research is a good deal more prone to human fallibility since both subjects and researchers are human beings.

Re:Obvious question ... (0)

Anonymous Coward | more than 5 years ago | (#25433569)

I'm not even going to bother reading the article, but something about "almost a third" and "most" don't add up. Anyway, the high level topic is rather self-evident: little, if any, science is conducted for the good of all man kind at the cost of great personal sacrifice. In the end it's about turning research into billions of dollars, so it stands to reason that some may be more eager than others in reporting their findings.

Irony? (1)

phatvw (996438) | more than 5 years ago | (#25432797)

Of course there is absolutely no chance that this particular piece of research is also wrong...

Re:Irony? (1)

Kneo24 (688412) | more than 5 years ago | (#25433303)

Yes, but to what extent? Is it the entire notion? Or just some numbers of his data? And how far off are those numbers if that's the case?

Misleading (5, Insightful)

Idiomatick (976696) | more than 5 years ago | (#25432819)

Title is wrong. It says that the FDA is corrupt. And that published papers take around 3years to get peer reviewed where the bad ones are removed. What a blatent attack on science generally. Sure paper publishing needs to be reviewed but 'most published research is false' is an outright LIE. 'Most published research' includes all of our basis of scientific knowledge. If most of our theories on biology were wrong really we realistically wouldnt have been able to move forwards into working with genes if we didnt know what a cell did.

Re:Misleading (0)

Anonymous Coward | more than 5 years ago | (#25433737)

Not sure about any corruption at the FDA, but the pharmaceuticals that provide them with the research are definitely corrupt in that they suppress results that are not conducive to the approval of their sometimes deadly concoctions. The pharmaceuticals have also succeeded in suppressing the publication of documentation that led to them losing lawsuits including the contents of judges decisions and opposing research results. Drugs are still advertised as "safe and effective" and widely prescribed even though research and court decisions have shown they can lead to the death of the patient or harm to those around them such as in the area of many anti-depressants and ADD/ADHD drugs but not limited to those areas.

So much of research these days is contracted out by corporations and that can lead to a certain amount of bias. Other funding can cause a certain amount of bias as well, unfortunately. Of course there is still much pure research going on as well which is less affected by bias. Science has a certain amount of nobility but human nature gets heavily involved too.

How universal is this. (5, Interesting)

sumdumass (711423) | more than 5 years ago | (#25432851)

At the risk of being modded down to oblivion, I am still curious to how this effects popular theories like global warming. We already has people claiming that the science is wrong and they are generally mocked and ignored because their works are published in major journals. Well, this story seems to indicate that publishing those claims will give them a larger change of it being incorrect.

Anyways, it seems that if you don't tow the line on climate change, there is no room for you anywhere. So where does this leave the accuracy of the claims in light of how common it seems that they can be wrong even when published in a respectable scientific journal. I know the IPCC looked at them, but they didn't validate any of the claims, they only looks at whether or not Humans were the cause (that was their charter and they acknowledged this in their reporting).

Re:How universal is this. (3, Insightful)

Samantha Wright (1324923) | more than 5 years ago | (#25433023)

The idea behind this is that pharmaceutical studies are difficult and expensive to perform, so they rarely get challenged for some time, and when they do, the challengers are more often in more obscure journals: so when people go to cite statistics and findings, they don't notice the fact that what they're quoting has been invalidated. Climate change has been studied over and over again and subjected to extensive analysis by many minds for many years, particularly because so many people have questioned it. To suggest what you are saying now is a little behind the times.

Re:How universal is this. (5, Interesting)

sumdumass (711423) | more than 5 years ago | (#25434261)

But the concept isn't unique to the pharmaceutical studies. With the general attitude towards dissenters of the Faith that has grew from global warming, I don't see why it isn't true here either. I mean the IPCC used faulty temperature data in their evaluations, Al Gore exaggerated quite a bit and used outdated charts because they proved his point better and Hansen, the guy who pretty much brought Global warming into the lime light admitted to exaggerating claims and justifying it by claiming it was necessary to make people aware of the problems.

I mean it is probably even more prevalent when the data sets used in studies aren't availible to people, the temp data that was proven to be wrong was reverse engineered because Hansen refused to disclose the data. People wanting to review these studies have been mocked and denied access to the data or had the data set obfuscated to make it even more difficult to work with. There was even one instance where someone was told that he couldn't have the data because he was going to pick the work apart and the author didn't want to help him do that. Not very scientific if you ask me. There is definitely room to question what is being said. Most people in disagreement today are in contention over the causes and the purposed solutions which to date, doesn't seem to be helping out in Europe.

Re:How universal is this. (2, Interesting)

wormBait (1358529) | more than 5 years ago | (#25433043)

The article is focused on largely medical studies, which are attempting to move toward curing people. Therefore, studies that cure people are interesting, studies that don't cure people aren't interesting. Global warming is different. Disproving global warming is VERY interesting, and would get published more readily than something supporting global warming.

Re:How universal is this. (1)

SL Baur (19540) | more than 5 years ago | (#25433591)

Global warming is different. Disproving global warming is VERY interesting, and would get published more readily than something supporting global warming.

Really? How about the recent stories which prove that glaciers in the north have been *growing*? How about followup stories to the ones we read some months ago claiming the North Pole ice cap was dying?

It's all politically motivated and has taken second stage to the banking crisis.

Re:How universal is this. (4, Informative)

Mr. Slippery (47854) | more than 5 years ago | (#25434317)

How about the recent stories which prove that glaciers in the north have been *growing*?

A handful of glaciers are indeed growing. The vast majority are shrinking [skepticalscience.com] , and they are shrinking much more than the handful of anomalous ones are growing.

A handful of unusual data points in a complex system does not prove a trend. It's as if you were to argue, "Scientists *say* that cigarette smoking will damage your health. But I know one guy who smoked and lived to a ripe old age. Therefore, these `scientific' findings are clearly the result of some politically-motivated anti-tobacco conspiracy."

Re:How universal is this. (2, Insightful)

shma (863063) | more than 5 years ago | (#25433081)

I fail to see how you can draw any conclusions about the reliability of atmospheric physics papers from a study of biomedical research papers.

Re:How universal is this. (1)

sumdumass (711423) | more than 5 years ago | (#25433139)

If X=Y then there is no reason that X or Y can equal Z either. The process is just as transparent in other areas and just because the limit in this particular article is with pharmaceutical studies, it doesn't mean that the same garbage in garbage out rules don't apply.

In fact, the IPCC did their reports using flawed temperature data and people are still pulling up the exaggerated Mann hockey stick graph as their proof even when there is a more accurate one availible. The inconvenient truth by Al Gore did this on purpose because the old graph showed more of a difference.

Re:How universal is this. (4, Insightful)

jmorris42 (1458) | more than 5 years ago | (#25433757)

> I fail to see how you can draw any conclusions about the reliability of atmospheric physics papers from a study of biomedical research papers.

Biomedical research is a lot more amendable to verification and falsification, thus an argument can be made that errors are getting corrected. Global Warming is faith based, it's predictions aren't made in anything resembling a controlled scientific environment and the only way to test it's predictions is to do nothing for twenty years and see if the disasters predicted come to pass. Now consider that rerunning a medical test and the origional paper wrong will get a researcher rewarded while writing anything whatsoever questioning human caused global warming gets a researcher labeled a whore of the oil companies and the argument that the science on GW might be at least as flawed as these biomedical papers grows.

He's merely observing the obvious, and no. (1)

plasmacutter (901737) | more than 5 years ago | (#25433387)

His study merely observes the obvious.

Scientific consensus in any field is reached by research, which is then published and subsequently challenged by other scientists.

At the point of initial research, the idea being tested is merely a hypothesis, otherwise known as "educated conjecture".

In the case of medicine, there are numerous uncontrollable variables which lead to a high degree of error in small case studies. (the placebo effect, environmental factors, the variable resilience of each patient, etc)

Global warming does not fall under this. It has been researched, and retested, and re-challenged numerous times. The resulting climate change predictions are presented in confidence intervals, and the usual results brought to policy makers are conservative figures from the lower third of that interval, which are still quite scary.

Re:He's merely observing the obvious, and no. (3, Insightful)

Daimanta (1140543) | more than 5 years ago | (#25434159)

"Global warming does not fall under this. It has been researched, and retested, and re-challenged numerous times.'

On this point you are walking on loose sands. What do you mean retested? You cannot test Global Warming. You can only make observations about it and then form your own opinion based on those facts. You cannot create 2 identical planets ,mess with their CO2 levels and then compare the results. All your data is based on your measurements and conclusions you draw from it. That is where the controversy lies. In order to test the Global Warming theory you need 2 carbon copies of 1900 earth and have only one use massive amounts of carbon fuels and the other next to none. Only then will you truely approach(you cannot mimick everything) how the system works.

And let us not talk about history because that is even more a topic of disputes.

Re:How universal is this. (1)

SL Baur (19540) | more than 5 years ago | (#25433539)

Anyways, it seems that if you don't tow the line on climate change, there is no room for you anywhere.

You mean "toe the line", but never mind.

Sadly, scientific progress tends to be made when dogmatic leaders die. 3 decades ago, scientists were worrying about a new ice age. We know from past evidence that the Earth naturally experiences "global warming" to melt ice ages and cools back down again.

Where are all the followup stories on the supposed North Pole ice cap melting that hasn't taken place because glaciers grew in the past winter? Hmmm?

Re:How universal is this. (2, Insightful)

kisak (524062) | more than 5 years ago | (#25434291)

At the risk of being modded down to oblivion, I am still curious to how this effects popular theories like global warming.

Global warming is not popular, it is down right scary. If a group of scientists disprove that our use of hydrocarbons have a significant effect on global warming, these scientists will be extremely popular and probably share a Nobel Prize.

What About Publish or Perish? (5, Interesting)

istartedi (132515) | more than 5 years ago | (#25432859)

I would think that "Publish or Perish" must contribute to a lot of crappy papers getting published. Shovel it out the door, somebody else says it's wrong, write another grant for a study to verify that, shovel that one out the door, rinse, lather, repeat...

Re:What About Publish or Perish? (4, Insightful)

Seakip18 (1106315) | more than 5 years ago | (#25433149)

How true that is.

The significant other is quitting grad school as soon as she gets her Master's in Neruoscience(she's in the PhD/Master Program). She can't stand the constant pressure of publishing nor the need constantly justify grant writing. She's not the best researcher, but the pressure is enough to drive her to not caring anymore. She'll get her consolation prize and get on with her life.

Maybe she's just not cut out for academia, though it's losing out on the great potential she has.

Re:What About Publish or Perish? (5, Insightful)

timholman (71886) | more than 5 years ago | (#25433969)

I would think that "Publish or Perish" must contribute to a lot of crappy papers getting published. Shovel it out the door, somebody else says it's wrong, write another grant for a study to verify that, shovel that one out the door, rinse, lather, repeat...

It does indeed. Thirty years ago an assistant professor could get tenure by publishing one good paper per year in an archival journal. Nowadays an assistant professor is expected to publish four or more journal papers per year. This leads to the well-known academic concept of the "MPU", i.e. the minimum publishable unit, or "just how many papers can I squeeze out of this one good idea?". This also leads to the backwards situation where a senior professor sitting on a Promotion & Tenure Committee may have fewer published papers (and fewer awarded research dollars) over his entire career than the assistant professor whose tenure he is voting on. Believe me when I say that the hypocrisy of this double standard is not lost on the junior faculty.

There's no doubt in my mind that the signal-to-noise ratio in archival journal papers has plummeted in the past two decades. 90% of all journal papers are superfluous, repetitive, or lacking in any significant advancement of the art, and I'll plainly admit that includes my own papers. Everyone in academia realizes what's going on, and knows it isn't good for the students or the faculty, but unfortunately that's the way the beans get counted in the academic world.

Re:What About Publish or Perish? (1)

kisak (524062) | more than 5 years ago | (#25434215)

Crappy papers don't get much citations or attention in general. Also, a scientific article does not need to be crap even though it is later shown not to be correct. Even the greatest scientists have made wrong theories and connections.

In a couple of years... (0)

Anonymous Coward | more than 5 years ago | (#25432881)

So... how do we know that John Ioannidis's research isn't false and likely to be refuted in a couple of years?

Internet to the Rescue (0)

Anonymous Coward | more than 5 years ago | (#25432899)

As the authors point out, published work is often wrong because journal editors know that sensational, wrong headlines garner much more information than mundane, correct headlines.

Their conclusion is that putting the articles on the internet would solve the problem.

Unfortunately, their solution is a like 10,000 spoons when all you need is a knife.

It's called (2, Informative)

assert(0) (913801) | more than 5 years ago | (#25432921)

publication bias.

Re:It's called (1)

arb phd slp (1144717) | more than 5 years ago | (#25433271)

Or the "Bottom Drawer Effect"
i.e. We never know how much valuable data is sitting in the bottom drawer of file cabinets unpublished because it returned a negative or no result?

We apologise again for the fault in the research (5, Funny)

ArbitraryDescriptor (1257752) | more than 5 years ago | (#25432945)

Those responsible for refuting the research of the people who have just been refuted, have been refuted.

Re:We apologise again for the fault in the researc (1)

albeit unknown (136964) | more than 5 years ago | (#25434201)

and then shot.

Follow the incentives (4, Insightful)

Badge 17 (613974) | more than 5 years ago | (#25432953)

Basic idea: high-profile journals want papers that are new and exciting. This means that scientists have an incentive to 1) rush their work, 2) choose fields that are popular, and 3) claim that their papers solve more than they actually do. This leads to sloppy, dishonest papers.

I'm not going to judge this paper - I haven't read it thoroughly - but to pair a title like "Why most published research findings are false" to a pretty well-known problem seems itself like an example of problem 3!

Re:Follow the incentives (1)

syousef (465911) | more than 5 years ago | (#25433083)

I'm not going to judge this paper - I haven't read it thoroughly

Chances are it's false.

If it's right the chance of it being wrong and claiming to solve more than it can is high, since surely the finding applies to itself.

If it's wrong, it's wrong.

Therefore chances are it's wrong.

All this circular logic is making me thirsty.

Re:Follow the incentives (0)

Anonymous Coward | more than 5 years ago | (#25434437)

I'm going to write a research paper whose conclusion is: "This research paper is incorrect." I will thereby form a world-destroying paradox. I mean, somebody's gotta destroy the world before the LHC does, right?

Re:Follow the incentives (1, Insightful)

Anonymous Coward | more than 5 years ago | (#25433111)

But this phenomenon can happen even if the research is done in good faith. Basically you can classify research as interesting/not-interesting and right/wrong. The problem is that those are not independent: It's much easier to be interesting (found 100x increase in something) if you make a mistake (confuse 0.01 with 0.01%). Since you don't publish something if it's not interesting, things that do get published have a higher chance of error.

Re:Follow the incentives (1)

Badge 17 (613974) | more than 5 years ago | (#25433175)

AC makes an interesting point, though any study that makes that basic an error will be corrected immediately and not cited hundreds of times, as in this paper. I suspect the errors induced are often more subtle - more a case of hopeful thinking than arithmetic ("Never believe a thing simply because you want it to be true" - Stephenson got that one right)

Context (2, Interesting)

edcheevy (1160545) | more than 5 years ago | (#25432973)

News flash! What works in one situation (or for one person) might not work so well in another. Too little research takes the context into account, particularly regarding any research that is human-related, and so it becomes easy to "disprove" prior findings.

More Irony (0)

Anonymous Coward | more than 5 years ago | (#25433015)

Ironically, this Economist article did a small, unrepresentative study and claimed that it has broad significance over a much larger topic. Its authors published the most compelling explanation they could think of for their results, ignoring more mundane ones (like, for example, the fact that a 'groundbreaking' study in Nature is going to spur more research which may well refute portions of the original study, whereas a study published in a secondary journal may not).

Refuting papers are correct? (1)

KevinIsOwn (618900) | more than 5 years ago | (#25433029)

Who's to say that the papers refuting this research are correct? It seems to be taken for granted that the dissenting papers are correct, and thus the original papers are wrong. It seems likely that the refuting papers may be wrong, or that there are complex situations in which both papers are correct (to differing degrees).

One third = Most (1, Informative)

Anonymous Coward | more than 5 years ago | (#25433041)

If one third of the papers are refuted, and this study concludes that "most" of the research is false, does that mean this study is part of the problem? No, i did not RTFA.

Re:One third = Most (1)

oddaddresstrap (702574) | more than 5 years ago | (#25433471)

No, it just means that you don't understand the "new math" used in the study.

Here come the global warming spooks (2, Insightful)

MosesJones (55544) | more than 5 years ago | (#25433047)

My prediction for this thread:

Several people will post about how this validates the TINY, TINY, TINY, number of scientists and LARGE number of completely uneducated "opinion" formers and MASSIVE number of people who think that "belief" is the same as fact.

What they will miss:

This article talks about how things are put out there then invalidated by SUBSEQUENT PUBLISHED RESEARCH, not about how there is a great conspiracy around something being "right" and everyone shouting down those who dare to disagree. Global Warming is something that has consistently been found to be happening and while certain bits have been revised due to subsequent research, most of that research has found that previously incorrect models were in fact too optimistic in their view.

This article doesn't strengthen your misguided, and uneducated, belief that Global Warming isn't real. When even the Republican candidate says its real then its time to let go and become part of the solution.

Re:Here come the global warming spooks (0)

Anonymous Coward | more than 5 years ago | (#25433221)

And if you want to, you can even forget "the Republican candidate saying it's real" (McCain isn't a very republican Republican, that's why he's been nominated). Even Libertarian magazines like Reason admit human-influenced global warming which will cause a variety of negative effects worldwide in 9 out of 10 of their articles, and they've really got nothing to gain by doing so.

Paper Only about "Therapy Effectiveness Research" (2, Insightful)

markk (35828) | more than 5 years ago | (#25433327)

Yes I agree - this paper is not about "Most Published Papers" in Science. It is about published papers in the area of therapy effectiveness. Especially those where we do not have a good model. Thus of course about half should be wrong I would guess, as established by later studies. This is statistics in action. When you are looking for high correlations and selecting for the positive, you will will get false ones. As long as this paper's authors could find LATER PUBLISHED RESEARCH showing this stuff was wrong, that is the scientific method. In fact if, say, 98 and 4/100's of papers were shown to be right later, I would smell something in the woodpile.

The meat of the article is the bias about reporting negative results. This is not a secret.

In regard to something like climate research, really it doesn't apply. but if you take the premise, it would generally bolster the 1000's (+) of papers over the years that show consistent effects and generally put more shadow on the couple showing otherwise. It would mean the papers the doubters bring up are wrong with even more percentage, since these are the papers with no validated mechanisms and generally many defects which immediately get pointed out. That is we would expect some wrong or null correlations to pop-up, given this paper and shouldn't put much support to isolated work that is not buttoned down to the max.

Thin Papers Hide Bad Work (0, Offtopic)

PingPongBoy (303994) | more than 5 years ago | (#25433101)

I really hate reading research papers. Many of them are cookie cutter papers to make it look like work was done - a bunch of text with a few graphs and lists thrown in, all followed by a massive list of references. The descriptions defy anyone to actually replicate the work. Theorems are proved on the basis of some other theorems in difficult-to-find references, but the logical steps (as you would see in a math textbook where theorems are proven in detail) are at best a few scattered mentions of the format "if x and y then z (and you better believe it because that's what happens when you do all the substitutions)."

In the past, people didn't have such a huge reference base so they could follow the logic, but now with computers, the Internet, and massive hard drives, papers ought to be much longer and more detailed. This would force researchers to not have the attitude of "because you are either studying for a Ph.D. or you have one, you should have the IQ to reverse engineer my logic".

When people start their education, they are told "show your work". Full credit is not given for just the final answer. Because there are more and more universities, institutes, students, problems, etc., people have less time to read about each project and follow every step of reasoning. It was necessary to keep papers short, and that wasn't such a bad thing when people took care to present well, but the system of trust can only persist as long as the trust is not broken. There are too many research areas now with their own little symbolisms and patterns of communications, as well as too many researchers who invent their own symbolisms and styles when they are unsure whether any standard exists. It's a Tower of Babel.

A solution in the computer age is quite simple. Computer storage is cheap enough to permit massive appendixes that give the details of derivations. The Internet can be used to distribute standard ways of expressing ideas and themes that are commonly found. The entire system needs to be more self enforcing by having papers widely available so that people can see what is the right way versus the wrong way. Then, the statistic of which papers have the most references can give a meaningful idea of which papers are the best.

When I see movies depicting life decades ago, I see that people presented themselves with greater complexity and attention to detail. Communications seem to be more bursty now, perhaps because everyone is trying to finish quicker with every objective. Often this leads to shallower thought though because there isn't time taken or given to ponder. So we may well be seeing "research" that just marginally advances a randomly selected result from someone else's papers, and that is the easy path to getting credit and "getting on with life". It's the whole attitude of "no one is going to care because there is so much going on and I'm just insignificant".

Computers can help here. If people want to achieve more signficance, they can produce more full-bodied writeups - this process itself forces them to think deeper and better, and if they have something worth telling, the world will find out.

Re:Thin Papers Hide Bad Work (1)

Badge 17 (613974) | more than 5 years ago | (#25433277)

In the past, people didn't have such a huge reference base so they could follow the logic, but now with computers, the Internet, and massive hard drives, papers ought to be much longer and more detailed.

While I agree with most of your post, I think there are real constraints on paper length. Mostly, these are researcher time - longer papers take longer to write, and to edit - and signal-to-noise - I need to know the basic idea of your paper *before* I decide to check your sign errors.

Of course, many papers in a high-profile journal have a more detailed, companion paper in some more specialized journal, which helps the situation - but you need to look for this paper! In some ways, it seems like we could profitably abolish "high-profile" journals and replace them with a combination of specialized journals and high-level overview articles like these for Physics [aps.org]

Re:Thin Papers Hide Bad Work (1)

CRCulver (715279) | more than 5 years ago | (#25433329)

I need to know the basic idea of your paper *before* I decide to check your sign errors.

Isn't that what the abstract is for?

IMPORTANT STUDY JUST PUBLISHED (0)

Anonymous Coward | more than 5 years ago | (#25433109)

Single malt scotch proven to be very effective in eliminating depression.

I now open the research to peer review of long-term efficacy studies.

Re:IMPORTANT STUDY JUST PUBLISHED (1)

arb phd slp (1144717) | more than 5 years ago | (#25433367)

Single malt scotch proven to be very effective in eliminating depression.

I now open the research to peer review of long-term efficacy studies.

Starting the grant application right now!
I'll be recruiting participants as soon as it passes IRB. Any takers?

Re:IMPORTANT STUDY JUST PUBLISHED (1)

nevillethedevil (1021497) | more than 5 years ago | (#25433839)

Count me in. Sounds like just what I need to take my mind of my crack habit.

Obvious (2, Insightful)

serviscope_minor (664417) | more than 5 years ago | (#25433117)

Many are wrong. This is pretty obvious, and it's also why science works. Eventually, the wrong ones will be replaced with something less wrong, and so on.

The flawed Method (1)

kitsunewarlock (971818) | more than 5 years ago | (#25433127)

This is in a large way because the empirical method is flawed for humans. It requires complete rationality and unaffected views, two things you just aren't going to find in human beings. Even our currently taught scientific method promotes going into an experiment with expectations. The fact these experiments cost money, time, etc... (many times in the forms of grants that will be pulled if you don't show significant progress), results are often overblown or outright false. Its not that the lofty ideals of the scientific and empirical methods aren't impressive--simply that they are as currently impossible to reach as the idea of being a perfect Christian.

I found a study (2, Funny)

Ranger (1783) | more than 5 years ago | (#25433131)

that proves most published research findings are true.

He is absolutely right (2, Interesting)

Anonymous Coward | more than 5 years ago | (#25433151)

Why is it that a large portion of scientific research today is garbage. Well one very powerful reason, money. I saw this firsthand working at a major university medical center on large scale behavior research projects. The outcomes of our studies directly effected existing and experimental drugs and the drug company representatives were right there alongside the researchers at all levels of the process. Professors received "gifts" and other unofficial incentives from them regularly. I saw at least one study where the results were out and out fabricated so that the results would support the effectiveness of a particular drug for treating a childhood psychiatric disorder. In other cases data was included after the fact or blanks were filled in by clinicians from memory. All practices that are highly unscientific. Many of these studies resulted in drugs and treatments for children that are in use today and based on research that is at best questionable and at worst fraudulent. When there is a profit motive behind science it becomes very difficult for it to remain true science and sadly that is the state of affairs in many fields today.

This statement is incorrect... (1)

Jorgensen (313325) | more than 5 years ago | (#25433169)

Strange... Let us see how this works out: 33% of papers are refuted in later-published papers. But of those later-published papers, 33% of those are refuted too. And of those, 33% are refuted... If my maths is right, then over time *all* research will have been refuted!? (I don't recall anybody refuting the general theory of relativity...) Luckily, this piece of science will then also be refuted - which then wipes out the basis of my argument... G'ah - this science stuff is complicated!

Your math is wrong (1)

roguegramma (982660) | more than 5 years ago | (#25433369)

Well I suspect you didn't do any math.

If 1/3 = 33.33% is refuted and 1/3 of the remaining 2/3 and so on, then eventually 1/2 = 50% will be refuted.

The formula for this is p/(1-p) cf. http://en.wikipedia.org/wiki/Power_series [wikipedia.org]

Re:Your math is wrong (1)

Jorgensen (313325) | more than 5 years ago | (#25433507)

If statements can only be refuted in the first "few years" - yes: you're correct. I assumed that this was similiar to the radioactive half-life: 33% refuted within a certain period => even more will be refuted during a longer period. And thus we will get closer and closer to 100% over time.

Uhm my math is wrong too (1)

roguegramma (982660) | more than 5 years ago | (#25433597)

Correct is that two thirds eventually will be refuted(like my first post is refuted) 1/3 + 2/9 + 4/27 = 1/3 *(1 + 2/3 + 4/9 + ..) = 1/3 * (2/3 / (1-2/3)) = 1/3 * 2 = 2/3 = 66.67%

Re:This statement is incorrect... (1)

arb phd slp (1144717) | more than 5 years ago | (#25433393)

I think it levels out to an asymptote eventually. But what do I know? I'm just a lowly social scientist and all of my findings are bullshit.

Re:This statement is incorrect... (0)

Anonymous Coward | more than 5 years ago | (#25433689)

But what do I know? I'm just a lowly social scientist and all of my findings are bullshit.

Why, yes, that is correct [xkcd.com] .

Not wrong, just misguided. (1)

Metasquares (555685) | more than 5 years ago | (#25433177)

I agree, but not for the same reasons (although the author's reasoning sounds plausible for big-name journals).

I've found that peer reviewers very seldom give good critiques of the methodology. Rather, most of the comments appear to be on the scope of the paper - comments of the variety "You're doing X. Smith et al. has done Y (which is tangentially and usually very weakly related to X) yet you don't mention this." I suspect that this is because most reviewers don't know enough about the research methods being used to provide a thorough, useful, and accurate critique, but still need to write something, so they take shots at the scope instead, trying to draw on what they do know.

If the methodology is not being effectively critiqued, there is little to no selection for sound research. And if each point you bring up causes reviewers to demand you bring up five more, this introduces selective pressure towards papers that either say far too much or far too little.

I still think a Digg-style system would work well for paper publication. Any half-sound research would be available, but research that people find "useful" would naturally rise to the top.

Paradox (1)

saxoholic (992773) | more than 5 years ago | (#25433203)

So... Since this study is published, I can assume that it's wrong... but then, if this study is wrong, it's right... but then... Excuse me while I pass out -- If it weren't for my horse...

And this is bad why? (3, Insightful)

PCM2 (4486) | more than 5 years ago | (#25433211)

Scientific research is just that -- research. If it were as easy as doing a couple of experiments, revealing the "truth" and moving on to the next thing, we'd all be living around Alpha Centauri by now. But science is hard and therefore a lot of conclusions are naturally going to be wrong. If that weren't the case then we wouldn't even need any scientific journals -- all we'd need would be newspapers.

Remember the whole "theory of evolution" issue that the creationists keep harping on? "They call it a theory so it must not really be true?" We all know that evolution is just about as "true" as any science gets -- and yet surely there are some portions of the current body of knowledge about evolution that will one day be falsified by later research. That's not a bad thing.

Notable research that has since been thought to be flawed or insufficient: Newtonian physics. Niels Bohr's model of the atom. Gregor Mendel's research into genetics. Einstein's theory of general relativity. Koch's postulates for determining disease causation. Quantum mechanics. And so on.

Correct the topic, please? (1)

argent (18001) | more than 5 years ago | (#25433217)

Please replace "false" with "incorrect". The word "false" implies deliberate fraud, and while that undoubtedly happens the cited articles do not suggest that fraudulent papers are in the majority.

Applies to all information (1)

moteyalpha (1228680) | more than 5 years ago | (#25433247)

New information supplants old ideas and how does this process become sensible? I often run into manuals that are a few versions out of date on the web and they never get removed. Until somebody can come up with a way that everyone can share information in a common data base that is version controlled, the problem will persist. Wikis and Wikipedia are good ideas, but the concept needs some kind of extension so that information and its corrections are connected in some way.

The fact that government / companies might screw with the data for personal interest is a separate issue of letting an inherently corrupt process manage your information.

I suppose the responsibility falls to the end user to deal with the inherent problems in managing and verifying the data that they use for their purpose. I doubt that a complete, secure, common, valid data base ( or system ) could be devised when more than one person is involved.

Unclear what they mean (5, Insightful)

Sapphon (214287) | more than 5 years ago | (#25433273)

Even after reading the article, I'm still not sure if the authors are saying:

A) Given that research has been published, it is more likely to be false than not; or
B) Given that research is false, it is more likely to be published than is the case for true research.

I mean, it says:

Dr Ioannidis made a splash three years ago by arguing, quite convincingly, that most published scientific research is wrong.

So, (Wrong Articles)/(Total Articles) = >=0.5, right?
But the only figures I can find in the same article are:

Dr Ioannidis based his earlier argument ... on a study of 49 papers ... (H)e found that, within only a few years, almost a third of the papers had been refuted by other studies.

So.. "most" is now "less than one third"?

I'm somewhat alarmed that The Economist lets people who don't seem to grasp basic statistics write their articles.

Really? (0)

Anonymous Coward | more than 5 years ago | (#25433317)

Well then this finding itself is more likely wrong than right, so we can just ignore it...

Publicity Bias (0)

Anonymous Coward | more than 5 years ago | (#25433395)

The problem is not even necessarily that any of the studies themselves are flaws, just that only the "exciting" results see the light of day. If 100 groups perform the same study, you would expect 10 of them to reject the null hypothesis with 90% confidence even if it were true. Now if only those studies that reject are published, it looks like 10 studies all rejected the null hypothesis with 90% confidence. On the other hand a meta-study that took into account the work of all 100 groups would reveal no statistically significant results.

Rutherford (1)

NoOneInParticular (221808) | more than 5 years ago | (#25433437)

I think most of the mess is caused by statistics forming the conclusion of the research.

"If your experiment needs statistics, you ought to have done a better experiment."

-- Ernest Rutherford (1871-1937)
[In N. T. J. Bailey: the Mathematical Approach to
Biology and Medicine]

Re:Rutherford (0)

Anonymous Coward | more than 5 years ago | (#25433645)

Rutherford worked in physics in the late 1800's to the early 1900's. Biology was still in its infancy.

Biological research necessarily relies on statistics. There few absolutes in biology. Almost every rule has an exception.

"most"? (2, Insightful)

jwilty (1048206) | more than 5 years ago | (#25433457)

When did "almost a third..." become "most?"

nothing to see here (1)

ReedYoung (1282222) | more than 5 years ago | (#25433473)

Like missing car keys, the correct/best theory of [_____] is in the last place we look -- because once we find them, we stop looking!

small sample size :-) (2, Insightful)

l2718 (514756) | more than 5 years ago | (#25433477)

So, a paper claiming that small sample sizes lead to wrong conclusions is based on analysing a sample of only 49 other papers? The mind boggles with the self-applicability ...

make me wonder (1)

FudRucker (866063) | more than 5 years ago | (#25433513)

what research findings were published to promote medicines in order to further the corporate profits (greed)...

Ugh. (1)

gsarnold (52800) | more than 5 years ago | (#25433625)

This sounds like a research paper written by an MBA.

0) The scientific method is a little more complicated than "trying things and observing what happens".

1) Procedures are scientific, not explanations or conclusions.

2) In science, repeatability trumps everything, and ALL RESEARCH ULTIMATELY HAS TO PASS THE REPEATABILITY TEST. If your results cannot be repeated, something was wrong with it.

3) The commercial medical field in particular has a history of forwarding the first round of research results straight through the marketing department instead of treating it as opportunities for further investigation.

-G.

P.S. No, I didn't RTFA.
P.P.S. ...I'm not going to, either.
P.P.P.S. ...On principle!
P.P.P.P.S. ...SCIENTIFIC principle!
P.P.P.P.P.S. ...Not gonna explain what I mean, either!
P.P.P.P.P.P.S. ... shoulda went with ////slashies!////, 'P.'s are hard to type!

umm? (1)

Jenos (1255810) | more than 5 years ago | (#25433635)

1/3 = Most?

Oh, the irony! (1)

AlgorithMan (937244) | more than 5 years ago | (#25433829)

Ioannidis argues that scientific research is so difficult -- the sample sizes must be big

Dr John Ioannidis bases his argument about incorrect research partly on a study of 49 papers

Oh, the irony!

The Proof is in the pudding (0)

Anonymous Coward | more than 5 years ago | (#25434077)

a study of 49 papers ... almost a third of the papers had been refuted by other studies

"Almost 1/3 of 49" = "Most Published Research Findings"

Nice math, Economist. I suppose Dr. Ioannidis does have something to gain by publishing inaccurate journal articles, though...

news at 11 (1)

kisak (524062) | more than 5 years ago | (#25434149)

Wow, headline grabbing, potential break-through scientific theories get a lot of scrutiny and citations from fellow scientists, and many of the theories fail the test of peer review. Daring theories are important for the further advances of science, but by definition most of them will fail.

Fixable? (2, Insightful)

philspear (1142299) | more than 5 years ago | (#25434229)

It's important to identify a problem no matter what, but are any of these biases fixable? I would argue that some of them, specifically the bias toward positive results, is not fixable and is inherent to how science works.

To quote one of the articles "...negative results are potentially just as informative as positive results, if not as exciting." But negative results often require much much more verification than positive results, if they can be verified at all, and are limited in how much they can tell you. The antidepressant studies mentioned, a negative result, that the antidepressants did nothing, only tells you that in the patients tested, the doses tested did not give you a noticeable positive result. Publishing a negative result on that would have very limited conclusions. The next year, they could find that doubling the dose was actually effective, making the writeup of the earlier negative result pointless and even more trivial. A waste of time, plus then you've published saying your own product doesn't work.

Negative results get even more pointless in other fields. If someone does a mutagenesis screen for a particular defect in C. elegans, and doesn't find any mutants affecting that, it could be noteworthy, indicating that any genes affecting that process were so vital that when you took one away you didn't get a worm at all, or it could just be luck that genes affecting the process were never mutated, or the researcher didn't do it correctly, or all genes involved were redundant, or some combination. What conclusions could you draw from that? It would be a negative result that would be nigh impossible to tell anything from. Without any positive hits, you could go to the trouble of making sure you did it correctly, but you're not going to make sure every gene got hit at least once, that would be impossible.

In still other cases, a negative result is often retrospectively found to be the fault of the researcher. Who wants to publish something that is basically telling your peers how dumb you are?

There's also that it requires a lot of extra work to make sure it's a negative rather than a null result. Usually when I hit a negative result, my inclination is to see if I did it wrong by repeating the experiment if possible, if it comes up negative again I usually take a different approach, if that also gets a negative result I re-evaluate. I don't ever do all the other supporting experiments that would be needed to convince a reviewer it's a real negative result. If I use an RNAi construct to knock down a gene, and it doesn't do what I'm expecting or anything else interesting, I don't verify the gene is actually knocked down, since that's more effort that would probably be a waste. I'm definitely taking a risk that it's a real result, but it's hard to prove a negative and there's also less motivation to do so.

The limited ability to make positive conclusions about negative results also limits where they could be published. There is a journal for negative results, but a publication there is not something I personally would put on a CV.

So while it is interesting that a bias against negative results may be throwing us off, it's not very usefull knowledge, because I don't see us able to do anything about it.

Publishing work that you claim doesn't work .... (1)

Blue Warlord (854914) | more than 5 years ago | (#25434319)

Now, I do wonder how many of my fellow scientists have ever published a paper with an approach, which they invalidate with their own validation in the same paper. I have the feeling the peer review process puts to much emphasize on positive results. Author perception is that reviewers easily ignore the fact that negative results are also significant. Hence, it's not worthwhile to publish something that doesn't work. However, imho knowing why something doesn't work is just as important as knowing how it should work...

Uhm... (0)

Anonymous Coward | more than 5 years ago | (#25434473)

Because they're PUBLISHED! They're after getting funding and positive reviews of the findings, and if that means fibbing them then so be it.
Why is this news?
Load More Comments
Slashdot Login

Need an Account?

Forgot your password?
or Connect with...

Don't worry, we never post anything without your permission.

Submission Text Formatting Tips

We support a small subset of HTML, namely these tags:

  • b
  • i
  • p
  • br
  • a
  • ol
  • ul
  • li
  • dl
  • dt
  • dd
  • em
  • strong
  • tt
  • blockquote
  • div
  • quote
  • ecode

"ecode" can be used for code snippets, for example:

<ecode>    while(1) { do_something(); } </ecode>