Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

Most Science Studies Tainted by Sloppy Analysis

ScuttleMonkey posted more than 6 years ago | from the devil-is-in-the-details dept.

Science 252

mlimber writes "The Wall Street Journal has a sobering piece describing the research of medical scholar John Ioannidis, who showed that in many peer-reviewed research papers 'most published research findings are wrong.' The article continues: 'These flawed findings, for the most part, stem not from fraud or formal misconduct, but from more mundane misbehavior: miscalculation, poor study design or self-serving data analysis. [...] To root out mistakes, scientists rely on each other to be vigilant. Even so, findings too rarely are checked by others or independently replicated. Retractions, while more common, are still relatively infrequent. Findings that have been refuted can linger in the scientific literature for years to be cited unwittingly by other researchers, compounding the errors.'"

cancel ×
This is a preview of your comment

No Comment Title Entered

Anonymous Coward 1 minute ago

No Comment Entered


How is this news? (0, Troll)

dmsuperman (1033704) | more than 6 years ago | (#20654229)

We all know that a good portion of all studies are based on bogus facts or on other studies based on other studies based on nothing.

Re:How is this news? (1)

SimonGhent (57578) | more than 6 years ago | (#20654257)

Is this not how it supposed to work? You publish and it is reviewed by peers who spot any mistakes or bias?

Re:How is this news? (5, Funny)

Leftist Troll (825839) | more than 6 years ago | (#20654349)

How can we even trust this study?

After all, studies show that most studies are wrong.

Re:How is this news? (2, Funny)

Garridan (597129) | more than 6 years ago | (#20655351)

Well, of course you can trust it, since somebody can verify this independantly. ;)

Re:How is this news? (5, Insightful)

Anonymous Coward | more than 6 years ago | (#20655741)

After all, studies show that most studies are wrong.


The fact is, good science is hard work. In fact, it is damn hard work, requiring not only a supremely keen intellect but a very high tolerance for tedium, great attention to detail, and usually a big fat wad of cash. Also, it requires a profound lack of ego (and the ability to cope with failure and keep trying), given that a trememdous amount of effort could (and frequently does) wind up being completely discounted by a peer-review or another study.

The endeavor of scientific research obviously provides us tremendous benefits, and is furthering the evolution of our species at a blindingly fast rate (depending on how you look at it, of course). It is very important, very hard, and very expensive.

There are many, many people who would like to be scientists but really don't have the brain for it (as I stated above, it isn't just intelligence that matters). Unfortunately, a lot of them wind up doing research anyway, and they cause problems. Hopefully there are enough good scientists with enough funding to clean up their mess.

Re:How is this news? (5, Insightful)

posterlogo (943853) | more than 6 years ago | (#20654691)

"We all know..." What are you basing this on??? As a postdoc, I've committed myself to a massive amount of work and I'm certainly not doing it for pay (which is meager), but a LITTLE amount of respect would be nice. I've published a few studies and it was incredibly hard work to do the kind of careful science that gets published. A small amount of scandals and people like you who swallow any sensationalist piece of news out there really cast things in an unfair light. I encourage you to read more scientific literature and actually try and understand how the scientific process works. Do you really think we live in the kind of technological age as we do in spite of "a good portion of all studies" being "bogus" or "based on nothing"? I find this incredibly insulting.

Re:How is this news? (5, Funny)

mh1997 (1065630) | more than 6 years ago | (#20654857)

"We all know..." What are you basing this on??? As a postdoc, I've committed myself to a massive amount of work and I'm certainly not doing it for pay (which is meager), but a LITTLE amount of respect would be nice. I've published a few studies and it was incredibly hard work to do the kind of careful science that gets published. A small amount of scandals and people like you who swallow any sensationalist piece of news out there really cast things in an unfair light. I encourage you to read more scientific literature and actually try and understand how the scientific process works. Do you really think we live in the kind of technological age as we do in spite of "a good portion of all studies" being "bogus" or "based on nothing"? I find this incredibly insulting.
It's a well known fact that 42% of all statistics are made up, and that over 70% of studies use these false statistics. Therefore, 89% of all studies are flawed.

You'd think a postdoc would have known this.

Re:How is this news? (1, Informative)

Anonymous Coward | more than 6 years ago | (#20655177)

As another post-doc, I have to say that while the article writer is blunt, tactless, and overdramatic, he has something of a point. Although the reviewers of papers submitted to conferences are usually diligent and careful (and give excellent feedback), I would be very surprised if any of them attempted to replicate the experiments described in the papers -- simply because it's an infeasible amount of work for them to have to do. So if the authors have made any mistakes that aren't obvious just from the text, they're unlikely to have be picked up by the reviewers. Most experiments never get re-performed elsewhere, so conclusions do not get confirmed as often as perhaps they should (academics are rewarded for new work, not reproducing and checking the work of others).

In fact on a good day, even if all experiments were carried out perfectly, the literature would still contain a lot of incorrect conclusions. Statistical tests in many fields are done at the .05 level -- ie a 5% chance that this result was due to random chance. So, up to 1 in 20 conclusions would be utterly wrong - it was just random chance after all. But since even conference papers cite a good 10 or 12 others, around half the papers would use an incorrect result in their discussion of an issue...

I'm painting the picture a little vaguely and perhaps just as overdramatically as the original author, but my point is that just because something is published in the literature does not mean you can check your critical judgement at the door. Scientists like you and me do not abandon their critical judgement, but sadly many science journalists (and hence the public at large) do. And that, I think, is the point the original author was trying to make.


Engineers must be fixing all your mistakes! (1)

tjstork (137384) | more than 6 years ago | (#20655367)

Ya know, you guys just write these papers that are all wrong and riddled with errors. Lucky for civilization, there are plenty of engineers that turn this slop into useful products for truth, justice, and the American way! :-)

Yes, science is worthless, look to Jebus. (0, Flamebait)

FatSean (18753) | more than 6 years ago | (#20654939)

When it becomes too difficult for a small mind to sort through all the data out there, there is a place where one can turn: the mindless belief of the religious.

Otherwise known as the ol' "If it isn't obvious to me, a god did it!" mis-thought.

Mod parent down (0)

Anonymous Coward | more than 6 years ago | (#20655501)

How is your comment in any way relevant? The story is about a tenured scientist complaining about the quality of verification of science results; there is no mention of religion anywhere in this thread, except in your post, where you insert some gratuitous flamebait (as if there aren't enough religious arguments on Slashdot yet).


Corrections aren't read (1)

Lucas123 (935744) | more than 6 years ago | (#20655121)

But once the study is out there, it's taken as gospel unless some other study comes along to discredit it. The vetting should come before the printing.

Re:How is this news? (2, Insightful)

m2943 (1140797) | more than 6 years ago | (#20655585)

I think the majority of working scientists don't "know" this and won't believe it either.

Personally, I think it's true. Between publish-or-perish, financial conflicts of interest, and various political and social movements trying to influence science, a lot of published science is worthless.

However, there's a difference between believing that and actually showing that. Anybody wanting to fix that needs clear and convincing proof first.

Yup. (3, Insightful)

Estanislao Martnez (203477) | more than 6 years ago | (#20654251)

And they are routinely reported sensationalistically in the media, and most of you people who are reading this right now swallow it all hook and sinker.

Re:Yup. (0)

Anonymous Coward | more than 6 years ago | (#20654275)

Must be how the global warming myth started.

Re:Yup. (2, Interesting)

AKAImBatman (238306) | more than 6 years ago | (#20654625)

Must be how the global warming myth started.

Actually, it is. According to the scientists who reported in "The Great Global Warming Swindle", the concept of CO2 warming was a fairly small area of research that wasn't taken very seriously (and actually seen as a benefit to combat Global Cooling!) until Margaret Thatcher decided to fund CO2 warming research. She was a big proponent of Nuclear Power and saw the CO2 warming research as another bullet point for the advantages of nuclear power. Once there was money on the table for the research, things went downhill from there.

If you haven't seen the documentary, I highly recommend it. One of the key issues they point out was that Gore's graph didn't show a slight problem with CO2 warming theory due to its scale of several million years. In specific, in the ice core samples used to create the data CO2 levels rose about 800 years AFTER the temperature rose. The reason? Because the ocean can hold less CO2 when it is warmed. (Which takes hundreds of years to cause an appreciable temperature change in a water body that large.) So it stops being a carbon sink and actually rejects already dissolved CO2. When the temperatures cool, the CO2 levels drop a few hundred years later as the ocean is able to reabsorb the CO2.

I came here for the global warming ref (3, Informative)

benhocking (724439) | more than 6 years ago | (#20654883)

And I wasn't disappointed. You also managed to please me with your inclusion of the bogus "The Great Global Warming Swindle", "Global Cooling", and conspiracy theories. Excellent!

As an interesting aside, I thought that this argument had been dropped because it was a little too easy to shoot down:

Because the ocean can hold less CO2 when it is warmed.
The interesting thing is that, despite warming temperatures, the oceans are holding more CO2 than before (which lowers their pH level as CO2 + H20 = C2H03, carbonic acid). This is possible because increasing the partial pressure of CO2 in the atmosphere (as we've done significantly) more than counteracts the decreased solubility due to temperature rises. It's possible that in the past this was a factor (although you should read up on those time courses and realize that your 800 year figure is also bogus), but it's clearly not true today. Global warming theories aren't based on correlations, they're based on fundamental principles of science.

Re:I came here for the global warming ref (1)

Arthur B. (806360) | more than 6 years ago | (#20655125)

Science only tells you that *ceteris paribus* an increase of CO2 in the atmosphere will increase the average temperature.

This is a rock solid fact but it's absolutely useless to predict anything about the temperature, because increase in CO2 do *not* happen ceteris paribus. There are so many feedback loops involved (growth of vegetation, oceans for example) that short of historical correlation there is little we can know or say about the effect of CO2 on temperature. Even something as simple as the ceteris paribus impact of CO2, while theoretically computable is unknown.

New tactic (1)

benhocking (724439) | more than 6 years ago | (#20655259)

I'll give you points for making an argument I hadn't heard before, even if it's wrong. The basic science is only a starting point, but detailed analysis allow you to make meaningful predictions about what happens when the concentration of CO2 increases. In fact, these predictions have borne out fairly well [sciencemag.org] .

Re:New tactic (0, Troll)

kickedfortrolling (952486) | more than 6 years ago | (#20655587)

Im not sure an effect predicted with hundreds of years worth of data can be tested in the 10 years since anthropogenic global warming became fashionable.. I think this may be an excellent example of basic science tainted by obscene analysis.

Something as chaotic as climate must be an easy target for tinkering?

Another global warming point: who is going to turn around and admit they were wrong if they find something contradictory? Denying AGW is basically denying the holocaust these days!

10 years? (1)

benhocking (724439) | more than 6 years ago | (#20655803)

Im not sure an effect predicted with hundreds of years worth of data can be tested in the 10 years since anthropogenic global warming became fashionable.. I think this may be an excellent example of basic science tainted by obscene analysis.
10 years?!? Here are a couple links you might want to look at:
http://www.msnbc.msn.com/id/20113753/site/newsweek/ [msn.com]
http://www.pbs.org/wgbh/pages/frontline/hotpolitics/etc/cron.html [pbs.org]
You'll note this bit from 1979 (nearly 30 years ago):

U.S. National Academy of Sciences reports that global temperatures could rise 1.5 to 4.5 degrees Celsius if carbon dioxide levels double. "A wait-and-see policy may mean waiting until it is too late," the group warns.
As early as the late 50s some scientists were already discussing how increased CO2 would lead to higher temperatures. This issue is not 10 years old.

Re:Yup. (1)

b17bmbr (608864) | more than 6 years ago | (#20654907)

well, you got the problem 110% exactly right: government funding. global warming became a problem once there was (taxpayer funded) research money aplenty. plus, when you consider the real motives behind all this crap, it's simply about how to control peoples' lives. enviros are just another radical religious group. be they fundamentalist christians or tree worshippers, they fancy themselves Plato's enlightened guardians. if they really cared about the environment, they'd lead by example and live ascetic lives.

Re:Yup. (3, Insightful)

CRCulver (715279) | more than 6 years ago | (#20655221)

It doesn't follow that environmentalists must live ascetic lives in order to be true to their beliefs. There is a significant subset of environmentalists who believe that development of technology should be persued, because sufficiently advanced technology used in the right way can benefit the environment. This line of thought is very visible in science-fiction. Kim Stanley Robinson, whose own Mars trilogy touched on the theme, edited a fine collection called Future Primitive [amazon.com] ("The New Ecotopias") where various writers speculate about how the advancement of mankind could lead to a stronger ecosystem.

Re:Yup. (1)

b17bmbr (608864) | more than 6 years ago | (#20655397)

I don't disagree. In fact, I do think that technology can lead to much healthier lives. I would love to see us leave the fossil fuel based economy and develop things like nuclear. And if you look at recent developments in coal, technology has made it much cleaner, and I believe cheaper too. as for the ascetic part, gore and his cohort travel around in private jets. edwards has his mansions and SUV's. etc. it's really no different than a senator who rails against homosexuals cruising in an airport bathroom. they're hypocrites. they don't believe what they preach for the rest of us. so in some sense, yes, they should be "role models".

Re:Yup. (2, Insightful)

mrseth (69273) | more than 6 years ago | (#20655475)

You really believe this? This is completely counter to everything I've experienced as a scientist. People become scientists because they love science not for the monetary rewards. If you want to be a rich climate scientist, all you need to do is get a job at the American Petroleum Institute. I am an "enviro" and could quite give a shit less in general about how you live your life, but yes, I will care what you do if it involves dumping dioxin into my local river. If that makes me a "nut" then I don't want to be sane.

Yeah, but its not "your" river.. (1)

tjstork (137384) | more than 6 years ago | (#20655861)

an "enviro" and could quite give a shit less in general about how you live your life, but yes, I will care what you do if it involves dumping dioxin into my local river.

It's not your river. It's mine.

signed, Jacks Nuclear Dumping.

PS. If we catch you on our river, we will shoot your for trespassing!

Re:Yup. (1, Interesting)

Anonymous Coward | more than 6 years ago | (#20654967)

This "lag" is well understood [realclimate.org] .

Re:Yup. (1)

MightyMartian (840721) | more than 6 years ago | (#20654979)

You are aware, I trust, that this program was pretty much discredited. Standard pseudo-scientific nonsense, not to mention taking a researcher out of context. Good to know you approve of liars.

Re:Yup. (0)

Anonymous Coward | more than 6 years ago | (#20655035)

And for those of you who don't know, TGGWS is just BS. Start to finish, pure crap. Just about every scientist in it has said that they were misrepresented, the director is known for being a fabricator. It's the "Dinosaurs lived with adam and eve" version of science. Utter shit. And you are below contempt for spreading it. See:

http://flet.org/node/20 [flet.org]
http://www.antarctica.ac.uk/about_bas/news/news_story.php?id=178 [antarctica.ac.uk]
http://folk.uio.no/nathan/web/statement.html [folk.uio.no]
http://www.royalsoc.ac.uk/news.asp?id=6089 [royalsoc.ac.uk]
http://environment.guardian.co.uk/climatechange/story/0,,2119695,00.html [guardian.co.uk]
http://news.independent.co.uk/environment/climate_change/article2368999.ece [independent.co.uk]

and TONS of others. The data are misrepresented, words taken out of context etc etc. Wall to wall shit from someone with a vested interest in spreading FUD. I know some of you still don't think global climate change happens, and some of you think the earth is flat too, but the HUGE SCIENTIFIC CONSENSUS is that the climate is changing, to some degree it's man made and we need to do something about it.

Re:Yup. (1)

AKAImBatman (238306) | more than 6 years ago | (#20655263)

And you are below contempt for spreading it.

Actually, you got a chance to attempt to discredit it publicly. Which is what the process is about. Scientists argue back and forth on the exact causes, and correct each other as necessary. Now I'm reviewing the data you provided that swings the other direction.

Just about every scientist in it has said that they were misrepresented

Unfortunately, that's one point you didn't make with your links. I've read through them and I don't see where the scientist who were present in the documentary disclaimed their roles. Not that I don't believe you, but do you have links that back that up?

Re:Yup. - Nup (0)

Anonymous Coward | more than 6 years ago | (#20655557)

> In specific, in the ice core samples used to create the data CO2 levels rose about 800 years AFTER the temperature rose. [etc]

This is a *very* highly disputed interpretation of the data. As I recall, the original study put an upper limit of around 800 years on the temperature rise, but later studies have narrowed that upper limit considerably. There are several that suggest it occured much more rapidly - within 2-3 years. And yes, ice core studies can be that accurate since the cores have layers of ice in them that accumulate every winter and are detectable like tree growth rings.

The particular "documentary" you mention has been widely criticised and soundly discredited now in many countries - the BBC has edited it several times to remove inaccuracies since its original airing and it is now substantially shorter (about 1/3 or 20 minutes) as a result.

Here it comes... (3, Funny)

Penguinisto (415985) | more than 6 years ago | (#20654253)

Insert politically charged science topics, point to 'em as examples, and launch into a stupid flamefest over it all in 3... 2... 1...


Re:Here it comes... (1)

MightyYar (622222) | more than 6 years ago | (#20654383)

Can't we skip that and jump right into how this supports intelligent design?

Re:Here it comes... (1)

allthefish (1158249) | more than 6 years ago | (#20654507)

Or, we could just skip that step, too, and just move right on to how this supports the fact that Microsoft is evil, as that's where every story on /. ends up eventually.

as a phd student (3, Insightful)

KeepQuiet (992584) | more than 6 years ago | (#20654295)

all i can say is "duh!". Everybody, being under pressure of "you have to publish", publish whatever they can. Sad but true

Re:as a phd student (1)

posterlogo (943853) | more than 6 years ago | (#20654553)

As a postdoc, I'd be interested to know what field you're in and where you study. If you actually believe what you just said, get out now. We don't want people who're so susceptible to fraud in science.

Re:as a phd student (0)

Anonymous Coward | more than 6 years ago | (#20654955)

In my field we don't want people who can't read. No one mentioned fraud, except to say it wasn't fraud.

Fairly common knowlege (5, Informative)

everphilski (877346) | more than 6 years ago | (#20654985)

It is fairly common knowledge that 3 things factor into tenure (in this order): (1) being published (2) bringing funding into the university and (3) teaching.

1. A good number to shoot for is 15 journal articles in your first 6 years. If you don't have tenure in 6 years chances are you are never going to get it. The point of being published is to get the name of the university out.

2. Should be self-explanatory. You need to bring in $$$ to the university. The more you bring, the more profitable you are and the more they need to keep you around. But publishing is still more important.

3. Teaching, while as students we all feel is important, is actually the least important thing towards tenure. A mediocre or even bad teacher who writes papers (that get accepted by excellent journals) at a rapid pace will get tenure where an excellent teacher who can't write for the life of him will not. This is why you often see people from industry teaching. They teach for the love, tenured professors are there for the research and for the higher level teaching (where it is more a relation of facts, not an educational process).

The 'sloppy analysis' referred to is not 'fraud' as you cite. There is a difference between fraud and sloppy analysis. The rush to put out papers (between 2 and three a year, by this guide, for tenure) causes some slop to occur. As a reference, I've been working on a paper with my advisor and a (yet-to-be-tenured) professor for almost a year already, and we are just submitting it to a major journal. And the paper is based mostly off of my thesis work completed a year ago! A good paper and good research takes time. But please, do not mistake sloppy analysis for fraud. Mistakes are one thing, deception entirely another.

SOURCE: Advice to rocket scientists: A Career Survival Guide for Scientists and Engineers. Dr. Jim Longuski, published by the AIAA in 2004. But again, this is fairly common knowlege and can be found anywhere you look. As a postdoc (I am too) I'm suprised you didn't know ...

I think teaching might come fourth (1)

benhocking (724439) | more than 6 years ago | (#20655385)

Number 3 is something akin to citizenship: participating in meetings, bringing in guest speakers, etc.

Sensationalist... (4, Insightful)

posterlogo (943853) | more than 6 years ago | (#20654317)

It is way off the base to say that "most published research findings are wrong". It is often the case that data analysis and interpretation for particular aspects of a research project (like 1-2 figures in a 7 figure paper) are up for vigorous debate. The scientific community can, in the long run, converge on very robust ideas, and drop those that are flimsy. To misleadingly imply that most research is wrong, which is exactly what the post suggests, is just poor interpretation of flimsy data, ironically.

Re:Sensationalist... (5, Insightful)

posterlogo (943853) | more than 6 years ago | (#20654509)

Furthermore, this epidemiologist primarily studied medically-related publications, and in fact focused mostly on high-profile research that make broad claims, or relied heavily on statistics to support a conclusion. Many research publications at the cell/molecular level do not rely on subtle statistical comparisons to prove a point. This guy is singling out research that is based heavily on correlations (like people with x, y, z are more likely to get a, b, or c diseases). He is only an expert in his own field, and I don't think he is qualified to judge every level of scientific publication, but he certainly doesn't mind the media attention.

Re:Sensationalist... (1)

Miraba (846588) | more than 6 years ago | (#20654741)

Oh, if only I had a modpoint...
Medical research != all scientific research; it's much more prone to errors due to how it's performed and analyzed. I hope your point doesn't get buried amongst all the charges of corruption.

Sloppy analysis (2, Insightful)

benhocking (724439) | more than 6 years ago | (#20654927)

So, is what you're basically saying is that this study was tainted by sloppy analysis?

Re:Sloppy analysis (1)

Miraba (846588) | more than 6 years ago | (#20655141)

No, the article in the WSJ is tainted by sloppy analysis. The paper itself (a metareview) shows that the authors recognized that their own biases could be a problem. From the abstract: "Two evaluators independently extracted data with a third evaluator arbitrating their discrepancies."

Re:Sensationalist... (0)

Anonymous Coward | more than 6 years ago | (#20654545)

Yes, it would be correct to say that most research results are irrelevant. The scientist then write the papers so that the results seem interesting. Most scientists have nothing to do with rockets these days, because funding is so easy to get.

Irony police... (1)

dj245 (732906) | more than 6 years ago | (#20655179)

To misleadingly imply that most research is wrong, which is exactly what the post suggests, is just poor interpretation of flimsy data, ironically.

Irony police, your analysis?

Medical research vs. basic research (3, Informative)

BWJones (18351) | more than 6 years ago | (#20654321)

It should be noted that "medical research" (epidemiology, clinical studies etc...) is very different from basic research (mechanisms, pathways, etc...) and the threshold for acceptance in journals that cover basic research is much higher than that for medical journals. i.e. There is significantly higher oversight and peer review criticism over basic research than there is medical research and the two fields should not be confused.

Re:Medical research vs. basic research (5, Interesting)

brteag00 (987351) | more than 6 years ago | (#20654605)

It's not just medical research. The scientific community works like any other community: the greater the implications, the greater the scrutiny, attempts to replicate, etc. The Huang embryonic stem cell study is a great case-in-point: the image-manipulation fraud was uncovered because of the vast number of researchers looking at the micrographs he published. (That sounds familiar, doesn't it: "Many eyes make all bugs shallow.") Global warming has many, many people working on models, taking ice cores, doing other analysis. Of course, the vast majority of published research isn't reported in Science [sciencemag.org] or Nature [nature.com] , and so it doesn't get as much exposure. That's why around here (the University of Wisconsin), it's standard practice that if your work depends on someone else's result, you first replicate her experiment and make sure you get the same result. (If you can't, you write a letter to the appropriate publication making note of your inability to replicate the result.) This means that eventually the mistake gets uncovered, and your research doesn't get burned because someone else has been sloppy.

Re:Medical research vs. basic research (2, Insightful)

Otter (3800) | more than 6 years ago | (#20654777)

That's why around here (the University of Wisconsin), it's standard practice that if your work depends on someone else's result, you first replicate her experiment and make sure you get the same result. (If you can't, you write a letter to the appropriate publication making note of your inability to replicate the result.)

Out of curiosity:

1) What is the usual failure rate for replication?

2) Do the letters routinely get published?

3) You just do that for work you're following up with experiments, not for everything you cite, right?

Re:Medical research vs. basic research (2, Interesting)

brteag00 (987351) | more than 6 years ago | (#20655145)

1) What is the usual failure rate for replication?

2) Do the letters routinely get published?

3) You just do that for work you're following up with experiments, not for everything you cite, right?

Unfortunately, I'm not in a hypothesis-driven lab [wisc.edu] , so I can't speak to any of these from direct experience. I know that I routinely see such letters published (frequently as "technical comments"), and I know that I go to seminars and routinely see people get raked over the coals for not having verified someone else's results. The only time you would do so, though, is for results on which your work depends directly.

Of course, there are perfectly valid reasons why that validation might fail, reasons that have nothing to do with someone being sloppy or deceitful. For example, many oft-used cell lines mutate as they are cultured, so your flask of MCF-7 breast cancer cells might not behave the same as the MCF-7 cells used across campus.

Re:Medical research vs. basic research (1)

flynt (248848) | more than 6 years ago | (#20655833)

Also, within medical research, a clear divide must be appreciated between randomized, controlled, clinical trials and epidemiology. A well-run clinical trial is about as good of an experiment as you can do. Patients and doctors remain blinded to the actual treatment so biases are not introduced.

Epidemiology is almost always done retrospectively, and while it may have its uses, there are *always* going to be possible confounding variables when patients are not randomized before receiving a treatment.

So please don't confuse clinical trials with epidemiological studies, the former are regarded as the "gold standard" for showing efficacy of new drugs and devices, while the latter would never serve that purpose.

Not so surprising (1)

allthefish (1158249) | more than 6 years ago | (#20654331)

With our current education system and a public that's not willing to scrutinize things carefully enough, I don't find it shocking at all that science journals are publishing bogus studies. It is a shame, certainly, but its inevitable until the scientific community as a whole moves in a different direction.

What About this Study? (5, Funny)

sarahbau (692647) | more than 6 years ago | (#20654351)

How do we know the study that shows that most studies are tainted isn't tainted?

Re:What About this Study? (1)

temcat (873475) | more than 6 years ago | (#20654445)

Damn, you beat me to it :-)

Yeah, general negative statements tend to negate themselves.

Re:What About this Study? (0)

Anonymous Coward | more than 6 years ago | (#20654795)

We read it very carefully.

Corporate Controlled Media Sucks Life (1, Troll)

twitter (104583) | more than 6 years ago | (#20654973)

Most "mainstream" news is tainted by big company advertising and influence. Examples:

Recent media consolidation [nwsource.com] has made things worse and the Bush administration has gone further in censoring government researchers than any in living memory. Articles like this represent the worst, a concerted attack on science itself.

Oschner [wikipedia.org] , who launched the "war on smoking" put up with a lot of the same kinds of harassment from the tobacco industry. Eventually, truth trumps profits but big tobacco is a disturbing example of how obvious and personal the harm has to be before it registers in the public imagination.

A larger number of reliable news sources helps make problems obvious. The big media counter will be an ever more shrill and less credible mainstream press. Their influence is waning.

Re:What About this Study? (1)

ch-chuck (9622) | more than 6 years ago | (#20654991)

Didn't you read the fine print - "Most scientific studies are sloppy and tainted, except for this one"

According to my research (3, Funny)

Archangel Michael (180766) | more than 6 years ago | (#20654375)

According to my research, most studies involve about 84% error rate due to flawed statistical analysis caused by people pulling statistics out of their arse. The other 16% are flawed due to NOT actually pulling statistics out of their arse.

My experience (2, Insightful)

Zelos (1050172) | more than 6 years ago | (#20654389)

This was certainly true in my experience. When I did a review of mathematical methods in my area a while back, most papers had basic calculation errors, missing information that made reproducing the work difficult or impossible, and they all used carefully selected examples to show their work in the best light.

Well use the Scientific Method then (4, Insightful)

Gallowglass (22346) | more than 6 years ago | (#20654417)

And one of the first rules is, "Never take a single study as proof of anything! Wait till the results are replicated before you even think of moving to a conclusion."

The major problem is really poor reporting on science research. The news media routinely blazon some **NEW * Scientific * Discovery!!!**. Then you read the story and somewhere around the 10th paragraph you might see that this is based on only one study - and oftentimes even before peer review.

Every scientists knows this. It's a shame the public doesn't. They wouldn't worry so much.

Re:Well use the Scientific Method then (1)

Lil'wombat (233322) | more than 6 years ago | (#20654581)

I agree. The problem with that approach is two-fold.
      1) Funding agencies will fund new research, but will not fund research to confirm previous research.
      2) The competition for funding, tenure, etc has led to an ever increasing specialization of expertise. Gone are the days of the gentleman scientists where anybody with a tower could replicate Galileo's experiments.

Wait for the meta-study (1, Insightful)

Crispy Critters (226798) | more than 6 years ago | (#20654647)

The sample sizes are often small in medical/health studies and human beings have a lot of extra variables that are hard to control. The best thing to do is to wait for the meta-study, where someone analyzes all the studies that relate to an effect. After many studies have been done, do they agree? Do they appear to have been well done? Adding the studies together creates a larger sample size and hopefully averages out some of the variation due to flaws in the method.

Re:Well use the Scientific Method then (1)

drfireman (101623) | more than 6 years ago | (#20654947)

The major problem is really poor reporting on science research.
This is indeed a major problem. But it doesn't have much to do with this article, which discusses poor analysis by scientists of their own data, more or less. When a scientists can collect data, fail to understand what can be learned from it, misreport it, and have that misreporting compounded by more misreporting by journalists, then you have... well, the world as we know it.

Slashdot taints article study finds (3, Funny)

packetmon (977047) | more than 6 years ago | (#20654467)

The secret is out! Investigators are looking into whether or not millions of scientists have been using modified versions of SCIgen [mit.edu] for their work. The FBI and Department of Termpaper Security have acknowledged the investigation but declined to speculate on the alleged ties between SCIgen and grammar terrorists citing a new law just passed by pResident Bush which allows warrantless underwear tapping.

Authorities are also investigating the connections between Malda, Bush Laden, Bill Gates, Dvorak and Borat [mit.edu] SCIgen is a program that generates random Computer Science research papers, including graphs, figures, and citations. It uses a hand-written context-free grammar to form all elements of the papers. Our aim here is to maximize amusement, rather than coherence.

o the irony (4, Funny)

Anonymous Coward | more than 6 years ago | (#20654475)

/. commentors commenting on sloppy submission about sloppy analysis

pot, meet kettle

"Most science..." (4, Insightful)

Otter (3800) | more than 6 years ago | (#20654477)

In research published last month in the Journal of the American Medical Association, Dr. Ioannidis and his colleagues analyzed 432 published research claims concerning gender and genes.

His work seems to focus on population genetics and epidemiology, which is notorious for having unreproducible claims due to a combination of uncorrected multiple testing, publication bias and statistical incompetence. This "gender and genes" is a perfect example: someone does a study, finds nothing, slices and dices the data until he gets p = 0.04 for females or Asians or smokers and publishes his breakthrough finding. I'd have been surprised if he hadn't found almost all of those to be wrong.

If you look at more in-vitro molecular biology and biochemistry work, I doubt if nearly as high a percentage of it is clearly "wrong", although quite a bit of it is worthless.

Re:"Most science..." (2, Interesting)

AJWM (19027) | more than 6 years ago | (#20654845)

Doesn't surprise me. Most people who go into the "fuzzy" vs the "hard" sciences do so because they're not good at or don't like math. (Yeah, I know, broad generalization.)

And as math goes, statistics can be pretty darn counterintuitive.

(I speak from experience - I worked for a few years in a university computer center's "academic support group", where among other things I help faculty and grad students with running statistical analysis packages. Some of the experimental designs were pretty bad, too.)

Re:"Most science..." (1)

eli pabst (948845) | more than 6 years ago | (#20655611)

To some degree that's why it's now harder to publish genetic epidemiology papers, especially things like association studies. Many journals are now requiring that you show some kind of functional effect of a mutation that's associated with a disease; gone are the days when you can publish something just because p0.05. And while many people assumed that the inability to replicate results was due to flawed statistics/methodology, we're also finding that there are large differences between different human populations so trying to replicate a study of an Amish population in Italians may not really be a true "replication" after all.

No Money in Replicating Results (5, Insightful)

cduck (9999) | more than 6 years ago | (#20654753)

I am not a scientist.

That being said, it's my understanding that most scientists work off of grants, and those grants fund novel research. Replicating results is of obvious importance in validating those results, but doing so seems at odds with the funding mechanisms that are the reality for what I would believe to be most researchers.

Are researchers supposed to replicate the experiments of others in their spare time and on their own dime?

(As rhetorical as that might have sounded, I actually welcome those with first-hand experience to respond to it)

Plenty of collisions (1)

everphilski (877346) | more than 6 years ago | (#20655381)

I'll talk about my master's thesis work. I was designing a guided bullet to be shot out of a 40mm cannon with a linearized guidance system and a pack of squibs. The combination of a linearized guidance system and a controller hooked up to the squibs would cancel out any pointing errors and initial guidance errors of the gun+RADAR system as the bullet was in flight, and then hopefully hit the target. I wrote a 6DOF simulation to model all this and that was the basis for my thesis.

Now, to answer your question, I am not personally aware of people getting funded to replicate results, although it would not surprise me. However, there is plenty of research that rubs shoulders. For example, in designing my guidance system, I came across tens of papers of people who had done it already. And there are many hundreds of people who have written 6DOF simulations at some point in time.

When papers go to be submitted, many times they are rejected (by the good journals, anyways) because 'they are too similar to something already submitted' or 'they do not really add anything unique to the body of science'. In other words, more or less, they replicate what has already been done. Whether intentional or not, engineers and scientists do from time to time re-invent the wheel, or come up with a solution very similar to something already done. Which isn't all bad, because then you can research what was done before and compare the existing body of results. (I did, in my case. I ran my numbers through a fellow researcher's sim [with his permission of course] to test one portion of the guidance model, and I ran my numbers through a commercial package to validate my core 6DOF model)

Re:No Money in Replicating Results (2, Insightful)

Anonymous Coward | more than 6 years ago | (#20655559)

I am a scientist (polymeric materials).

Are researchers supposed to replicate the experiments of others in their spare time and on their own dime?
You are correct that no grant money is specifically allocated for reproducing other's results, nor for generating "null results" (showing that somethings isn't the case, e.g. that a particular methodology *won't* work). This is a problem, because it means that some important findings that are "uninteresting" don't get studied (or, worse, the data exists but never gets published).

However it's not as bad as it initially sounds. Although we don't receive direct funding to reproduce results, that kind of thing frequently (but not always) happens anyway. If you're building on someone else's work, you will inevitably reproduce some of their experiments. Another example would be that you measure the same thing as someone else, but using a different technique. This kind of corroboration is sometimes even better than directly reproducing their method, because it shows that you can arrive at the same conclusion from a variety of techniques.

So, truthfully, any of the important (and certainly all of the amazing/surprising) results do end up being reproduced in one way or another. But, we never write grant proposals that are simply "we aim to reproduce the work done by X" ... rather we say "we intend to extend upon the work done by X by attempting Y."

I wouldn't object to a tweak to the grant system that gave more recognition to reproducing results and obtaining null-results. But I don't view it as a huge shortcoming of science as it is currently practiced, since we scientists have enough discretion with how we design our experiments that we can put in various checks when they are required.

Bad Dates (2, Informative)

HTH NE1 (675604) | more than 6 years ago | (#20654769)

Quoting http://www.prophet.phlegethon.org/Fiction/Mines/bd.htm [phlegethon.org]

EA Wallace Budge is one of the great authorities on Egyptology, but his work is badly out of date, and was actually never all that good. If nothing else, he has a tendency in his translations to treat Egyptian theology as monotheistic in the model of Christianity. In the Stargate movie, Daniel Jackson says 'I don't know why they keep printing him': The simple answer is that the copyright is expired, so it's cheap, and his name still shifts copy.

My favorite (0)

Anonymous Coward | more than 6 years ago | (#20654827)

My favorite scenario is when the author makes claims that are technically true but which a naive reader will misconstrue as meaning something more. The naive population typically consists of the vast majority of readers, but in some cases, it may consist of everyone except two people in the world, one of which is the author. A retraction is irrelevant, because the claims are "true", and clarifications just don't happen in the scientific literature.

No better example than climate science (0)

Anonymous Coward | more than 6 years ago | (#20654913)

At least an interested group of amateurs are working to replicate some of the more controversial research results (e.g. the Britta reconstruction, MBH hockey stick, and the GISS temperature record). See http://www.climateaudit.org/ [climateaudit.org] and http://www.surfacestations.org/ [surfacestations.org] for more.

It must be so... (2, Insightful)

Actually, I do RTFA (1058596) | more than 6 years ago | (#20654919)

"There is an increasing concern that in modern research, false findings may be the majority or even the vast majority of published research claims," Dr. Ioannidis said. "A new claim about a research finding is more likely to be false than true."


Since the criterion is that the claim is published, someone had to find the study new and interesting. Most new ideas are going to be wrong, especially true the more significant it is. After all, how many crackpot theories were postulated between Newtonian and Relativistic physics? On the other hand, most things easily verifiable, etc, are too obvious to me considered new and interesting. Note, while I find this interesting, I did not come up with this idea. Some economists published a similar study over a year ago postulating this as a reason. Of course, it's probably wrong.

More classes in statistics needed! (1)

jjh37997 (456473) | more than 6 years ago | (#20654969)

The secret shame of the scientific community is that statistical analysis is the foundation of all good research but few Ph.D programs offer more than a single semester worth of train in the subject. Truth is, training in statistical analysis should start in grade school but I doubt that will happen any time soon. One solution is dropping high school and college requirements of calculus and replace them with a year of statistics, which would be useful to more students.....

Re:More classes in statistics needed! (1)

everphilski (877346) | more than 6 years ago | (#20655543)

Problem with that is most of the people who need statistics will need some knowledge of calculus. Maybe they won't be explicitly composing integrals or differential equations but they will need a good understanding of rates, areas under the curve, gradients, etc... but I do agree statistics in high school beyond mean/median/mode would be useful.

Re:More classes in statistics needed! (1)

MontyApollo (849862) | more than 6 years ago | (#20655643)

I agree. I was a physics major and never took a single statistics class. They would throw in some probability and statistics in other classes, but it always seemed to be something you were just supposed to pick up on your own. You would just be taught random tools as the situation arose, but never really learned what tool to apply to what situation.

I think life science majors got a much better education in statistics than physical science majors.

I think every science major should required to take a class in probability and statistics, to be followed by a class in data analysis.

strong variation with fields (3, Interesting)

call -151 (230520) | more than 6 years ago | (#20654981)

There are a lot of different attitudes about the role of the anonymous referee, in different fields and in different settings. In computer science and mathematics, where most of my publications are, the role of the referee depends upon a number of things. A few comments relevant to my disciplines:

  • The responsibility for correctness lies with the author, not the referee. It is good if the referee spots problems but it is not the obligation of the referee to certify that every last detail is correct.
  • Often, the primary responsibility of the referee is to comment on the importance, priority, relevance and how much interest there is in the work.
  • In the CS world of conference refereeing (as opposed to CS journals) there is often absurd time pressure. Articles/abstracts are due at midnight local time on some date, so things are typically hastily written, and referees must review things in a very short timeframe and practically never get a chance to check things carefully. As far as I am concerned, the conference publication model in CS is terribly broken. There have been some calls for reform, but those have been coming for at least the last 10 years or so and over that period it's gotten worse, not better.
  • In math, it can take a year for a referee to work through something techical, so the process is slow.
  • Typically, referees are uncompensated for their work. Some people take their refereeing duties seriously, and some do not. Generally, those who do a good job in a timely fashion are asked more often to referee more things, which is not exactly a reward.

Partly because Medical Doctors dont know math (1, Insightful)

Anonymous Coward | more than 6 years ago | (#20655059)

The study does not refer to science in general, only to medicine. Medicine is hardly a typical example of standard science.

1) Physicians are mostly trained to be healers (practitioners) not scientists.
2) Medical data is very inaccurate compared to the data in standard science (physics, chemistry, geology, etc)
3) Dealing with inaccurate data requies advanced knowledge of matematical statistics and most medical doctors do not have a basic grasp of this field.
4) Many MAJOR ERRORS in the medical literarure are due to the ignorance of basic principles of statistics.
5) Sure ignorance in the field of statistics is only one cause of poor medical research; however it cannot be ignored.

Re:Partly because Medical Doctors dont know math (1)

Dunbal (464142) | more than 6 years ago | (#20655653)

OK I feel I have to nit-pick a little.

1) Physicians are mostly trained to be healers (practitioners) not scientists.

      I think that most physicians start out as "scientists", after all, most of us come from science majors. However the biggest mistake that medical students make is thinking in absolutes. Yes, anyone can memorize anatomy, or physiology, or pharmacology, etc. After all, there are only so many anatomic variations in the population, a person's nerves will always transmit a signal the same way, and warfarin/coumadin will always interact with pretty much any other medication you prescribe.

      But something happens during the career. When you actually get a patient in front of you, suddenly all those absolutes go out the window. Everyone is different. Rarely will you see the "classic" or "textbook" or "tv medical show" presentation of a disease. Many patients come with very vague or mysterious complaints. An example, which happened in my practice a month or so ago- who would have guessed that a patient who was treated successfully 2 times elsewhere for a recurring skin infection actually had diabetes - which was CAUSING the infections? Only a physician. After all, if it was THAT easy who needs doctors - we have wikipedia. Right? WRONG.

      So I must state that medicine IS a science. Filled with facts. You can memorize the pathology. You can study algorithms (they do exist) to reach diagnoses. Patient comes in with X, Y and Z, just follow the algorithm. And you'll be right, a lot of the time. But the SUCCESSFUL practice of medicine - how to APPLY the science, is an art. Just like Engineering is an art. Any fool can build a bridge, given enough scientific knowledge of materials and forces and tensile strengths, etc. But the guy who builds a good bridge (except in Minneapolis) where no one thought he could - he is an artist. Does that make him less of a scientist?

I agree with your second point. We're dealing with the normal curve ALL the time. Most patients will have response "X", but you'll get some with "A" and "Z" and there's always one with "4".

3) Dealing with inaccurate data requies advanced knowledge of matematical statistics and most medical doctors do not have a basic grasp of this field.

      We don't need it. That's what we have peer review for - supposedly. To save us from all the BS and pet theories and make sure that only decent studies get published. It doesn't always work, though. But this is also why things like the Cochrane studies [cochrane.org] are important. Your average doc in the street doesn't need to battle with statistics. (S)He just needs to know if what (s)he's going to or what (s)he's been taught do WORKS. Studies like Cochrane let us look back and see if what we're doing actually WORKS or not. Despite whatever was published/hyped/etc.

I agree with points 4 and 5. Anyone who is going to publish must consult with a statistician. Medical practice is interdisciplinary. Why should medical research be any different?

Is WSJ academic? (1)

micromuncher (171881) | more than 6 years ago | (#20655073)

[The hotter the field of research the more likely its published findings should be viewed skeptically, he determined.] ...
[No one knows how much shoddy research is out there.]

I'm not sure what the point of this article is besides fear mongering. The goal of most scientific research is to prove a set of assertions - and sure this set may not be fully encompassing or comprehensive - but you've got a model and you try see what fits - and its not always exact.

Take for example the recent debacle /wrt Steve McIntyre and climateaudit.org. The point was "NASA's stats and calcs are wrong." This started an anti-global warming fiesta and many neoconservatives/anti-Dole hopped on the bandwagon to discount, rather than disprove, McIntyre's findings. NASA Hansen responded with http://www.columbia.edu/~jeh1/realdeal.16aug20074.pdf [columbia.edu] pointing out that the error was insignificant in overall trending and a correction was posted. But very little media attention was given to that.

So now I see a new wave of WSJ luddites missing the point.

(Oh please tag this as a troll, I've bashed WSJ again.)

good science and bad reporting (1)

kurthr (30155) | more than 6 years ago | (#20655101)

Hmmm... what fraction of news published and edited in reputable journals contains factual errors? That would be another interesting story.

But come on... 90% of everything is crap. This is no more or less true in medical research, which is a fraction of the $50 billion total spent on research. OMG that's like "One MILLLLIIION DOLLLLARS". That's like 0.5% of GDP so don't be surpised when it's bunk, it's drop in the bucket compared to development costs. If you haven't figure this out you're still a little naive.

So what do real scientists and engineers (that includes doctors) do? They build something that works, and it's a long hard slog, and most of the problems you encounter aren't the cool sexy ones you thought they would be at the start. They don't make big headlines, and they've mostly been discovered (and solved) by a zillion people before you. The key is solving a problem so well that nobody has to solve it ever again! Then the other bad solutions slowly go away, and people can work on other problems (yes QWERTY is good enough).

Now with medicine, things were made a bit better because of the FDA and the clinical trials model, before the more significant political interference of the last decade. Unfortunately, there's the issue that people really want to believe in magical cures that will save uncle Milt, or their cat so they'll pressure doctors and believe con-men for the rest of time. This is fed by the journalistic-industrial lie complex.

Sure scientists are wrong all the time, or certainly less accurate than they could be, but not usually about big things. More often than not they miss the real interesting results making things come out the way they think they should. It takes someone who really believes in their ability to do something right to discover something really new... and if it's truely astounding they better be prepared to spend the next decade proving it and developing some cool new tools with it. If those things work as they predicted and not the guys before... then hey they must be on to something!

The problem is that there are all sorts of quick-fix "cranberries cure cancer" quacks out there. They're in sales where it doesn't matter if it works. That's why your computer doesn't run any faster with the new whizzy RAM than the old stuff, but it doesn't mean Moore's Law doesn't progress. The key is the guys doing the development and implementing the new ideas, and proving they work. They keep making better faster cooler stuff !THAT WORKS!

The BS you read in most hype journals (EETimes are you listening?) is mostly unproven tripe. It occasionally has a grain of truth, along with a lot of interpretation bias, and if you bothered to read the original article (did you?) and some of their other articles (hah that's harder) you can often tell what the nuance is, how careful the methods were, and whether it's worth trying to replicate yourself. Guess what if you're not in the field that paper really wasn't meant for you (it was meant for the tenure comittee or the guys who gave you a grant etc). The stock market guys may go crazy, but that's because they don't give a damn about science or the truth. Same thing with most journalists these days, frankly. They care about money, and figure their reputation won't be much worse than anyone elses.

Go read the actual article:
http://medicine.plosjournals.org/perlserv/?request=get-document&doi=10.1371/journal.pmed.0020124&ct=1 [plosjournals.org]

It's not nearly as interesting as reading a sensationalistic WJ article, but I give them props for linking to it.

This explains... (1)

Schnoogs (1087081) | more than 6 years ago | (#20655119)

last weeks article about Liberals being smarter than Conservatives. I kid!! I'm a moderate so what do I care! ;)

Most Science News Tainted by Sloppy Journalism (1, Interesting)

Anonymous Coward | more than 6 years ago | (#20655175)

That might be a better title for this article. Journalists love to take findings from scientific literature and mangle, misconstrue, and misinterpret them. Let's see... a journalist finds a biologist who looks at a particular subfield of genetics, and somehow manages to blow that up into "most published research findings are wrong."

Never mind that there are thousands of journals in enormously diverse fields - biology, chemistry, physics, all the branches of engineering. We can take a sample of 432 articles on a single subject, and extrapolate that to state that ALL scientific research is 'probably' wrong.

Way to further undermine the value of science in the eyes of the layman, jackass.

Cargo Cult Science (5, Insightful)

ObsessiveMathsFreak (773371) | more than 6 years ago | (#20655265)

People need to realise that a lot of those calling themselves scientists are not really scientists at all. They don't apply the scientific method. They massage data regularly. They misapply statistics constantly. They don't subject their theories to falisfiability [wikipedia.org] . They waffle, hand wave, engage in rhetoric, and generally do just about everything except an honest to goodness, old fashioned solid, scientific experiment.

Feynman spotted them over 30 years ago. He called them Cargo Cult Scientists [wikipedia.org] . They put on the appearance of science, but have none of its substance. They give a good performance, like an actor playing a scientists on TV. They wear the clothes, speak the language, seemingly apply the methods. But it's all empty. There's no rigor. There's no insight. There's no real testing going on. It's all just people waving around graphs, and lines, and their qualifications, and formulae they don't understand, to support the theories they want to be true, regardless of whether they are true or not.

It's because in this day and age, you can't be a witchdoctor. You can't appeal to spirits, or gods, or karma, or any of the other philosophical reason thrown up in past ages. We live in "The Age of Reason", and people expect things to be proven to them "scientifically". So all the people who in the past would have risen high by browbeating, appealing to authority and writing great prose, are forced to dress themselves up in white coats and go through the motions of an experiment before they proclaim their great revelations to the world. The experiments however, are just as empty as all the old techniques, and bear only superficial relation to actual science.

Personally, I think it's gotten worse over the last 30 years. The unwillingness of actual scientific communities to challenge the misapplication of their methods by unscientific ones has lead to a dilution of the authority of science as a whole. Under the current regime any half baked psychiatrists can show pictures to 20 undergraduates, record a few squiggles on an MRI, run the numbers through R over and over until he gets what he wants, and proclaim to the world just about whatever he likes, and still be called a scientist! No wonder it's all too easy for the Intelligent Design movement to pose as "real science". Just look at how low the threshold for real science is.

There's only one way to deal with Cargo Cult Scientists. You have to call them out. You have to show how flimsy and false their supposed science really is. You also need to learn all the old rhetorical techniques, because faced with someone who actually knows what they're doing, the Cargo Culter will fall back to very old and time honored methods which enable him to win from a weak or false position. I think the real scientific community owes it to itself to show up these charlatans for what they really are, Con men. If they don't, science will just become more diluted in the long run until the public regards it in the same way it regards homeopathy.

PhD != Research Scientist (3, Interesting)

rumblin'rabbit (711865) | more than 6 years ago | (#20655327)

Universities are pumping out PhD's at a prodigious rate. As a manager of R&D, I've interviewed and hired more than my share. Virtually all say they want to do research.

Here's my problem. Only a fraction (I'm guessing 1 out of 5) are actually capable of doing good research. The rest are competent employees for developing other people's research into useful products, but aren't terribly original thinkers, nor show a lot of initiative, nor show the rigour and clarity of thought one wants to see in a researcher.

Frankly, when I "unleash" employees on open-ended problems without much guidance, the majority soon begin to flounder.

There is nothing wrong with getting advanced degrees, but many then feel they are obliged to do original research when in fact they really aren't up to it. This may be one reason why the quality of papers isn't where it should be.

An invitation to physicists and mathematicians (1)

Sir Holo (531007) | more than 6 years ago | (#20655449)

Is this Wall Street Journal article on the credibility of (medical) scientific research an invitation to scientists to publish in Nature that modern economic theory is grounded in the flawed assumptions that individuals will make choices based on the most desirable outcome in every instance? Or their insistence on ignoring that an economic system is inherently finite, and thus unlikely to settle into an optimal state? Or how about the time scale of settling, which in many cases would be longer than human lifespans and thus unrealistic in the light of basic human psychology? Oscillatory systems?

They are inviting physicists to take aim at the flaws in modern economic theory. This is sorely needed.

yeah but... (2, Funny)

Kelexel (1158529) | more than 6 years ago | (#20655465)

[Speaking to self...]
A scientific study published that most scientific studies are wrong... therefore there is a good change of it being wrong.... Which means that most scientific studies are right... But if most studies are right then this one is also right... which means...
c.. an.. t take .... this...
[Head explodes]

Even perfectly honest scientists have this problem (0)

Anonymous Coward | more than 6 years ago | (#20655711)

Even if scientists were perfectly honest and disinterested, new results in a "hot" field should be suspect.

Most publications require that results happen at the p <= 0.05 level: that is, that the chance that the result happened by chance is less than one in twenty. For every twenty experiments, one will show the result. If a field is "hot" and one hundred groups of honest and disinterested scientists work on a problem, five would have publishable papers, no matter whether the result is real.

A better way to do science would be to publish a proposed experiment before any data is collected, then publish the results -- no matter whether the experiment succeeds or fails.

Flawed medical studies != all of science (2, Informative)

Durandal64 (658649) | more than 6 years ago | (#20655735)

This guy's main beef appears to be with medical studies and other sciences which rely heavily on statistics (sociology, psychology and the other wannabe-sciences). This is not surprising, to be honest. Statistical analysis isn't difficult, but I've known many social science students. They consider statistics to be extremely advanced and have no other mathematical background. As a result, they don't have a very deep understanding of how to mathematically model a system. Naturally, this will lead to bogus conclusions and incompetent analysis work. Medicine has a similar problem, albeit on a smaller scale. Most of the time, statistical analysis will yield correlations, but they won't tell you anything about the mechanism behind what you're seeing, which is what's important in science.

I'd expect the rate of error for physics experiments to be much lower than that of, say, sociology.
Load More Comments
Slashdot Account

Need an Account?

Forgot your password?

Don't worry, we never post anything without your permission.

Submission Text Formatting Tips

We support a small subset of HTML, namely these tags:

  • b
  • i
  • p
  • br
  • a
  • ol
  • ul
  • li
  • dl
  • dt
  • dd
  • em
  • strong
  • tt
  • blockquote
  • div
  • quote
  • ecode

"ecode" can be used for code snippets, for example:

<ecode>    while(1) { do_something(); } </ecode>
Sign up for Slashdot Newsletters
Create a Slashdot Account