Beta
×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

How To Better Verify Scientific Research

samzenpus posted 1 year,3 days | from the checking-it-twice dept.

Science 197

Hugh Pickens DOT Com writes "Michael Hiltzik writes in the LA Times that you'd think the one place you can depend on for verifiable facts is science but a few years ago, scientists at Amgen set out to double-check the results of 53 landmark papers in their fields of cancer research and blood biology and found only six could be proved valid. 'The thing that should scare people is that so many of these important published studies turn out to be wrong when they're investigated further,' says Michael Eisen who adds that the drive to land a paper in a top journal encourages researchers to hype their results, especially in the life sciences. Peer review, in which a paper is checked out by eminent scientists before publication, isn't a safeguard because the unpaid reviewers seldom have the time or inclination to examine a study enough to unearth errors or flaws. 'The journals want the papers that make the sexiest claims,' Eisen says. 'And scientists believe that the way you succeed is having splashy papers in Science or Nature — it's not bad for them if a paper turns out to be wrong, if it's gotten a lot of attention.' That's why the National Institutes of Health has launched a project to remake its researchers' approach to publication. Its new PubMed Commons system allows qualified scientists to post ongoing comments about published papers. The goal is to wean scientists from the idea that a cursory, one-time peer review is enough to validate a research study, and substitute a process of continuing scrutiny, so that poor research can be identified quickly and good research can be picked out of the crowd and find a wider audience. 'The demand for sexy results, combined with indifferent follow-up, means that billions of dollars in worldwide resources devoted to finding and developing remedies for the diseases that afflict us all is being thrown down a rathole,' says Hiltzik. 'NIH and the rest of the scientific community are just now waking up to the realization that science has lost its way, and it may take years to get back on the right path.'"

Sorry! There are no comments related to the filter you selected.

Scientists == Always Right (-1)

Anonymous Coward | 1 year,3 days | (#45257487)

You can always believe what a scientist tells you. Science is infallible. Albert Einstein would kick Jesus Christ's ass.

Re:Scientists == Always Right (5, Insightful)

Anonymous Coward | 1 year,3 days | (#45257507)

Science is infallible.

More like science is always wrong. Scientists always set out to be less wrong than the last guy, though.

Re:Scientists == Always Right (1)

Sockatume (732728) | 1 year,3 days | (#45258359)

Every great scientist's career is built on the cold, dead corpses of his peers' and antecedents' own work.

Re:Scientists == Always Right (0)

Anonymous Coward | 1 year,3 days | (#45257671)

Only if it's a climate scientist. For all other you're allowed scrunity without calls for being put to the torch for blaspheming and denying The Three-Letter Diety

Re:Scientists == Always Right (4, Insightful)

Vanderhoth (1582661) | 1 year,3 days | (#45257743)

Took 29 minutes to get from the story being posted to "CLIMATE SCIENTIST ARE LIREZ!!11!!1". You know there are a lot of other branches of science, many of which are far more subjective than climate science.

There's also plenty of data and models out there if you wanted to run your own experiments to confirm or disprove a particular paper or claim. I'd be very interested in reading your counter paper.

Re:Scientists == Always Right (0)

Anonymous Coward | 1 year,3 days | (#45257831)

They are also far less doomsday oriented and easier to ignore without being compared to a nazi apologist.

Re:Scientists == Always Right (1)

Vanderhoth (1582661) | 1 year,3 days | (#45258491)

As mentioned in a thread below, Sociology. They make stuff up, but if you disagree then you're going against what's considered politically correct and you're sexists, and/or racist. Climatology is based on real numbers of things that can be measured.

Like I said it's fine if you don't agree with it, but what ends up happening is people go on message boards and start screaming and making outrageous claims against the popular literature and data, but then have absolutely nothing to back them up other than "Fox news said so!!!"

So what do I believe the lying climatologists that have reproducible facts and figures supporting their claims, or some nobody screaming that I'm an idiot because I'm not outraged that there's evidence to support climate change is real?

Re:Scientists == Always Right (2)

musterion (305824) | 1 year,3 days | (#45257841)

In addition to questioning "Climate Change", one could also look at the "Science" of may other fields as suggested. Especially crucial for examination are the many psychological and sociological studes that are used to "guide" public policy. I venture that many are complete loads of crap designed specifically to influence public policy.

Re:Scientists == Always Right (5, Interesting)

Vanderhoth (1582661) | 1 year,3 days | (#45258411)

Completely agree, I have a friend who's getting a doctor in Sociology with a concentration in women studies. Some of the crap she's made me read is ridiculous. She stopped talking to me for awhile after I said what she does isn't science. She can't even replicate an "experiment" from one group to another let alone across a generational, cultural, or geographic gap, and yet some of these "studies" are used to set employment policies that discriminate against majorities and created the "we don't care if you're qualified to do this if you don't help us meet our quota" environment.

Re:Scientists == Always Right (1)

Oligonicella (659917) | 1 year,3 days | (#45258617)

psychological and sociological studes

These are not fields of science other than as repositories of anecdotal evidence. The vast bulk of each discipline's studies and experiments cannot be reproduced. You venture correctly.

Re:Scientists == Always Right (4, Insightful)

Oligonicella (659917) | 1 year,3 days | (#45258571)

Took 29 minutes to get from the story being posted to "CLIMATE SCIENTIST ARE LIREZ!!11!!1"

He neither said that nor implied it. What he said was that any criticism of AGW is met with a defense akin to a religious fervor. This is a true statement.

As demonstrated.

Slashdot for scientists (2)

Sattwic (545957) | 1 year,3 days | (#45257525)

So basically they want to introduce a Slashdot for scientists..

Prepare for a brand new style of flame-wars!

Re:Slashdot for scientists (1)

Anonymous Coward | 1 year,3 days | (#45257563)

So basically they want to introduce a Slashdot for scientists..

Prepare for a brand new style of flame-wars!

Do you have any evidence for that claim?

Or how about data?

You call THAT data?!

Plah-ease!

A 95% confidence interval?! What?! Couldn't handle 99%?

Loser.

Re:Slashdot for scientists (2)

Opportunist (166417) | 1 year,3 days | (#45257805)

And in Soviet Russia, science disproves YOU.

Re:Slashdot for scientists (2)

Austrian Anarchy (3010653) | 1 year,3 days | (#45257603)

So basically they want to introduce a Slashdot for scientists..

Prepare for a brand new style of flame-wars!

Since Popular Science [popsci.com] dropped the ball, the government had to take over.

Eyeballs and Bugs (1)

Sattwic (545957) | 1 year,3 days | (#45257533)

Reminds me of Eric Raymond's 'Given enough eyeballs, all bugs are shallow' law.

Re:Eyeballs and Bugs (1)

Anonymous Coward | 1 year,3 days | (#45257781)

which is plain old wrong. review depth is not distributed like a statistical distribution. You need to have a combination of subject matter expertise, and the desire to use that expertise, and the skill to do the deep review (of software, or a paper, or whatever). There are actual bounds on some of this stuff.

Sure, I can use a penny as a shim under one leg to stabilize a rocking table. If I have enough pennies, I can find one that is "exactly" the right size, as long as the gap (bug) is within the range of penny sizes. But if the gap is 5 mm (1/4"), there are no pennies available: there are none that thick.

Re:Eyeballs and Bugs (0)

Anonymous Coward | 1 year,3 days | (#45257827)

is not distributed like a statistical distribution.

Fascinating. A distribution that is not distributed like a distribution. Let me add that to my list of things not on my list.

Re:Eyeballs and Bugs (5, Interesting)

Sattwic (545957) | 1 year,3 days | (#45257839)

The problem with the present method is that each paper is scrutinized before publication only by a very small select cohort of experts. And once this decision is taken, its 'published and stays published for ever' (in most cases, discounting the outright fraudulent ones that are retracted)'.

I am a professor of pharmacology and we do critical appraisal of scientific papers in our department all the time for symposiums. You won't know what kind of mistakes my undergrads pick up in journal clubs, of papers published in prestigious journals.

By enough eyeballs, I do mean qualified eyeballs. Not just eyeballs.

Re:Eyeballs and Bugs (2)

Immerman (2627577) | 1 year,3 days | (#45258039)

I think the point is that *some* of those eyes will have the requisite expertise to catch subtle flaws. And perhaps just as valuable *lots* of those eyes will have enough expertise to catch the simplistic flaws - the sorts of things so obvious that the real experts aren't even looking at that area because they assume no expert would make such an obvious error. But since we're all human, occasionally we do.

Re:Eyeballs and Bugs (0)

Anonymous Coward | 1 year,3 days | (#45257857)

Reminds me of Eric Raymond's 'Given enough bugs, all eyeballs are shallow' law.

You raise an interesting point. On the other hand, eww!1 Once they finish with the eyeballs, prepare to have your brain eaten!!1!

Capitalists doing science. (-1)

Anonymous Coward | 1 year,3 days | (#45257535)

That evil for-profit capitalists drug company Amgen should be banished frmo science because all progressive "hate speech" buzzwords are encapsulated in one organization.

How do we know the new study is correct? (0)

Anonymous Coward | 1 year,3 days | (#45257547)

How do we know the new study is correct?

problems (5, Insightful)

Anonymous Coward | 1 year,3 days | (#45257557)

I think the real problem is that scientists aren't lending any prestige to reproducing experiments so nobody bothers. Journals want to publish new results, not confirmation. Advisors discourage students from reproducing experiments, which makes sense since they won't be published.

Re:problems (2, Informative)

Anonymous Coward | 1 year,3 days | (#45257683)

This. No one wants to read: "We've confirmed this."
Actually, I might want to read it, and even write it, but good luck getting it into ieee or acm.

Re:problems (5, Interesting)

SirGarlon (845873) | 1 year,3 days | (#45258173)

That is the journals' problem. One could imagine devoting a quarter of each issue to one-page papers confirming previously-published results. In fact, that could be a great way for graduate students to break into prestigious journals. In my not-so-humble opinion, the fact that most or all journals don't make efforts to publish corroborating studies is biting criticism of the journals and their role in undermining the proper scientific method.

Re:problems (3, Insightful)

Anonymous Coward | 1 year,3 days | (#45258275)

No, it's not the journals' problem. It's a funding problem.

Nobody wants to pay for scientists to reproduce and verify each others' work.

Re:problems (4, Insightful)

Curupira (1899458) | 1 year,3 days | (#45258433)

No, it's not the journals' problem. It's a funding problem.

Nobody wants to pay for scientists to reproduce and verify each others' work.

It is both: a funding problem AND the journals' problem. They are not contradictory (far from it, actually).

Re:problems (3, Insightful)

celticryan (887773) | 1 year,3 days | (#45258445)

As AC said below - there is no funding to do this. In addition to no funding, there is no incentive. Speaking in generalized terms, scientists are judged on their research record. That is a combination of:
1. How much money they have brought in through grants.
2. How many papers they have published.
3. The prestige of the journals they have published in.
4. How many times their papers have been cited by other researchers.
So, if you keeping your job depends on those 4 things, where is the incentive to check the work of someone else? Especially large, difficult, and expensive experiments. At best, you get a quick "Comment on XYZ" paper that questions some findings and the authors reply with a "Reply to Comment on XYZ" telling you why your comment is rubbish and you didn't understand what they were saying.

Re:problems (5, Interesting)

Opportunist (166417) | 1 year,3 days | (#45257883)

Someone tack an insightful to that guy? Why don't get I Modpoints when I need them?

Because this is exactly the source of the problem: All the credit, all the fame, all the limelight and of course in turn all the grant money goes to whoever or whatever organization publishes it first. Adding insult to injury, since they also got the patents, of course.

We need to put more emphasis on proof. I keep overusing the "Assertion without proof" meme lately, but it fits in science as much as it fits for the NSA, just because you say so doesn't make it so, unless someone else who has an interest to debunk your claims has to confirm that you're right your assertion is essentially worthless.

And yes, a confirmation from a buddy isn't much better either.

Re:problems (1)

SirGarlon (845873) | 1 year,3 days | (#45257939)

There's more than one problem at work here. You've identified one, but I think the lack of quality review is also serious. The reviewers' names and their recommendations (accept/reject) should be published along with the paper when it is accepted, to give them an incentive to reject garbage instead of rubber-stamping it.

Re:problems (1)

pixelpusher220 (529617) | 1 year,3 days | (#45258031)

I like the idea. Only concern might be it would cause the number of people willing to do the reviews to drop... quality goes up but quantity of reviewed material goes down - which isn't good for anybody. Not sure which is worse though.

Re:problems (2)

SirGarlon (845873) | 1 year,3 days | (#45258123)

I think an ineffective review is worse than no review, because an ineffective review imparts a false sense of confidence in the paper's methods and findings. So a sharp decline in the number of reviewers could be a good thing, if it weeds out the liars who say that have rigorously criticized the manuscript when they haven't.

Re:problems (1)

Oligonicella (659917) | 1 year,3 days | (#45258821)

Actually, I think that would be good. I'm pretty sure the majority of papers would drop as well because shoddy work would get you the wrong kinds of headlines and those submissions would dwindle. You can ensure that by providing in synopsis form the papers that you rejected and the reasons.

Re:problems (0)

Anonymous Coward | 1 year,3 days | (#45258417)

One solution to this is the Journal of Negative Results in Biomedicine. It is literally a journal of "we tried to replicate the reported result and weren't able to." Those of us with longer beards pretty much ignore a published result that isn't an order of magnitude better than the control.

Re:problems (0)

Anonymous Coward | 1 year,3 days | (#45258693)

"Those of us with longer beards pretty much ignore a published result that isn't an order of magnitude better than the control."

And why do you think that has something to do with how reliable the result is? Really if no other information is available I would assume huge results in biology are most likely not translatable to different conditions.

One dogma in a list of many...... (1)

m.shenhav (948505) | 1 year,3 days | (#45257579)

...... and I am happy its finally being acknowledged and tackled more openly.

Don't be too happy. (0)

Anonymous Coward | 1 year,3 days | (#45257629)

Because you WILL see - like on Fox News - these articles and studies showing science is "full of it" and that things like Global Warming are false.

See what happened there? They'll take this information and instead of saying, "Hey look science is addressing a problem.", they'll turn it into science is full of shit and evolution, global warming, or whatever scientific discoveries that contradict their World view and narrative as being false.

Re:Don't be too happy. (1)

Opportunist (166417) | 1 year,3 days | (#45257843)

...and all the while leaving out that their claims don't even offer any kind of falsification chance, i.e. being nonscientific in the first place.

Re:Don't be too happy. (0)

Anonymous Coward | 1 year,3 days | (#45258723)

Nearly all science based on p values is non falsifiable. They are disproving the opposite of their research hypothesis. That practice is what needs to go away.

Peer review isn't about validation (5, Insightful)

Anonymous Coward | 1 year,3 days | (#45257581)

Follow-up studies are where the validation/replication/testing happens. This is not new. Any decent scientist knows this. Peer review is a filter, but it's a pretty basic sanity check, not a comprehensive evaluation of the work. Once published, that opens a paper and the ideas within it to critique by ALL readers, not only the reviewers. Thus, post-publication is when the real scientific review happens. Peer review merely removes the stuff that isn't formulated, measured, and organized well enough to bother reading it in the first place (i.e. it gets rejected). It's an imperfect process, so sometimes stuff slips through anyway. That's what the follow-up papers are for.

Re:Peer review isn't about validation (4, Insightful)

Chris Mattern (191822) | 1 year,3 days | (#45257713)

Which is exactly the problem. Nobody's doing follow-up papers. Follow-up papers aren't sexy, don't get published in top-line journals and don't get a lot of cites. In short, they don't advance your career. So scientitsts don't do them.

Re:Peer review isn't about validation (2)

dkf (304284) | 1 year,3 days | (#45257833)

Which is exactly the problem. Nobody's doing follow-up papers. Follow-up papers aren't sexy, don't get published in top-line journals and don't get a lot of cites. In short, they don't advance your career. So scientitsts don't do them.

Follow-up papers are the usual things that a doctoral student starts out their career writing. Sometimes even masters students (depending on the discipline and the difficulty of conducting the experiments). The grad-school grunts don't have a lot of expectation of being heavily cited, so the risk to them is much lower. Once they've duplicated someone else's results, they can start thinking about what they'd do differently...

Re:Peer review isn't about validation (0)

Anonymous Coward | 1 year,3 days | (#45257961)

As a current grad school student, I disagree with this statement. I'm currently on the masters no thesis track, but my adviser is trying to get me to switch to a thesis track and work on some sub-portions of her research. And to quote "It's something that you could get published!"

Re:Peer review isn't about validation (0)

Anonymous Coward | 1 year,3 days | (#45257985)

The grad-school grunts don't have a lot of expectation of being heavily cited, so the risk to them is much lower.

And to quote "It's something that you could get published!"

Yes. Published is different than cited.

Re:Peer review isn't about validation (0)

Anonymous Coward | 1 year,3 days | (#45257849)

In short, they don't advance your career. So scientitsts don't do them.

What do you think grad students are for?

Re:Peer review isn't about validation (1)

Immerman (2627577) | 1 year,3 days | (#45258135)

To do the grunt work on the projects the scientist they're working for wants to do.

Re:Peer review isn't about validation (1)

dkf (304284) | 1 year,3 days | (#45258697)

To do the grunt work on the projects the scientist they're working for wants to do.

Replicating others' work is usually the introduction to that. After all, if you can have your minion show that a rival falsified their results, that'll be one less group competing with you for the money in the next call for grant proposals.

Re:Peer review isn't about validation (1)

Immerman (2627577) | 1 year,3 days | (#45258815)

But for that to be worth it you have to have reason to believe that your rival's results are invalid. Just randomly replicating results in the hope of stumbling on such a case consumes a lot of time, money, and lab space that could instead be focused towards advancing your own projects, after all replicating an experiment can take as many resources as the original experiment did.

Re:Peer review isn't about validation (1)

z3r0w8 (664036) | 1 year,3 days | (#45257745)

So, scientific papers shouldn't try to do the best work they can before publication? The article(to me) seems to insinuate that just like "news", paper publishing is rushing to get out regardless of the claims. True, this sample size is no where near what it needs to be to imply anything factual, but a 10% verification rate seems a little bit more than "a few slipping through." I guessing if you did a follow-up study of how many of the unverified papers had follow-on studies, the rate would be alarmingly low. So, we end up with a lot of published scientific papers that people can point to that are inaccurate and kicking the verification can down the road. The just seems bad all around.

Re:Peer review isn't about validation (1)

Opportunist (166417) | 1 year,3 days | (#45257941)

And don't you think that's a tad bit dangerous? Imagine you're building your research based on the findings of antigravity, only to find out (after investing a lot of dough into the whole process) that your foundation is completely bogus?

Now imagine the danger inherent to such an approach when it comes to human medicine.

When you build upon a foundation, you don't test the foundation beyond the obvious points, i.e. whether it can hold your building, you don't really stress test it. You neither have the time nor the resources for it, you simply assume that it holds and only if it crumbles you find out whether it does. Now, if you don't happen to put stress on the "false" spots, it may even hold up and even reinforce its credibility, despite being bogus.

That's a pretty dangerous way of verification. It kinda feels a bit like a scholastic approach where you eventually have to start working around "proven" principles because they fail at the facts that you needed from them while they worked out for others and now you'd have to stand alone against hundreds of "important" people trying to show them their error, which they will fight tooth and nail simply because their reputation is now on the line, they accepted a false theory as "true" because it worked out for their own works and would now have to reevaluate their whole system, let alone accept the humiliation of supporting something that is then obviously false (once you look at a false theory from the right angle, it is very often very obviously false).

I wouldn't be surprised if the whole "dark energy" building eventually runs into one of those problems...

Atheistic pseudoscience is the problem (-1)

Anonymous Coward | 1 year,3 days | (#45257583)

One way to fix science is to stop the atheists pushing pseudoscience like the theory of evolution in direct defiance to the Word of God. Sorry atheists but you still haven't proven that two monkeys can give birth to a human being.

Re:Atheistic pseudoscience is the problem (0)

Opportunist (166417) | 1 year,3 days | (#45257971)

Oh yeah, instead we'll take the bible, it's a hell of a well peer reviewed work, not to mention that its primary protagonist has a really swell rep for cooperating with colleagues.

Re:Atheistic pseudoscience is the problem (1)

pixelpusher220 (529617) | 1 year,3 days | (#45258055)

A scientific 'theory' is proven [wikipedia.org] . A 'hypothesis' is still uncertain.

Or do you not believe in the theory of gravity either?

Medicine is not a science (0)

Anonymous Coward | 1 year,3 days | (#45257587)

In any case it is not exact and all those studies they have depend on their interpretation of the statistics. There is usually a very low chance of repeatability and a large dependence on individual judgement. There is a large chance of bias and their error bars are probably way too small.

Replication (1, Insightful)

Hognoxious (631665) | 1 year,3 days | (#45257601)

The best way of checking spurious, biased, or erroneous results is for someone else to independently do the same experiment. However there's no money or glory in replication. So nobody does it.

I wonder which will be most amusing, Fox's interpretation of this story or the tardbaggers' interpretation of that. I've already assigned "herp", "derp" and "6,000 years" to hotkeys.

Re:Replication (2)

gtall (79522) | 1 year,3 days | (#45257639)

Yes, there's no money in replication. More importantly, there is no money for replication. Who's going to fund replication studies? And it wouldn't be small amount of money.

Maybe what is needed is a tax on research to fund replication studies. That opens up another can of worms, is one replication study enough? Who decides? Whether the Tea Baggers like it or not, this seems like an area that will require government intervention. Taxing research is unlikely to bring in enough money. Taxes will have to be raised somewhere to pay for it. (Several Tea Bagger angels were sacrificed in the writing of the previous sentence.)

Funding for replication (4, Informative)

edremy (36408) | 1 year,3 days | (#45257721)

Interestingly, the Economist's article on the same points this weeks notes that there is a group specifically devoted to doing replication- the Reproducibility Initiative [scienceexchange.com] from PLOS One. They've got a $1.3 million grant from the Arnold Foundation to look at 50 high profile papers in cancer research.

Re:Replication (1)

BitZtream (692029) | 1 year,3 days | (#45257983)

You do realize every time you do something like call people 'tea baggers' you make yourself look like a douche to everyone around you, right? What you find cute and clever, most of the rest of us equate to an ignorant 12 year old who doesn't have any clue what he's talking about.

Name calling is for ignorant children, grow up.

Re:Replication (2)

cryptolemur (1247988) | 1 year,3 days | (#45257673)

The *best* way would be to do a different experiment with the expectation of getting the same results if the original research was valid and understanding of the studied phenomena good. Then, regardless of whether the second study validates the first one or not, we would actually have more data and better understanding of the issue and problems regarding it's study.

Invalidating shoddy research would be a bonus.

Re:Replication (0)

Anonymous Coward | 1 year,3 days | (#45258831)

The problem with this is that if there is failure to replicate it can always be chalked up to the difference in the experiment. We need direct replications to understand if the effects that have been reported are consistent.

Re:Replication (1)

ApplePy (2703131) | 1 year,3 days | (#45257987)

Indeed, replication is good. One should always look askance at "hot new science" until it's repeated enough. But then you just go off the rail and make it political.

Fox's interpretation of this story or the tardbaggers' interpretation of that. I've already assigned "herp", "derp" and "6,000 years" to hotkeys.

Right... so believing that the federal government is too big and out of control, equates, in your mind, to a complete lack of scientific understanding to the point of mental retardation.

Got it.

Just about everyone believes in something nonsensical and unscientific. Whether it's the 6,000 year nonsense of the religious wingnuts, or the notion that all humans are somehow equal in ability, per the lefty wingnuts. We'll probably also find that most First World people are firmly behind the concept of science -- that is, as a tool -- but more skeptical of the results sometimes generated. It's far too often nowadays that science is bent to the will of the politician or the ad-man.

Re:Replication (0)

Anonymous Coward | 1 year,3 days | (#45258115)

It's far too often nowadays that science is bent to the will of the politician or the ad-man.

For a moment there I thought you meant "mad-man," then I noticed the annoying Allstate commercial in the corner. (no, I do not use an adblock compulsively, mostly just when visiting a new site to make sure they don't get any ad revenue from me until I think a page has some value)

Re:Replication (1)

jythie (914043) | 1 year,3 days | (#45258239)

There is some money in replication, but funding does tend to be harder to get. It can be hard to explain to politicians and board members why one wants money to do what has already been done. Which is a pity given how important it is.

Re:Replication (1)

Sockatume (732728) | 1 year,3 days | (#45258387)

There's a lot of money in replicating results that big business has a vested interest in disagreeing with. Chew on that the next time somebody tells you that climate scientists only support global warming to get grant funding.

'Sexiest claims' (1)

Anonymous Coward | 1 year,3 days | (#45257613)

New Paper Idea: The study of light waves as they traverse the complex structure of a Brazilian volleyball team

in vivo biology is not all science (3, Insightful)

methano (519830) | 1 year,3 days | (#45257615)

It's important to remember that in vivo biology is not all of science. It's a lot harder to know what you're doing in biology. If you want excellent reproducible science, let's just roll balls down inclines, measure that and hope we don't get sick.

Half right (4, Interesting)

SirGarlon (845873) | 1 year,3 days | (#45257625)

Well, it's good to see a major scientific institution waking up to a phenomenon Richard Feynman warned about in the 1970s [columbia.edu] . Yet it seems to me the proposed solution is a little ad hoc. If scientists want to restore integrity to their field(s) -- and I applaud their efforts to do so -- why aren't they using an experimental approach to do so? I think they should try several things and collect data to find out what actually works.

Re:Half right (0)

Anonymous Coward | 1 year,3 days | (#45258289)

...why aren't they using an experimental approach to do so?

Because there is a consensus. Validation and experimentation go out the window when that happens. Consensus is a bigger threat to science than anything else today. Consensus has become it's own religion. The mentality that people cannot be wrong, that they should not be questioned, that they should not have to prove their claims, because of their position and title. That's not science, but it happens every day, more and more. Consensus breeds stagnation. Consensus destroys the quest for the unknown. Consensus imprisons the mind. It binds good men with mental chains. Consensus is a deep seated faith that what is known is known, it cannot be wrong.

Re:Half right (2)

sackvillian (1476885) | 1 year,3 days | (#45258291)

If scientists want to restore integrity to their field(s) -- and I applaud their efforts to do so -- why aren't they using an experimental approach to do so? I think they should try several things and collect data to find out what actually works.

That's exactly what's happening. Different groups of scientists, journalists, university-groups and so forth are trying to implement a variety of systems. [arstechnica.com]

Of course, like real science, each group tends to only focus on one approach with the hope that their results will emerge as the best amongst the competition. You're not referring to "scientists" as some kind of monolothic entity, are you?

mod@ Down (-1)

Anonymous Coward | 1 year,3 days | (#45257637)

We''l[ be able to

Biology's problem (2)

ITEM-3 (3348273) | 1 year,3 days | (#45257651)

Whenever one of these stories is posted about inaccurate and falsified research papers, it's always a field related to biology. This doesn't seem to be nearly as much of a problem with the hard sciences (physics, chemistry). We should avoid rhetoric like "science has lost its way" since the problem is mostly isolated to one branch of science and such statements only serve as ammo for the anti-science crowd. Disclaimer: I'm a physicist.

Re:Biology's problem? Hard sciences, too. (2)

Nightlight3 (248096) | 1 year,3 days | (#45257945)

Physics is not immune to parasitic and mercenary research phenomena either, especially in more exotic areas with great funding potential, such as quantum computing & crypto where exaggerations and self-puffery are common. One might say the whole field is of that kind, since their whole theorizing (which is all they got) rests on the speculative aspects of quantum measurement theory, the foundations of which are still awaiting unambiguous experimental demonstration (such as the "loophoole free" violations of Bell inequalities [arxiv.org] ), for over half century already. Should the experimental failure to confirm the fundamental conjectures persist, the whole field will be recognized as fancily relabeled analog computing (such as D-Wave system).

Infected to (1)

z3r0w8 (664036) | 1 year,3 days | (#45257699)

So, this information source is infected the same as any other information source today. No one cares if they are right or true anymore, just if it gets views. I think we all know most science isn't sexy...

Say... (0)

Impy the Impiuos Imp (442658) | 1 year,3 days | (#45257785)

It's almost as if behavior is evolving to maximize success at sucking on the tit of government.

Nah.

So now we're all skeptics... (0, Flamebait)

hsthompson69 (1674722) | 1 year,3 days | (#45257795)

...except about something like catastrophic anthropogenic global warming :)

It's so funny that the left wing can be so insightful about certain things, but then engage in utter, mind blowing hypocrisy counter to their rational, reasoned argument.

The answer in this case is simple - make every damn scientific paper start with an intro section called "necessary and sufficient falsifiable hypothesis statement". Too much speculative navel gazing and fuzzy study setup leaves the room open to fiction writing, rather than cold hard scientific scrutiny.

Remember, first and foremost, science is a way for us to prove ourselves *wrong* - it's a way of knocking down ideas, and only grudgingly giving acceptance to the ones that survive the contest. The best scientists ruthlessly try to find every possible hole in their ideas, rather than glossing over contradictory evidence or alternatives.

Re:So now we're all skeptics... (1)

ApplePy (2703131) | 1 year,3 days | (#45258045)

It is amazing how much "it's settled science!" sounds like "God said so!" isn't it?

I know people who regularly drop the "settled science" quips on social media (average intellectual level: ZOMG AGW is totally fer realz!!) -- people I personally know who could not even define the word "science". It's important to remember that there's a gulf between those who understand the method, and those who mindlessly parrot whatever is popular. The latter we call religion.

Re:So now we're all skeptics... (1)

Sockatume (732728) | 1 year,3 days | (#45258459)

It's more like "you'll need to provide counter-evidence at least as strong as the concensus".

Re:So now we're all skeptics... (0)

Anonymous Coward | 1 year,3 days | (#45258563)

I think in the case of AGW you have on one side an army of climatologists who are almost all in agreement (plus their armchair experts) versus an army of assorted unrelated fields (plus their armchair experts) who are all 100% in agreement but can't agree on the reason they agree. So I won't say it's settled science, but I will say that the good science is heavily in favor of one view.

Re:So now we're all skeptics... (1)

m.shenhav (948505) | 1 year,3 days | (#45258109)

Oh so Popper's Falsificationism is the be-all and end-all of what constitutes science? I guess I was mistaken when I thought there is far more subtlety and detail in the philosophy of science.....

Re:So now we're all skeptics... (2)

hsthompson69 (1674722) | 1 year,3 days | (#45258423)

Yes, in fact, Popper was right - falsification is the bedrock of science.

Without falsification, you simply have religion, no matter how fancy the lab coat you dress up in looks like :)

Just because you use maths doesn't mean it's not religion.

Re:So now we're all skeptics... (1)

Anonymous Coward | 1 year,3 days | (#45258481)

Oh so Popper's Falsificationism is the be-all and end-all of what constitutes science? I guess I was mistaken when I thought there is far more subtlety and detail in the philosophy of science.....

If there is philosophy in science, you aren't doing science correctly. As a practicing (and grumbling and struggling) scientist, the original poster is correct - real science is all about proving things wrong because it is impossible to prove something right. The best we can do is to say 'we haven't proven it wrong yet.' This is why in my eyes large portions of biology and sociology and psychology isn't science. Or to quote a famous scientist:

All science is physics, the rest is stamp collecting. - E. Rutherford

Re:So now we're all skeptics... (2)

JoeDuncan (874519) | 1 year,3 days | (#45258117)

...except about something like catastrophic anthropogenic global warming :)

Oh man, you are totally right! How could I have been so blind! We should be more skeptical about shit like evolution and gravity too! Down with close minded dogma!

Re:So now we're all skeptics... (1)

hsthompson69 (1674722) | 1 year,3 days | (#45258477)

Evolution (or more specifically, natural selection), and gravity have necessary and sufficient falsifiable hypothesis statements.

Astrology does not have a necessary and sufficient falsifiable hypothesis statement.

Intelligent design does not have a necessary and sufficient falsifiable hypothesis statement.

AGW does not have a necessary and sufficient falsifiable hypothesis statement.

If you want to understand how to discern pseudo-science from science, look for the necessary and sufficient falsifiable hypothesis statement.

Re:So now we're all skeptics... (0)

Anonymous Coward | 1 year,3 days | (#45258649)

...except about something like catastrophic anthropogenic global warming :)

Oh man, you are totally right! How could I have been so blind! We should be more skeptical about shit like evolution and gravity too! Down with close minded dogma!

Yes, be skeptical about everything. Newton's gravity, as taught in low level physics classes, is wrong - which was one of the reason why Einstein proposed Relativity. Yet we are already seeing some evidence of where Relativity is breaking down. So what is gravity? As for evolution (and I work in the field), many of their predictions are vague or more qualitative than quantitative. Add in the fact - which version of evolution? Are we talking about Wallace's version? Or Darwin's version? Or what about Lamarck's version which seems to have some evidence when metagenomics are considered? Or what about Fisher's work?

So yes, be skeptical about shit like evolution and gravity. The theories we have that explains or describes them may be wrong. They have not been proven wrong yet, but who knows what the future holds.

Re:So now we're all skeptics... (1)

Immerman (2627577) | 1 year,3 days | (#45258419)

>Remember, first and foremost, science is a way for us to prove ourselves *wrong* - it's a way of knocking down ideas, and only grudgingly giving acceptance to the ones that survive the contest. The best scientists ruthlessly try to find every possible hole in their ideas, rather than glossing over contradictory evidence or alternatives.

Yes it is, which is why I'm inclined to take seriously steadily expanding body of data that almost entirely supports catastrophic AGW. Are there occasional points of conflicting data? Of course - there always are in any experiment. When almost everything fits together consistently within the framework of a broadly accepted model, then the outliers are quite likely to be errors, or possibly things which we simply do not yet understand well enough to see how they fit within the model.

Of course it's also always possible that they are the signs of something completely unexpected that's going to turn the accepted model on it's ear - as quantum mechanics did to physics. The thing is though that even then the prior model tends to be mostly valid, there's just other factors at work that allow for additional behaviors not predicted by the original model. So yeah, it may be that there's some other way to interpret the data that makes for a much rosier climate picture, but nobody has yet managed to create another model that holds together in the face of available data. And not for lack of trying by various researchers funded by those with a vested interest in the status quo, who have mostly produced science so laughably bad that it's obviously a PR tool never intended to be plausibly presented to other scientists.

Re:So now we're all skeptics... (1)

hsthompson69 (1674722) | 1 year,3 days | (#45258515)

When almost everything fits together consistently within the framework of a broadly accepted model, then the outliers are quite likely to be errors, or possibly things which we simply do not yet understand well enough to see how they fit within the model.

The same can be said about astrology.

A "consistent with" model isn't science - hell, the bible gives us plenty of "consistent with" observations...the key to science is falsifiability, period.

If your model predicts that a coin flip will be either heads or tails, 100% of the time, it's not much of a model. Heads I win, tails you lose is a sucker bet, not a scientific proposition.

Put another way, can you name or cite any catastrophic AGW studies that ever stated a necessary and sufficient falsifiable hypothesis statement? Can you quote that statement?

Re:So now we're all skeptics... (1)

Sockatume (732728) | 1 year,3 days | (#45258429)

Having lost the argument on GW and then AGW, denlialists have now invented catastrophic AGW as their new talking point? Good grief.

Re:So now we're all skeptics... (1)

hsthompson69 (1674722) | 1 year,3 days | (#45258537)

Wait, are you saying you deny the last 17 years of no statistically significant warming? (GW)

Or are you saying you deny the last 150 years of natural warming coming out of the little ice age? (AGW)

Or are you saying that AGW is true, but we don't need to worry about it, because on the whole increased temperatures are better for the biosphere? (CAGW)

What part of "climate always changes" don't you understand?

Re:So now we're all skeptics... (1)

Sockatume (732728) | 1 year,3 days | (#45258581)

I can't say that I, personally deny any of those things, as I'm unqualified. However I have the sense to side with a broad spectrum of independent, competing researchers in a wide variety of fields with decades more experience than me whose work all points in a direction contrary to the argument your furthering.

Re:So now we're all skeptics... (1)

Sockatume (732728) | 1 year,3 days | (#45258539)

You understand that you're basically calling for the elimination of explorational science, right? No more observational science, no more materialistically inventive science, no more methodologically inventive science, no more science but that which can be boiled down into a child's pat hypothesis-test-result-conclusions science lesson.

You're basically saying that we should obliterate science as a creative endeavour.

Science has not lost it's way (0)

Anonymous Coward | 1 year,3 days | (#45257809)

Science is a method of doing things. Some scientists in some research fields are failing to follow through with the science, which is to verify results.

How To Formulate Subject Lines Better (1)

gnomeza (649598) | 1 year,3 days | (#45257869)

Or "How Better To Formulate Subject Lines". But "How To Better Formulate Subject Lines"? Ugh. Have to read that three times to parse it.

Religion (1)

BitZtream (692029) | 1 year,3 days | (#45257951)

Far too many people treat science like a religion. If a scientist says it, it must be true, which ironically is the exact opposite of science. As has already been pointed out, all science can do is tell you something DOESN'T work that way. Instead what happens is people latch on to stupid things as if its carved in stone, regardless of how many times over the years it gets proved to be untrue or not entirely correct, they'll latch on to current theory and treat it as if its a law, and won't even blink an eye when the current theory turns out to be wrong and needs modified, the next time around ... mysteriously, it can't be wrong now!!@%!

People who treat science like a religion are just as bad as religious nut jobs, arguably worse since at least the religious nut job is aware that they are basing their thoughts on faith rather than proof.

It's all about incentives (3, Interesting)

voislav98 (1004117) | 1 year,3 days | (#45257997)

What is not discussed is that in science as in life it's all about incentives. All you have to is look at who is paying for these studies, directly (through research grants) or indirectly (speaking or consulting fees), and things will become much clearer. The biomedical and life sciences are most vulnerable to corruption because the incentives are very high, successful drug/treatments are worth a lot of money. Even unsuccessful ones, given the proper appearance of effectiveness are worth money.

Other sciences are less susceptible because there is no incentive to hype the results, not because those scientists are more ethical. There is two solutions for the problem. One is to remove incentives, which would mean overhauling the whole system of scientific funding. The other is to mandate raw data sharing. This would make it easier for people to reanalyze the data without actually redoing the experimental parts.

A good example of this is Reinhart-Rogoff controversy in economics, where they claimed one thing in their widely publicized 2010 paper (high debt levels impede growth), but their statistical analysis was shown to be riddled with errors, skewing the data to the desired conclusion. This was discovered the when they shared their raw data with a University of Massachusetts grad student. While data sharing would not eliminate these issues it would make is harder to perform "statistical" analysis that introduces biases.

There's a reason for that (1)

aaaaaaargh! (1150173) | 1 year,3 days | (#45258065)

Really there is a simple reason for this. (Of course, it's not the only one but presumably the primary one.)

Tenure-track positions and funding are to a large extent determined on the basis of the number of publications weighted by the reputation of the journals, not by the quality of publications.

The idea is that good journals will reject bad papers, which doesn't work as well as is desirable due to the extreme amount of submissions the journals receive, which have to be reviewed by relatively small numbers of unpaid voluntary reviewers.

There are many ways this problem could be alleviated and I have no idea which would be the best one. For example, hiring comittees could be encouraged to only take a look at 10 papers chosen by the applicant and disregard all others including their total number. But it's doubtful they would follow this advice in practise. Or, "allowed" publications per average year could be limited to a minium of n and a maximum of m papers. So for example, to keep funding you need to publish (on average, over a larger period of time) at least 1 peer-reviewed article and no more than 3 per year in average. Sounds crazy and I don't know how to enforce this, but it would increase the quality of papers if m is chosen sufficiently low. Or, get more stringent peer reviewing, although it's a mistery how you'd obtain that in the current system. Perhaps open access journals with crowd reviewing/ranking and meta-moderation would work, as long as mechanisms are held in place to weed out sockpuppets and trolls - difficult, though.

Anyway, it's mostly the publication pressure, in terms of numbers, that causes bad publications.

Now back to work... I need to finish a hastily written paper.

Re:There's a reason for that (1)

Sockatume (732728) | 1 year,3 days | (#45258679)

In my experience good journals aren't about rejecting bad papers - once you get above a certain level in a subject each journal is sending its papers to the same reviewers - but that good journals reject unimportant papers. That's why things like publications in big-name journals are significant, they imply important work. Not, necessarily, correct work, but anyone who's unwilling to publish research that turns out to be wrong shouldn't be a scientist in the first place.

Science isn't broken. (3, Insightful)

Chalnoth (1334923) | 1 year,3 days | (#45258333)

The correct take-away from this kind of study is not that a specific field of science is "broken" (also, cancer research is not all of science), but rather that there is room for improvement.

There is no question whatsoever that cancer research has made leaps and bounds over the last few decades in terms of improving the lives of many people with cancer, both by helping them to live longer, and by helping them to live better. What this kind of study shows is that we can do even better still, if we can find ways to fix the flaws that remain in cancer research.

Papers may wrong but truth is decided by consensus (2)

umafuckit (2980809) | 1 year,3 days | (#45258583)

It has been said in other papers too that a lot of the literature is wrong (http://www.plosmedicine.org/article/info:doi/10.1371/journal.pmed.0020124) and that this is more likely in higher impact journals and for papers with lower sample sizes (http://www.nature.com/nrn/journal/vaop/ncurrent/full/nrn3475-c6.html). The idea is that a smaller sample size is more likely to lead to a Type I error (incorrectly finding a statistically significant result) or over-estimating the size of an effect. Consequently, these smaller sample size studies find what looks like a stunning effect but what they're really seeing is an outlier. The paper looks awesome so it gets published somewhere high impact, where it is sensationalised. This effect is exacerbated by the "publish or perish" mentality, where researchers are pressured to produce many high impact papers in order to get grants. It's also a function of the fact that a lot of research is being done, so the high volume increases the odds of this shit happening. Cancer biology is particularly prone to this sort of effect because it's very competitive, there's a lot of interest in it and so it generates high impact papers, and there are a lot of big screening studies that depend heavily on statistics to confirm effects. In some branches of biology you hardly need a stats test because variables are few in significance is obvious. However, when you're screening vast numbers of drug targets then you have all sorts of problems with multiple comparisons and the like. You need elaborate stats tests and they have to be done right. Overall, however, whether the community as a whole believes something is determined by state of the literature in general and not just a single study. What we consider true or false is influenced by the politics of science as well as the data. This is nicely reviewed in the controversial book, "The Golem", by Collins and Pinch (http://www.amazon.com/The-Golem-Should-Science-Classics/dp/1107604656).
Load More Comments
Slashdot Login

Need an Account?

Forgot your password?