Beta

×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

How Science Goes Wrong

Soulskill posted about 9 months ago | from the frankenstein's-monster,-skynet,-and-bacon-perfume dept.

Science 316

dryriver sends this article from the Economist: "A simple idea underpins science: 'trust, but verify'. Results should always be subject to challenge from experiment. That simple but powerful idea has generated a vast body of knowledge. Since its birth in the 17th century, modern science has changed the world beyond recognition, and overwhelmingly for the better. But success can breed complacency. Modern scientists are doing too much trusting and not enough verifying — to the detriment of the whole of science, and of humanity. Too many of the findings that fill the academic ether are the result of shoddy experiments or poor analysis (see article). A rule of thumb among biotechnology venture-capitalists is that half of published research cannot be replicated. Even that may be optimistic. Last year researchers at one biotech firm, Amgen, found they could reproduce just six of 53 'landmark' studies in cancer research. Earlier, a group at Bayer, a drug company, managed to repeat just a quarter of 67 similarly important papers. A leading computer scientist frets that three-quarters of papers in his subfield are bunk. In 2000-10 roughly 80,000 patients took part in clinical trials based on research that was later retracted because of mistakes or improprieties. Even when flawed research does not put people's lives at risk — and much of it is too far from the market to do so — it squanders money and the efforts of some of the world's best minds. The opportunity costs of stymied progress are hard to quantify, but they are likely to be vast. And they could be rising."

cancel ×

316 comments

Sorry! There are no comments related to the filter you selected.

Greed (0)

Anonymous Coward | about 9 months ago | (#45158747)

This is what makes thinks go wrong.

Re:Greed (5, Funny)

Anonymous Coward | about 9 months ago | (#45158775)

FYI, 'trust, but verify' is also a great rule of thumb for spell-check.

Re:Greed (1)

approachingZero (1365381) | about 9 months ago | (#45159339)

Me thinks you correct.

Re:Greed (0)

Anonymous Coward | about 9 months ago | (#45159489)

Methinks feta means
youmakecheesyjokessofastyoucan'tevenhitthespacebar

Peer review stretched to its limit by money (5, Interesting)

Paul Fernhout (109597) | about 9 months ago | (#45159025)

http://www.its.caltech.edu/~dg/crunch_art.html [caltech.edu]
"The crises that face science are not limited to jobs and research funds. Those are bad enough, but they are just the beginning. Under stress from those problems, other parts of the scientific enterprise have started showing signs of distress. One of the most essential is the matter of honesty and ethical behavior among scientists.
    The public and the scientific community have both been shocked in recent years by an increasing number of cases of fraud committed by scientists. There is little doubt that the perpetrators in these cases felt themselves under intense pressure to compete for scarce resources, even by cheating if necessary. As the pressure increases, this kind of dishonesty is almost sure to become more common.
    Other kinds of dishonesty will also become more common. For example, peer review, one of the crucial pillars of the whole edifice, is in critical danger. Peer review is used by scientific journals to decide what papers to publish, and by granting agencies such as the National Science Foundation to decide what research to support. Journals in most cases, and agencies in some cases operate by sending manuscripts or research proposals to referees who are recognized experts on the scientific issues in question, and whose identity will not be revealed to the authors of the papers or proposals. Obviously, good decisions on what research should be supported and what results should be published are crucial to the proper functioning of science.
    Peer review is usually quite a good way to identify valid science. Of course, a referee will occasionally fail to appreciate a truly visionary or revolutionary idea, but by and large, peer review works pretty well so long as scientific validity is the only issue at stake. However, it is not at all suited to arbitrate an intense competition for research funds or for editorial space in prestigious journals. There are many reasons for this, not the least being the fact that the referees have an obvious conflict of interest, since they are themselves competitors for the same resources. This point seems to be another one of those relativistic anomalies, obvious to any outside observer, but invisible to those of us who are falling into the black hole. It would take impossibly high ethical standards for referees to avoid taking advantage of their privileged anonymity to advance their own interests, but as time goes on, more and more referees have their ethical standards eroded as a consequence of having themselves been victimized by unfair reviews when they were authors. Peer review is thus one among many examples of practices that were well suited to the time of exponential expansion, but will become increasingly dysfunctional in the difficult future we face."

I've collected some other quotes on social problems in science here:
http://www.pdfernhout.net/to-james-randi-on-skepticism-about-mainstream-science.html#Some_quotes_on_social_problems_in_science [pdfernhout.net]

Re:Peer review stretched to its limit by money (1)

riverat1 (1048260) | about 9 months ago | (#45159125)

So the question is what to do about that? It's easy to point out the faults but much more difficult to come up with constructive suggestions. In most cases the only people qualified to judge the work are others in the field. I just don't see how you get away from peer review despite its warts.

Re:Peer review stretched to its limit by money (2)

narcc (412956) | about 9 months ago | (#45159233)

Making people aware of the problem is a good first step.

Re:Peer review stretched to its limit by money (0)

Anonymous Coward | about 9 months ago | (#45159469)

Peer review is usually quite a good way to identify valid science.

There is absolutely no evidence for this claim at all.

Re:Peer review stretched to its limit by money (1, Troll)

fermion (181285) | about 9 months ago | (#45159477)

Also cancer research/biotech is barely science. No one would say Thomas Edison was a scientist, he was just someone who tried a bunch of stuff until something worked, with not necessarily a great deal of understanding of the basic science. We can argue the specifics of the points, but most of these people are just highly trained technicians paid to write papers that say what they are supposed to. Remember all the paper than came out supporting tobacco?

Most of this 'research' has significant consequences if they do not play out. No one is going to rock the boat by verifying results because such knowledge will keep the product from coming to market. Pharmaceuticals has a built in buffer to pay for future deaths, but those cannot be made if the product is never on the market.

Now for real science, this is not an issue. Researches will regularly ignore bad research, but the results are not often the most critical thing. It is the methods. Even bad result can lead to innovative experiment. I recall one case where the results of a technique were bad for years, but the technique proved very useful, and the errors were eventually discovered.In fact the damaging papers are sometimes where bad techniques are promulgated.

Re:Peer review stretched to its limit by money (1)

geekoid (135745) | about 9 months ago | (#45159495)

Shut up, you have no idea what you are talking about. I just can't tell if you don't know what science is, or are just a pig headed ass.

Re:Greed (-1, Flamebait)

Anonymous Coward | about 9 months ago | (#45159081)

This story is exactly why so many people do not believe the myth of evolution. And yet when you call out the "science" as being unreproducible you are called a loon and painted as anti-science. Looks like maybe the evolutionists need to spend more effort on doing repeatable and reproducible science rather than extrapolating entire species from bone fragments.

Re:Greed (-1)

Anonymous Coward | about 9 months ago | (#45159155)

How predictable. "Flamebait" mod used simply because one disagrees. At least have the balls to respond to me instead of downmodding me.

Re:Greed (1)

Anonymous Coward | about 9 months ago | (#45159281)

...myth of evolution.

Evolution and speciation have both been directly observed and are well documented, no bone fragments required.

Re:Greed (-1)

Anonymous Coward | about 9 months ago | (#45159377)

Sure it has, hence why there's no repeatable experiments and data, no? You know, the whole point of the article.

Re:Greed (0)

Anonymous Coward | about 9 months ago | (#45159515)

Except there is. A colossal wealth of data, actually, on everything from microbial cultures to insects, to the flu shots we receive each year.

Sounds Like Work... (0)

Anonymous Coward | about 9 months ago | (#45158765)

Ain't nobody got time for that!

Re:Sounds Like Work... (5, Insightful)

lgw (121541) | about 9 months ago | (#45159217)

It's not about being lazy. Feynman famously addressed this [youtube.com] in his "Cargo Cult Science" rant in his Caltech commencement address given in 1974. (There's no recording AFAIK, that link is to someone reading the transcript).

He makes very good points: funding is for new results. Attempting to repeat another scientists published work is not a new result (unless you can't), and many places won't even allow you to try, unless it's something very sexy like observing the Higgs boson or something. It's an important structural problem, and it was worth calling attention to forty years ago.

There's no doubt that some unscrupulous researchers have noticed this and are gaming the system. The incentives to do so are particularly high in biochem.

Re:Sounds Like Work... (2)

acscott (1885598) | about 9 months ago | (#45159385)

bump. Gaming the system is smart behavior (maybe unethical though). Just provide incentives to do the work in the best way. Just an idea, for example, and in no way perfect: If a scientist publishes (peer-review and all) results they get a credit, but if another scientist cannot replicate results they get 90% of that credit, leaving the former with 10% of the credit. Also in an experiment you have inputs, transformations and outputs. Inputs being the data, transformations being the analysis, and the outputs being the result that tests a hypothesis. The Inputs (i) and transformations (t) must be made available for other scientists to reproduce the results. This too, can be gamified. If a scientist provides i they get a bigger credit. If they provide t, they get an even bigger credit. In another way, the scientific process as implemented is also subject to the scientific method. It should be anyway.

Anti-science? See, now you have proof! (1, Insightful)

Anonymous Coward | about 9 months ago | (#45158767)

Aaaand this is exactly the kind of thing that young-earth creationists and climate change deniers will jump on to show that science (and scientists) can't be trusted.

Re:Anti-science? See, now you have proof! (0)

Anonymous Coward | about 9 months ago | (#45158839)

TFA is a Homeopathy fan's wet dream.

Re: Anti-science? See, now you have proof! (1)

crdotson (224356) | about 9 months ago | (#45158957)

I would think it is their nightmare instead. I've never seen a homeopathy result verified.

Re:Anti-science? See, now you have proof! (3, Insightful)

Chas (5144) | about 9 months ago | (#45158979)

No. Actually it isn't. Because homeopathy can't even replicate their own results in controlled environments.

It SHOULD, however, be a wakeup call to scientists all over that their chosen fields are more caught up in the "publish or perish" mentality than they should be.

Between this, and others willing to take these unreplicated (and possibly unreproducible) studies as "Holy Writ", what people think of as science IS becoming as sloppy as religion.
Which makes it harder for the people who actually DO the grunt work and the follow-up to receive their just due.

Re:Anti-science? See, now you have proof! (0)

Anonymous Coward | about 9 months ago | (#45159313)

There are numerous clinical trials showing benefit for homeopathic treatments. The truth is that clinical trials are very weak forms of evidence. Comparing two averages of >10,000 people each is so far away from any causal mechanism as to be meaningless in my opinion.

Re:Anti-science? See, now you have proof! (2)

Desler (1608317) | about 9 months ago | (#45159415)

There are numerous clinical trials showing benefit for homeopathic treatments.

And yet you've failed to even link to the results of even a single one out of the supposed "numerous" clinical trials that have supposedly shown what you claim.

Re:Anti-science? See, now you have proof! (3, Informative)

LurkerXXX (667952) | about 9 months ago | (#45159335)

I'm not sure what 'publish or perish' has to do with it.

I do research. I can get funding from NIH from a well designed, well reasoned approach to learn something new. What I can't get is funding to replicate some other researcher's finding.

  I'd be happy to do replication work in addition to novel research, but it's a simple fact that no one will pay for salary of lab techs, lab equipment, or reagents in order to replicate something, even if I'm willing to donate my own time.

Re: Anti-science? See, now you have proof! (0)

crdotson (224356) | about 9 months ago | (#45158939)

If you would be so kind as to replicate the catastrophic prediction results -- oh, wait, those can't be tested?

I trust the results that can be verified. Add co2, heat increases. I don't trust the out-of-your-ass guesses on co2 sensitivity, nor the maybe-it-is thinking linking every bit of bad weather to AGW.

Re: Anti-science? See, now you have proof! (1)

Anonymous Coward | about 9 months ago | (#45159007)

It's not a fucking guess. You just proved the GP's point.

Part of the problem is you're treating as a defensible status-quo position: "unprecedented human activity cannot cause unprecedented environmental responses", especially when we have evidence that other unprecedented human activities have caused more local environmental catastrophes.

Where are your tests for the counter-theory that there is no global climate change, in the face of data such as http://www.skepticalscience.com/pics/SkepticsvRealistsv3.gif [skepticalscience.com] ?

Re: Anti-science? See, now you have proof! (2)

crdotson (224356) | about 9 months ago | (#45159187)

Yes, I've seen the 'escalator' animation and have read the relevant articles at skepticalscience. The real question isn't whether there's warming, it's what the slope of the red line actually is, when we add CO2 the way we have been.

I'm not saying that "unprecedented human activity cannot cause unprecedented environmental responses". I'm also not stating that "unprecedented human activity MUST cause unprecedented environmental responses", which you are. Look, from the IPCC AR5 report:
---
The equilibrium climate sensitivity quantifies the response of the climate system to constant radiative forcing on multi-century time scales. It is defined as the change in global mean surface temperature at equilibrium that is caused by a doubling of the atmospheric CO2 concentration. Equilibrium climate sensitivity is likely in the range 1.5C to 4.5C (high confidence), extremely unlikely less than 1C (high confidence), and very unlikely greater than 6C (medium confidence)16. The lower temperature limit of the assessed likely range is thus less than the 2C in the AR4, but the upper limit is the same. This assessment reflects improved understanding, the extended temperature record in the atmosphere and ocean, and new estimates of radiative forcing. {TFE6.1, Figure 1; Box 12.2}

16 No best estimate for equilibrium climate sensitivity can now be given because of a lack of agreement on values across assessed lines of evidence and studies.
---
So the ever-so-sophisticated estimate (sorry, not 'guess'!) is from 1.5C to 4.5C, and unlike previous reports, they declined to give a best estimate (previously mentioned at 2.5C with high confidence). I believe that most people would agree that we don't have much of a problem at 1.5C and that we have a big problem if it's 4.5C, so their estimates don't really tell us much at this point. Add to that the fact that the observed temperature trends have been way at the bottom of what most of the models predict, and I don't think it's a stretch to say that alarmists like you have been seriously overestimating the dangers here.

But that wasn't the original point of the post. The point is that you cannot replicate or test the results in most of the current climate papers, other than to wait 50 years. We've waiting around 15 years since the initial predictions, though, and the initial ones weren't very good.

Re: Anti-science? See, now you have proof! (4, Insightful)

riverat1 (1048260) | about 9 months ago | (#45159201)

If you would be so kind as to replicate the catastrophic prediction results -- oh, wait, those can't be tested?

We're in the process of testing them right now. We should have results in 50 or 100 years although chances are we don't have to run the full experiment to see where it's heading.

What makes you think the results on CO2 sensitivity are "out-of-you-ass guesses" rather than just an expression of the uncertainty of the results? Where have you seen a scientist that links every bit of bad weather to AGW? There are some non-scientists who may do that but that's not science.

Re:Anti-science? See, now you have proof! (1)

schwep (173358) | about 9 months ago | (#45158953)

Nor can the results be verified in an independent lab environment.

Re:Anti-science? See, now you have proof! (-1)

Anonymous Coward | about 9 months ago | (#45158975)

Precisely...all those nasty rethuglicans and god fearing people should just learn to shut up and just accept everything proclaimed from the Church of Gore. It's people like you who will be the first in line to start shoving people into the ovens, I'm afraid.

Re:Anti-science? See, now you have proof! (1)

uCallHimDrJ0NES (2546640) | about 9 months ago | (#45158999)

Right, coward...because EVERYTHING is an "us and them" issue. Let's not talk about the economic difficulty of funding proper research. Let's talk about your pet peeves instead.

Re:Anti-science? See, now you have proof! (0)

Anonymous Coward | about 9 months ago | (#45159075)

Unfortunately, almost everything is becoming an "us and them" issue. What do expect when the conversation starts with accusations of being some kind of denier? Seems like rational discourse, right off the bat, is unlikely.

Re:Anti-science? See, now you have proof! (2)

MtHuurne (602934) | about 9 months ago | (#45159067)

The whole point of science is using a process that will lead to more reliable results. If we stay quiet about weaknesses of the process or how it's executed, what is left will be science in name but not actually valuable.

Re:Anti-science? See, now you have proof! (5, Insightful)

phantomfive (622387) | about 9 months ago | (#45159117)

If you are trying to prove to young-earth creationists that the earth is old because they should trust scientists, you're doing the wrong thing. If you do that, you turn it into a fight about, "the guy I trust" vs "the guy you trust."

Instead, if you really want to talk to a young earth creationist (I don't know why you would), you need to show them the evidence. Really dig deep. If they want to discuss carbon dating, then dig in and show the evidence we have of why carbon dating works. Eventually, if they are willing to go along with you (and it will take a lot of work so they might not), they will turn into an old-earth creationist.

And you will absolutely learn something along the way. Never turn the discussion into an argument about "the guys I trust" vs "the guys you trust" because that argument is never won, by either side.

Re:Anti-science? See, now you have proof! (0, Informative)

Anonymous Coward | about 9 months ago | (#45159279)

Try the same thing with an AWGer and you will get disasterous results. Point out the Phil Jones ignored FOIA requests and when it looked like he would be forced to hand over his data for peer review he had it all deleted. Point out how the predictions of the last 12 years have been completely wrong. Point out how despite being wrong and there being no warming for 15 years, the IPCC concluded their research was 96% accurate and ignored how far off their previous predictions were.

AWG has become a laughing stock to people who understand science, yet no matter how much you point it out the "true believers" still call you names because they fail at discussing the facts.

Even previous IPCC members have claimed the latest IPCC report is a joke [dailycaller.com] yet I'll be called names for point that out.

Re:Anti-science? See, now you have proof! (1)

mcgrew (92797) | about 9 months ago | (#45159457)

if you really want to talk to a young earth creationist (I don't know why you would)

To teach, of course. And I echo everything you said.

Re:Anti-science? See, now you have proof! (1)

narcc (412956) | about 9 months ago | (#45159251)

Aaaand this is exactly the kind of thing that young-earth creationists and climate change deniers will jump on to show that science (and scientists) can't be trusted.

So we should just ignore the problem because a few loons will use it to justify their crazy beliefs? Brilliant.

I weep for humanity.

Re:Anti-science? See, now you have proof! (0)

Anonymous Coward | about 9 months ago | (#45159337)

I'm the OP for this subthread. My point was exactly the opposite of what you're suggesting. It was more intended as "Dammit, people, see what you've done? Now you've given them ammunition!" So no, I absolutely do not think the problem should be ignored.I'm just sad it happened in the first place.

Re:Anti-science? See, now you have proof! (0)

Anonymous Coward | about 9 months ago | (#45159287)

``The results of our work are mostly fictional, but we're super smart and highly educated, so trust us anyways — especially the part where you downsize yourself to some arbitrarily pathetic level of subsistence to `save the Earth' because we say so.''

Good luck with that.

Re:Anti-science? See, now you have proof! (1)

Anubis IV (1279820) | about 9 months ago | (#45159359)

Science is a method, not a religion. Trust has nothing to do with it. As a believer in a particular religion, I trust that where modern science seems to disagree with my beliefs, it will eventually be demonstrated to be incorrect, but if that happens we wouldn't say that my trust had anything to do with it. It'd simply be the scientific method working as it should, and has nothing to do with whether or not I or anyone else trusted in it.

Re:Anti-science? See, now you have proof! (1)

mcgrew (92797) | about 9 months ago | (#45159491)

That was insightful. If your understanding of science conflicts with your understanding of your religion, you either misunderstand the science or the religion, because they are not in conflict.

Can someone verify the numbers? (5, Funny)

ChronoReverse (858838) | about 9 months ago | (#45158803)

Are the numbers from this article just pulled out of a hat?

Re:Can someone verify the numbers? (0)

Anonymous Coward | about 9 months ago | (#45158873)

Yes.
(Trust me, but verify.)

Re:Can someone verify the numbers? (2)

WillAffleckUW (858324) | about 9 months ago | (#45159059)

Are the numbers from this article just pulled out of a hat?

But it was the very best kind of hat ...

(apologies to a certain British mathematician)

replication (2)

jennings (105869) | about 9 months ago | (#45158849)

No results that has not been replicated should be trusted. The problem is not that errors are made but rather that results are trusted before they are replicated.

Re:replication (2, Informative)

Anonymous Coward | about 9 months ago | (#45159071)

Exactly. As a long term Computer Scientist, everyone in my field (Computational Linguistics) knows that published results are only good if they sound plausible and can be reproduced. This boils down to citations. Bad papers (on average) tend to have low citation counts.

Re:replication (0)

Anonymous Coward | about 9 months ago | (#45159449)

I'm a grad student in natural language processing and my sense is that you have a mixture of people who don't understand math and those who don't understand language each trying to do both - the result isn't pretty. Even the good conferences accept a ton of rushed junk science.

0_o (0)

Anonymous Coward | about 9 months ago | (#45158853)

Oh crap. Don't let conservatives in the U.S. see this...

I see what's coming (1)

TheloniousToady (3343045) | about 9 months ago | (#45158861)

Next, you're gonna tell me you can't reproduce my cold fusion results. So, whose fault is that?...

There more being wasted (2)

Arduenn (2908841) | about 9 months ago | (#45158871)

In science, a lot of time and resources are wasted on testing ideas that other scientists had tested before, but who never published the results, simply because they weren't spectacular enough. Even though some negative results are obfuscated in papers reporting findings 'more worthwile' reporting, and even though some of the negative results are discussed at science meetings, many scientists have been trying to reinvent the wheel. Now how much is wasted? I don't know, but the result could be shocking.

Re:There more being wasted (1)

bob_super (3391281) | about 9 months ago | (#45158989)

10 years of my friend's failed PhD.
Essentially proved that something just couldn't be done with known tweaks on the usual biotech methods, but the committee decided that it wasn't valid to publish the failure to do something.

So some other poor guy out there is probably again going to waste 3/5/7/10 years to get the same results.

Repeat ad infinitum.

Science should be abandoned. (0)

Anonymous Coward | about 9 months ago | (#45158877)

I have been unable to personally reproduce Usain Bolt's result of 9.63 seconds for the 100m dash. Dr. Bazarov recommends this data should be rejected.

Re:Science should be abandoned. (2)

phantomfive (622387) | about 9 months ago | (#45159135)

Now you're talking about history, not science.

When things go really wrong... (1)

Anonymous Coward | about 9 months ago | (#45158883)

is when people (especially the media) assume that published studies represent the truth rather than something that deserves more investigation.

"trust but verify" (1)

roc97007 (608802) | about 9 months ago | (#45158885)

"trust but verify" is a good idea in many areas -- relationships, law, security -- not just science. But it's especially important in areas where published results establish precedent and serves as the basis of new results. Else we end up with baggage that hampers future efforts. It's not just a matter of saying "oops, those results are invalid", we also have to ask "ok, what other research has those results affected, and how does invalidation change things?"

All scientific conclusions should be questioned... (-1, Troll)

Anonymous Coward | about 9 months ago | (#45158899)

Except those from Climate scientist... All of those conclusions should be taken on faith!!!

Re:All scientific conclusions should be questioned (1)

riverat1 (1048260) | about 9 months ago | (#45159243)

You know, if you don't have enough scientific knowledge to even begin to understand what the scientists are talking about how can you do anything other than take it on faith that they know their stuff?

Re:All scientific conclusions should be questioned (-1)

Anonymous Coward | about 9 months ago | (#45159393)

You know, if you don't have enough scientific knowledge to even begin to understand what the scientists are talking about how can you do anything other than take it on faith that they know their stuff?

My doctorate is in physics... I am CERTAIN that my scientific credentials are on solid ground... Climate science is just a bunch of BULL SHIT!!!

For you Incompetent scientist want to bees.... here is a simple test to understand how CO2 is NOT the threat it is made out to be.
First when analyzing a system it i typical to determine the important contributions and the negligible contributions to a system and then neglect the negligible to simplify the problem.

So a simple experiment... Record the temperature at sunset and at sunrise. Make a note at to the amount of cloud cover there is during the night. What you will find is that clear nights get colder than cloudy nights... That is water acting as a green house gas. Now, do the same experiment right under the smoke stack of a coal fired power plant. What you will notice is that there is a significant temperature change related to the amount of cloud covert and NO temperature change related to the CO2 emission. That means the contribution from water as a green house gas is much greater than that from CO2 and therefore, CO2 can be neglected from consideration. Yep, you can verify that water is a green house gas but even when in a location where CO2 concentrations are maximized its effect as a green house gas cant be measured...

If your own verifiable experimental evidence doesn't convince you that this climate nonsense is just that nonsense then you are an idiot!!

Re:All scientific conclusions should be questioned (2, Funny)

Anonymous Coward | about 9 months ago | (#45159245)

Except those from Climate scientist... All of those conclusions should be taken on faith!!!

When I read this comment, I was laughing until I realized that it was modded down to -1. Clearly, you struck a nerve here at Slashdot.

Let it be know: On Slashdot, YOU SHALL NOT QUESTION THE FAITH OF CLIMATE CHANGE!!! So it is written so shall it be!!!

As always (0)

lesincompetent (2836253) | about 9 months ago | (#45158903)

Belief in something is, again, detrimental to mankind. News at eleven.

Trust?! (1)

deathcloset (626704) | about 9 months ago | (#45158907)

That's faith man.

How about actual proof?

Seriously, how do these studies get published? I mean this "trust" thing goes both ways and is probably why some really cool results don't get published - like the famous http://en.wikipedia.org/wiki/Belousov%E2%80%93Zhabotinsky_reaction [wikipedia.org]

From the wikipedia: "Belousov made two attempts to publish his finding, but was rejected on the grounds that he could not explain his results to the satisfaction of the editors of the journals to which he submitted his results."

What the hell? Who needs to explain results when one can just perform the experiment!?

Trust? That's for friendships and financial dealings.

Proof (or disproof as you fancy). That's for science and knowledge.

Alright, now someone enlighten me and fix my apparently skewed view on this matter please because I don't get how this crap is happening.

Re:Trust?! (1)

John Allsup (987) | about 9 months ago | (#45159185)

Imagine your the referee for a paper on an experiment at CERN.  Either you work at CERN and are not independent, or else you do not and do not have access to the resources to replicate the experiment.

Re:Trust?! (1)

deathcloset (626704) | about 9 months ago | (#45159249)

That's a very good point. I'm sure you see where I'm coming from though - after all, the results of any experiement support or oppose the results of others. If a collider result opposes a non-collider experiement then the non-collider experiement can be modified in light of the collider results. Sure that would a bit inductive and fuzzy, but it's better than just trusting. Regardless, your argument is strong and illustrates the fundamental challenges of cutting-edge discovery and research. Thank you for your succinct answer.

This is a real problem and conflict of interest (5, Insightful)

shaitand (626655) | about 9 months ago | (#45158915)

All researchers in the sciences have a motivation to be published, in the form of recognition, academic progress, and financial motivation. Not many of them have an incentive stop working on looking great for producing results and check the work of someone else.

Re:This is a real problem and conflict of interest (0)

Anonymous Coward | about 9 months ago | (#45159089)

Exactly! Every researcher in US which relies on grants is lying when he/she declares "No conflict of interest" when submitting a paper to a journal. If you don't publish, there is no way that you are going to get funded. And the pressure to get funded is enormous: employment, tenure and salary directly depend on the so called "independent funding".

Re:This is a real problem and conflict of interest (2)

Alef (605149) | about 9 months ago | (#45159273)

But you gain recognition and get published if you prove someone else wrong. And your academic progress is hampered if someone shows your results to be flawed. I think you are ignoring the competitive element.

That said, there is a problem with the current trend of grants being based strongly on the number of published papers, as it waters down the content of each paper and gets in the way of basic, long term research where there is no guarantee for "quarterly research results".

arrogance (0)

slick7 (1703596) | about 9 months ago | (#45158921)

The arrogant like Edison who play plug and chug until they come up with something gain noteriety while true genius, like Tesla are pushed to the backwaters of anonymity or forever locked up in the prison of national security. People like Wilhelm Reich or Royal R. Rife were ruined by the mediocre. Also Pons and Fleischman. And even the unknown inventors, like Stanley Meyers are killed when they cannot be bought off. What ever happened to those that were bought off?

Money (4, Insightful)

jasnw (1913892) | about 9 months ago | (#45158935)

Assuming TFA's numbers are correct, I'd bet that much of the problem is that no agency, be it government or commercial (and particularly commercial) wants to spend it's money seeing if published results are reproducible. Additionally, no one ever won a Noble Prize for excellence in reproducing others' results. Verification of results is key to science, but this is one of several aspects of doing science right that the funding agencies either don't want to, or can't (as in Congress looking over the shoulders of managers at the NSF), pay for. Everyone wants "everything, all the time" without paying for it, and this is the sort of thing that happens when decisions are driven by the money people (who may be scientists, to be fair) and not the people who know what the hell is going on.

Re:Money (5, Interesting)

blueg3 (192743) | about 9 months ago | (#45159179)

This is spot on.

It may be true that we spend too much time doing the initial work and not replicating results. That's not what the article shows, though.

It conflates the reproducibility rate of publications with some idea of "trust" versus "verification". There's no evidence (presented) that this means that scientists believe what is published. The author seems to think that papers should be verified before they're published, but that's not the point of scientific publication. The publication reports what the authors did and what their results are. It is nothing stronger: it does not represent (despite authors' bombastic claims) that what they found is actually hard scientific fact. That's only accepted (in theory) when those results are reproduced. Papers about reproducing the experiments are (in theory) also published, so that a critical scientist can evaluate the body of literature about how a hypothetical scientific fact has been tested. For this reason, the first publication of some new potential fact is naturally before anyone has verified it.

Without some evidence that paper results are being widely accepted into the "scientific canon" without verification, this is just an author being confused about science. That's a bit fair, though, because the press tends to focus on first publications (they're more interesting) and reports them as if they are fact. A scientist knows better, but the public at large generally does not. It's very disingenuous of the press -- but it sells.

In fact, the only evidence presented sounds like the process works just fine. A first publication of a new thing in biotech is a potential huge advancement and gold mine. Investors, scientists, and engineers all seem to know that the rate of the first publication actually being something as opposed to spurious is low, so the first thing they do apparently is try to verify it and make sure it's really a thing. That's pretty much what you want to happen.

This is because ..... (2)

prasadsurve (665770) | about 9 months ago | (#45158949)

People just cant accept that the fact that an experiment (which in they have invested time and money) can produce no conclusion even though from the point of view of science this is completely valid result. In today's goal oriented world, people want every experiment to mean something; to prove/disprove some hypothesis. And so they seem to find patterns in things which dont have any.

Re:This is because ..... (1)

HiThere (15173) | about 9 months ago | (#45159315)

More to the point, negative results have a very hard time getting published, even though they are nearly as valuable as positive results.

Also, replication of a result doesn't usually bring much in the way of kudos, even though it's an essential part of the scientific method. I won't say that a replicated result has a hard time getting published, but I will say it has a hard time of getting many inches of print.

Blame the patent system (1)

Opportunist (166417) | about 9 months ago | (#45158977)

If you can patent any kind of harebrained crap someone came up with in a pipe dream without having to provide a working model, that's what you get.

As soon as someone dreams up something that might kinda-sorta work in some sorta way, he will rush to the patent office. Should someone else finally come up with a working model, he'll rip the real inventor off with it.

And nobody is concerned with theoretical science? (1)

xyourfacekillerx (939258) | about 9 months ago | (#45158981)

You know, the field that verifies theories without experiments? How many experiment designs are published that conclude the only outcome of the experiment, hypothetically speaking, without real world results? Photons that split into alternate dimensions (carrying energy with them) and also send messages backwards in time, it's all on paper and considered fact, but no experiments have been done to prove it... does that now sound like craziness to anyone but me?

Re: And nobody is concerned with theoretical scien (2)

crdotson (224356) | about 9 months ago | (#45159041)

Many of those papers do lead to predictions which can be tested. Witness the Higgs particle prediction and later verification.

It's dangerous to make a broad statement, but in general I would say that if you don't have a verifiable/falsifiable theory, you're dealing with philosophy and not science. Both are valid endeavors, but we shouldn't mix them up.

slowth (1)

harvey the nerd (582806) | about 9 months ago | (#45158995)

The US spends too much time dithering on "proving" new discoveries and processes before taking useful, competitive actions. There is period between discovery and generally agreed development, before extensive verifications that used to be a tremendous competitive advantage for successful companies in the US. You're first, making billions with something cheaper, faster and better, while the competition's politico-bs "proovers" enjoy their sinecure 10-20-30 years. Now the proovers have everything stopped out in the economy. Enforcing excess verifications is one means that slower, technologically impaired companies steal from innovative individuals, either by forced co-option, "an offer you can't refuse," or bankruptcy. Grinding, pettifogging verification often needs to occur, but often later in the ramp up and production cycles.

Yeah, but it does depend on the area of science (5, Interesting)

Anonymous Coward | about 9 months ago | (#45159005)

A big part of the problem is null results. Getting and reporting a null result is SUPPOSED to be good science. And in a lot of areas of science, its OK - you'd obviously prefer to find something cool, but you don't kill your career by not finding something. But in some fields, if you do a study that doesn't find something, you can literally set your career back a decade or. Guess what this leads to?

I was talking to somebody was getting her PhD in Biochem. She was in the midst of a 5 year study on the effects of some drug. A condition to get her PhD was that she must publish a "substantial" peer review result. And her department had gone out of their way to define null results as not substantial. This meant that if her study found that the drug wasn't effective, she didn't get her PhD - she would have literally had to start over and had wasted 5 years of her life.

This is common in some areas of science, but not others.

So the first thing to do is get rid of garbage policies like this. My understanding is that its much more common in biology related fields, but that might just be my bias (I'm from a physics background, so I have an admitted bias here).

Until you fix this policies like this, you will always have people getting "creative" with their statistics or just outright making up data. For some reason, a lot of these biology related fields don't seem to care about policies like this, which I just don't understand. I mean, we know that these policies lead to bad behavior, but nothing is done to fix it. Maybe somebody in these areas can explain the rational to me.

Re:Yeah, but it does depend on the area of science (2, Funny)

John Allsup (987) | about 9 months ago | (#45159219)

That's why if you look through, say the TRIP database of medical studies, what you see is synopses of what look to be 'it really works, honest!' stories reminiscent of fake Amazon reviews, and little in the way of 'research downers' to make it look legit.  The kind of success rate you see in these research papers is like the poll outcomes you get in stuffed ballot box elections in the middle of some countries whose dictators don't quite understand how democracy is supposed to work.

An even worse mix: science and politics (0, Offtopic)

scottbomb (1290580) | about 9 months ago | (#45159013)

In other words, "global warming", now called "climate change". Junk science and perhaps the greatest hoax ever, complete with expensive policies pushed on us by greedy politicians.

Re:An even worse mix: science and politics (1, Informative)

riverat1 (1048260) | about 9 months ago | (#45159277)

The scientists are the ones who brought politics into it.

Re:An even worse mix: science and politics (3, Informative)

riverat1 (1048260) | about 9 months ago | (#45159311)

All my mod points for the ability to edit. Of course what I meant was:

The scientists are not the ones who brought politics into it.

Lord Forgive me, but (5, Interesting)

Ol Olsoc (1175323) | about 9 months ago | (#45159033)

I read TFA.

Really everyone should. Because while some of the points are good, The Economist misses the biggest one of all

In the University environment of today, the scientists and researchers are hamstrung by Non-Disclosure agreements. How does one share experimental information when to do so will cause you and your University great problems? One of the biggest offenders is the Biotech industry. Talk to someone, lose your funding and probably your job.

This is just the culmination of the past several decades shift from Government sponsored research to industry dominated research. It's a completely understandable position - industry wants return on it's investment, and research that doesn't generate profit might be good research, might be groundbreaking, but to the industry sponsoring the research it is a failure if they don't profit from it.

I'm pretty certain that industry would consider completely flawed and incorrect research as successful if it generated money for the company sponsoring the research.

So they draw the conclusion that scientists are lazy. I draw the conclusion that this is what happens when making money is the most important factor, and the scientists are bound by their contracts.

Re: Lord Forgive me, but (0)

Alex Cane (3296683) | about 9 months ago | (#45159363)

Surely, the real problem is- are we limiting science to those who do it as a job, professionally, for money? Exibiit A: Einstein, whose work was done alongside a full-time job and hardly published in peer-reviewed journals. Is it the professionalisation of science that is tarnishing the 'brand'?

Re: Lord Forgive me, but (1)

LurkerXXX (667952) | about 9 months ago | (#45159501)

Einstein was a theoretical physicist. That's kind of a small group. For the vast, vast majority of science, you need equipment, supplies, reagents, etc. That can cost from many thousands to millions.

Academics are Cowards (0)

Anonymous Coward | about 9 months ago | (#45159037)

Academics are just looking for grant money from the government or private foundations or trying to get tenure. That's all they care about. Whether they produce anything useful is completely secondary. That means that they want to avoid doing anything politically controversial or that steps on anyone's toes. Having real verifiable research is secondary to all that.

A simple idea underpins science? (3, Insightful)

fisted (2295862) | about 9 months ago | (#45159043)

trust, but verify?

We must be talking about different sorts of science, because from what i know, the simple idea rather is
"be objective, and be sure to keep it falsifiable

Re:A simple idea underpins science? (1)

HiThere (15173) | about 9 months ago | (#45159379)

You are talking about different parts of the process. The person performing the experiment needs to be objective and keep it falsifiable. The person reading a report of the experiment should "trust but verify". (I'm not real sure about that "trust" word in there. It sort of depends on how unreasonable the finding is, and what the source is that claims it.. And those that trust least will be most inclined to verify.) Remember, though, that we are talking about a population of scientists (i.e., all those who read the article making a particular report), not a particular individual scientist. If everybody had to replicate every finding from scratch it would be really difficult to make any progress.

Journals don't encourage followup articles (2)

WillAffleckUW (858324) | about 9 months ago | (#45159049)

One of the key problems is that doing the replicative experiments and publishing that data, as well as any divergence from the initial data of the first study by another author, is hard to get published in peer-reviewed scientific journals, as are replicative studies where the results do not concur with the original study.

It's also hard to get funding for this.

Fix that and you fix the observed problems, which mostly crop up in certain scientific cultures that tend not to encourage juniors from challenging senior scientists. Cases in point: Asian cultures for the most part, particularly South Korea and China, where we find a lot of problems with such scientific studies.

(this is my personal observation and has not been peer reviewed nor scientifically tested)

This would suggest ... (1)

Evil Pete (73279) | about 9 months ago | (#45159061)

That there is a name to be made in debunking landmark studies in the biosciences because they cannot be reproduced. That should be part of the scientific process.

Feynman said something similar (4, Informative)

phantomfive (622387) | about 9 months ago | (#45159077)

Richard Feynman pointed out something similar in his Cargo Cult speech [columbia.edu] . Here is an excerpt:

When I was at Cornell, I often talked to the people in the psychology department. One of the students told me she wanted to do an experiment that went something like this--it had been found by others that under certain circumstances, X, rats did something, A. She was curious as to whether, if she changed the circumstances to Y, they would still do A. So her proposal was to do the experiment under circumstances Y and see if they still did A.

I explained to her that it was necessary first to repeat in her laboratory the experiment of the other person--to do it under condition X to see if she could also get result A, and then change to Y and see if A changed. Then she would know the the real difference was the thing she thought she had under control.

She was very delighted with this new idea, and went to her professor. And his reply was, no, you cannot do that, because the experiment has already been done and you would be wasting time. This was in about 1947 or so, and it seems to have been the general policy then to not try to repeat psychological experiments, but only to change the conditions and see what happened.

Nowadays, there's a certain danger of the same thing happening, even in the famous field of physics.

Re:Feynman said something similar (0)

Anonymous Coward | about 9 months ago | (#45159417)

I remember the day when people thought mass created gravity, but it only influences it.

Funding bodies and presure for progress (1)

John Allsup (987) | about 9 months ago | (#45159103)

Too much funding is too dependent on satisfying funding bodies' requirements for the quantity of published research.  Failure to publish jeopardises one's research career.  But funding bodies do not have the resources or expertise to verify the correctness of publication, and nor do the referees.  It doesn't take a rocket scientists to work out what happens next.

The other issue with much of modern science (2)

John Allsup (987) | about 9 months ago | (#45159143)

Is that many experimental results are too resource intensive for most people to replicate.  Just try replicating the findings that support the existence of Higg's Boson.

Re:The other issue with much of modern science (1)

John Allsup (987) | about 9 months ago | (#45159153)

And for that matter, a mathematical result whose informal proof in a published paper relies on ten other informal proofs in ten other papers, who in turn rely on others, etc. until to make sure you have to check through tens of thousands of pages of informal proofs and verify that everything can be made logically rigorous on suitable foundations.

No... (1)

abroadwin (1273704) | about 9 months ago | (#45159167)

This is not science going wrong, this is wrongly calling something science.

This undermines all "science" (1)

AmazinglySmooth (1668735) | about 9 months ago | (#45159333)

When results cannot be trusted, all "science" loses credibility.

Oxymoron (0)

Anonymous Coward | about 9 months ago | (#45159367)

"Trust but verify." I hate that phrase. It's used all over the place, but it is meaningless. In any situation where verification is appropriate, trust is not. I guess people just don't know what "trust" means any more.

Modern Science (1)

EmagGeek (574360) | about 9 months ago | (#45159467)

No longer "trust, but verify." More like "correlate, and blame."

Science in real life vs how people think of it (1)

Anonymous Coward | about 9 months ago | (#45159499)

People think science goes something like this: Think of an idea, see if that idea works, if it doesn't, try again a different way or give up. If it works, perform more test runs and then send it off to a practical science lab where they figure out how to use it practically and patent the shit out of it. Once that's done, the product goes through regulation channels and then it gets released to the public.

What actually happens: Ideas get tested, if it works it gets patented. If it doesn't, it gets patented. Once enough tests are run, they go through regulation channels, but only if they bribed the right people. Once that's done, and they covered their tracks so that they can't be sued by anyone, they release the product half-tested and let humans be guinea pigs. When people are affected by it, they try to sue but can't because there's a law that prevents them from suing and science gets richer and richer. Any employee that opposes this is hunted down and anyone leaking any information about how politics dictates science gets sent to prison.

Further trials are conducted by foreign agencies and they make claims that it's more harmful than helpful and the local corporation silences them one way or another. Anyone that attempts at making a legitimate version on whatever they were working on will get sued and the government will back them up on it. This is basically modern science in a nutshell.

Load More Comments
Slashdot Login

Need an Account?

Forgot your password?
or Connect with...

Don't worry, we never post anything without your permission.

Submission Text Formatting Tips

We support a small subset of HTML, namely these tags:

  • b
  • i
  • p
  • br
  • a
  • ol
  • ul
  • li
  • dl
  • dt
  • dd
  • em
  • strong
  • tt
  • blockquote
  • div
  • quote
  • ecode

"ecode" can be used for code snippets, for example:

<ecode>    while(1) { do_something(); } </ecode>