×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

Positive Bias Could Erode Public Trust In Science

Soulskill posted about 2 years ago | from the but-that-doesn't-sound-positive-at-all dept.

Science 408

ananyo writes "Evidence is mounting that research is riddled with positive bias. Left unchecked, the problem could erode public trust, argues Dan Sarewitz, a science policy expert, in a comment piece in Nature. The piece cites a number of findings, including a 2005 paper by John Ioannidis that was one of the first to bring the problem to light ('Why Most Published Research Findings Are False'). More recently, researchers at Amgen were able to confirm the results of only six of 53 'landmark studies' in preclinical cancer research (interesting comments on publishing methodology). While the problem has been most evident in biomedical research, Sarewitz argues that systematic error is now prevalent in 'any field that seeks to predict the behavior of complex systems — economics, ecology, environmental science, epidemiology and so on.' 'Nothing will corrode public trust more than a creeping awareness that scientists are unable to live up to the standards that they have set for themselves,' he adds. Do Slashdot readers perceive positive bias to be a problem? And if so, what practical steps can be taken to put things right?"

cancel ×
This is a preview of your comment

No Comment Title Entered

Anonymous Coward 1 minute ago

No Comment Entered

408 comments

Feelings are more important than science (5, Insightful)

Gothmolly (148874) | about 2 years ago | (#39965477)

Right? isn't that what American schools and TV have been teaching for the last 30 years? Nerds aren't cool - facts are open to interpretation - everyone is special - you can eat more than you grow... When you have a society rewarding irrationality, what do you expect? Rigorous science?

Re:Feelings are more important than science (-1)

Anonymous Coward | about 2 years ago | (#39965595)

Nerds aren't cool - facts are open to interpretation - everyone is special - you can eat more than you grow...

I do eat more than I grow. There's nothing controversial about that.

Re:Feelings are more important than science (-1, Flamebait)

Muramas95 (2459776) | about 2 years ago | (#39965641)

We need to go back to survival of the fittest.

Re:Feelings are more important than science (3, Insightful)

r1348 (2567295) | about 2 years ago | (#39965689)

We still are, it's just the definition of "fit" has shifted since when we started creating the environment we live in.

Also, I personally don't miss the "good old times when we were all starving".

Re:Feelings are more important than science (3, Interesting)

Sique (173459) | about 2 years ago | (#39965987)

About every living creates the environment it lives in. Some in very obvious ways (ants, bees, beavers, moles...), some pretty subtle (grazing animals of the open plains tend to hinder the growing of shrubs and trees and thus keep the plain open). Same goes for plants, which change the immediate environment in a way to hinder concurrent species by shadowing the ground, changing water levels and chemical properties of the soil, and fend off enemies.

Re:Feelings are more important than science (5, Insightful)

AwesomeMcgee (2437070) | about 2 years ago | (#39965771)

I wonder how many of these "positive bias" results come from the fact that if you publish results that disagree with the bias of those who are paying for the study, they'll probably ensure it's never published and you'll find yourself no longer running studies on their dollars.

In the tech industry we all deal with non-technical managers who drive the technical direction and often times define the message to the clients. Does science suffer the same unskilled managerial types pushing scientists to interpret results in a particular way perhaps?

I have a hard time believing a professional scientist doesn't know how to apply the scientific method, but then again incompetence is rampant in every other industry I guess, why not the scientific one..

Science comes when results are confirmed (5, Insightful)

Geoffrey.landis (926948) | about 2 years ago | (#39966075)

Actually, science is stll working; the real trouble comes with the publicity of the science.

You should never believe the results of any single study. Every scientist knows this; or should know this. Science comes when results are confirmed, not when somebody publishes the first paper. The real work of science just starts when somebody publishes a study saying "we show that x has the effect y." That initial paper really is no more than "here's a place to start looking." However, newspapers want to publish news, and they need to publish whatever's hot and interesting and being done today, not "well, scientist z had his team take a look at the xy phenomenon to see if there was anything interesting there, and they couldn't really find anything there, although maybe some other research lab might have different results."

And, I suppose that somebody should post a link to the obligatory xkcd: http://xkcd.com/882/ [xkcd.com]

Re:Feelings are more important than science (0)

Anonymous Coward | about 2 years ago | (#39965803)

Right? isn't that what American schools and TV have been teaching for the last 30 years? Nerds aren't cool - facts are open to interpretation - everyone is special - you can eat more than you grow... When you have a society rewarding irrationality, what do you expect? Rigorous science?

Yes, yes I do

Re:Feelings are more important than science (4, Interesting)

SirGarlon (845873) | about 2 years ago | (#39965911)

This is a dilemma that is really goes to the heart of the philosophy of government. If the majority is irrational, is it better to give them self-determination and accept they will make frequent bad decision, or have the enlightened few rule them and impose better-informed decisions upon them?

Hint: there is no correct answer. I am not an historian but as far as I know this debate between a pure democracy and some form of republic goes back to Rome and Greece.

What is kind of weird is that the two major parties in America have developed into philosophies that are kind of opposite their names: the Democrats favor the paternalistic nanny state governed by the enlightened few (what I would call a "republic"), and the Republicans favor the ignorant mob ("democracy").

As an aside, when America was a young nation many of her leaders advocated public education as a way to narrow the gap between the elite and the general population. That does not seem to be working out real well, though.

Re:Feelings are more important than science (3, Insightful)

ArcherB (796902) | about 2 years ago | (#39965925)

Right? isn't that what American schools and TV have been teaching for the last 30 years? Nerds aren't cool - facts are open to interpretation - everyone is special - you can eat more than you grow... When you have a society rewarding irrationality, what do you expect? Rigorous science?

Considering that I'm done growing, if I didn't eat more than I grow, I'd die of starvation.

Phrasing (0)

Anonymous Coward | about 2 years ago | (#39965483)

Well you could have phrased that friendlier. Maybe positive bias only needs a little spin to leave the bad rep. behind?

Slashdot is pants (-1)

Anonymous Coward | about 2 years ago | (#39965491)

Sharpie in pooper. Shoe on head.

Wow! I guess Science HAS become a religion (0, Insightful)

Anonymous Coward | about 2 years ago | (#39965511)

Nuff' said

Re:Wow! I guess Science HAS become a religion (5, Insightful)

AwesomeMcgee (2437070) | about 2 years ago | (#39965811)

Science is a method dingbat, anyone who puts faith in a scientist however is practicing demagoguery.

Practice science, not demagoguery.

Re:Wow! I guess Science HAS become a religion (2)

samjam (256347) | about 2 years ago | (#39965913)

I've not seen it put more concisely than that yet - well done.

In relation to your second line: Practice of science is more engineering that science -
In theory there is no difference between theory and practice, but in practice there is.

Re:Wow! I guess Science HAS become a religion (4, Interesting)

BobMcD (601576) | about 2 years ago | (#39966141)

Saying 'science is a method' is like saying 'Christ was a Jew'. True, but it doesn't change what happened.

Science was an idea designed to seek empirical truth. To find things in such a way that those who followed after could find them again. Then people got a hold of it and started using it as a means to control one another.

Christ (even from the atheist point of view, so bear with me) had a simple message of love being service to your fellow man. Then people got a hold of it and we get monstrosities like the Crusades.

That's where the 'HAS become' part of the above phrase kicks in...

Obvious Complex System (5, Interesting)

Anonymous Coward | about 2 years ago | (#39965525)

Positive Bias is another word for Group Think. I guess it could also mean deception [bishop-hill.net]

There types of articles are moronic. (4, Interesting)

atlasdropperofworlds (888683) | about 2 years ago | (#39965539)

There are "studies", and then there is observation, modelling, prediction, model testing which is this thing called science. "Studies" are bullshit. Scientific research functions as it should. I believe the OP's article is just a chunck of sensationalist BS, or utterly ignorant of what science is (and is not).

Re:There types of articles are moronic. (3, Insightful)

jeffmeden (135043) | about 2 years ago | (#39965569)

There are "studies", and then there is observation, modelling, prediction, model testing which is this thing called science. "Studies" are bullshit. Scientific research functions as it should. I believe the OP's article is just a chunck of sensationalist BS, or utterly ignorant of what science is (and is not).

You forgot to mention that it is yet another piece of published work that suffers from positive bias...

...and inaccuracies (1)

Roger W Moore (538166) | about 2 years ago | (#39966035)

You forgot to mention that it is yet another piece of published work that suffers from positive bias...

Nevermind that what about the inaccuracies? Economics is not a science and 'biomedical research' (where the article claims is where the biggest problems lie) and epidemiology are medicine. Perhaps the solution is to educate people as to what science is because it is not defined as any subject which publishes research papers containing numbers and/or long sounding names.

Re:There types of articles are moronic. (4, Interesting)

fearofcarpet (654438) | about 2 years ago | (#39966007)

There are "studies", and then there is observation, modelling, prediction, model testing which is this thing called science. "Studies" are bullshit. Scientific research functions as it should. I believe the OP's article is just a chunck of sensationalist BS, or utterly ignorant of what science is (and is not).

That is not really what TFA is talking about. Daniel Sarewitz is re-phrasing a long-known problem with "studies," as you call them, which is that complex systems are--by definition--too complex to study as a whole. I am a physical scientist, which means that I typically make or measure something in a well-controlled experiment and then change variables in order to test a hypothesis. I can basically publish a paper that says "we tried really, really hard to find it, but it wasn't there." In the life sciences, they are trying to answer vague cause-effect questions like "does this drug affect a particular type of tumor more than a placebo." Thus researchers in those fields have to create models in which they can control variables. He gives the example of mouse models, which are obviously imperfect models for human physiology. How imperfect is the question. The creeping phenomenon that he is addressing is the tendency to relax the standards for what counts as positive evidence--and I'm grossly oversimplifying--by waving your hands around about how mouse models are imperfect, but that there is definitely "a statistically significant trend." The root cause is simply the ridiculous amount of pressure that life science researchers are under to publish, which requires results, because their methodology is standardized. Those poor bastards can spend eight years on a PhD project that goes nowhere or burn four years of their tenure clock figuring out that their experimental design was flawed. *Poof* no funding, no tenure, no degree, time to consider a new career. That sort of potential downside creates the sort of forced-optimism that TFA describes.

Re:There types of articles are moronic. (1)

ceoyoyo (59147) | about 2 years ago | (#39966019)

We generally call observation, modelling, prediction and model testing "studies."

The problem is that a lot of the observation is flawed, the modelling based on that observation may be flawed, the predictions are unreliable and the model testing is insufficient.

The basic problem is not any kind of bias, it's that the majority of working scientists don't know how to do adequate stats. It's quite simple to fix. Most scientists could probably learn the majority of what they need to know in an afternoon seminar focusing on the actual problem.

Re:There types of articles are moronic. (2)

Will.Woodhull (1038600) | about 2 years ago | (#39966137)

Both studies and science are affected by bias inducing forces, such as 'publish or perish' policies of institutions and grant availibilty from stakeholders with an economic interest in the results. Elsevier, et al, raise similar problems with their control of the distribution of knowledge pipelines.

About a hundred years ago traffic problems became so bad that government had to step in and legislate which side of the road drivers had to use, and who had right of way at intersections. It might be that similar legislation is needed to curb the biasing influences on research today.

It is a mistake to think that this is about the science itself. This is about controlling the environment in which the science is done so that we benefit from good science rather than science steered by whoever has the deep pockets to pay for it. The laboratory needs to be isolated from outside economic and political influences, and those influences have found too many ways to get around the barriers that used to work, sort of, in a kind of "it is insulated enough that sometimes good work can be done" way.

Wait, what? (2, Insightful)

bmo (77928) | about 2 years ago | (#39965553)

'Nothing will corrode public trust more than a creeping awareness that scientists are unable to live up to the standards that they have set for themselves,' he adds.

No, the corrosion of public trust is the incessant idiocy coming from Fox and other Murdoch properties exclaiming "oh those silly scientists got it wrong again!" when the story is about a refinement of a model or something.

Scientists are losing the credibility war because scientists are not PR flacks and are unable to counteract the "we don't have to report actual news, we got a court order saying we don't" assholes at Fox.

There is a concerted effort to discredit scientific research no matter what it is.

--
BMO

Re:Wait, what? (-1)

Anonymous Coward | about 2 years ago | (#39965619)

angry much?

Re:Wait, what? (4, Insightful)

jo_ham (604554) | about 2 years ago | (#39965857)

angry much?

Damn right scientists are angry. The pervasive anti-intellectualism and well-funded attacks on science to suit political ends are getting ridiculous. Positive bias in peer review is a problem that needs to be addressed, but it is all part of science - studies that cannot be replicated are examined in detail and their models either rejected or refined to be tested again. Just because something is published does not make it "the truth" - it means it is a set of conclusions based on experiments that were done by a particular team of scientists. If it's not repeatable then future publications will say so - that's how the back and forth and refining of models happens. Things don't get held back until the issue is "settled" then get published, it simply doesn't work like that.

Where things start to fall down (and where the anti-science folk with an agenda have such an easy time) is that it can be hard for the layman to sort out what to trust in a scientific publication, since the well-hammered, repeated-by-many-groups stuff is sitting alongside single publications claiming XYZ based on PQR from a single data set in a single research group. It's so easy to wade in there and say "look at this! see! scientists can't agree! it's all a big con to get more money! they'll say anything to further their agenda!".

Again, I'm not dismissing the problems of positive bias - it's a factor of peer review and the way human beings approach the reporting of science, but the job is made much harder by a barely-science-literate blogosphere working full steam to discredit anything they can to make it look like there's some sort of global conspiracy of scientists working against the general public in whatever field the propaganda machines are working against. Climate change is clearly the biggest one at the moment, but it's not restricted to that. There's almost anything related to things that challenge fossil fuels and energy research towards energy independence, then there's anything in the biology field relating to stem cells, vaccinations, pharmaceuticals, etc.

The rise of public opinion that science is somehow something to be regarded with great suspicion and that scientists are actively working against the public good is not only troubling, it's highly counterproductive. It's time we started getting angry about it, it's just almost impossible to fight back - scientists are not equipped to do so against a highly organised and well funded group of individuals who do that sort of character assassination and propaganda for a living.

Re:Wait, what? (-1, Offtopic)

Anonymous Coward | about 2 years ago | (#39965627)

Will you fucking quit putting your username in your sig when everyone can blatantly see your username at the top of your post? You're not that important! Damn!

Re:Wait, what? (0)

Anonymous Coward | about 2 years ago | (#39965643)

There is a concerted effort to discredit scientific research no matter what it is.

--

Now now, you know that's not true. Any scientific research that can be distorted and manipulated to suit Fox and other right-wing agendas is given entirely too much credit.

Like say, if there was some badly conducted study showing that white men were more intelligent than white women, black men, Asian women. They'd try to prove they were smarter than Asian men or even Jewish Men, but they'd rather just exploit prejudices against them. Still, such a study will be trumpeted as if it validated every prejudice they had and made to show why they need to discriminate more!

And they're probably still looking for "scientific" evidence that gays are more likely to cause Global Warming.

Re:Wait, what? (1, Insightful)

SteelKidney (1964470) | about 2 years ago | (#39965655)

Or the hordes of bloggers, forum commenters, and internet shills claiming that "This is absolutely, unalterably correct and anybody entertaining even reasonable skepticism is an IDIOT".

Re:Wait, what? (1)

bmo (77928) | about 2 years ago | (#39965973)

You mean like the anti-vaxers?

Two years ago last February, the Lancet retracted "Doctor" Wakefield's "study" and you still see this shit.

--
BMO

Re:Wait, what? (3, Insightful)

Anonymous Coward | about 2 years ago | (#39965657)

Oh yes, blame it all on fox, the ultimate evil in the universe. It has nothing to do with studies being published making lavish claims which are then later proved false, or so wildly overblown that it's almost embarrassing. Of course, the conduct of the scientists themselves couldn't possibly be at fault and it must all be an even republican conspiracy.

Grow up, pull your head out of your ass, and realize that fox and republicans aren't the only source of evil in the world.

Re:Wait, what? (0)

Anonymous Coward | about 2 years ago | (#39966115)

Grow up, pull your head out of your ass, and realize that fox and republicans aren't the only source of evil in the world.

But they are.

Re:Wait, what? (1)

tp1024 (2409684) | about 2 years ago | (#39965667)

Not being an American, I don't know what FOX is putting out. But if they are pointing out that any arbitrary number of "X causes cancer", "Y prevents cancer" and "Z causes heart disease" studies are bullshit, then it is not merely their perfect legal right to do so, but they are also delivering an accurate describtion of such "research".

Just because somebody is a murderer and thus a bad man, doesn't mean he also picked your pocket, ran a red light and parked his car in front of your garage.

Re:Wait, what? (3, Informative)

AlecC (512609) | about 2 years ago | (#39965851)

The trouble is that most such "X causes cancer" statements come from the media themselves, recklessly shortening research results saying "consumption of X correlates with a positive increase in cancer", where the nature of the correlation is unknown and the increase is very small. So it is all to often not an accurate prediction of the research. Particularly, there can often be a chinese whispers effect, where the researcher publishes a paper, the University PR department publishes a precis edited for PR purposes, a popular science journal then reports with its own bias, and it is then taken by mainstream media and truncated again.

A particular example, as often reported b y Ben Goldacre, is the Daily Mail, which seems to summarise everything into causing or curing cancer, and will report the same substance on both sides within days.

Re:Wait, what? (0)

Anonymous Coward | about 2 years ago | (#39966011)

No they have been found to be deliberately lying, and that it is in fact part of their corporate policy. They have however not been convicted since they argued their channel "Fox News" is not a news network but a news-themed entertainment channel, so they don't have to report factual news. That is the loophole that allows them to have news-like programming that spreads obvious falsehoods.

Re:Wait, what? (4, Insightful)

gandhi_2 (1108023) | about 2 years ago | (#39965729)

This is a story about actual bias in scientists which is affecting the quality of their research.

And you completely gloss over that to take issue with Fox, which is no more ore less biased than msnbc, et al...
Look, people seek an echo chamber. "News" companies of all types just supply the demand.

I suppose YOU don't see a problem with some news organizations taking biased scientific output and unquestioningly running with it as though it were the concrete truth for ever more.

Re:Wait, what? (5, Insightful)

thegreatemu (1457577) | about 2 years ago | (#39965929)

While I agree that models are frequently refined, leading to new results, there is a disturbing trend I see, not having to do with positive bias necessarily, but with uncertainty estimation.

One thing that I've found incredibly hard to beat into undergrads taking my physics lab courses is that getting your uncertainties (or error bars) right is far more important than getting the right central value. This is because uncertainties are the only way that two experiments can be compared against each other, or the only way to compare experiment to theory. If I have two models of climate change, one of which predicts a temperature rise of 3 C ± 5% and another that predicts 4 C ± 7%, those results are in large disagreement, whereas two studies that predict 20 C ± 15% and 40 C plusmn 35% are in much closer agreement.

But I see it seems much more frequently, especially in fields like astronomy, too little thought goes into the systematic uncertainties, and you'll get 4 experiments measuring the same thing with results that cannot be reconciled if you take their statistics at face value. This was a huge problem with many of the early global warming predictions as well; every year a new estimation would come out that was completely incompatible with the previous one. Yes, these models are insanely complicated, and it's damn hard to understand all the systematics. And of course you can't put in error bars for plain old mistakes. But do it too many times, and people begin to lose any faith that your estimates can be relied on for anything.

This is the problem I see; not necessarily bias toward a positive result, but a bias toward underestimating the uncertainty of your measurement, which I suppose could be different sides of the same coin. (E.g., a result of 2 ± 0.1 is a positive result; a result of 2 ± 5 is not!).

Re:Wait, what? (1)

invid (163714) | about 2 years ago | (#39965945)

In order to understand what this bias means, people have to understand the scientific process, and many people don't. Such people would be more convinced by scientists pontificating on high that they are always right, the way religious leaders do.

Re:Wait, what? (0)

ArcherB (796902) | about 2 years ago | (#39966063)

'Nothing will corrode public trust more than a creeping awareness that scientists are unable to live up to the standards that they have set for themselves,' he adds.

No, the corrosion of public trust is the incessant idiocy coming from Fox and other Murdoch properties exclaiming "oh those silly scientists got it wrong again!" when the story is about a refinement of a model or something.

Scientists are losing the credibility war because scientists are not PR flacks and are unable to counteract the "we don't have to report actual news, we got a court order saying we don't" assholes at Fox.

There is a concerted effort to discredit scientific research no matter what it is.

--
BMO

God forbid you get more than one side of any story. You should thank God that FoxNews exists. They are the only news organization that actually reports on things that might make the current government look bad. For example, yesterday there was a report out about stuff Mitt Romney did in high school, 45 years ago. Can anyone tell me what grades Obama got in college 20 years ago? FoxNews is the only organization that even looks for these answers while the rest of the media is busy showing Obama swat at flies, say that he supports gay marriage (again), report the good news that gas prices have dropped a few cents (after going up a dollar), and reporting the winners of the latest "Dancing with the Stars".

The job of the press is to look for information, report what it finds, and act as a government watchdog. FoxNews is the only news organization doing that job. FoxNews is not Pravda. They'll actually report some of things you don't want reported. I'm sorry if you are mad that I'm getting information you don't want me to know, but I'd prefer to make up my own mind instead of being told what to believe without any counter argument.

Just because they report stuff you don't like does not mean that they should be silenced. There is that whole Bill of Rights thing that is supposed to protect them. Do you agree with the First Amendment or not?

data point (5, Insightful)

Anonymous Coward | about 2 years ago | (#39965555)

I received my PhD in physics, and the thesis was measuring a number, in which I measured zero within the error bar. Not particularly interesting, but valid science. My wife was in a PhD program in Biology, she also did valid science, novel measurement technique, came up with an uninteresting result, therefore was not able to publish, therefore was unable to graduate. It would have been extremely simple to fudge the result to a 2-3 sigma result 'hinting' at an interesting answer, which would have gotten published. I think certain sciences have gotten to a point where they have forgotten that if you do valid work in a novel way, then that is science and you should not be punished for the conclusion of the measurement. Most measurements you do of the natural world should probably end up being unsurprising, and thus uninteresting, but you don't graduate or get tenure with those kinds of results. I think this is the mechanism for the positive bias. That is why I do not take results from certain branches of science at face value.

Re:data point (4, Insightful)

spiffmastercow (1001386) | about 2 years ago | (#39965697)

This. Until we start giving PhDs for finding expected results and/or verifying the results of others, we're going to have this problem of research 'exageration'.

Re:data point (3, Interesting)

Hatta (162192) | about 2 years ago | (#39966077)

Which is exactly what we need to do. We need an entire independent arm of the sciences dedicated to confirming established results. This is something that needs to occur separate from the pressure to come up with novel results to please grant reviewers.

Re:data point (1)

jholyhead (2505574) | about 2 years ago | (#39965755)

It's valid science, but unless it conveys important information to other people, why would other people be interested in reading it in a journal?

No one remembers the 100s of ways Edison found not to make a lightbulb - even though they were each as important as his one successful attempt.

Re:data point (2, Insightful)

Anonymous Coward | about 2 years ago | (#39965877)

Original commenter here:

It shouldn't be in the highest tier journals, but should be able to be published somewhere so people can find the technique and that someone had done the measurement, with result x. And if the idea and methodology is sound, it shouldn't prevent someone from getting a PhD. But it does. And it filters out a lot of people who would choose to publish results that do not further their own career, as well as filtering in people willing to do things like playing with how they cut their data until the results start looking interesting. (it does not even require faking data, just someone willing to stop looking for ways to cut out 'bad' data when the results start looking interesting.)

Re:data point (3, Insightful)

AwesomeMcgee (2437070) | about 2 years ago | (#39965903)

If it's valid good science it speaks to the competency of the individual who practiced it, if we aren't graduating these folks, then we're encouraging graduation of incompetent or sensationalist scientists.

Re:data point (1)

jholyhead (2505574) | about 2 years ago | (#39965949)

Agreed.

The science should be of a publishable quality, but that doesn't always mean the result is publishable. AC's wife got screwed.

Re:data point (0)

Anonymous Coward | about 2 years ago | (#39965791)

Perhaps she could publish in JASNH: http://www.jasnh.com/

Re:data point (1)

Anonymous Coward | about 2 years ago | (#39965813)

It's also hard to publish failures, unless they are spectacular. If research doesn't pan out, most shelve it and move on, leaving the next poor bastard to go down the same dead end in the same dead way instead of avoiding it or trying it in a different way.

Re:data point (0)

Anonymous Coward | about 2 years ago | (#39965901)

So, how does a layperson like myself figure out what studies are valid? Is there a BS detector cheat sheet to help us figure out what we can dismiss? What are these "certain" branches?

Re:data point (0)

Anonymous Coward | about 2 years ago | (#39965933)

While I agree that negative results could receive more attention, your story is lacking two key elements.

First, was your wife involved in a single project for the entire duration of her PhD? That's tremendously risky, because it means that if the project delivers a negative result, then she has nothing to show for 5+ years of work. It's also strange, because I don't understand how someone could spend their entire PhD doing a single thing unless they had other obligations which didn't leave them time for a PhD (suggesting they shouldn't be doing one in the first place), or procrastinated a lot.

Second, where was your wife's advisor? An advisor's major responsibility is to make sure his/her students don't screw up by choosing ridiculously difficult topics (like a Millenium Prize problem in mathematics), things that show little promise (that other groups have tried and failed, and the student doesn't know because the result wasn't published) or (to a lesser degree) things that have little to no impact.

Thanks for that. (0)

Anonymous Coward | about 2 years ago | (#39966079)

In my early middle age, sometimes when looking back I wish I went into the natural sciences. My current business/tech career seams really meaningless most of the time and I think if I did something more "worthwhile" like research, then things would be better - you know that whole the grass is greener over the septic tank mind tricks.

When I see things like what you posted, I realize that things turned out OK. The World didn't lose a brilliant scientific mind, either - I honestly think I'd be a mediocre scientist at best. I had this dream briefly in college of being an experiemental physcist, limnologist or a botanist - I'm sooo glad I dodged that bullet.

It's just not your post either. I hear things from rank and file researchers about their own misgivings about their work, the profession, and the BS they have to deal with.

Of course it is a problem (4, Insightful)

tp1024 (2409684) | about 2 years ago | (#39965561)

The solution lies in a reformation of research finance that is not focussed on how many papers X published compared to Y, but also takes into account whether they are consequential or not and if they actually comply with at least basic scientific attributes such as repeatibility, verifiablity, falsifiability, accessibility of all data and all conducted research, as well as actually conducted verification of research by independent third parties.

There should also be an outright condemnation of data mining, where data bases are checked only for the existence of attributes and correlations that happen to affirm the researchers opinion and leave all others untouched.

Fields like economics, medicine and climate have long since deteriorated to mere cargo cults due to those failings.

Re:Of course it is a problem (0)

Anonymous Coward | about 2 years ago | (#39965685)

Medicine is a cargo cult, really? You're one of those guys that refuses to get his children vaccinated too?

Re:Of course it is a problem (1)

tp1024 (2409684) | about 2 years ago | (#39965815)

I'm one of those guys who is fed up with "studies" that "indicate" that X prevents Y just after another study showed that X causes Y. I'm one of those guys who read the news and knows that studies showing that some new pill is usually tested for effectiveness not against accepted treatments but merely against placebos.

Vaccination is well established and has a firm scientific grounding that can be replicated any number of ways, be it epidemiology, in-vivo or in-vitro experiments

Re:Of course it is a problem (0)

Anonymous Coward | about 2 years ago | (#39965867)

Medicine is a cargo cult, really? You're one of those guys that refuses to get his children vaccinated too?

Maybe s/he is just familiar with the increasing body of evidence that MOST published medical research is false.

http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1182327/

Or perhaps s/he is familiar with the rife corruption in basic research and financial pressure to confirm certain results, whether true or not, because those results will benefit certain biomedical companies.

Re:Of course it is a problem (0)

Anonymous Coward | about 2 years ago | (#39965759)

Each article and journal has an "impact factor", a metric that quantifies the number of times a source is referenced. This rewards individuals with solid work. Whether or not all institutions weight such things appropriately is another matter.

Re:Of course it is a problem (1)

codegen (103601) | about 2 years ago | (#39965843)

Impact factors are also misleading. They consider positive and negative references the exact same. They also don't measure industry impact.

Re: (3, Interesting)

Reibisch (1261448) | about 2 years ago | (#39965915)

And ignores the fact that once published, there's no reliable corrective mechanism to propagate those results down beyond a standard literature search. I'm posting as AC because quite a few years ago I published results that I believed at the time to be correct, but were shown to be wrong in a subsequent paper. Despite this, I'm *still* being cited in new papers while the paper that refuted mine is seldom cited. Science isn't some infallible field. We make mistakes; Sometimes those mistakes are accidental, sometimes they're sloppy, and yes, sometimes they're even intentional. That doesn't reduce the validity of science, but it requires us to be more vigilant.

Re:Of course it is a problem (1)

rwv (1636355) | about 2 years ago | (#39965783)

So would you have scientists publish fewer original research papers and more papers that attempt to reaffirm or disaffirm research that's been published by their peers? I'd be surprised if there isn't *some* resource that links research publications with a list of secondary papers that "Support the Same Conclusion" and "Refute the Conclusion". Given that scientists are looking to publish (churn out) a lot of papers... it seems low-hanging fruit would be studying and trying to repeat the Research/Conclusion of a peer.

On the other hand, I'm an engineer and not a scientist so I take a practical view of applying what's coming out of the scientific community... and the most practical view truly is ignoring 95% of it and letting the "really good stuff" percolate out.

Re:Of course it is a problem (2)

tp1024 (2409684) | about 2 years ago | (#39965885)

Even if there were such lists, they are not what research funding is based on.

What funding is based on is which journal published your article and how many citations it received in other papers - whether the author citing your paper has actually so much as read (much less made use of) your paper or not. Just being popular with peers helps, because you can always find some excuse or other to cite a paper even if it is semi-relevant at best.

Re:Of course it is a problem (1)

slimme (84675) | about 2 years ago | (#39965939)

Data mining is indeed a very mediocre scientific activity. Correlation on itself means nothing at all. If you want to proof something the correlation should be 100% and you should be able to explain why the correlation exists and replicate it in controlled experiments. The problem is that those slam dunk scientific discoveries are all or mostly allready found. And nowadays the poor scientists need to find something to bolster their path to glory.

Good science could be: find a correlation an proof the causality. But a lot of studies stop at the correlation. That's what fills newspapers nowadays. 'You get fat from diet coke since most people that drink diet coke are fat'.

Some scientist try to eliminate all other reasons and then decide that their causality is the only one that explains the correlation. But in effect they say: those things correlate and I 'the superintelligent scientist with multiple PhD's' cannot find another explanation and that is why my explanation must be true.

For background you should listen regularly to 'more or less: behind the stats' http://www.bbc.co.uk/programmes/b006qshd [bbc.co.uk]
You can listen to podcasts that are interesting and fun to listen to. And some of the older ones are absolutely great. They gave me great insight in the workings of media (and science).

The only downside is that if your girlfriend tells you something she heard on the radio and you answer her: correlation is not causality, she gets upset.

Re:Of course it is a problem (3, Interesting)

ceoyoyo (59147) | about 2 years ago | (#39966069)

"also takes into account whether they are consequential or not"

That makes the problem worse. We need to evaluate papers based on whether they are good science or not, and not publish the ones that are bad science. Currently the "negative results" which are actually inconclusive results, are not published because they are, well, inconclusive. Unfortunately a lot of the "positive" results are also inconclusive, but they ARE published. The solution is not to publish more "negative" results, it's to stop publishing the flawed "positive" ones.

Replication is king... (2)

Life2Short (593815) | about 2 years ago | (#39965563)

Some sort of redirection towards findings that can be verified by independent labs would seem to be an improvement on the current system. But that would require a focus non science as a system and less of the "great researcher" emphasis we see today.

Artifact of specialisation. Part of the problem. (1)

fleeped (1945926) | about 2 years ago | (#39965601)

The more specialised scientists become, the more difficult a proper and thorough peer review is to do. That's the 'innocent' side. And then, you have money and politics..

This isn't a new problem (0)

Anonymous Coward | about 2 years ago | (#39965665)

I thought positive bias was just one of those things part of the human brain that we just have to deal with. I mean, this isn't some kind of new thing. Look at all the miracles that have been attributed to God over the course of human history. I doubt that old research was any better. Cognitive science has just made us more aware of positive bias.

Sure it does ! (0)

Anonymous Coward | about 2 years ago | (#39965677)

If what is reported by scientist is wrong, regardless of whether it is too optimistic or too pessimistic, then the public has to be losing faith. IMOO, what contributed the most to that trend is the growing pressure put on scientist to obtain results. Positive ones. Not in 1 or 10 years, but now, otherwise you might be looking for another job pretty quickly.
What is the more puzzling to me being that the said pressure has been put on the scientists mainly by the public itself, so well, I guess you reap what you sow.

Positive bias is the wrong term. (5, Informative)

Karmashock (2415832) | about 2 years ago | (#39965681)

They're overstating precision. Which rather then a forgivable error is an elementary mistake no trained scientist should ever make.

A VERY basic concept they teach at the lowest level of science education is the distinction between accuracy and precision. This is science 101.

Accuracy is whether or not a given conclusion is correct.
Precision is to the degree of specificity.

Typically you run into problems on complex subjects because they overstate the precision of their data or their ability analyze the data.

This can boil down to simple thinks like significant digits.

For example, I'm measuring volume to two significant digits in a giant data set with thousands of measurements. When and if I average those numbers the final average can't have more then two significant digits. That sounds elementary but you see this error made on some big studies. You'll have a situation where something is being measured in a crude sense by many sources and then in the analysis a much higher degree of specificity is implied.

Often that degree of specificity is required to make certain conclusions which is why they break the rule. This is lazy and a breach of scientific ethics. What they need to do is collect the data all over again this time to the level of specificity they need.

Simply saying its too hard to collect the data properly so they're going to make assumptions is not reasonable or ethical. I suppose you could do it so long as you kept an asterisk next to the data and the findings to make it very clear throughout that the conclusion is a guess and not in any way empirical science since at some point people were guesstimating results.

Re:Positive bias is the wrong term. (0)

Anonymous Coward | about 2 years ago | (#39965875)

No, they're blatantly lying so that they can get publishable results. Sorry, seen it here. I really wish we had a peer reviewed journal of uninteresting results to prevent repetition of negative results and so that folks who do good science and find something uninteresting can, say, get thier PhD. We've been lying for decades now, and that's why industry is doing most of the interesting applied science now.

Re:Positive bias is the wrong term. (2)

medcalf (68293) | about 2 years ago | (#39966009)

That's one of the problems I have with global warming. The precision of the best made, best sites thermometers is about a degree. One common class is +/- 5 degrees. Yet these are being used to derive results in tenths of a degree. Satellite proxy measures help, but that's a very short record in climate-significant time frames. Is the Earth warming? Almost certainly. Is that outside the norm? Hell, we don't even know if it's outside the margin of error.

Re:Positive bias is the wrong term. (5, Informative)

ceoyoyo (59147) | about 2 years ago | (#39966139)

"When and if I average those numbers the final average can't have more then two significant digits."

Yes, the average can have more precision than the individual measurements. That's actually kind of the point of an average. It can't improve accuracy though, for the most common definitions of accuracy.

Your definitions of accuracy and precision are sort of right, but also sort of misleading. And the way you use specificity is incorrect. But as you correctly point out, a lot of working scientists are a bit fuzzy on all these concepts too.

No secret in academia (1)

crazyjj (2598719) | about 2 years ago | (#39965687)

I used to be in academia, and it was no secret that researchers almost always found the results that they had planned to find from the beginning in their studies. I don't think I ever once saw a case where a researcher started out with a hypothesis, found it was completely wrong through the research, and then let it go. There are a million ways to cook the numbers to reach the conclusion that you want to (and the one that gets you the grant money and publication). If a certain position is popular (and well funded), expect everyone to jump on it and start producing paper after paper confirming it.

Re:No secret in academia (1)

hey_popey (1285712) | about 2 years ago | (#39965845)

This seems quite complicated! The right way to do stuff is to first do it, and write your hypothesis afterwards according to your results. The same goes for engineering: you make the product, and then write the specification. Otherwise you would be wrong / request for deviation too often, and that would be depressing!

Apophenia (3, Insightful)

concealment (2447304) | about 2 years ago | (#39965725)

When you have a hammer, everything looks like a nail.

Anything can become a religion, as a result. We're less critical of our data when that happens, and we "nudge" it into place.

The problem is not "science" per se but our social approach to it.

Wrong Angle (1)

wisnoskij (1206448) | about 2 years ago | (#39965739)

I would say that the bias is not so much positive as it is "get noticed and get more grant money" biased.
Unfortunately, you get noticed more and get more grant money if you find what you were looking for instead of disproving your initial hypothesis.

Also this article seems to imply that the problem that needs to be fixed is that of public trust, while I would argue that the public should distrust a community that gets it wrong so often (it is the skeptical, scientific thing to do). Unfortunately, far too many people have the opinion that science is absolute truth and always right; but science is not an "exact science" so to speak, scientists overlook variables all the time and prove what they want to prove just as readily as everyone else.

The "science is settled" (-1)

Anonymous Coward | about 2 years ago | (#39965799)

Global warming: HOAX with a liberal political agenda
Evolution: HOAX with a liberal political agenda

This is why academics and their "advanced degrees" are considered MORONS by the rest of us. Scientists are just a bunch of dumbass liberals who think big words can hide their incompetence.

Re:The "science is settled" (1)

benjfowler (239527) | about 2 years ago | (#39965953)

I was gonna say, "Give it time before all the flat-Earthers showed up".

Slow news day on Drudge Report, huh?

I left academia and science because of this. (1, Insightful)

Anonymous Coward | about 2 years ago | (#39965805)

Academia is a Ponzi scheme built on fudged statistics, unreported failures and outright forged results.

Unfortunately, Publish or Perish and the overadmission of PhD candidates has resulted in a system where getting a paper out is more important than what goes into the paper.

Science is about money now (2)

hshana (657854) | about 2 years ago | (#39965821)

Like just about everything else in this world, science is about money. And how do you get money in science? By finding and/or hyping the next leap forward. Being successful in science is all about getting grants. You don't get tenure without bringing in grant money, you don't get grant money without publishing in the best journals, you don't publish in the best journals without finding the next leap. Your typical PhD finishes school in their late 20's, probably with significant school loan debt. He or she then gets a postdoc where they can barely afford to live in the city with the prestigious school that they think they need to further their career. At the same, it's probably time to think about starting a family (especially if you are a woman). And as a postdoc, the pressure to publish is even greater in a more compressed time frame. There is so much pressure there financially, emotionally and mentally, that is it is no wonder that some people cave and take shortcuts and fudge results. And then, if you do make it to a tenure track position, you don't do much science any more. Instead you spend all your time writing grants and churning through postdocs, who may or may not be fudging their results to get a recommendation to get a better position...

Positive bias in engineering research (2, Insightful)

l00sr (266426) | about 2 years ago | (#39965831)

In engineering research, there is definitely a positive bias; in fact, negative results are rarely published at all. This is both because negative results have less sex appeal than positive results and because peer reviewers are trained to outright reject publications without positive results. Although there is huge pressure to publish positive results, I'm not aware of systemic fraud in the literature. What does happen, however, is roughly this: 1) researcher gets great idea. 2) researcher tries idea. 3) idea fails to produce state-of-the-art results. 4) researcher adds hacks and kludges to marginally improve performance. 5) repeat steps 2-5. So, what you get in the end are journals filled with "positive results" that mean nothing and a bunch of "scientists" who make a living doing things that do not really resemble science at all.

Re:Positive bias in engineering research (1)

GJSchaller (198865) | about 2 years ago | (#39966039)

In engineering research, there is definitely a positive bias; in fact, negative results are rarely published at all. This is both because negative results have less sex appeal than positive results and because peer reviewers are trained to outright reject publications without positive results

This. "People" don't want to hear things perceived as "negative," "failures," or "I don't know," even when it's accurate. The common person thinks science has all of the answers, and that if a scientist doesn't know the answer, it's the fault of the scientist, not science as a whole. Obviously, if problem X can't be solved, it's because the scientist working on it is lazy / stupid / biased, and not that society as a whole doesn't have the answers yet.

I can link this to two common phenomenon in current society: 1) TV / Movie science, in which the fictional scientist does something risky and cool, and solves the problem of the week using "SCIENCE!". See also, the CSI effect in court rooms. 2) Corporate culture, where people don't want to hear "I don't know" for an answer. Most corporate people are conditioned to have an answer, ANY answer, rather than say "I don't know yet." (Pet peeve of mine - being asked to solve a problem I haven't been informed of yet, and / or then being asked why I don't know the answer when I don't know the parameters of the issue / question.)

Until the concept of "It's OK to admit you don't know everything, and learning stems from this" is common place, there will be positive bias in published results, and other facets of society, even when the results are faulty.

Duh (-1)

Anonymous Coward | about 2 years ago | (#39965871)

I spent years riding the wave ahead of bleeding edge technology, working with NASA, Universities, Fortune 500 companies, national laboratories. Education in the United States sucks. Communication in the United States suck. While morons in the States complain about spelling or grammar in communication scientists from other countries are focusing on communicating ideas. While scientists in the States are focusing on proving their own ideas and calling it research, scientists in other nations are focusing on real research. This problem is really confined to scientists educated in the States, maybe seeping into Western Europe. It is primarily based on ego and it was primarily prevalent in politics and religion. People in the States refused to change their political or religious ideas when confronted with facts. Now scientists in the States are refusing to change their ideas ideas when confronted with facts. It sucks, but, poor education and a total disregard for facts is a typical precursor to the fall of a nation.

R&D (1)

Lord Lode (1290856) | about 2 years ago | (#39965907)

I'm not sure if it's just me, but I've got the feeling that these days, most useful and ground breaking new research and technology comes from companies, not universities.

What do you think?

Publication bias (1)

overshoot (39700) | about 2 years ago | (#39965909)

Publish or perish. And journals prioritize first results on a research question, just like newspapers want to scoop the competition. On top of that, even first news on a hypothesis is less likely to be published if it's negative.

So what do you expect academics to do? Reinvestigating a previously reported research result is unlikely to get funded, it's unlikely to be published even if it is researched. They live and die by publications.

Unless and until the publication system makes it possible for academics to check each others' work without killing their own careers, we won't see it happen.

Who cares? (1)

DogDude (805747) | about 2 years ago | (#39965951)

This article is about trying to convince religious nutjobs that science is valid? WTF is the point of that? People, in 2012, who don't understand the scientific method are hopeless. Ignore them and move on.

Same Old, Same Old (1)

scruffy (29773) | about 2 years ago | (#39965981)

This has always been the case. Science is not a uniform march to the Truth. There is a difference between well-verified and understood results (think engineering) and working at the margins with not much data and the usual human failings (the vast majority of publications). Scientists are humans, not gods. It takes a lot of effort and error to get to the well-verified and understood part.

Scientists and Catholic Priests (1)

wcrowe (94389) | about 2 years ago | (#39966021)

This is pretty serious. I know that my own confidence in scientific research has been eroded quite a bit in recent years. I can only imagine how it vindicates those who despise science. As far as harming public perception, this is about the equivalent of Catholic priests diddling little boys.

Lack of funding has already killed it (1)

sl4shd0rk (755837) | about 2 years ago | (#39966031)

When you consider what the US gives to it's science budget, the only thing eroding public trust is the paid-for blathering by bought-off scientists. Don't try and hide the real problem with "Positive Bias". That's what you get when you hire someone at a pharmaceutical company to write up a paper for Nature. You don't get independent viewpoints you get advertisement with a scientific undertone.

"A half a penny... The most powerful agency of the dreams of a nation is currently underfunded to do what it needs to be doing."
  - Neil deGrasse Tyson

http://www.tor.com/blogs/2012/04/neil-degrasse-tyson-on-the-nasa-budget [tor.com]

Papers are advertising (1)

gr8_phk (621180) | about 2 years ago | (#39966065)

Most papers I've seen make magnificent speculative claims.

This novel material could usher in a new era in...
This novel method could become...
OMG this material has an IOR of 1.00000001 for gama rays - this will enable.... (yes this was a few days ago).
OMG jelly spread on glass can produce a voltage when exposed to sunlight - organic solar cells could change the world but more research will need to be done to get the efficiency above 0.3 percent.

Anything that isn't in a textbook is "novel" because that's an important word in patents and it's important to seem new. Then the grand claims are made either to make the research seem relevant or to line up funding for future work. Papers have become promotional material.

It happens in Slashdot too... (4, Interesting)

elsurexiste (1758620) | about 2 years ago | (#39966067)

I remember a few days ago someone submitted a story about piracy for "The Avengers" being low compared to potential profits from them. A few high-ranked comments were like "This is yet another proof that [insert common /. parlance here]". I saw very few comments that stated the most plausible reason: a camcorded action film, with crappy audio and a shaking image, can't compete against the real thing. I thought the same thing: confirmation bias.

People do it all the time. If something can somehow support their views (specially if they don't RTFA) they'll use it as yet more confirmation. "I still don't get why this piece of evidence is discarded by everyone else! They must be delusional or have bad intentions". For example, I imagine this article will be used as evidence for: lack of funding, falling standards in the US, the demise of education, lack of scientific reasoning (maybe they'll even extend it to scientists themselves), and other common /. utterances. I wonder how many of them will actually say what I found out after RTFA...

So, everyone is playing the same game, and scientists are no exception. But hey, that study has numbers on it. At least you can try to replicate the findings, if only the entry barrier wasn't so high: these tests are *hugely* expensive. More collaboration may be a good idea. Shared laurels are better than none, right?

P.S., a nice article on confirmation bias (and other goodies) here [youarenotsosmart.com].

What ever happened to trying it yourself? (0)

Anonymous Coward | about 2 years ago | (#39966143)

Get an independent lab to confirm your results if you want to be published. Ftfy

Hypothesis: Personification improves receptivity. (1)

VortexCortex (1117377) | about 2 years ago | (#39966149)

Materials:
You will need an inanimate entity or concept, an emotional situation, a forum of discourse, and a mind with approx 80 to 120 billion neurons.
Procedure:
0. Select commonly known, generally accepted concept or phrase.
1. Personify the inanimate subject.
2. Rephrase the concept or phrase including the situation designed to invoke an emotion, such as sympathy.
3. Await until an opportunity with sufficient relevance presents itself.
4. Present the phrase in the forum of discourse.
5. Cite a nebulous entity, so as to lend artificial credence.
6. Observe the Science!
7. ...
10. Prophet!

------

"If you torture the data long enough, it will confess to anything."
-- Unknown Econometrician.

Load More Comments
Slashdot Account

Need an Account?

Forgot your password?

Don't worry, we never post anything without your permission.

Submission Text Formatting Tips

We support a small subset of HTML, namely these tags:

  • b
  • i
  • p
  • br
  • a
  • ol
  • ul
  • li
  • dl
  • dt
  • dd
  • em
  • strong
  • tt
  • blockquote
  • div
  • quote
  • ecode

"ecode" can be used for code snippets, for example:

<ecode>    while(1) { do_something(); } </ecode>
Sign up for Slashdot Newsletters
Create a Slashdot Account

Loading...