Beta

×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

The Neuroscience of Screwing Up

samzenpus posted more than 4 years ago | from the nobody-is-right-all-the-time dept.

Science 190

resistant writes "As the evocative title from Wired magazine implies, Kevin Dunbar of the University of Toronto has taken an in-depth and fascinating look at scientific error, the scientists who cope with it, and sometimes transcend it to find new lines of inquiry. From the article: 'Dunbar came away from his in vivo studies with an unsettling insight: Science is a deeply frustrating pursuit. Although the researchers were mostly using established techniques, more than 50 percent of their data was unexpected. (In some labs, the figure exceeded 75 percent.) "The scientists had these elaborate theories about what was supposed to happen," Dunbar says. "But the results kept contradicting their theories. It wasn't uncommon for someone to spend a month on a project and then just discard all their data because the data didn't make sense."'"

cancel ×

190 comments

Sorry! There are no comments related to the filter you selected.

Why most scientists and engineers screw up (-1, Flamebait)

Anonymous Coward | more than 4 years ago | (#30601514)

It's because they are not very good. We could use 75% fewer, and we'd get more done. They picked the wrong major in college.

Re:Why most scientists and engineers screw up (0, Troll)

Anonymous Coward | more than 4 years ago | (#30601602)

Disclaimer: I am an academic research associate

Here's a quote from the article:

"But the results kept contradicting their theories. It wasn't uncommon for someone to spend a month on a project and then just discard all their data...The details always changed, but the story remained the same: The scientists were looking for X, but they found Y."

The dirty little secret is that the Y is not always unexpected, just too politically incorrect and dangerous to be released to the public. For example, my team at Rutgers just completed a comprehensive experiment measuring a variety of factors including intelligence and genetic makeup(read: race). What we discovered would have caused a political shitstorm orders of magnitude worse than that of Don Imus when he referred to our Women's basket ball team as "Nappy-headed Ho's", so we declared it unsucessful and quietly buried it.

We tried, we really did. We developed formulae which would account for environmental/nutrure factors. We were very forgiving with the fairness of our methods, and yet the numbers still added up in a way that was unflattering to our hypotheses.

Oh, well. Maybe they'll finally figure it out when a monkey-ass coon tries to blow up a plane and ends up lighting his nuts on fire. Wait, what? HA HA! Oh, man! What a Gorilla!

Re:Why most scientists and engineers screw up (3, Insightful)

A beautiful mind (821714) | more than 4 years ago | (#30601684)

I think the parent post is a brilliant example of what happens when someone perfects trolling to a science.

Re:Why most scientists and engineers screw up (1)

Hurricane78 (562437) | more than 4 years ago | (#30602090)

If he is, then he got nothing on you, who is also an expert in using reverse psychology for trolling. ^^

Re:Why most scientists and engineers screw up (2, Interesting)

Alien Being (18488) | more than 4 years ago | (#30602240)

Really? Or is it that you are SO politically correct that you cannot see truth.

I happen to have mod points and my on-the-fly ranking went from insightful to interesting to troll and back to interesting.

I've lived long enough to understand that each of the 6 billion people on this earth is different than every other. Some are remarkably good and some are remarkably bad. Most of us are just average in our own interesting ways.

But still, I do believe that genetic differences affect what we are and that genetic differences can be attributed to where our genes came from.

Those who choose to never risk offending anyone are perhaps the most intellectually dishonest among us.

Should I post this or should I mod YOU the troll?

Re:Why most scientists and engineers screw up (2, Insightful)

TapeCutter (624760) | more than 4 years ago | (#30602354)

"Those who choose to never risk offending anyone are perhaps the most intellectually dishonest among us."

All fine and good, except the OP does not contain anything more intellectual than a bunch of bald assertions wrapped in the emotions of a xenophobe. In other words, you should have modded the GP informative, the OP is a well formed troll.

Re:Why most scientists and engineers screw up (2, Informative)

NeutronCowboy (896098) | more than 4 years ago | (#30602454)

But still, I do believe that genetic differences affect what we are and that genetic differences can be attributed to where our genes came from.

The theory that race has nothing to do with intelligence has nothing to do with political correctness, and all with science: specifically, the scientific discovery that the taxonomy of human races is not definitive, not specific and has no basis in genetics. Which in turn means that the ggp's assertion that race was a statistically significant factor in their research means that their research was utter crap to begin with.

So let me ask you this then: what makes you think that race is the same as genetics, or that you can even reliably a race? I mean, outside of some outdated and non-scientific notions of physiognomy and phrenology?

Re:Why most scientists and engineers screw up (1)

ResidntGeek (772730) | more than 4 years ago | (#30601698)

Any chance you have a preliminary write-up, or even raw data I could read? Sounds exquisitely interesting. ResidntGeek@gmail.com if you can.

Re:Why most scientists and engineers screw up (0)

Anonymous Coward | more than 4 years ago | (#30602486)

Are you really asking an AC troll for sources?

You're wasting your time.

Re:Why most scientists and engineers screw up (3, Interesting)

Jurily (900488) | more than 4 years ago | (#30601754)

The dirty little secret is that the Y is not always unexpected, just too politically incorrect and dangerous to be released to the public.

So, when reality is racist, you change it?

Re:Why most scientists and engineers screw up (1)

lq_x_pl (822011) | more than 4 years ago | (#30601950)

No, when reality is politically incorrect, you bury it. Just imagine if the ACLU sued life, the universe, and everything...

Re:Why most scientists and engineers screw up (4, Funny)

rikkitikki (91982) | more than 4 years ago | (#30602010)

They'd get 42 dollars?

Re:Why most scientists and engineers screw up (4, Interesting)

Anonymous Coward | more than 4 years ago | (#30602176)

Or when the results you get aren't acceptable to the people responsible for continued funding.

Years ago, I worked for months trying to reproduce the Polywater research,
http://en.wikipedia.org/wiki/Polywater
and eventually reported that I was unable to do so.
The department considered my work a failure (as in, I must have been incompetent) and did not publish my findings. When, years later, the publications reporting successful discovery/creation of Polywater were shown to be fraudulent, and my results were correct, I did not even receive an apology.

Throwing out results is unethical as well as irresponsible. Many discoveries have come from re-evaluating what appears to be "bad" data. It might not be possible to use it now, but it should be at least stored.
For instance, it has been reported that the "bit of "scruff" on her chart-recorder papers that tracked across the sky with the stars"[1] looked like bad data to Jocelyn Bell Burnell's supervisors. Today we call the phenomenon a pulsar.
[1] Wikipedia

Racism and science. (1, Flamebait)

tjstork (137384) | more than 4 years ago | (#30602000)

So, when reality is racist, you change it?

I thought World War II empirically proved that the master race is not all its cracked up to be. American mutts and Soviet subhumans kicked the living shit out of the master aryan race. The whole concept of NAZI ideology was that they were the master race, they were not only deserving of victory, but destined, thus, by the most racist rules there are, they proved themselves inferior.

PS. Polish women are the hottest of all European women.

Re:Racism and science. (1)

Tynin (634655) | more than 4 years ago | (#30602254)

PS. Polish women are the hottest of all European women.

Pics or it isn't true! That said, I'm sure the GP was trolling. But due to the PC movement (more specifically, our backlash to the PC movement) it is too easy to claim to be an authority of something like this and get people to believe their might be a hint of truth to it without him even providing any real details. If you condense the post down to its finer points, it quickly becomes obvious he is bashing people with brown skin, especially by the time you get to the end of the post.

Re:Racism and science. (1)

Jurily (900488) | more than 4 years ago | (#30602338)

it quickly becomes obvious he is bashing people with brown skin, especially by the time you get to the end of the post.

Without wanting to delve into the finer points of troll detection, that post does have a point: what happens when either the people working on the data have strong opinions about the outcome, or the people giving the funding do?

Re:Racism and science. (1)

Tynin (634655) | more than 4 years ago | (#30602500)

No quick answers to that one other than to allow the research to kill itself overtime. Since no one but funders/researchers would be aware of any bias, no one could do much about it until they publish their findings. But once they publish, if it is bias then an independent review with repeated testing and evaluation against the original research should either show it to be legitimate or not. If the research was obviously faked, then I hope the "scientists" behind it and any other research they have done are closely reviewed as well.

Re:Racism and science. (1)

Tynin (634655) | more than 4 years ago | (#30602532)

As soon as I hit submit, I realized I answered the wrong part of this question. I'm expecting them to publish, but what likely trolling AC was saying is they did the research but now armed with their findings they aren't publishing due to possible bias of the funders not wanting to hear that answer. I don't think we can combat this, as once again no one would know about it as it would be an inhouse secret. Unless a researcher on the team decided to go rogue and release it to wikileaks, I think no one would be the wiser.

Re:Racism and science. (1)

Jurily (900488) | more than 4 years ago | (#30602284)

Seriously, what the fuck?

Re:Racism and science. (0)

Anonymous Coward | more than 4 years ago | (#30602664)

Just 'cause the Aryans weren't the so-called "master race" doesn't mean there isn't one. I think right now, it's a toss up between money and women.

Re:Why most scientists and engineers screw up (1)

joocemann (1273720) | more than 4 years ago | (#30602364)

The dirty little secret is that the Y is not always unexpected, just too politically incorrect and dangerous to be released to the public.

So, when reality is racist, you change it?

Don't attempt to reason or argue with Anonymous Cowards. There is no way to corroborate anything he/she said.

Just a little advice.

Re:Why most scientists and engineers screw up (1)

Ethanol-fueled (1125189) | more than 4 years ago | (#30602508)

The dirty little secret is that the Y is not always unexpected, just too politically incorrect and dangerous to be released to the public.

Eh, so the scumbag may have been racist. Dosen't make the first 5/6 of his post any less true. In science, the experiment comes first. The troll is probably just disappointed that science didn't prove his racist views wrong, as well as advance his career and get him more funding.

HehehehehHAHAHAHAHAHAHAAH! [alanisgood.com]

Re:Why most scientists and engineers screw up (1)

indiechild (541156) | more than 4 years ago | (#30602702)

A troll from Anonymous Coward suddenly equals reality?

How the fuck did that get modded interesting?

Re:Why most scientists and engineers screw up (1)

spiffmastercow (1001386) | more than 4 years ago | (#30601946)

That's what wikileaks is for, man. If you're telling the truth, then you have an obligation to see that the data is released so that it can be evaluated, tested, and verified (or debunked).

Re:Why most scientists and engineers screw up (2, Insightful)

bzipitidoo (647217) | more than 4 years ago | (#30602404)

Are groups of people from very different locations, such as whites and blacks, different? Of course they are! Not so different as to be separate species (yet, and with global communication, maybe never), but evolved in different directions to adapt to different conditions. Just being different means there are activities for which one group will be better suited than the other group. There has to be, otherwise they aren't different, are they? Get over both the racism and the political correctness, and admit this basic fact. Skin color is a fairly superficial difference. Africa has by far the most diversity-- the Eastern African perhaps has less in common with the Western African or the Pygmy than with whites.

Much more harmful is the tendency to oversimplify fitness to a single, grossly over broad measure of a difficult to define concept that is not universally relevant, such as IQ, and declare one group superior to another based on only that. When a declaration of superiority is made, you may be sure it is for purposes of propaganda. Geniuses (defines as people with IQ > 140, or perhaps > 160) make their share of fatal mistakes, have flaws that can render their supposed advantages much less valuable, take risks and sometimes lose, sometimes let success go to their heads (most recently, Mike Leach), just like everyone else. Bobby Fischer was a genius, but that monomania which made him able to be World Chess Champion hurt him in so many other ways. The Soviet Union really bought into the idea of chess (and other contests such as the Olympics) as a good measure of a society's fitness, and devoted so much effort to it that except for Fischer, they pretty much mopped the boards with other nations' best players. And what was it all for? Propaganda that ultimately proved empty when Communism collapsed. I expect the Space Shuttle astronauts all do very well on IQ tests, but is that a smart gamble, risking their lives on that thing, for the fame and money they get? For other sorts of fitness, there are many fantastic athletes, rock stars, leaders, and the like who ended tragically. A small change in conditions can at a stroke reverse the fitness of most any trait.

ObClimate reference (0)

Anonymous Coward | more than 4 years ago | (#30601530)

"The science is settled!" :P

Re:ObClimate reference (0)

Anonymous Coward | more than 4 years ago | (#30601612)

Hide the decline!

Sometimes screwing up leads to success ... (3, Informative)

xmas2003 (739875) | more than 4 years ago | (#30601542)

The WIRED piece threads what is written in the summary around the story of how Arno Penzias and Robert Wilson at Bell Labs discovered Cosmic Radiation [wikipedia.org] after being puzzled for a year about background noise on their radio telescopes ... even scraping pigeon poop off their gear as a possible source until they realized the signal was real - Homer Simpson would have said D'OH! ;-) [komar.org]

Ridiculous (4, Interesting)

MrMista_B (891430) | more than 4 years ago | (#30601566)

"It wasn't uncommon for someone to spend a month on a project and then just discard all their data because the data didn't make sense."

That doesn't mean the data is wrong, it means the /hypothesis/ was wrong, if not the theory, and needs to be modified.

If they're really throwing out date just because it 'doesn't make sense', they're doing religion, not science.

Re:Ridiculous (5, Insightful)

wizardforce (1005805) | more than 4 years ago | (#30601636)

If your equipment is malfunctioning, you may end up with data that is fairly random where there should be some pattern or your measurements on your controls don't remotely match the values they should be. As an example, a standardized solution tests for a markedly different concentration than it should; a good sign that something is wrong. Things go wrong occasionally. That is why it is imperative that experiments be repeatable and have good experimental design.

Re:Ridiculous (3, Interesting)

MyLongNickName (822545) | more than 4 years ago | (#30601648)

And this is what bothers me. If you are willing to run an experiment enough times, you will eventually get data to support your assertions. Get a statistical 90% certainty, and it could be that you ran the scenario 100 times, and throw out the 99 times that did not give you this certainty. The scientific process is bullet proof. The folks who "do science" not necessarily so.

Re:Ridiculous (0)

Anonymous Coward | more than 4 years ago | (#30601750)

I'd like you to take your scientific method fail and go test it by jumping off a building. Don't worry, if you repeat it enough times you'll get data to prove that you can fly.

Re:Ridiculous (1)

stms (1132653) | more than 4 years ago | (#30601916)

Scientist are bullet proof, I have a 90% certainty of that.

Re:Ridiculous (2, Interesting)

TapeCutter (624760) | more than 4 years ago | (#30601956)

"If you are willing to run an experiment enough times, you will eventually get data to support your assertions."

Yes, I belive Edison tried over 5000 different hand made bulb/filiment combinations before he found one that supported his assertion.

Thowing out data is not about proving pet theories, it's about admitting you cocked up the experiment. eg: Prof Sumner Miller [abc.net.au] never edited out failed demonstrations from his TV show, nor did he claim the failed demo proved accepted theories of physics were wrong, rather he would simply exclaim - "Experiments never fail, it is I who have failed to set the right conditions for nature to cooperate" and then try again.

Re:Ridiculous (1)

wizardforce (1005805) | more than 4 years ago | (#30601974)

If there's nothing wrong with your equipment or procedures and your experiment fails to find X then it's indicative that your hypothesis is wrong and any group that repeats your experiment is going to know about it immediately.

Re:Ridiculous (0)

Anonymous Coward | more than 4 years ago | (#30602036)

That's ridiculous. Experiments are designed to test hypotheses. However not all experiments succeed, where success means "provides significant information confirming or contradicting the hypothesis". Designing and executing meaningful experiments is hard, and it's something that smart people routinely fail to pull off.

Re:Ridiculous (1)

KliX (164895) | more than 4 years ago | (#30602030)

If the search space was that simple, we'd know everything by now.

Re:Ridiculous (3, Funny)

Culture20 (968837) | more than 4 years ago | (#30602108)

The scientific process is bullet proof. The folks who "do science" not necessarily so.

What exactly are you advocating?

Re:Ridiculous (2, Funny)

Thing 1 (178996) | more than 4 years ago | (#30602516)

The scientific process is bullet proof. The folks who "do science" not necessarily so.

What exactly are you advocating?

That, in battle against an opponent armed with a firearm of some sort, one should adorn one's self with the scientific process, and not the folks who are following it, for greatest effect?

Re:Ridiculous (0)

Anonymous Coward | more than 4 years ago | (#30602438)

It makes no difference unless you can get EVERYONE ELSE to agree to throw out each of their 99 times as well.

Until then, you're just a Crackpot.

Re:Ridiculous (3, Informative)

Shadow of Eternity (795165) | more than 4 years ago | (#30601660)

Not always, sometimes your data doesn't make sense because you made a mistake somewhere that wound up turning your results into garbage.

Re:Ridiculous (1)

steelfood (895457) | more than 4 years ago | (#30602584)

Sounds like they're in need of engineers to design those experiments.

Re:Ridiculous (1)

Shadow of Eternity (795165) | more than 4 years ago | (#30602650)

And then eventually they'll have a child and that child, born of science and engineering, will be our new god.

Re:Ridiculous (1)

labnet (457441) | more than 4 years ago | (#30601694)

Often the data is crap, because the measurements are so hard to make.
For example, you would think measuring temperature is easy. Not so.
Lets say you wish to determine the cooling capacity of an airconditioner.
How do you measure the temperature and air velocity gradients across both the return and supply air streams. Do I use 1 sensor, 10 sensors, 100 sensors. Do you create turbulence or laminar flow? How accurate is the humidity measurement?
The point is, the data is often crap, because measurements are hard to make, time is limited, can't afford the right equipment, not enough labour, could not fully simulate the enviroment etc etc.

Re:Ridiculous (0)

Anonymous Coward | more than 4 years ago | (#30601912)

Yeah, I remember when I was in college I used to throw a lot of dates out because they didn't make any sense.
Usually the dates threw me out, but that is another thing.

Re:Ridiculous (1)

nedlohs (1335013) | more than 4 years ago | (#30602016)

It can also be that their methodology is wrong, their equipment is wrong, they are simply incompetent.

I suspect that the main thing that differentiates scientists who make a major lasting contribution to science and those that just push knowledge slowly forward is whether they treat such problems as "Oh... that is interesting" or "Crap, must have screwed it up better start it over".

Of course I'm sure those in the first category spend a lot of their time chasing ghosts because most of the time they did in fact screw something up...

Re:Ridiculous (2, Insightful)

rbannon (512814) | more than 4 years ago | (#30602022)

Not religion, but federally funded dogma. More than 20 years ago I became aware of how dogma gets grounded in fundamental research: you need to write grants that fit the dogma. One hapless soul actually stood up during a big AIDS conference and suggested that the researchers were mere lemmings. He, of course, was shouted down, but he was only trying to tell the lemmings to keep an open mind. Fast forward 20+ years and the lemmings are still in control.

Our educational system is totally broken when the educated just want things to fit. Even in mathematics, we're promoting a crop of "just tell me what to do!"

Re:Ridiculous (4, Insightful)

BlueParrot (965239) | more than 4 years ago | (#30602140)

"It wasn't uncommon for someone to spend a month on a project and then just discard all their data because the data didn't make sense."

That doesn't mean the data is wrong, it means the /hypothesis/ was wrong, if not the theory, and needs to be modified.

If they're really throwing out date just because it 'doesn't make sense', they're doing religion, not science.

a) You've clearly never done any real research or you would be well aware of the hundreds of millions of ways you can screw up an experiment and get nonsense data ( bad machinery, you wired up a detector wrong, the cell lines you were feeding vitamin K happened to get contaminated by bacteria halfway through etc... )

b) There is almost never a clear difference between data and theory. The only raw data you have is a bunch of numbers on a piece of paper, in order to determine if they correspond to your theory or not you need to interpret the numbers somehow, and it may just as well be the interpretation that is wrong as is the theory you were trying to test using the interpreted data.

c) Because you are often restricted by cost and time it's often not feasible to do a full analysis of why your experiment did not work. Hence if you did not get any useful results ( uncertainty was too large, it seems obvious you must have messed up somewhere etc.. ) then frequently the only sane option is to conclude your experiment was a failure.

d) If scientists followed your advice we would never have got the electronic equipment you used to make your post.

Basically your ideas about what science is or should be are extremely naive and to anybody who has done even a high school chemistry experiment it should be clear you have no idea what you're talking about.

Re:Ridiculous (2, Informative)

RobertF (892444) | more than 4 years ago | (#30602250)

Anyone who has spent time working in a lab knows that not all data is equal. You can get useless results if something isn't quite as clean as necessary, or perhaps you were in a bit of a rush and didn't connect everything perfectly. Any interesting experiment usually has numerous points at which humans can mess things up. Errant data is usually a sign that you have improperly set up the experiment, so you'll spend most of your times reviewing and fixing procedures until you get what you expected.

Re:Ridiculous (0)

Anonymous Coward | more than 4 years ago | (#30602464)

The bulletproof, 'scientific' statement is something like "if you hook up the apparatus exactly so, then you will get these results," when you really want a meaningful statement like "cosmic ray muons travel at 0.99c." These are very different claims.

The whole point throwing out data is because "hooking up the apparatus exactly so" does not seem to correspond to measuring the speed of cosmic ray muons.

Re:Ridiculous (2, Informative)

Arancaytar (966377) | more than 4 years ago | (#30602524)

It is possible to end up with crap data because the premise of your experiment is wrong. You can ignore a variable that should have been controlled or kept equal, or you can measure the wrong variable.

You can also end up with data that neither confirms nor denies your hypothesis, because it allows no statistically significant conclusion.

Re:Ridiculous (0)

Anonymous Coward | more than 4 years ago | (#30602540)

Lesson #1 for any scientist:

Nature does whatever the hell it wants to. The best that a scientist can ever do is try to model it. If the model doesn't fit, it is wrong. Nature is always right.

Lesson #2 for any scientist:

Don't discard data. If there is a systematic error, then verify that it skewed the data and try to get data without that error. If there is no significant systematic error, then the data is correct.

Lesson #3 for any scientist:

Be humble. Pride, past successes, and the need for it to be right do not affect whether a theory is correct.

Re:Ridiculous (0)

Anonymous Coward | more than 4 years ago | (#30602568)

I'm a scientist. Throwing out data is not good practice. Instead, some view 'weird' data as a great opportunity for further theorizing, which is probably how it should be considered.

Nonetheless, under some conditions I could understand throwing out data. When there's too much of it, you need to focus analysis on that of which you can make sense. If nothing makes sense, then there's nothing to contradict or confirm.

You never discard the data (5, Interesting)

techno-vampire (666512) | more than 4 years ago | (#30601586)

If the data don't make sense according to your theory, you don't discard the data, you discard the theory and work out a new one that fits the facts as you've observed them. TFA says that Dunbar was watching postdocs doing research, and if so, they should have known better. Alas, too many people who call themselves scientists are more interested in proving their pet theory true than in finding out what's actually going on.

Re:You never discard the data (1)

Tablizer (95088) | more than 4 years ago | (#30601712)

Alas, too many people who call themselves scientists are more interested in proving their pet theory true than in finding out what's actually going on.

Are you trying to tell me that Underoo-wearing mice don't jump higher?

       

Re:You never discard the data (2, Insightful)

A beautiful mind (821714) | more than 4 years ago | (#30601768)

Alas, too many people who call themselves scientists are more interested in proving their pet theory true than in finding out what's actually going on.

It's just a result of how science is performed. Science doesn't have low hanging fruits anymore, consequently any problem that someone investigates takes dedication, because it's intellectually hard or takes lots of effort or both. Most people aren't going to be motivated enough to put that much effort into it without already having an axe to grind, a point to prove, a pet theory to push into the limelight.

Also, in a lot of cases you don't know there is something interesting in the area you're looking at. I think what separates bad scientists from good scientists is how you realize when something doesn't match up to your preconceived notions and how you recover from conflicting data.

Re:You never discard the data (2, Insightful)

dwguenther (1100987) | more than 4 years ago | (#30601998)

The people who are not motivated enough to put in the effort are not scientists - they are pundits. Researchers who are truly interested in their work - and that would be most of them - put in decades of observation and analysis looking for some truth, because simply grinding an axe would never be personally satisfying. It is lazy and disrespectful of you and other armchair commentators to simply dismiss all that work with a three-line opinion.

Re:You never discard the data (1)

Metasquares (555685) | more than 4 years ago | (#30602420)

It is lazy and disrespectful of you and other armchair commentators to simply dismiss all that work with a three-line opinion.

Doing precisely this is one of the more distasteful parts of most scientists' jobs.

Re:You never discard the data (0)

Anonymous Coward | more than 4 years ago | (#30602362)

Recovering from conflicting data often (though not always) requires discarding bad data, because things go wrong---samples get contaminated, measurement apparatuses get miscalibrated, the planets don't line up just right (yes, I'm kidding---except maybe if it's some kind of astrophysics experiment...), etc. In those situations the conflicting data are simply invalid, being taken from a (necessarily) flawed apparatus/environment, and it's perfectly reasonable to discard them (or more likely, label them "bad", keep them somewhere out of the way, but not publish them), make any fixes you can, and try again.

The question is how to recognise when your conflicting data are actually real. This is what takes so much time and effort to eventually determine, and while it does take longer, compared to confirmation of existing theories, science inevitably gets there eventually.

Re:You never discard the data (0, Troll)

topham (32406) | more than 4 years ago | (#30601816)

The problem is you generally do not get money to simply study X.

You get money to show that X affects Y in manner Z. If X doesn't affect Y in manner Z they pull your funding, give you a failing grade, or otherwise find ways to punish the results.

They do this over and over again and then wonder why researches fake data, toss good data out and re-do the study looking for results they want instead of what is.

Want to do a drug study that says Drug A is safe to fight cancer? Got results that indicate an increase risk of heart attack? Have the study declared flawed, re-do the study with a slightly different mix of subjects and repeat. With luck your new study shows the heart attack risk is below the error threshold of your study and you can ignore it. Release your drug, make your millions and, after you leave have the real-world implicates show up on the 5 o'clock news.

Re:You never discard the data (1)

dwguenther (1100987) | more than 4 years ago | (#30602096)

This comment displays a vast ignorance of how research actually gets done. Most research is funded to gain knowledge, regardless of the result (otherwise it would simply be called 'knowing', not research). Note for instance that the drug companies continue spending millions dollars on basic research year after year, even when they don't get an immediate result. So don't broad-brush the dedicated work of hundreds of thousands of scientists with your own questionable view of ethics.

Re:You never discard the data (4, Insightful)

caramelcarrot (778148) | more than 4 years ago | (#30601850)

As other people have pointed out - sometimes the data is just crap due to the difficulty of making measurements. Sometimes you've measured something other than what you actually need to compare to theory, sometimes there's too much noise. The skill of a great experimentalist is being able to take good enough data that you can't justify ignoring it if it comes out different to what you expected.

Re:You never discard the data (5, Insightful)

bcrowell (177657) | more than 4 years ago | (#30601872)

If the data don't make sense according to your theory, you don't discard the data, you discard the theory and work out a new one that fits the facts as you've observed them. TFA says that Dunbar was watching postdocs doing research, and if so, they should have known better. Alas, too many people who call themselves scientists are more interested in proving their pet theory true than in finding out what's actually going on.

This is a beautiful explanation of how science is supposed to work. In reality, science doesn't really work this way. It doesn't work this way in my experience as a scientist, and it doesn't work this way if you read the history of science.

For some good historical examples, see Microbe Hunters, by de Kruif (one of the best science books of all time, although you have to look past the racism in some places -- de Kruif was born in 1890). A good example from physics is the Millikan oil-drop experiment, where he threw out all the data that didn't fit what he was trying to prove -- but then claimed in his paper that he'd never thrown out any data. Galileo described lots of experiments as if he'd done them, even though he didn't actually do them, or they wouldn't have actually come out the way he described.

Michelson and Morley set out to prove the existence of the aether, published their results believing they must be wrong. Nobody else believed them, either. Various people then spent the next 30 years trying to fix the experiment by doing things like taking the apparatus up to the top of a mountain, or doing the experiment in a tent, so that the aether wouldn't be pulled along with the earth or the walls of a building. By the time Einstein published special relativity in 1905, most physicists had either never heard of the MM experiment, or considered it inconclusive.

When your results come out goofy, 99.9% of the time it's because you screwed up. You don't publish it, you go back and fix it. If every scientist published every result he didn't believe himself, the results would be disastrous. If you try over and over again to fix it, and you still fail, only then do you have to make a complicated judgment about whether to publish it or not.

The way science really works is not that scientists are disinterested. Scientists generally have extremely strong opinions that they set out to prove are true using experiments. The motivation is often that scientist A dislikes scientist B and wants to prove him wrong, or something similarly irrational, personal, or emotional. The reason this doesn't cause the downfall of science as an enterprise is that there are checks and balances built in. If A and B are enemies (and if you think the word "enemies" is too strong, you haven't spent much time around academics), and A publishes something, B may decide just to see if he can screw that sonofabitch A over by reproducing his work and finding something wrong with it. It's just like the adversarial system of justice. Society doesn't fall apart just because there are lawyers willing to represent nasty criminals. Einstein was famously asked what he would do if a certain experiment didn't come out consistent with relativity; his reply was that then the experiment would be wrong. Einstein fought against Bohr's quantum mechanics for decades. Bohr fought against Einstein's photons for decades. They were bitter rivals (and also good friends). It didn't matter that they were intensely prejudiced, and wrong 50% of the time; in the end, things sorted themselves out.

Re:You never discard the data (1)

dwguenther (1100987) | more than 4 years ago | (#30602124)

You may be the first actual scientist I've seen post in one of these discussions. That's a very well written explanation of the realities of research.
Mod this guy up!

Re:You never discard the data (1)

rkfig (1016920) | more than 4 years ago | (#30602228)

Wow. Well spoken, good information, and level headed. If I hadn't noticed the user ID, I would say that you are new here. Thanks for the input.

Seconded. (2, Insightful)

Estanislao Martnez (203477) | more than 4 years ago | (#30602232)

If the data don't make sense according to your theory, you don't discard the data, you discard the theory and work out a new one that fits the facts as you've observed them. TFA says that Dunbar was watching postdocs doing research, and if so, they should have known better. Alas, too many people who call themselves scientists are more interested in proving their pet theory true than in finding out what's actually going on.

This is a beautiful explanation of how science is supposed to work. In reality, science doesn't really work this way. It doesn't work this way in my experience as a scientist, and it doesn't work this way if you read the history of science.

Indeed. The sort of thing being discussed in TFA is one of the classic themes of late 20th century philosophy and history of science: the disconnect between traditional philosophy of science and the actual practice of science.

Kuhn's Structure of Scientific Revolutions is a good place to start. Just one tiny example of the book: Kuhn goes on about how during normal science, scientists perform experiments to confirm the results that they expect to get. When an experiment contradicts the theory, they don't automatically assume that the theory is wrong; on the other hand, they assume that the experiment was flawed.

Feyerabend and many other philosophers of science take a complementary stand to this by stressing the theory-ladenness of "facts." The claim that the "facts" contradict a hypothesis is never a theory-independent observation, but rather, the conclusion of a different theory that we may overthrow. Feyerabend's classic example is the Tower Argument that Aristotle used to refute the theory that the Earth moves. Wikipedia's article on Paul Feyerabend [wikipedia.org] has a decent, if terse, explanation of this:

"The tower argument was one of the main objections against the theory of a moving earth. Aristotelians assumed that the fact that a stone which is dropped from a tower lands directly beneath it shows that the earth is stationary. They thought that, if the earth moved while the stone was falling, the stone would have been "left behind". Objects would fall diagonally instead of vertically. Since this does not happen, Aristotelians thought that it was evident that the earth did not move. If one uses ancient theories of impulse and relative motion, the Copernican theory indeed appears to be falsified by the fact that objects fall vertically on earth."

Feyerabend goes on to argue that many of our most successful contemporary scientific theories (e.g., heliocentrism and geodynamicism) became so because their Renaissance and Enlightenment proponents held on to them and continued to elaborate on them despite them being contradicted by "the facts," as judged by the application of theories that were better established at the time (e.g., Aristotelian mechanics). That is, new scientific theories often succeed because their proponents keep working on them and improving them despite being contradicting by the "facts"; then as the new theories become stronger and better accepted, people start juding the "facts" by the lens of the new instead of the old, and forget the problems that the new theories were judged to have and never resolved (e.g., things like Newtonian physics not having the same explanatory range as Aristotelian physics).

Re:Seconded. (2, Insightful)

bcrowell (177657) | more than 4 years ago | (#30602324)

Feyerabend and many other philosophers of science take a complementary stand to this by stressing the theory-ladenness of "facts."

Yep, they're totally right. For example, this 2003 paper [arxiv.org] claimed to have empirically verified the prediction of general relativity that gravitational forces propagate at the speed of light. The authors made some technical errors, which were rapidly pointed out by others in the field. The final answer is that actually nobody has the faintest clue how to test this specific, century-old prediction by Einstein. The reason is that nobody has figured out any alternative theory of gravity that (a) fits presently known experiments, and (b) predicts that gravitational forces propagate at some other velocity than the speed of light. There are other theories of gravity that satisfy a, and are inconsistent with general relativity, but they are all consistent with general relativity on b. Since there is no alternative theoretical framework, there is no way to design or analyze an experiment to test the question.

Re:You never discard the data (1)

Rising Ape (1620461) | more than 4 years ago | (#30602056)

If the data don't make sense according to your theory, you don't discard the data, you discard the theory

Not really, assuming the theory is something well-established and tested. Popper oversimplified things - experimental data is rarely so unambigous that you can outright discard a reliable theory. It's much more likely that you messed up than you proved it wrong, or maybe the theory needs a fairly minor modification rather than complete rejection.

That's no reason to discard data though - not until you understand *why* the discrepancy arises, or at least have established that the data is unreliable in some other way. If, despite diligent effort, you can't find anything wrong with your analysis, then you publish and see if anyone else can explain it. The evidence may well then accumulate to the point at which the original theory is untenable, or alternatively it may demonstrate a different explanation for the original result that doesn't invalidate the theory at all.

Re:You never discard the data (1)

nedlohs (1335013) | more than 4 years ago | (#30602064)

Most of the time that the data doesn't make sense it's because the scientist fucked up and didn't calibrate the sensor, or mislabeled a sample, or made a tpyo.

But yes, discarding it is the wrong thing to do. Repeating it again (and if you now get what you were expected, doing it a third time) is usually the thing to do.

Of course if it is expensive, then just write down the results the theory predicts ith a fudge factor for error. Better to retard humanity's knowledge of the world than to maybe look silly.

Re:You never discard the data (1)

syousef (465911) | more than 4 years ago | (#30602156)

If the data don't make sense according to your theory, you don't discard the data, you discard the theory and work out a new one that fits the facts as you've observed them.

If it's a well established theory, you want to eliminate sources of the error before trying to overthrow it. For every Einstein that moves us to the next level from a well established theory there are 3 million cranks that just can't set up a well controlled experiment to save themselves. If you've conducted the experiment sufficiently badly the chances of working out what happened are nil. What you do is repeat the experiment correcting the errors and see what you get.

Alas, too many people who call themselves scientists are more interested in proving their pet theory true than in finding out what's actually going on.

It's a human frailty. Einstein wasted the later half of his career because he believed "God does not play dice", rather than accept Quantum theory. What a pity superstition had to come into it. A decade after his death Bell's Theorem has proven him wrong.

Re:You never discard the data (0)

Anonymous Coward | more than 4 years ago | (#30602200)

If the data don't make sense according to your theory, you don't discard the data, you discard the theory and work out a new one that fits the facts as you've observed them. TFA says that Dunbar was watching postdocs doing research, and if so, they should have known better. Alas, too many people who call themselves scientists are more interested in proving their pet theory true than in finding out what's actually going on.

Not a climate "scientist", are you?

Re:You never discard the data (1)

techno-vampire (666512) | more than 4 years ago | (#30602340)

Not a climate "scientist", are you?

No, I'm not, and I don't play one on TV, either. I am, however, glad you asked, because soi disant "climate scientists, especially those of the CRU are exactly the people I was thinking of.

Experimental error (1)

dbIII (701233) | more than 4 years ago | (#30602694)

It depends. If most of your data is noise it's fairly worthless anyway and you are better off trying to limit the sources of error and try again.
For example consider seismic data. You've got 50Hz or thereabouts induced in the cables near powerlines, you have wind blowing on the geophones, passing cars or trains, differences in soil above the rock and other sources of noise. A lot of seismic data processing seems to be about throwing away the noisy data and stacking up what is left to limit the effect of noise even furthur.
For other things there are different sources of error which may not be obvious. It's tempting to think it really is 27.23 Celcius becuase the digital thermometer say so, but the little semiconductor measuring probe may be out a full half a degree or more even if it does spit out numbers that fool people into thinking it is more accurate. Sure enough ten seconds later it could be telling you it is 26.8 Celcius when nothing has changed.

Alas, too many people who call themselves scientists are more interested in proving their pet theory true than in finding out what's actually going on.

If what is actually going on is that a train went past when the reading were taken or if the mains power had a minor spike then nobody really cares. It can take a while to set up a good experiment or set of measurements and some of the initial information collected may be rubbish. I've had bits of mid range steel tested where the results came back with large amounts of tungsten - and instead of compiling some theory about how it got there I've told the lab to kick the new kid off the machine, clean the electrodes and spark test it again.

obligatory (insert famous movie here) comment (0)

Anonymous Coward | more than 4 years ago | (#30601594)

Question *Everything*.

Throw away data? (0)

Anonymous Coward | more than 4 years ago | (#30601656)

Proving a theory incorrect is often just as valuable as proving a theory correct.

Re:Throw away data? (2, Insightful)

MathiasRav (1210872) | more than 4 years ago | (#30602070)

Proving a theory incorrect is often just as valuable as proving a theory correct.

I'd rather say proving a theory incorrect is just as valuable as proving a *hypothesis* correct. If it's a hypothesis, it's no fun proving it wrong (it wasn't established anyway, it might go against your intuition but nobody cares), and if it's a theory, it's no fun proving it right (what are you talking about, of course it's right, we already knew that).

I would elaborate on this but that would just be filler.

Re:Throw away data? (1)

Thing 1 (178996) | more than 4 years ago | (#30602684)

(Your parentheses elaborated.) And -- completely agree. :)

Good! (4, Interesting)

RyanFenton (230700) | more than 4 years ago | (#30601662)

If problems occur as you postulate elaborate hypothesis, then stop piling up the elaborate hypothesis! But be sure and still make available your existing (complex) hypothesis, methodology and unexpected data - preventing others from going down the same path with the same methodology is still highly valuable!

Let's say you're looking at a production and consumption cycle involving neurotransmitters and neuroreceptors of some sort, and the various channels of input and output involved. Your starting presumption you base your hypothesis on is that there is a buildup which triggers an electrical signal to stop consumption and clear the channel. The only evidence you can realistically gather for now is protein density at a certain output channel - but others have worked to ensure this is a reliable approach specifically under these circumstances.

So, you do the specific experiment, trigger the signal, but you get a wildly different result - the stop in consumption occurs, but the protein density does not change at all in the output channel. What actually happened is still unknown, only you haven't verified any correlation with your hypothesis. You still have valuable data, but no mechanism to verify under the circumstances. Either your methodology failed, or you misunderstood what was happening - and the world of knowledge is made larger by either... even if your paymasters won't get happy about the result.

Science is often like throwing pebbles in complete darkness - it takes a lot of stones and close listening to make out a mental picture of the scene - especially when there's a lot of noise already around. Everyone would love it if we could just flip the lights on - but we have yet to invent a light that can see into the inner workings of the functioning brain very well. Gotta keep throwing those pebbles for now.

Ryan Fenton

Re:Good! (2, Funny)

dcmoebius (1527443) | more than 4 years ago | (#30602076)

Neurotransmitter consumption cycles? Come on, if you want the +5 Insightful, you really need to couch your hypotheticals in terms of cars.

Re:Good! (1)

Thing 1 (178996) | more than 4 years ago | (#30602592)

[...] preventing others from going down the same path with the same methodology is still highly valuable!

Exactly. Thomas Edison "discovered" over 5,000 ways how not to create a light bulb. Had he published each and every one of them, perhaps the light bulb would have been invented sooner -- perhaps by someone else, or perhaps by him, collaborating with someone else who had read his published accounts of "how not to create a light bulb."

Or you can edit your data.... (2, Insightful)

sl149q (1537343) | more than 4 years ago | (#30601784)

Is it just me or does this sound like an explanation for some of the Climategate science... But in that case they just massaged or ignored data that didn't agree with their conceptual framework of CO2 causing global warming.

Not that the skeptics are all that immune. They seem to cherry pick data almost as well (just not quite as successfully from the POV of selling their story to the media and political left ..)

Re:Or you can edit your data.... (3, Informative)

dwguenther (1100987) | more than 4 years ago | (#30601942)

Yeah, it's just you. AP News found no evidence of massaged or ignored data (http://news.yahoo.com/s/ap/20091212/ap_on_sc/climate_e_mails). So climate science is a poor example of this thesis.

Re:Or you can edit your data.... (1)

khallow (566160) | more than 4 years ago | (#30602006)

Where's the AP story for the computer code?

Re:Or you can edit your data.... (2, Insightful)

DNS-and-BIND (461968) | more than 4 years ago | (#30602098)

The Associated Press are scientists now? Oy vey, what is the world coming to when commenters on science.slashdot.org quote the media as an authoritative source?

Re:Or you can edit your data.... (1)

NeutronCowboy (896098) | more than 4 years ago | (#30602494)

As opposed to a random blogger? Really? Be careful throwing around the authority argument, it might come back to bite you.

Re:Or you can edit your data.... (0, Troll)

Anonymous Coward | more than 4 years ago | (#30602138)

no evidence of massaged or ignored data

Well, not really. Quoting from the article you linked, they used tree rings when it supported AGW and ignore them when it didn't:

The "trick" that Jones said he was borrowing from Mann was to add the real temperatures, not what the tree rings showed. And the decline he talked of hiding was not in real temperatures, but in the tree ring data which was misleading, Mann explained.

Sometimes the data didn't line up as perfectly as scientists wanted.

Re:Or you can edit your data.... (0)

Anonymous Coward | more than 4 years ago | (#30602604)

no evidence of massaged or ignored data

Well, not really. Quoting from the article you linked, they used tree rings when it supported AGW and ignore them when it didn't:

The "trick" that Jones said he was borrowing from Mann was to add the real temperatures, not what the tree rings showed. And the decline he talked of hiding was not in real temperatures, but in the tree ring data which was misleading, Mann explained.

Sometimes the data didn't line up as perfectly as scientists wanted.

Well, technically the GP post is correct.

That's neither "massaged" or "ignored".

However, "buried" would be an accurate characterization. Although I don't think that GP poster realized that when he fell for the cleverly-worded handwaving.

Re:Or you can edit your data.... (0)

Anonymous Coward | more than 4 years ago | (#30602386)

You mean besides how NASA's big charts of the world's average annual temperature from the past 100 years look vastly different now from how they did five years ago? Even if it's just being fixed due to error correction, it obviously isn't stable enough yet to draw conclusions from.

Re:Or you can edit your data.... (3, Insightful)

Rising Ape (1620461) | more than 4 years ago | (#30602002)

By "almost as well" I assume you mean "all the time". The "sceptic" arguments are nothing but a parade of cherry picking with little attempt at genuine investigation.

And there's no real evidence of the proper scientists massaging or ignoring anything. Just because a detailed, written account of everything doesn't exist in stolen, incomplete private documents doesn't mean it doesn't exist at all.

Re:Or you can edit your data.... (3, Informative)

J Story (30227) | more than 4 years ago | (#30602298)

And there's no real evidence of the proper scientists massaging or ignoring anything. Just because a detailed, written account of everything doesn't exist in stolen, incomplete private documents doesn't mean it doesn't exist at all.

The behaviour surrounding the data is certainly indicative of a lack of confidence in the findings. Refusing FOI requests and claiming that "the dog ate it" do not show a group filled with the belief that their research is unassailable.

Better than discarding data (0)

Anonymous Coward | more than 4 years ago | (#30601902)

"It wasn't uncommon for someone to spend a month on a project and then just discard all their data because the data didn't make sense."

No need to discard perfectly good data when all you need to do is adjust it a little. Don't they know about Mike’s Nature trick [wattsupwiththat.com] ?

The problem... (maybe?) (4, Insightful)

pieisgood (841871) | more than 4 years ago | (#30601988)

I can't help but think that Neuroscience needs to calm down, sit back, and take a deep breath. We are examining a system and we are trying to reverse engineer it. We can't start out by trying to create elaborate hypothesis for large systems, we need to go low level and examine the simpler systems. I really think they should hold on to the higher cognitive models for a later time because we can't even completely model C. Elegans and it has the least neurons of any, current, living organism. The way I see it, I total expect their hypothesis to be wrong, because they don't thoroughly understand the low end of the system.

posting to undo bad mod (-1, Offtopic)

chris mazuc (8017) | more than 4 years ago | (#30601990)

...

What a good article (1)

zogger (617870) | more than 4 years ago | (#30601992)

It came late in the year but I would nominate this linked article as the best of slashdot, 2009.

Integrity? (0)

Anonymous Coward | more than 4 years ago | (#30602028)

Throw out data or Reform Hypotheses should not be the first step when you collect data that doesn't make sense. This is the last step, not the first.
1)Check your instruments.
2)Check them again.
3)Check the calibration of said instruments.
4)Check system again.
5)Check the software, what assumptions are being made in the software? Are there misplaced decimals? Are your variable selections all appropriately classed/typed?
If you complete this checklist without coming across any failures, RECOLLECT DATA.
Compare new data with old data. What are your applicable/appropriate error measurements?
Perform steps 1 - 5 again.
Your sets of data should help you decide whether it is your hypotheses, or first data sets that need to be thrown out.

If you have been hired to prove something that is mostly false: gut check time. Are you a scientist or a really smart P.R. hack?

Anonymous Coward (2, Insightful)

Anonymous Coward | more than 4 years ago | (#30602034)

As a researcher myself, I certainly hope they don't throw out data too often. There is occasion to do so...sometimes, when trying to establish correlations (admittedly the weakest form of describing a phenomenon, etc), you learn that there is not one. There are times you obtain data that simply says, "These two phenomenon do not strongly affect each other" or "Something we do not know about or have not accounted for is happening all over this mess."

This data could be kept forever in the unlikely event it will prove useful, especially if there is something else going on...could be as simple as a RF/EM noise (which actually happened to a coworker of mine, though I helped to figure out the issue and make alterations to block/filter this noise out.) In previous years, data storage was sometimes at a premium, although lately this is not an issue as HDD climb to extraordinary capacities (until that capacity becomes the norm, then it is merely ordinary.) My point is that rejected or discarded data, at least in my experience, is due to situations such as these.

Things such as "massaging" or ignoring data are not only horribly bad scientific practice, they are a tremendous drag on humanity's progress...you usually learn through failure, but we are led away from the truth by practices such as those.

Bugs (5, Insightful)

graft (556969) | more than 4 years ago | (#30602208)

If the data doesn't fit your theory, the problem is most likely neither with the data (which is fine) nor with your theory (which may also be fine) but with the method you used to produce your data. You probably wired in an incorrect resistor, forgot to close a parenthesis in your Perl code, forgot to add the correct amount of EDTA to your reaction, etc. Then your results ended up looking like shit, and not surprisingly. Doing science is hard.

There's no need to postulate any grand conspiracies or take pot-shots at science in general. This paper is examining real people doing real shit. Most of the time we fuck up, and we're not smart enough to figure out where we made the error.

Exceedingly common (2, Interesting)

pigwiggle (882643) | more than 4 years ago | (#30602268)

in my experienced - I'm a physical chemist doing atomic resolution condensed phase computer modeling. It's so common that I am troubled when the first analysis gives the answer I expected. I likely spend more time looking for errors when the answer makes sense the first go through. Really.

Re:Exceedingly common (1)

NeutronCowboy (896098) | more than 4 years ago | (#30602512)

in my experienced - I'm a physical chemist doing atomic resolution condensed phase computer modeling. It's so common that I am troubled when the first analysis gives the answer I expected. I likely spend more time looking for errors when the answer makes sense the first go through. Really.

This. Getting everything right the first time is like winning the lottery - you don't believe it, and you shouldn't. People doing experiments is a messy thing. Isolating variables is difficult, and much more difficult than just making something happen.

Load More Comments
Slashdot Login

Need an Account?

Forgot your password?
or Connect with...

Don't worry, we never post anything without your permission.

Submission Text Formatting Tips

We support a small subset of HTML, namely these tags:

  • b
  • i
  • p
  • br
  • a
  • ol
  • ul
  • li
  • dl
  • dt
  • dd
  • em
  • strong
  • tt
  • blockquote
  • div
  • quote
  • ecode

"ecode" can be used for code snippets, for example:

<ecode>    while(1) { do_something(); } </ecode>