Beta

×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

Psychologists Strike a Blow For Reproducibility

Soulskill posted about 8 months ago | from the science-is-self-correcting dept.

Science 138

ananyo writes "Science has a much publicized reproducibility problem. Many experiments seem to be failing a key test of science — that they can be independently verified by another lab. But now 36 research groups have struck a blow for reproducibility, by successfully reproducing the results of 10 out of 13 past experiments in psychology. Even so, the Many Labs Replication Project found that the outcome of one experiment was only weakly supported and they could not replicate two of the experiments at all."

cancel ×

138 comments

Sorry! There are no comments related to the filter you selected.

Good news, everyone! (2, Interesting)

Anonymous Coward | about 8 months ago | (#45534537)

We ought to be encouraging this sort of thing.

Re:Good news, everyone! (-1)

Anonymous Coward | about 8 months ago | (#45535067)

my ass pussy itches. Confucious say: man who go to sleep with itchy asshole wake up with stinky finger. Any ideas?

Re:Good news, everyone! (1)

i kan reed (749298) | about 8 months ago | (#45537999)

Yes, that's a good point. What measures do you think "we," which presumably refers to mild-to-moderately informed laypeople, can put into place to help?

Not trying to be sarcastic, I genuinely wonder what practical options exist.

Psychology (5, Funny)

Anonymous Coward | about 8 months ago | (#45534539)

If only Psychology was a science.

Re:Psychology (0)

turkeydance (1266624) | about 8 months ago | (#45534559)

is to.

Re:Psychology (0)

oldhack (1037484) | about 8 months ago | (#45534743)

Nuh-uh.

Re:Psychology (0)

Anonymous Coward | about 8 months ago | (#45535263)

If only Psychology was a science.

is to.

Nuh-uh.

as "the Riemann hypothesis" is to "Your mom and a bowl of potatoes"?

It operates the scientific method (1)

Anonymous Coward | about 8 months ago | (#45536413)

But isn't really able to hold on to much of the scientific method because there's so much we don't know.

Alchemy isn't a science, but it led to chemistry which was.

Astrology isn't a science, but it led to astrophysics which was.

Because the methods Alchemy and Astrology used to get BETTER predictions was the nascent scientific method and keeping what worked and ejecting what didn't was part of that method (which a lot of social science doesn't do, worse luck). And the remains were the germs of the science that later came forward to replace it.

Re:Psychology (0)

I'm New Around Here (1154723) | about 8 months ago | (#45534603)

Yes, Dr. Cooper. We agree with you. But at least they weren't liberal arts majors.

Re:Psychology (-1)

Anonymous Coward | about 8 months ago | (#45534841)

If only Psychology was a science.

Lol -- psychiatrists and psychologist doing experiments. It's is a weak science at its best; a modern day priesthood at its worst.

Re:Psychology (5, Funny)

FatdogHaiku (978357) | about 8 months ago | (#45535001)

If only Psychology was a science.

Lol -- psychiatrists and psychologist doing experiments. It's is a weak science at its best; a modern day priesthood at its worst.

For your heresy and disbelief you have been diagnosed as a paranoid schizophrenic!
We will monitor you to see if medication will be adequate to silence you, I mean, control your symptoms...

Re:Psychology (1)

Anonymous Coward | about 8 months ago | (#45536425)

it was is easy to reproduce , they just made it up the same way they made up the the first result. psychology you'd have to be out of your mind to believe it.

Re:Psychology (1)

Anonymous Coward | about 8 months ago | (#45535049)

How do you feel about political science or economics?

Re:Psychology (3, Funny)

Anonymous Coward | about 8 months ago | (#45535777)

Angry.

Mostly angry.

Re:Psychology (1)

Anonymous Coward | about 8 months ago | (#45535831)

How do you feel

I see what you did there.

Re:Psychology (5, Interesting)

Michael Woodhams (112247) | about 8 months ago | (#45535053)

Psychology is a huge field. Perception, experimental analysis of animal behaviour, clinical psychology, cognitive biases etc. etc. (Note that only one of those involves psychiatrists.) Some bits allow for harder science than other bits.

I personally don't know enough about psychiatry to form a judgement on how scientific they are, but unlike you, at least I know what a psychologist is (or something of the range that they could be.) Your trite dismissal says much about your ignorance and nothing about psychology.

Re:Psychology (-1)

Anonymous Coward | about 8 months ago | (#45535283)

Your trite dismissal says much about your ignorance and nothing about psychology.

Calm down Touchy, it wasn't meant to be a factual dissertation.

Your ivory tower sniff redefining opinionated rhetoric as ignorance says much about the politicization of these two disparate medical fields and their lack of interest in keeping the less virtuous in their field from finding and abusing power due to the stronger sociological nature of one of those professions (William James aside).

Re:Psychology (0)

Anonymous Coward | about 8 months ago | (#45535791)

Psychiatry is a branch of medicine, and psychiatrist is (I believe) a protected term, so it can't be legally used by anyone without a medical degree.

Expect them to be as scientific as most medics - better than the general population, but most of them probably don't have formal research training.

Re:Psychology (4, Interesting)

steelfood (895457) | about 8 months ago | (#45535961)

Psychology is a soft science because of the numerous variables that in studies are often simplified into a constant often for simplicity's sake and nothing else. Economics and politics are the same, mostly because they're based on psychology.

It's an inexact science because the human condition is imperfect. As opposed to the hard sciences, which are exact, because the universe around us is "perfect". And then, there's computer science, which is a mathematical, computational science that's absolute. It's not even "perfect" anymore; it's exactly what the maths say it is, and any failure sits between keyboard and chair.

Anyway, psychology is important, because the only way to truly understand the imperfect conditions of humans is via an inexact science. And it's something only fully understood by humans (computers can simulate the hard sciences to a calculable degree of accuracy, but they'll never be able to simulate the soft sciences in the same way), and innately at that.

The way to think about psychology is using fractals. X% | X is > statistical significance, of the population behaves in manner a. X * (100-X)% of the population behaves in manner b. X * (100 - X * (100-X))% of the population behaves in manner c. Etc. a, b, c, etc. are up to you to figure out. And when you change the test, the individual that falls into one category is not guaranteed to fall into the same category again.

Note that the human mind can comprehend infinity (poorly for most, but very possible for a few), both countable and uncountable variants, but a computer will never be able to calculate it. So the fractal analogy works really, really well.

Re: Psychology (0)

MickLinux (579158) | about 8 months ago | (#45536813)

The universe only appears to be perfect to macroscopic viewers, because the time dimensions are held comparatively constant due to between small particles.

In other words, and electron -- which may well "see" 1 spatial dimension and three time dimensions, appears to behave statistically, not predictably. Therefore, quantum mechanics behaves like an imperfect science, as you call it.

That said, psychology, economics, and sociology attempt to make their science more perfect, often, by binding large numbers of subjects together, to eliminate uncontrollable variables and discover the elementary laws. Yes, they are behind natural philosophy and alchemy in that, but only because it takes much more energy and time to study large numbers of people.

Re:Psychology (2)

VortexCortex (1117377) | about 8 months ago | (#45536739)

Psychology is a huge field. Perception, experimental analysis of animal behaviour, clinical psychology, cognitive biases etc. etc.

No, the field you're thinking of is Neuroscience and Cybernetics -- These have evidence based on observation and models which have predictive power. Psychology is just confirmation bias. [wikipedia.org] You must prove the null hypothesis more implausible than the original hypothesis, yet Psychology does not do this. For every ridiculous Sexual Epistemology, there's an equally valid Scatological Epistemology.

The truth is that neurons fire in brains, and that complexity gives rise to emergent behaviours. Leaping the gulf in understanding to arrive at the explanations that Psychology and Philosophy give is akin to claiming a God in a Chariot pulls the Sun across the sky.

An apt analogy (1)

PPalmgren (1009823) | about 8 months ago | (#45537291)

Good post Woodhams, I'll use an analogy I formed when discussing Psychology with my girlfriend whose been in the field a while: Psychology today is like studying Chemistry in the bronze age. Back then, they didn't have the means to understand the why of this chemical working with this chemical, they just knew it worked and did Chemistry via trial and error and guessing. Today, psychology is classifying things based on relations and forming best practices, but we don't understand why things are the way they are because of our limited understanding of the brain.

Maybe things will change in 100 years, maybe not. I think the field is worth its weight in gold though, there's a lot of good that can be/is being done and a lot of progress still to be made.

Re:Psychology (5, Informative)

Shavano (2541114) | about 8 months ago | (#45534875)

If the experiments are reproducible, it's science.

Apparently it's biochemistry that is not a science.
http://online.wsj.com/news/articles/SB10001424052970203764804577059841672541590 [wsj.com]

Re:Psychology (5, Informative)

rtb61 (674572) | about 8 months ago | (#45535853)

Psychology ran in a major hiccup, as many of it's experiments are no longer reproducible not because of bad 'science' but because they are considered naughty and not something that should really be done to people to test out psychological theories, as in http://www.bps.org.uk/what-we-do/ethics-standards/ethics-standards [bps.org.uk] (I used British standards rather than US ones, as the US ones have so badly been mauled by the US government and their fully medically and psychological researched mass torture facility at GITMO that the US ones are rules that 'should be' broken as defined by the US government) and http://mentalfloss.com/article/52787/10-famous-psychological-experiments-could-never-happen-today [mentalfloss.com] .

Huge mistake here (0)

Anonymous Coward | about 8 months ago | (#45537065)

reproducibility is the ability to be reproduceable. It may not be *reproduced*, but it is reproducible. Contrasted that with a lot of result in psychology being ad-hoc, and having no reproducibility, in other word, you cannot ever reproduce the results no matter how you would try.

The article you linked does not say anything about reproducibility, it only state experiment were not reproduced. The problem of psychology is that it has a lot of stuff which has zero reproducibility !

Re:Psychology (0)

Anonymous Coward | about 8 months ago | (#45537625)

Sometimes even reproducibility does not help:

In Retrospective gambler’s fallacy they ask to estimate number of throws before 3 sixes and 2 sixes and 1 three.

This is a very poor choice of experiment.

1. requires assumption from people how this was happenning
2. requires assumption whether people do understand 6,6,6 is 3 times less likely than 6,6,3

It is either non-science or a very, very bad one.

Re:Psychology (5, Interesting)

Lamps (2770487) | about 8 months ago | (#45535027)

The ironic thing about statements like these is that they usually come from people with no scientific training in any field, nor any meaningful training in statistics, but only a "sciency" inclination and questionable, popular distillation-derived knowledge of some principles from what they consider "the hard sciences".

Sadly, this irony will be lost on the people making such statements, who will, for some unfathomable reason, continue to disparage people doing meaningful work in the sciences, while never coming close to accomplishing anything of the sort themselves.

Actual academics have an idea of the hard work involved in contributing to the human knowledge base in all scientific disciplines, and thus, tend to respect each other's work (as long as others don't step on their own toes in their particular area of specialization, in which case, prepare for turbulence).

Re:Psychology (-1, Troll)

oldhack (1037484) | about 8 months ago | (#45535063)

The ironic thing about statements like these is that they usually come from people with no scientific training in any field, nor any meaningful training in statistics...

See, spewing bogus claims like this is why people come to see you as the fraud that you are.

Re:Psychology (0)

Lamps (2770487) | about 8 months ago | (#45535243)

The ironic thing about statements like these is that they usually come from people with no scientific training in any field, nor any meaningful training in statistics...

See, spewing bogus claims like this is why people come to see you as the fraud that you are.

Wow - you seem to have a profound knowledge of me. Should I check the bushes for a creepy basement-dwelling type with a set of binoculars?

Either way, you might want to try to satisfy your desperate craving for any sort of human contact through some other activity than trolling.

Re:Psychology (1)

Anonymous Coward | about 8 months ago | (#45535093)

"popular distillation-derived knowledge "

That dash implies the knowledge comes from a crowd at a bar, is this correct?

A quick question (1, Insightful)

Okian Warrior (537106) | about 8 months ago | (#45535535)

The original model held that psychotherapy could cure depression. Talk to your analyst once a week and after years of treatment you got better.

Then it was discovered that low norepinephrine caused depression, and tricyclics fixed that and cured depression.

Then it was discovered that low serotonin caused depression, and SSRIs fixed that and cured depression.

Then it was discovered that low dopamine caused depression, and MAOIs fixed that and cured depression.

And recently, the The New England Journal of Medicine reported depression meds have no effect [huffingtonpost.com] .

One last question... just one*.

Is psychology evidence-driven, or belief-driven?

(*) This isn't just me asking. Here's a quote from the The New England Journal of Medicine [nejm.org] article:

Evidence-based medicine is valuable to the extent that the evidence base is complete and unbiased. Selective publication of clinical trials — and the outcomes within those trials — can lead to unrealistic estimates of drug effectiveness and alter the apparent risk–benefit ratio.

(**) Also, I have no meaningful training in science or statistics. If you want, you can win the argument by pointing this out in your response.

Re:A quick question (2)

dabadab (126782) | about 8 months ago | (#45535825)

And recently, the The New England Journal of Medicine reported depression meds have no effect.

That's patently untrue. The Huffingtonpost article you link to is wildly inaccurate, self-contradictory and much more about sensationalism than the actual NEJoM article.

Re:A quick question (0)

Okian Warrior (537106) | about 8 months ago | (#45535917)

And recently, the The New England Journal of Medicine reported depression meds have no effect.

That's patently untrue. The Huffingtonpost article you link to is wildly inaccurate, self-contradictory and much more about sensationalism than the actual NEJoM article.

I'm just pointing out what doctors are saying. If they are wrong, please tell us what the study actually means. In particular, this line from the NEJM study:

According to the FDA, 38 of the 74 registered studies had positive results (51%; 95% CI, 39 to 63)

Doctors, including the one in the linked HufPo article, are stating specifically that antidepressants don't work base don this study. In so many words, directly from the article.

Since doctors are wrong, presumably because they don't have meaningful training in science or statistics, who then should be the go-to expert to interpret these findings?

Re:A quick question (0)

Anonymous Coward | about 8 months ago | (#45535977)

Not only do you not have any meaningful training, your logic is also flawed. That you find what appears to you one example of flawed science does not invalidate all of modern psychology. Sorry.

Re:A quick question (1)

maxwell demon (590494) | about 8 months ago | (#45536039)

If about half of the studies showed a positive effect, then it is hardly a proof that there is no effect. It may not be sufficient to show that it has an effect, but it's a clear hint that it might have. To make better statements one would have to have a closer look at the studies in question, because not every study has the same quality. In the extreme case, all of the studies showing a positive effect might be flawed, while those showing no effect might be sound (in which case, the claim that there's no positive effect would be justified). In the opposite extreme case, all the studies showing no effect are flawed, and all the studies showing an effect are fine (in which case the effect would be proven). But even if no single of the studies is flawed, the different studies most likely have different significance levels which have to be used to weight those studies. A highly significant result is more important than less significant results.

Whom to ask? Well, at first, the authors of the study, because they should know best what they wrote. Then, some well-published researchers who already demonstrated that they know their shit.

A doctor is specialized in practising medicine. I'd also not go to an experimental physicist to let me explain anything non-basic about a physics theory, or to a theoretical physicist for an explanation of the finer points of an experiment (or to an engineer for both; he would, however, be the one to ask about the details of some engine, or about engineering practice). Which doesn't mean that there are none of them who could give a good answer. But unless you've got independent evidence that you would, you better ask those who are doing exactly what your question is about.

Re:A quick question (3, Informative)

steelfood (895457) | about 8 months ago | (#45535993)

Medication belongs to the field of psychiatry. And for the most part, they do have an effect. But it's only temporary, and the human body gets used to it after a while. So in the long term, medicine is largely useless, and in fact is counterproductive, as they tend to cause other, worse effects ("side" effects). But in the short term, it helps.

All systems have a state of equilibrium, a state of stability. The same holds for the body and the mind, two different but related and dependent systems. They'll always tend towards the state of equilibrium because that's the path of least resistance.

Psychological ills are not the equilibrium being tipped, but the point of equilibrium itself changing. To truly "cure" someone of depression or OCD or bipolar, you have to change the point of equilibrium itself. That's much, much harder than you can imagine, and a far greater challenge than any pill will ever resolve. Those whose equilibrium were changed by an event in their life are easier to change back than those who are born with a certain equilibrium. Some people call the former nature vs. nurture. I call it, again, the past of least resistance.

Psychology is not attempting to medicate everyone. It's attempting to explain in terms familiar to the scientific-minded humanity.

Re:A quick question (1)

rvw (755107) | about 8 months ago | (#45536233)

One last question... just one*.

Is psychology evidence-driven, or belief-driven?

(*) This isn't just me asking. Here's a quote from the The New England Journal of Medicine [nejm.org] article:

Evidence-based medicine is valuable to the extent that the evidence base is complete and unbiased. Selective publication of clinical trials — and the outcomes within those trials — can lead to unrealistic estimates of drug effectiveness and alter the apparent risk–benefit ratio.

(**) Also, I have no meaningful training in science or statistics. If you want, you can win the argument by pointing this out in your response.

Read this book: Why zebras don't get ulcers [amazon.com] . It explains how stress influences our life, and how complex the system works that tries to regulate this. It shows that it all works beautifully, for people living in the wild, but the system is not so good for us.

This book won't give you the solution to depression, but it will show that the body uses many methods to accomplish several things, like redirecting sources when in danger or in rest. There is not one solution - any solution will have side effects, and this goes for the solutions of the body as well.

Re:A quick question (0)

Anonymous Coward | about 8 months ago | (#45537205)

Ironic given that ulcers have been found to be caused by bacteria and can be cleared up with a course of antibiotics. They're not caused by stress or depression.That was an old belief.

http://en.wikipedia.org/wiki/Helicobacter_pylori

Re:Psychology (0)

phantomfive (622387) | about 8 months ago | (#45535609)

Without a doubt, there is a LOT of unscientificness going on in the field of psychology. Look at the satanic ritual abuse situation of the 80s [wikipedia.org] for an obvious example, especially when you realize that even today there are still psychologists treating patients for this 'malady.'

Another example is the DSM, which is a joke from a scientific perspective [wikipedia.org] (though this is perhaps an indication of the difficulty of the field of psychology as much as anything).

Re:Psychology (0)

Anonymous Coward | about 8 months ago | (#45535965)

The ironic thing about statements like these is that they usually come from people with no scientific training in any field, nor any meaningful training in statistics, but only a "sciency" inclination and questionable, popular distillation-derived knowledge of some principles from what they consider "the hard sciences".

Could you please show me a reasonable experiment with proper statistics supporting this claim?

Re:Psychology (1)

rvw (755107) | about 8 months ago | (#45536967)

The ironic thing about statements like these is that they usually come from people with no scientific training in any field, nor any meaningful training in statistics, but only a "sciency" inclination and questionable, popular distillation-derived knowledge of some principles from what they consider "the hard sciences".

Could you please show me a reasonable experiment with proper statistics supporting this claim?

Well to be honest - we have to confess something to you. Slashdot was once setup as a psychological experiment.... Just take a look at the users and comments, no more proof needed!

Re:Psychology (0)

Anonymous Coward | about 8 months ago | (#45536253)

they usually come from people with

Ad hominem based on assumptive pretext! That's two drinks in the Slashdot drinking game. Slow down cowboy!

Re:Psychology (0)

Anonymous Coward | about 8 months ago | (#45536397)

Then again there are people like me. Out of the 4 psychologists I personally know one is neurotic, one is constantly murmuring to himself and stalked a woman in our department until she officially pressed charges against him, one is considered completely bonkers by anyone who has ever met him and women are leaving his talks because he's pathologically mysogynic, and only one seems to be halfway normal (if not a bit autistic).

Anecdotical evidence, of course, but it has casted some personal doubts about psychology in me. The two most crazy ones do not even recognize that people conceive them as such. One of them is working as a psychiatrist to evaluate criminals (murderers, etc.) who have no means of complaining about his assessments, drives a Porsche and is an active member of a far right-wing party.

Re:Psychology (0)

Anonymous Coward | about 8 months ago | (#45537159)

Except economics, it's shit.

Re:Psychology (2)

antifoidulus (807088) | about 8 months ago | (#45535121)

Unfortunately there are 2 major hurdles that limit all but the most groundbreaking experiments:

1. Money
2. Glory

Money is probably the biggest factor, there just isn't enough money allotted to trying to reproduce experiments. Most budgets only exist for new/continuing research, not verifying experiments done by others. And as the cost of doing experiments rises(more sophisticated equipment necessary, lots of paid "volunteers" etc) this is only going to get worse.

Second, although not as important, is the "glory" factor. Very few talented scientists want to spend their time and research money on reproducing experiments done by others. There aren't a whole lot of publishing opportunities in doing so, esp. if you cannot refute what they have done. You can see this to a certain extent in the open source world as well, for all but the most famous of projects you tend to have a very large # of projects that essentially do the same thing. Why? Because a lot of people want to "invent" a new program rather than improve upon what is there. Fortunately in the OSS world money isn't nearly as much an issue, so you do have large #s of people improving OSS rather than trying to re-invent the wheel.

Re:Psychology (0)

blue trane (110704) | about 8 months ago | (#45535505)

There's no shortage of money. From http://www.bis.org/publ/otc_hy1311.pdf [bis.org] :

The latest BIS statistics on OTC derivatives markets show that notional amounts outstanding totalled $693 trillion at end-June 2013.

In the face of so much global liquidity, it's ridiculous to say there's not enough money for food stamps, or science, or free health care.

Re:Psychology (0)

khallow (566160) | about 8 months ago | (#45535667)

Notional amounts of derivative securities have nothing to do with money.

For example, I recently traded some stock options. The actual value was around $600 at the time of the sale. The notional amount, which happens to be the stock's price trigger at which the options activate times the number of options, was $13,000. You do the math.

Re:Psychology (0)

blue trane (110704) | about 8 months ago | (#45536097)

Even if you reduce it by an order of magnitude, that's still $60 trillion. And estimates of the shadow banking system [slashdot.org] are $100 trillion.

There's no shortage of money. The only scarcity is the lack of political will to fund basic social services and science. There is no economic or physical necessity preventing us from funding food stamps, science research, health care, a basic income.

Re:Psychology (0)

Anonymous Coward | about 8 months ago | (#45535813)

You'd take away people's BMWs? Think of the children!

Re:Psychology (1)

blue trane (110704) | about 8 months ago | (#45536017)

No, create more. Let the Fed fund the govt by buying T-bills. Since the Fed returns the interest to the Treasury, the borrowing cost is zero.

Give Republicans low (or no) taxes, but in return have them eliminate the debt ceiling laws.

Re:Psychology (1)

bakes (87194) | about 8 months ago | (#45535661)

You forgot 'Politics'. Not sure where that fits in the ordering though.

Re:Psychology (1)

VortexCortex (1117377) | about 8 months ago | (#45536869)

If only Psychology was a science.

The sad thing is that it's on par with the level of TV Tropes. They first use their pattern matching brains to notice some pattern, then go seek out quantifications for it. That's ridiculous. That's why literally everything is a trope -- even the tropeless story is a trope. The same goes for psychological classifications and categorizations of behaviours. Some psychologists claim to study cognitive bias -- Yet their own confirmation bias has blinded them to the fact that their distinctions themselves were biased to begin with.

The answer is to start at the bottom -- Understand through observation the mechanisms of neurobiology. Model them through cybernetics. Verify them mathematically, experimentally, and link the behaviours to the physical world, instead of guessing from the top down through the fog of their own cognition.

The primary problems humans face is that their minds are too small, and they don't live long enough. They must specialize if they are to become experts in an area of study before their lives run out. Contrast this with the real world, existing for billions of years, timeless on human scales, and the fact that there are no divisions in sciences. The quantum physics of spin giving predictive power to macroscopic magnetism is linked to human behaviour via eddy currents in their electrochemical neural network brains.

The sciences remain separated quite foolishly. Eg: The Philosopher should start with Information Theory and Entropy and Cybernetics and build their epistemology on mathematically provable principals and cybernetic models with entropic parameters set by observations in quantum physics. THEN you they can say what knowing means. Instead they get continually tripped up by one aspect of a self reflecting cybernetic being: Their data could be tampered with. Yes, indeed, troves are dedicated to the simple subject, and still they have not done as I propose and utilize information theory and observations of thermodynamics to quantify the apparent degree of which knowing can exist. Fools.

One must make due with the planet one finds oneself on. Taking cognitive limits into concern they need not specialize deeply in each field, but no Psychiatrist or Philosopher or Psychologist should be ignorant of the governing field of Cybernetics -- Especially not the cybernetically provable social dynamics that result from information disparity: The Mathematics of Secrets. I would venture to say they should be able to explain the uncertainty principal as well -- since it can be directly applied to perception.

Re:Psychology (2)

rtaylor (70602) | about 8 months ago | (#45537411)

Science is a process, not a field of study or a result.

That process can be applied to anything where you want to find a fact.

Re:Psychology (1)

LearningHard (612455) | about 8 months ago | (#45538011)

That is a very disingenuous statement. While I would agree that there are many aspects of Psychology/Psychiatry that are not very scientific, there are some areas that are rigorous. Neuropsychology can be a good example. Testing and measurement is another good example. There is a lot more to Psychology than the hokey therapy that we think of.

Re:Psychology (1)

wisnoskij (1206448) | about 8 months ago | (#45538241)

How is this rated Funny? Psychology is not a Science.

Not bad at all (4, Interesting)

TwineLogic (1679802) | about 8 months ago | (#45534549)

So 2 or 3 out of 13 were not reproduced in these attempts. I imagine the standard was "P 0.05." If you then consider ANOVA, the collection of 13 studies did not perform poorly at all.

Re:Not bad at all (0)

sporting go4 (3445497) | about 8 months ago | (#45534775)

yes I agree

Re:Not bad at all (4, Insightful)

Anonymous Coward | about 8 months ago | (#45534805)

The fact that people are trying to reproduce the experiments is good news in and of itself.

Re:Not bad at all (1)

sandytaru (1158959) | about 8 months ago | (#45534835)

I agree. Science goes through the upgrade of hypothesis, to tested results, to verified results, to working theories, and eventually laws (although the line between the last two is arbitrary in modern science, really.) The more results that are tested again and again, the better science is as a whole.

Re:Not bad at all (5, Insightful)

bondsbw (888959) | about 8 months ago | (#45534885)

If the scientific community valued reproducibility as much as original work, we would solve 2 problems:

1) Science without confirmation can lead us astray for years.
2) There are plenty of scientists who a great at experimentation but lousy at coming up with new ideas, and these scientists (or potential scientists) may not be finding their full potential.

And while we're at it, let's value failed experiments as much as successful experiments.

Re:Not bad at all (1)

drinkypoo (153816) | about 8 months ago | (#45537367)

1) Science without confirmation can lead us astray for years.

It's not science if it's not tested. It's just making assumptions.

It's also not science if you just take their word for it when they said they tested it. It might have been originally, but not any more.

Re:Not bad at all (0)

Anonymous Coward | about 8 months ago | (#45535213)

And why do you think a p value is related to the probability a study will be replicated?

Re:Not bad at all (0)

Anonymous Coward | about 8 months ago | (#45535601)

radiolab had an awesome episode about a study that the first few time they did the experiment it worked, but as they performed it on more groups of people over time it worked less and less..

Re:Not bad at all (4, Interesting)

phantomfive (622387) | about 8 months ago | (#45535639)

No, unfortunately, because they didn't choose the studies to reproduce randomly. FTA:

[The studies chosen for reproduction] included classic results from economics Nobel laureate and psychologist Daniel Kahneman at Princeton University in New Jersey

At least some of these were fairly important research, which ideally would have been verified more than once. That there was any doubt that they would be reproducible is worrisome in itself.

Re:Not bad at all (1)

fatphil (181876) | about 8 months ago | (#45536639)

That 20% failed to be reproduceable is just as worrying.,

Re:Not bad at all (2)

toQDuj (806112) | about 8 months ago | (#45536851)

This is very important, as they did not randomly pick studies but rather chose the ones they "Deemed Worthy". As they did not want to be proven bad scientists (I assume), their conscious or unconscious bias will have been towards sound or easy studies.

Re:Not bad at all (0)

Anonymous Coward | about 8 months ago | (#45536369)

Sorry, what? The probability of getting 3 or more unreproducible with p(unreproducible)=0.05 is 0.003. The probability of getting 2 or more unreproducible is 0.025. I reject the hypothesis that p(unreproducible) is 0.05, at a confidence level of P=0.03.

Clearly (1, Funny)

msobkow (48369) | about 8 months ago | (#45534579)

Scientists must have *more sex* if they are to reproduce... :P

Re:Clearly (0)

sporting go4 (3445497) | about 8 months ago | (#45534739)

LOL you are clever too

It's hopeless. (1)

Confusedent (1913038) | about 8 months ago | (#45534791)

I've been barking up that tree for years.

Re:It's hopeless. (0)

Anonymous Coward | about 8 months ago | (#45534809)

Not even dogs attract mates that way.

Science has a reproducibility problem? (2)

TheloniousToady (3343045) | about 8 months ago | (#45534783)

I'll believe it when I see it.

Re:Science has a reproducibility problem? (1)

Anonymous Coward | about 8 months ago | (#45535785)

And then see it again.

The problem isn't necessarily reproducibility (4, Interesting)

Dahamma (304068) | about 8 months ago | (#45534797)

The "problem" with experiments that aren't reproducible may not be with the experiments as much as with the popular media that decides to make sweeping generalizations based on one result. Though I guess some blame definitely needs to be applied to the researcher who allows unverified results to be misrepresented to get that 15 minutes of fame in a quote in The Guardian or USA Today...

Re:The problem isn't necessarily reproducibility (2)

Impy the Impiuos Imp (442658) | about 8 months ago | (#45534899)

The whole point of an experiment is to run enough trials to gain statistical confidence. It's supposed to be it's own validation in that sense.

So it's either a systematic error in their experiment, or fraud.

Re:The problem isn't necessarily reproducibility (0)

Anonymous Coward | about 8 months ago | (#45535123)

The whole point of apostrophes is to either mark a contraction or a possessive of a noun. Possessive pronouns are already possessive in that sense. So it's either a systematic error in your post, or ignorance.

Re:The problem isn't necessarily reproducibility (1, Funny)

Oligonicella (659917) | about 8 months ago | (#45535209)

The whole point of karking on a single misplaced apostrophe is to elevate yourself with pomposity.

Re:The problem isn't necessarily reproducibility (1)

fatphil (181876) | about 8 months ago | (#45536669)

Something passed through our hands here at the office recently. A "scientific" study, in the very soft field of human behaviour, where their sample size was 27. And that set was split into 4 groups. Absolutely any result from that experiment was possible and could be explained as pure random chance, not deviating from the null hypothesis.

Reading works like that, and others from the same authors and institutes, we got the feeling that it was mostly a delusion that they were doing science. Like how when a 4-year-old is given a tub of water, some washing up liquid, and various containers and vessels - there's a delusion that it's actually doing the washing up, and you really don't want to spoil its fun by breaking that illusion - it's not doing any harm, is it?

Re:The problem isn't necessarily reproducibility (1)

drinkypoo (153816) | about 8 months ago | (#45537383)

there's a delusion that it's actually doing the washing up, and you really don't want to spoil its fun by breaking that illusion - it's not doing any harm, is it?

The harm comes when they "dry" the dishes with the dish towel, getting it all nasty and dirty, and then put the dishes away in the cupboard still dirty, contaminating the other dishes they come into contact with. There's a direct analogy to be made here.

Re:The problem isn't necessarily reproducibility (1)

Chris Mattern (191822) | about 8 months ago | (#45538217)

But don't be quick to assume it's fraud, because it's very, very easy to make a systematic error in designing an experiment.

Re:The problem isn't necessarily reproducibility (1)

Lamps (2770487) | about 8 months ago | (#45534927)

That's not really a problem from the perspective of scientists - in the fields of psychology, cog sci, and neuroscience, I've never encountered an instance of a researcher using any popular media distillation of some study as a meaningful source of info on that study (aside from making them aware of the study's existence).

Also, you seem to be assigning some a priori status of reproducibility or lack thereof to some studies, which really confuses the issue. For example, what does it mean for an experiment to be "not reproducible"? You can fail to reproduce the results of an experiment, but proving that a result is not reproducible, or somehow knowing that it isn't, is a different issue altogether.

Re:The problem isn't necessarily reproducibility (1)

Oligonicella (659917) | about 8 months ago | (#45535265)

For example, what does it mean for an experiment to be "not reproducible"?

It means that, as a scientist, you have failed to produce a worthwhile experiment. Hypothesis is first. A road map for reproducibility is next. If said map does not function, you have failed.

Re:The problem isn't necessarily reproducibility (1)

Lamps (2770487) | about 8 months ago | (#45535425)

I guess I should clarify the larger epistemic point at which I was hinting. That others may not, in some reasonable number of attempts, reproduce an experiment does not mean that the experiment is categorically not reproducible. Any number of things, such as lab conditions (which are not, in practical, absolute terms, reproducible within a lab, much less between different labs), can influence the results of experiments, and while adhering to certain sound methodological principles abstracts away a lot of these real-world complications, and while statistics can tell us a certain probability of one thing or another, these are idealizations. There's a distinction between an experiment as an abstraction (i.e. that thing that gets written up in a methods section) and an instance of an experiment (i.e. that thing that gets written up in the results section) - only the latter exists in the tangible world, while the former exists as an idea. Thus, while we can comfortably talk of experiments that haven't been reproduced, it's another thing to casually throw around terms such as "not reproducible". We should be much more cautious about the latter term.

Re:The problem isn't necessarily reproducibility (1)

femtobyte (710429) | about 8 months ago | (#45535545)

Isn't the ability to reproduce results based on the "idea" in the methods section central to the concept of scientific "reproducibility"? If I claim I applied one set of methods and got a certain result --- but those methods are different from the physical reality of my setup that reasonable adherence to the stated methods will produce a vastly different result --- then I have failed at publishing a "reproducible" experiment.

Example:
"By the method of releasing lead spheres at rest into the air, I have observed that lead spheres have a tendency to rise upward (padhering to methods as written, which need to be sufficiently precise to capture all relevant "lab-specific" conditions which would significantly effect the result?

I think if differences between abstract ideal and physical implementation are sufficient to preclude replication of results, then this renders an experiment "not reproducible," rather than "explaining away" the inability of following experimentalists to produce consistent answers.

Re:The problem isn't necessarily reproducibility (1)

femtobyte (710429) | about 8 months ago | (#45535569)

Sorry, I have no idea how Slashdot managed to completely mangle my post, cutting out a big chunk in the middle, after previewing and submitting. I don't have the patience to re-write it, so just ignore the garbled mess left after Slashdot's unexpected redaction of a whole middle paragraph.

Re:The problem isn't necessarily reproducibility (1)

Lamps (2770487) | about 8 months ago | (#45537257)

Don't worry about the post getting cut off - the point you were trying to make is clear. To address the idea that "the methods are different from the physical reality of my setup" - it's simply not possible to be comprehensive in the instructions you provide when you write up your methods, and furthermore, in reality, variables are often introduced in a lab which impact experimental results, but which are not accounted for in the methods writeup because the authors are not aware of these variables (this is known to impact the likelihood that results will be reproduced). Adhering to certain methodological principles may mitigate the latter issue, but you can't be sure that the issue has been resolved.

Consider that it's believed that many older studies in psych, if conducted today, would produce non-significant p-values simply for the reason that over time, the population from which participants will be drawn has become different from the population from which participants were drawn for the original study (the changes may be cultural, but may be also be something that we cannot readily pinpoint). Methods sections cannot be comprehensive in addressing all concerns, and don't account for this sort of thing. Are the original studies failed studies? Should the conclusions of the original studies simply be revised to qualify the generalizability of these studies, and if so, at what point can we be reasonably confident that the conclusions of a study are generally applicable? So that's a commonplace scenario illustrating that the notion of being able to reproduce results is more slippery than we may have thought.

Re:The problem isn't necessarily reproducibility (2)

_Shad0w_ (127912) | about 8 months ago | (#45535163)

If you're going to pick a paper then The Guardian was not a good choice, given Dr Ben Goldacre writes a regular column for the called "Bad Science" where he critiques terrible science reporting in the media (amongst other things).

I read the headline... (3, Funny)

trudyscousin (258684) | about 8 months ago | (#45534965)

...and thought, "Now there's a ray of sunshine for Slashdotters."

As a group, that is.

Re:I read the headline... (1)

steelfood (895457) | about 8 months ago | (#45536007)

Funny, because there's another study recently published that said technology is killing everyone's sex lives. It's probably old news here, as it's just a downward extrapolation of the extreme case found here.

I thought at first... (1)

ysth (1368415) | about 8 months ago | (#45534989)

I thought at first it was saying 36 groups each tried to reproduce the results of 13 experiments, and all 36 were successful with 10 of 13 (though not necessarily the same ten), successfully reproducing the results of a reproducibility meta-experiment.

reply (0)

Anonymous Coward | about 8 months ago | (#45535011)

I also agree with your opinion

Not a science (0)

Anonymous Coward | about 8 months ago | (#45535057)

Okay, so... I'm going to preface this with: I have a degree in Economics (blood type hater) and support many heterodox ideologies as well as much of modern microeconomic theory. I'm a strange duck.

So while I firmly believe psychology is not a science and that's because, I believe, often it's not viewed in a mathematical context. This weakness is generally rectified by behavioral economists which synthesize the results into a more "probable" (I.e. Being able to compute a probability); which, alludes and often precludes a truth (however you want interpret "truth").

In the end it all ends up in the realm of biologists, and neurologists, who discover the true nature (causality, not correlation) that will solve the symptom at hand.

I say symptom because I believe if we better understood how to understand this (the root) these symptoms would fade.

Eh (1)

sociocapitalist (2471722) | about 8 months ago | (#45535775)

So...they couldn't reproduce the reproducibility problem...?

"not been able to reproduce" (1)

bickerdyke (670000) | about 8 months ago | (#45535845)

Were they not able to reproduce the outcome of an experiment, or were they not able to reproduce the whole experiment (as in "We assume that during a total eclipse in the month of may" or... "to reproduce this, take any old Large Hadron Collider lying around...")

Re:"not been able to reproduce" (1)

fatphil (181876) | about 8 months ago | (#45536751)

Apparently, there were no "experiments", in the lab/contrivance sense - it was just a questionaire.

Regarding the failures:
"Of the 13 effects under scrutiny in the latest investigation, one was only weakly supported, and two were not replicated at all. Both irreproducible effects involved social priming. In one of these, people had increased their endorsement of a current social system after being exposed to money[3]. In the other, Americans had espoused more-conservative values after seeing a US flag[4]."

[3] Caruso, E. M., Vohs, K. D., Baxter, B. & Waytz, A. J. Exp. Psych: Gen. 142, 301â€"306 (2013).
[url was a redirect loop for me]

[4] Carter, T. J., Ferguson, M. J. & Hassin, R. R. Psych. Sci. 22, 1011â€"1018 (2011).
http://pss.sagepub.com/content/22/8/1011
"""
A Single Exposure to the American Flag Shifts Support Toward Republicanism up to 8 Months Later

      1. Travis J. Carter[1],
      2. Melissa J. Ferguson[2] and
      3. Ran R. Hassin[3]

      1. Center for Decision Research, Booth School of Business, University of Chicago
      2. Department of Psychology, Cornell University
      3. Department of Psychology and The Center for the Study of Rationality, Hebrew University

Abstract

There is scant evidence that incidental cues in the environment significantly alter peopleâ€(TM)s political judgments and behavior in a durable way. We report that a brief exposure to the American flag led to a shift toward Republican beliefs, attitudes, and voting behavior among both Republican and Democratic participants, despite their overwhelming belief that exposure to the flag would not influence their behavior. In Experiment 1, which was conducted online during the 2008 U.S. presidential election, a single exposure to an American flag resulted in a significant increase in participantsâ€(TM) Republican voting intentions, voting behavior, political beliefs, and implicit and explicit attitudes, with some effects lasting 8 months after the exposure to the prime. In Experiment 2, we replicated the findings more than a year into the current Democratic presidential term. These results constitute the first evidence that nonconscious priming effects from exposure to a national flag can bias the citizenry toward one political party and can have considerable durability.
"""

The Decline Effect (0)

Pino Grigio (2232472) | about 8 months ago | (#45535847)

Is this news? The Decline Effect [newyorker.com] has been known about (and generally ignored) for years.

Reproducibility (0)

Anonymous Coward | about 8 months ago | (#45536393)

Some scientists should not be allowed to reproduce. The problem will take care of itself.

due to incomplete info. (0)

Anonymous Coward | about 8 months ago | (#45537769)

I worked in a bio tech lab for a while. Optimizing reactions is a lengthy and expensive process. Because of that, scientists like to keep at least part of the process secret and not let their competitors know all the minute dteaila required. Papers very rarely conatain all of the nitty grtitty tricks required to reproduce the result.

One of my favorite experiments (1)

erroneus (253617) | about 8 months ago | (#45537881)

I have a particular experiment which, in my mind, highlights an aspect of human nature we would all prefer to deny. We all have within us the capacity to ruthlessly abuse others, even and especially friends and family when given the opportunity. This famous experiment [wikipedia.org] a group of peers where some were assigned the role of prison guard and others that of prisoner. It really didn't take long before things went really bad.

I have often heard that corruption is a problem of opportunity more than of character. I believe it is generally true. I believe this has been common knowledge for centuries if not thousands of years. I believe the reason the US constitution was written as it is was to delay if not entirely prevent certain inevitable human behaviors. Our very nature as humans is our undoing. What enables us to survive beyond our primitive selves is our recognition and understanding of our natures and to inhibit and limit those aspects which are the most harmful.

"The goal of a good society is to structure social relations and institutions so that cooperative and generous impulses are rewarded, while antisocial ones are discouraged. The problem with capitalism is that it best rewards the worst part of us: ruthless, competitive, conniving, opportunistic, acquisitive drives, giving little reward and often much punishment to honesty, compassion, fair play, many forms of hard work, love of justice, and a concern for those in need...the enormous gap between what US leaders do in the world and what Americans think their leaders are doing is one of the great propaganda accomplishments of the dominant political mythology ...the real danger we face is not from terrorism, but what is being done under the pretext of fighting it." ~ Michael Parenti

I am not a socialist or a communist. I believe we should all be allowed to seek our fortunes and to build lives of our choosing. However, we can only get to a certain point in development before we start tearing each other down. We have exceeded that point. We need to return to some middle ground.

Hold on a second (1)

GrumpySteen (1250194) | about 8 months ago | (#45538253)

"Science has a much publicized reproducibility problem" links to an article that says "as many as 17â"25% of such findings are probably false", but a group reproducing 10 out of 13 experiments (23% not replicable) is striking a blow for reproducibility?

Load More Comments
Slashdot Login

Need an Account?

Forgot your password?
or Connect with...

Don't worry, we never post anything without your permission.

Submission Text Formatting Tips

We support a small subset of HTML, namely these tags:

  • b
  • i
  • p
  • br
  • a
  • ol
  • ul
  • li
  • dl
  • dt
  • dd
  • em
  • strong
  • tt
  • blockquote
  • div
  • quote
  • ecode

"ecode" can be used for code snippets, for example:

<ecode>    while(1) { do_something(); } </ecode>