Beta
×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

Peer Review Highly Sensitive To Poor Refereeing

Soulskill posted more than 4 years ago | from the i-blame-jim-joyce dept.

Science 233

$RANDOMLUSER writes "A new study described at Physicsworld.com claims that a small percentage of shoddy or self-interested referees can have a drastic effect on published article quality. The research shows that article quality can drop as much as one standard deviation when just 10% of referees do not behave 'correctly.' At high levels of self-serving or random behavior, 'the peer-review system will not perform much better than by accepting papers by throwing (an unbiased) coin.' The model also includes calculations for 'friendship networks' (nepotism) between authors and reviewers. The original paper, by a pair of complex systems researchers, is available at arXiv.org. No word on when we can expect it to be peer reviewed."

Sorry! There are no comments related to the filter you selected.

This reminds me of something else (2, Interesting)

Pojut (1027544) | more than 4 years ago | (#33611148)

I can't quite remember what it was, but I seem to remember seeing it everywhere. It was exactly like TFA article, though. [wikipedia.org] Damn, what was that place called again?

Re:This reminds me of something else (2, Interesting)

ObsessiveMathsFreak (773371) | more than 4 years ago | (#33611500)

Wikipedia is completely different. There, you submit your work in whole and anonymous "referees" proceed to secularly mutate your effort into an intellectual monstrosity of its former self. Nothing is sacred. Some ignoramus actually removed all chemical equations from the Smelting article [wikipedia.org] . At least in peer review, referees simply make suggestions which you yourself implement.

Re:This reminds me of something else (3, Funny)

Farmer Tim (530755) | more than 4 years ago | (#33611740)

Some ignoramus actually removed all chemical equations from the Smelting article.

At least they didn't replace the whole thing with "Who smelt it dealt it".

Re:This reminds me of something else (0)

Anonymous Coward | more than 4 years ago | (#33611634)

I can't quite remember what it was, but I seem to remember seeing it everywhere. It was exactly like TFA article, though. [wikipedia.org] Damn, what was that place called again?

Because something on Wikipedia is just as reliable as a published journal.

yeah (-1, Offtopic)

Anonymous Coward | more than 4 years ago | (#33611158)

thats what always happens to my comments on here... frosty pist!

Pharmaceutical "Studies" and Climate "Science". (0, Flamebait)

Jeremiah Cornelius (137) | more than 4 years ago | (#33611588)

After reading the article, you can add these terms to George Carlin's list of oxymoronic phrases.

Well now... (0)

Anonymous Coward | more than 4 years ago | (#33611162)

I saw that last sentence coming. I can imagine $RANDOMLUSER's supervisor won't accept his excuse for said supervisor's paper failing peer review.

Sounds familiar (-1, Troll)

Anonymous Coward | more than 4 years ago | (#33611164)

Does this remind anyone of Climategate?

Anyone?

Bueller?

Bueller?

Peer review (-1, Offtopic)

Anonymous Coward | more than 4 years ago | (#33611192)

Result: -1 Troll

Now mod me up for factual accuracy or face infinite recursion

The climate skeptics will have a field day (4, Interesting)

Anonymous Coward | more than 4 years ago | (#33611210)

This is precisely what the global warming skeptics say is happening with the global warming alarmist community. ie. scientists review each others' papers, in a 'co-operative' manner as it were.

I think I'll point some skeptics at this paper and then sit back with a bowl of popcorn and watch what happens.

Re:The climate skeptics will have a field day (1)

Pojut (1027544) | more than 4 years ago | (#33611264)

They probably won't believe it's real, and will accuse you of trying to make them look foolish.

Re:The climate skeptics will have a field day (2, Insightful)

ByOhTek (1181381) | more than 4 years ago | (#33611510)

The funny thing is, the skeptics suffer from the same problem.

I hope the moderates don't, otherwise were borked.

Re:The climate skeptics will have a field day (0)

Anonymous Coward | more than 4 years ago | (#33611950)

With the key difference that the skeptics haven't had their emails leaked in which they whine and howl whenever they don't get to peer review and discard every article that critizes their work.

You mean whine when a POS paper is printed (0)

Anonymous Coward | more than 4 years ago | (#33612014)

You mean whine when a POS paper is printed.

G&T's paper in E&E refuses to accept that the presence of another object nearby at temperature will cause the temperature of a hotter object to be higher than it would without. Making a mockery of the laws of thermodynamics which REQUIRE that this be true.

E.g. If you have one end at 300C and the other end at 200C, at some time later, say, 1 minute, the hotter end will have cooled to, say, 280C. If the other end was at 100C rather than 200C, G&T would have the temperature of the hotter end at 280C still.

But that makes a mockery of the thermal conduction laws which include the temperature DIFFERENCE between two ends.

It also makes a mockery of the radiative loss equations for an object in a hot enclosure:

t1 = temperature of hot body
t2 = temperature of enclosure

power loss = sigma * (t1^4 - t2^4)

Yet G&T's paper in E&E would say that this is impossible! Since the temperature drop of the hot body cannot, to them, change if the enclosure is hotter.

Think also that according to G&T, you end up with the hot end colder than the cool end, because it cannot be affected by the temperature of the cool end, therefore it must cool to the same temperature as the situation of absolute zero at the other end. That means it cools to absolute zero. Which is colder than the other end.

How does G&T's paper work even at teenager school physics level of understanding?

It doesn't.

But it got through to be published?

Yet so many are bleating here about how it must be the IPCC "warmists" suppressing papers opposing them. Here we have

a) no opression, the paper is printed
b) the paper is OBVIOUSLY crap

yet somehow, (b) doesn't mean that denialists promote crap papers to bolster their position. According to some, anyway.

How does THAT work?

Re:You mean whine when a POS paper is printed (4, Insightful)

oiron (697563) | more than 4 years ago | (#33612380)

I agree with all that you said, but for heaven's sake, use full-forms at least once

For those as confused as I, the paper the parent refers to is "Gerhard Gerlich, Ralf D. Tscheuschner. Falsification of the atmospheric CO2 greenhouse effects within the frame of physics.".

Contains such howlers as "There's no such thing as average temperature"... RC Wiki page [realclimate.org]

Re:You mean whine when a POS paper is printed (1)

ByOhTek (1181381) | more than 4 years ago | (#33612464)

Lacking mod points, or even the ability to mod this thread at this point, I'll just say: thank you.

Re:The climate skeptics will have a field day (2, Insightful)

jfengel (409917) | more than 4 years ago | (#33611870)

I think I'll point some skeptics at this paper

Let me know when you find some. I mostly meet deniers, with a deep ignorance of climatology or any other science and a deep conviction of a conspiracy.

If you locate some actual skeptics, people capable of analyzing the evidence, who have come to the opposite conclusion of the vast majority of actual climatologists, I'd love to hear from them.

Re:The climate skeptics will have a field day (1)

IndustrialComplex (975015) | more than 4 years ago | (#33612424)

If you locate some actual skeptics, people capable of analyzing the evidence, who have come to the opposite conclusion of the vast majority of actual climatologists, I'd love to hear from them.

What about people who aren't skeptics, but are damned tired of the whole thing being hijacked as a way to sell people on junk ideas.

All the good intention in the world won't do you any good if the 'fix' isn't practical.

Re:The climate skeptics will have a field day (1)

oiron (697563) | more than 4 years ago | (#33612444)

There were quite a few, like Friis-Christensen [wikipedia.org] who actually raised real scientific questions, but most of those have since come to the conclusion that global warming is indeed caused (primarily) by human influence

On the other hand, every Real Scientist is a skeptic; if someone claimed, whether in a scientific paper or otherwise, that global warming would cause the ice-caps to melt by Dec. 21 2012 or something like that, you can bet that the entire scientific community would pretty much laugh them out of the room. Somehow, I don't think that was what GP meant by "skeptic".

So, good luck on that Great Skeptic Hunt!

Highly political subjects? (4, Insightful)

AnonymousClown (1788472) | more than 4 years ago | (#33611228)

"The system provides an opportunity for referees to try to avoid embarrassment for themselves, which is not the goal at all," he says.

So, if a reviewer sees a paper that has actual data and a conclusion that goes against the consensus of the scientific community, the reviewer may reject it for fear of appearing foolish? Or rejecting someone just because of their publicized personal beliefs?

Here's a hypothetical, a climate scientist who's an openly devout Christian finds data that sheds doubt on human caused global warming will be rejected because someone's afraid of looking foolish.

That's the way I'm interpreting this study.

Re:Highly political subjects? (1)

AnonymousClown (1788472) | more than 4 years ago | (#33611250)

Shit, I forgot...

Then there's the risk of letting shit go through to appear "unbiased".

Re:Highly political subjects? (5, Insightful)

MozeeToby (1163751) | more than 4 years ago | (#33611266)

Can anyone give me a good reason why the reviewers get information about the author in the first place? Granted, there are disciplines that are closed knit to the point that the reviewers would recognize the author based on their past work, but in most cases I would think not knowing who the author is would address at least some of the issues that they highlighted here. It's hard to obscure the rest of the review process but limiting nepotism should be relatively simple.

Re:Highly political subjects? (5, Informative)

prefect42 (141309) | more than 4 years ago | (#33611380)

It is done as you've guessed, but it's still often obvious who the author is. Don't forget that sometimes a bad review has nothing to do with knowing who the author is. If you come across a paper that's done almost exactly the same work as you have done, or criticises your work, you could choose to give it a false bad review to try to prevent it from being published. I've seen papers that have received three reviews, two that say it's good, and one that says it's nowhere near worthy of being published. You often question the outliers.

Confirmation bias (1)

grimJester (890090) | more than 4 years ago | (#33611708)

The outliers might not be due to conscious suppression of competing research. People just have some ways of thinking that make their subjective opinions sometimes contrast with what an objective observer would think.

Re:Highly political subjects? (1)

martyros (588782) | more than 4 years ago | (#33612252)

In Computer Science, top-tier conferences are higher quality than most journals. The admission is determined by a program committee, whom are carefully selected by the program chair because of what they'll bring to the table.

I've only served on one PC, but can't imagine trying to serve on a program committee they describe in the the paper, with 1/3 "rational" (i.e., self-serving) people and 1/3 "random" (i.e., can't tell a good paper from a bad). Of course you'd get essentially random. But if that happened, the program chairs have been failures at finding a good PC.

One of the comments later in the article addresses this -- talks about how the role of the journal editor in choosing reviewers is important. I bet if they added some kind of feedback mechanism, they'd get a system much more resilient to bad reviewers.

Re:Highly political subjects? (4, Interesting)

Trepidity (597) | more than 4 years ago | (#33612382)

As a computer scientist, my impression is that the program committees really are pretty random, or at least based on some sort of preference other than a widely agreed "quality" standard. Try it sometime: resubmit a paper rejected from a top CS conference verbatim to another top CS conference. The correlation between the reviews is usually quite low, both in terms of the numerical scores, and especially in terms of what they liked / complained about.

Re:Highly political subjects? (2, Informative)

pz (113803) | more than 4 years ago | (#33612268)

It is done as you've guessed, but it's still often obvious who the author is. Don't forget that sometimes a bad review has nothing to do with knowing who the author is. If you come across a paper that's done almost exactly the same work as you have done, or criticises your work, you could choose to give it a false bad review to try to prevent it from being published. I've seen papers that have received three reviews, two that say it's good, and one that says it's nowhere near worthy of being published. You often question the outliers.

Whether the authors are revealed to the reviewers or not varies from journal to journal. All of the large handful of reviews that I've done had the author information presented to all of the reviewers; I've not reviewed for really big name journals though (at least not yet). The reviewers' identities are not made known to the authors, though. It is often, however, rather easy to identify the reviewers because my field is not that large, and personalities can shine right through unedited writing like reviewer's comments. Similarly, even if the author were to be anonymized, it's normally pretty easy to identify the laboratory the work came from based on the references cited, since most labs build on previous work in the lab, so cite their own papers more than others.

Re:Highly political subjects? (2, Insightful)

DerekLyons (302214) | more than 4 years ago | (#33612458)

I've seen papers that have received three reviews, two that say it's good, and one that says it's nowhere near worthy of being published. You often question the outliers.

With such a small sample size - there's no such thing as an outlier. There is still selection bias and confirmation bias though, as you so aptly demonstrate.

Re:Highly political subjects? (0)

lgw (121541) | more than 4 years ago | (#33611386)

Well, often you could guess the authors with reasonable accuracy by knowing the field well (a paper about X that says Y must be from Z or one of Zs grad students). Also, those climategate emails described lining up the proper reviewers for certain papers. I don't know how real or widespread that is (hey, they examined themselves for corruption and found none so it didn't really happen, right?) but it does illustrate a failure mode: if those choosing who reviews what paper are themselves corrupt, papers can be steered to carefully chosen reviewers to ensure the desired result.

Fundamentally, the peer review system has no defense against deliberate misfeasance during the review of papers. That lies only in skeptics reproducing the original results (or failing to). That's a good reason to be greatly skeptical of any claim where the originato can't or won't share the data or methods needed to reproduce his results from scratch - you can't depend on the referee system alone, as tha was always just the first step of the process.

Re:Highly political subjects? (2, Informative)

Shrike82 (1471633) | more than 4 years ago | (#33611388)

If you're a reasonably active researcher in a specific discipline (even more so if you work in a small sub-field) then you'll likely get to know your peers when you meet them at conferences and when you collaborate with other groups in projects. These same people will be the first to be asked to review a paper in their (and your) field, and will either recognise your work or simply see your name at the top. Now if they have no specific involvement in the work then ethically they're not in the wrong for reviewing it. They of course must be unbiased, but that's a subjective term in the world of paper reviewing.

Disclaimer: I both write and review journal articles in a few fairly narrow Computer Science sub-disciplines.

Re:Highly political subjects? (1)

darien.train (1752510) | more than 4 years ago | (#33611460)

Malcolm Gladwell [wikisummaries.org] would agree with you.

The ironic part of this is that you need not look further than human behavioral scientists to help solve this problem. It is also possible that the whole idea of anything human-based being "non-biased" is a fantasy made up to represent an ideal that will never happen. Humans are just biased to their physiology and environment. End of story.

Re:Highly political subjects? (2, Informative)

TheRaven64 (641858) | more than 4 years ago | (#33611704)

It's often difficult to hide it. Someone qualified as a referee has to be familiar with the state of the art in a subject, and when it comes to journals the field is usually a very specialised subset of a broader field. The people qualified as a referee will generally recognise the work of their colleagues, and also of their competitors. People can also communicate out-of-band. If you say to the top dozen people in your field 'I submitted a paper about X to this journal / conference this year' then the odds are that one or two of them will be reviewers.

Re:Highly political subjects? (3, Informative)

ElektronSpinRezonans (1397787) | more than 4 years ago | (#33611344)

A referees rejection can be overruled by the editor. It's his job to choose referees that will understand the research and make sure they are just.

Re:Highly political subjects? (2, Interesting)

commodore64_love (1445365) | more than 4 years ago | (#33611382)

>>>a climate scientist who's an openly devout Christian finds data that sheds doubt on human caused global warming will be rejected because someone's afraid of looking foolish.

Nothing that extreme. More like they would reject papers that claim "global warming caused by natural causes" and accept papers that say "global warming caused by man", in order to protect their Own beliefs. A guy named Thomas Kuhn wrote about this very phenomenon (protecting the current paradigm aka worldview) several decades ago, about why the particle theory of light was initially rejected in favor of the existing "light passes through a medium" theory.

Basically the scientists/reviewers rejected papers as "hogwash" simply because they don't fit the accepted scientific theory. It can be a real challenge for new ideas to overcome this censorship.

Re:Highly political subjects? (2, Interesting)

Bigjeff5 (1143585) | more than 4 years ago | (#33611800)

I remember hearing about a nutrition paper that was rejected from a medical journal for a reason along the lines of "That can't possibly be true." So the guy updated the paper with an explanation of the basic bodily functions involved and how they work, which shows exactly why it could happen, and still rejected. He submitted it to a different paper where they basically said "This looks sound, we'll publish on the condition that you remove the explanation. Any doctor would know this already." The paper didn't fit in with the first reviewer's beliefs on nutrition, so he rejected it outright. A journal that was less biased on the subject approved it.

Now imagine that your research goes against the beliefs of all the referees. How do you get published then?

That same guy figured out how to get all his papers accepted without fail: simply load it down with so much math that the referee won't want to take the time to check your work. Since they can't find anything wrong with it because they didn't do the math, it's an automatic pass almost every time (except for cases like the above).

Climategate for example (-1, Redundant)

Anonymous Coward | more than 4 years ago | (#33611230)

Just as the Climategate e-mails showed, a corrupted peer reviewed process produces all kind of junk passing as science.
A small group of buddies agree to let their own articles through, and delay or thwart any paper that contradicts or challenges their agenda.

Re:Climategate for example (4, Informative)

oiron (697563) | more than 4 years ago | (#33611618)

Broken record time, but yes. Such subversion of the peer review process did show up. The culprits weren't the ones you expect. [csicop.org]

In general however, I think that this study is rather pessimistic. And anyway, it hasn't been peer reviewed, so who knows... ;-)

(yes, I did read TFA, but not the paper

So, umm, was this, you know, peer-reviewed? (-1, Offtopic)

Anonymous Coward | more than 4 years ago | (#33611234)

(no text)

Just like the Slashdot moderation system (-1, Offtopic)

Old97 (1341297) | more than 4 years ago | (#33611254)

Ok, mod me down.

Re:Just like the Slashdot moderation system (-1, Troll)

Jeng (926980) | more than 4 years ago | (#33611294)

The Slashdot moderation system does work, mod me up.

Re:Just like the Slashdot moderation system (5, Interesting)

spikenerd (642677) | more than 4 years ago | (#33611492)

Slashdot let's you publish first, and be reviewed later. The peer-review system used by scientists forces them to work on their papers until someone finally "mods" it acceptable. Imagine how much faster science could advance if we had a system that actually let scientists focus on research, let people trained in technical writing do the reporting, and let Google design a post-publication moderation system to sort out the useful advances from the career posturing. Science could learn a lot from Slasdot. It is simply ignorant that we continue to put up huge barriers to publication.

Re:Just like the Slashdot moderation system (3, Insightful)

thePsychologist (1062886) | more than 4 years ago | (#33611968)

Letting technical writing people doing the writing won't work. A large part of the scientific writing is the discussion of the experiment, which not only helps the scientist clarify his or her own thoughts and gives insight into future experiments, but also really only is worth reading if the scientist or members of the experimental team do it themselves. Technical writers really only would have the ability to write the experimental procedure, and even then it would be hard. Since science is so specialized you'd have to have technical writers for thousands of subdisciplines, etc. This goes especially true for mathematics, where the writing procedure is very closely related to doing mathematics.

Already because of this, no time for the scientist would be saved. A Google moderation system would have two problems. First, it wouldn't save any time because you still have to have some person doing the reviewing, and secondly you have to have someone qualified doing the reviewing whom you can trust to some extent to review in confidence, for otherwise if there are certain major problems with the paper but a few good ideas, they can be "stolen" by others, which may become a problem.

Re:Just like the Slashdot moderation system (1)

labcoatless (1902340) | more than 4 years ago | (#33612212)

One problem I see with the approach you suggested is that work is already visible to many but may or may not be accepted in the end. Publications are often rejected not due to a lack of innovation but because of other aspects (evaluation, comparison to other work, ...). A new idea appearing in this paper will then be available to the world while the author is not (yet) credited for his work. Someone else might just pick up his idea, fix the problems, and publish it himself.

Re:Just like the Slashdot moderation system (1)

TheRaven64 (641858) | more than 4 years ago | (#33612328)

This is more or less how it works in Physics, and an increasing number of other projects. The paper in TFA is an example of this. It's published on the preprint archive (arXiv.org - the X is meant to be a chi), where it can be read and commented on. A final version, incorporating feedback, may later by submitted to a more traditional journal.

Re:Just like the Slashdot moderation system (0, Redundant)

commodore64_love (1445365) | more than 4 years ago | (#33611858)

Yeah Slashdot is a peer-reviewed website, and has exactly the same flaws as described in the article. You've been assigned a (0). Slashdot's mod system is basically just "+1 I Agree" or "-1 I Disagree". I don't see why it even exists, because the same purpose kind be achieved via posting.

just like /.? (1, Troll)

grub (11606) | more than 4 years ago | (#33611256)


Peer Review Highly Sensitive To Poor Refereeing

Much like /.'s moderation system. No Citation Needed for this one.

Re:just like /.? (5, Insightful)

alvinrod (889928) | more than 4 years ago | (#33611564)

As much crap as /.'s moderation system gets, it actually tends to be one of the better systems I've seen on the net. First of all, it's highly customizable so if I wanted to I can easily set it to add or remove value from certain types of moderations. I usually bump flamebait and troll up a few pegs just so I can see the posts that do occasionally get unfairly moderated. I can also add other posters I find interesting to a list and bump up their post value so if I'm interested in what they have to say I can always make sure I'll see it.

I also think that the community goes a long way towards making the system work well. Sure there will always be people who abuse the system and moderate posts with which they disagree as flamebait, etc. but the community as a whole does a good job of promoting interesting lines of conversation and for any given topic there are probably a few people in the community who specialize in that area and can provide some excellent commentary.

It's not perfect, but it's probably one of the best systems in actual practice that's currently being used.

Re:just like /.? (1, Insightful)

Anonymous Coward | more than 4 years ago | (#33611756)

What I find sad about the /. system is that it works very well as you describe, it was one of the first, and now, 10+ years later, most sites have no moderation system, or terrible moderation system, or terrible and very complex moderation systems.

Because who would want to use what works and not reinvent their own!?

Re:just like /.? (1)

grub (11606) | more than 4 years ago | (#33611782)


It's good but the system is useless when people's opinions overrule common sense.

Take comments about evolution/creationism/god/no-god. In those cases people often moderate based on their agendas, not common sense.

Re:just like /.? (5, Insightful)

MozeeToby (1163751) | more than 4 years ago | (#33611892)

I think it comes down to two things.

First, the relative rarity of mod points encourages people to take it seriously. It also encourages the most active, most interesting posters to give up posting once in a while to moderate. Most other sites use a mod system that allows so many votes per article but still don't allow you to post and mod the same article. That means that the most frequent posters will seldom mod and that there can be people who only ever mod articles without ever commenting on them.

Second, attaching a reason to the mods encourages people to actually think about why they are modding the way that they are. As many people say, there is no "-1 I Disagree" mod, in order to mod someone down you have to be saying that they are actively trying to derail the conversation. Of course, lots of modders will ignore that and mod however they want, but I think that it does make at least some people stop and think before they accuse someone else of being flamebait or a troll.

Re:just like /.? (-1)

commodore64_love (1445365) | more than 4 years ago | (#33611976)

I think Slashdot's system has the exact same flaw as Web Of Trust (WOT) which blocks foxnews.com on my browser.

The system is abused by people pushing an agenda, with the intent of censoring what they don't like.

Re:just like /.? (4, Interesting)

Trepidity (597) | more than 4 years ago | (#33612422)

I've long thought there should be a "-1, Disagree" option in the drop-down box that takes a mod point but has no effect.

Re:just like /.? (1)

commodore64_love (1445365) | more than 4 years ago | (#33611924)

>>>it actually tends to be one of the better systems I've seen on the net

As compared to what? I think it's junk. It has exactly the same flaws as described in the article, and is basically just "+1 I Agree" or "-1 I Disagree". The same purpose could be achieved via posting.

Of course I grew-up in age of Fidonet and Usenet, where pretty much anything was allowed. I prefer that non-censorship, even if it means you sometimes encounter a conspiracy nut (who can easily be ignored).

Re:just like /.? (0)

Anonymous Coward | more than 4 years ago | (#33612364)

No, Slashdot's system is far worse: the reviewers (moderators) range from completely retarded to semi-intelligent, and comments that get moderated as +4 or +5, Insightful, are usually demonstrably incorrect.

Your post exemplifies this.

Re:just like /.? (1)

Bigjeff5 (1143585) | more than 4 years ago | (#33611806)

Basically, yeah.

It falls prey to human nature.

Review content matters (3, Informative)

lymond01 (314120) | more than 4 years ago | (#33611282)

When you're talking about scientific papers, a "bad apple" reviewer may be able to skew the record in terms of 1-10 scales, but reviewers also do a qualitative write-up of the material. That's really the only important part and if one or two people fall outside the line of general consensus, they'll just be ignored.

Re:Review content matters (5, Interesting)

Shrike82 (1471633) | more than 4 years ago | (#33611470)

Well that's not always the case. Different journals have different review processes. Some ask for numerical choices on a scale, others want choices in terms of "strongly agree", "somewhat agree" etc. for specific questions, others want only written comments and a final choice. Even this final choice is different in many cases, sometimes restricted to Accept, Accept with minor corrections, Accept with major corrections, Invite for resubmission and simply Reject, while others take the final choice as an aggregate of multiple choice responses or numerical averages. Some systems are obviously easier to be biased with than others.

Regardless of all this though, sometimes you'll find out that only two of three reviewers responded, and at least one of those probably got one of their postdocs or even a PhD student to do the review. Some reviews will have empty parts where a reviewer was supposed to write a paragraph but couldn't be bothered, or because they didn't want to reveal the fact that they were totally unfamiliar with the subject matter. Getting a journal paper published is more hit and miss than you'd think. I used to think that a good paper with good ideas was enough, but it's not always the case.

Re:Review content matters (0)

Anonymous Coward | more than 4 years ago | (#33611722)

Oh, there are well-known (and used) tricks for that. "The paper needs to be re-submitted (i.e. is not accepted this time) with more experimental data". "There were similar ideas developed in the 1960's in an unrelated field, and the paper should include a comparison". "The level of English is not appropriate".

If you have decided to shoot down an article in a way that referee members who are only lukewarm about the paper will not dare to oppose, you can easily. If your paper is very good you may get a white knight to defend you. Otherwise, you'll be booted out together with the bad ones. Too many submissions, too little time.

Re:Review content matters (1)

Bigjeff5 (1143585) | more than 4 years ago | (#33611824)

We're not talking about the reviewers, we're talking about the referees. The guys who let papers in the journal so that they can be reviewed.

If the paper never gets published, how will reviewers ever see it?

Re:Review content matters (1)

Shrike82 (1471633) | more than 4 years ago | (#33611972)

I think you're not entirely clear on how peer review works. Once a paper is published the peer review process is over, bar someone seriously questioning the results or conclusions and forcing the paper to be withdrawn post-publication. This is pretty rare. The usual peer review process goes something like this: Someone submits a paper to a journal. The editor will sometimes take a cursory look to determine if it's totally crap, but often will simply assume that people wouldn't waste their time by . If it's OK then they'll send it to reviewers which they have on record, often people who themselves have submitted papers to the journal and have agreed to be reviewers. These reviewers receive the paper, read it and then make comments and/or score it and make their reccomendation as to whether or not the paper should be published. There are often conditions set by reviewers that the authors have to comply with, such as "do this part again, tidy up this figure, expand this section" etc.

A referee is something else entirely. They make sure sports are played according to the rules.

Re:Review content matters (0)

Anonymous Coward | more than 4 years ago | (#33612300)

I call them reviewers and AFAIK most other people do as well.

Re:Review content matters (0)

Anonymous Coward | more than 4 years ago | (#33611998)

Yeah, but sometimes (I should say always, in my experience) your paper gets peer-reviewed by 2-3 people tops. What happens if one of them does not event understand the point of the paper or has lots of rambling comments on something somewhat unrelated or simply downplays the impact the paper could have because it dislikes the topic (or the exact opposite, which in fact might be even more damaging to the system)? The editors will not disregard 33%-50% of the reviews they get...

Will this change anything? (1, Troll)

iONiUM (530420) | more than 4 years ago | (#33611284)

Since the scientific community is so very obsessed with peer review, will this study actually modify the standard procedure?

Of course, all I can think of is: gee, I wonder if this has had any impact on all the climate change studies that are constantly contradicting each other...

Re:Will this change anything? (2, Insightful)

arivanov (12034) | more than 4 years ago | (#33611528)

There is an aforism: Democracy as a form of government is riddled with problems. However we are yet to invent anything better.

Same with the peer review. It has its problems. However, we are yet to invent anything better

Re:Will this change anything? (0)

Anonymous Coward | more than 4 years ago | (#33611608)

"aphorism", doofus.

Re:Will this change anything? (0, Troll)

mcgrew (92797) | more than 4 years ago | (#33612006)

I had to google to find out what an "aforism" was, and found that there is no such thing. Google asked "Did you mean: aphorism?"

Ok, now I know what an aforism is. It's an aphorism that's misspelled so badly that I couldn't figure out WTF you were talking about. Have you ever even seen the word "aphorism" in print before?

Some of us don't move our lips when we read, nor sound out words, so when you do that you lose us completely. Some of us, when we read, don't even see the words; we see only what the words convey, like the guy in The Matrix who said "I don't even see the code any more. I just see blondes, brunettes, redheads..."

Re:Will this change anything? (2, Funny)

Adambomb (118938) | more than 4 years ago | (#33612026)

Well, I thought i had developed a better system but the thesis was shot down in peer review.

Re:Will this change anything? (1, Insightful)

Anonymous Coward | more than 4 years ago | (#33611812)

For any journals which use the model in the paper

- 2 reviewers;
- only accept or reject decisions; and
- no editorial board member involved

they might do something.

However, the author's model does not fit ANY credible scientific journal.

In general at least 3 reviewers are used. Further, the editorial member checks the reviews and is an expert in the field.

In other words, the normal journals will only publish junk papers if ALL 3 reviewers AND the editorial board member are poor.

Actually, I can't think of a single journal in science which uses the author's model.

Damn, now what do I do? (1)

codewarren (927270) | more than 4 years ago | (#33611328)

I can't decide if I should care whether this was peer reviewed or not.

Re:Damn, now what do I do? (1)

jeffmeden (135043) | more than 4 years ago | (#33612204)

Do your own peer review... toss a coin and decide if it passes muster.

Curse (1)

srussia (884021) | more than 4 years ago | (#33612490)

...and then curse again. Let the recursion begin!

surprises... (1)

l3v1 (787564) | more than 4 years ago | (#33611374)

Almost always - no, that's not a scientific deduction, it's just coming from skewed subjective personal experience - the ones who most complain about problems with article peer review systems are those who have the most problems publishing decent articles at decent places. Also, nepotism? Ever heard of [single/double] blind reviews? I guess this must be one of those slow news days.

Not to mention "autarkistic" research communities (3, Interesting)

Anonymous Coward | more than 4 years ago | (#33611404)

I mean scientists who publish among themselves, i.e. inside their narrow specialty, in their own journals, without checking whether the problem at hand has been solved elsewhere. This is more and more common as people get more specialized, and can lead to very basic errors propagated inside the whole community, like rheologists believing in the existence of pure elongational flow (a trivial misunderstanding of tensor algebra). Since the peers reviewing the papers are members of the same community, those errors usually get unnoticed.

It's part of automating the process. (5, Interesting)

Remus Shepherd (32833) | more than 4 years ago | (#33611420)

Just this week, I was asked to peer review a paper in which I was mentioned in the Acknowledgments. The request was sent out automatically -- the journal has records of all their authors, and the keywords for this paper matched the keywords in my profile, so I was picked to review it.

I recused myself, but really I should never have been asked. If they're going to handle the peer review process automatically, the artificial intelligence that makes the decisions needs to be improved.

Re:It's part of automating the process. (5, Insightful)

starless (60879) | more than 4 years ago | (#33611820)

Except that I've heard of people deliberately adding people to acknowledgements to try to make sure they don't get those people as referees (and it hasn't worked)!

Re:It's part of automating the process. (2, Insightful)

Shrike82 (1471633) | more than 4 years ago | (#33611994)

If they're going to handle the peer review process automatically, the artificial intelligence that makes the decisions needs to be improved.

I don't think it's a massive problem, it just relies on people being ethical about declining to review something if they have an interest (in the legal sense) in the work. People have a lot to lose if they try and cheat the system and get caught.

The Social Text Affair (2, Interesting)

dazedNconfuzed (154242) | more than 4 years ago | (#33611432)

The great anecdote demeaning peer-reviewed journals is The Social Text Affair [nyu.edu] , where a prominent peer-reviewed journal published with enthusiasm the article "Transgressing the Boundaries: Toward a Transformative Hermeneutics of Quantum Gravity", only to be informed it was, in fact, computer-generated gibberish submitted as a joke.

Re:The Social Text Affair (1)

sribe (304414) | more than 4 years ago | (#33611550)

Peer review only works in fields where the "peers" are not gibbering idiots ;-)

Re:The Social Text Affair (4, Informative)

iris-n (1276146) | more than 4 years ago | (#33611810)

Social Text is emphatically not prominent nor was peer-reviewed at the time of the affair.

http://en.wikipedia.org/wiki/Sokal_affair [wikipedia.org]

While the Sokal affair is interesting, it has nothing to do with the matter at hand.

Re:The Social Text Affair (1)

Trivial Solutions (1724416) | more than 4 years ago | (#33611904)

Fucken morons. Do you think theys guys were just having a bull-shit session? What's you're explaination for this?

http://www.usccb.org/nab/bible/1corinthians/1corinthians14.htm [usccb.org]

You gotta read between the lines

http://www.usccb.org/nab/bible/daniel/daniel5.htm [usccb.org]

They took the ouijii board from the temple.

The precident is for God to speak in riddles and stuff and in multiple languages

http://www.usccb.org/nab/bible/acts/acts2.htm [usccb.org]

It's mean to tickle the president.

When doing random passages from the Bible, the proper names speak like MENE TIKEL PERES speaks. That's the rosetta stone.

Interpretation is like looking at cloud shapes. God knows what you're gonna think.

You guys are hopeless, fools.

http://www.usccb.org/nab/bible/1corinthians/1corinthians2.htm [usccb.org]

Modern Christianity is secular humanism. "If today you hear his voice harden not your hearts, morons!" Really fucked up.

God says...
accusing garland enters Praise login fitted straitly will
rank vex framed slightly guarded parity disguise bulky
sorrowed moreover cleaved flagitiousness initiated Happy
confessing insight Whosoever digested slain Guardian ambition
agony sublimities sweeten lessons heavily volume trouble
seeth Suppose Thither Else hereunto refresh too astrologers
errors journey emotions shut bud dispense kindled restest
sun answerest revealing endurance contracted prairienet
arising obedient outset deprived Never cedars imparts
reasonable goes Therefore whomever Nor softened pleasest
bringeth fabulous bloody cleansing swaying login infirmities
enticed except needeth suppressed burnest pieces when
accepted whencesoever remission littles syllable flux
downward Senators gathers Redemption preached translated
lost grandchildren using approbation clay gatheredst endued
subvertings Monnica fetched forethought

Read a nansosecond clock when you press a key and use it to pick random words, letters, phrases, passxages, notes or whatever. You gotta hold-up your end of the conversation. Love God, not just neighbor. Do offerings. There are no offerings anymore. God wants praise.

9
I need no bullock from your house, no goats from your fold.
10
For every animal of the forest is mine, beasts by the thousands on my mountains.
11
I know every bird of the heavens; the creatures of the field belong to me.
12
Were I hungry, I would not tell you, for mine is the world and all that fills it.
13
Do I eat the flesh of bulls or drink the blood of goats?
14
Offer praise as your sacrifice to God; fulfill your vows to the Most High.
15
Then call on me in time of distress; I will rescue you, and you shall honor me."
16

Use your head -- obviously he doesn't need animal sacrifices. Love Him. What do you think He wants. I do hymns and comics and stuff.

Re:The Social Text Affair (0)

Anonymous Coward | more than 4 years ago | (#33612106)

computer-generated gibberish submitted as a joke

Sokal's paper was not computer generated. It was not submitted as a joke; the submission was a deliberate deception.

Important limitations in the Model... (2, Interesting)

nweaver (113078) | more than 4 years ago | (#33611456)

There are a couple of significant and important limitations in the model:

a) It assumes only two reviews per paper, and that the reviews are pure boolean, and that reviewer types are also pure and reviewers are randomly selected (when two of the classes of reviewers, 'mythantropes' (always reject) and 'altruists' (always accept) are specifically selected against by editors and PC chairs based on reputation).

b) It does not consider the cases (such as conferences) where there is a program committee meeting and the papers are not just considered on their own, but gone through a relative ranking process.

Re:Important limitations in the Model... (1)

ironjaw33 (1645357) | more than 4 years ago | (#33611890)

a) It assumes only two reviews per paper

I don't think I've ever seen a conference or journal that didn't assign fewer than three reviewers per paper. Furthermore, reviewed papers at some conferences may go through an additional review process by a committee, consisting of 10-20 people. It would be interesting to see the effect of the number of reviewers per paper on the quality of the refereeing.

Re:Important limitations in the Model... (1)

fotoguzzi (230256) | more than 4 years ago | (#33612222)

...'mythantropes'...

Google has three hits for this word, and one of the hits is the parent posting.

change the system (1)

chichilalescu (1647065) | more than 4 years ago | (#33611536)

just tweak arXiv so that reviews of posted articles are linked to in the abstract page. or something like that. why the need for anonymous reviewers? as long as the discussion is objective and to the point, there's no need for anonymity. and we are far past the point where it costs money to attach comments to an article...

Re:change the system (1)

oiron (697563) | more than 4 years ago | (#33611678)

Good point, though for anything serious, I'd think that you need a way of verifying a given reviewer's credentials. Just imagine if a young-Earth big-C Creationist kept trolling on a paper discussing (say) the selective advantage of different colouring in a desert lizard...

Re:change the system (2, Informative)

iris-n (1276146) | more than 4 years ago | (#33611884)

And it would also help the readers understand the article, a good referee report is quite illuminating. However, this has already been tried out by Nature in 2006

http://www.nature.com/nature/peerreview/ [nature.com]

and didn't work so well. Apparently, scientists are somewhat reluctant to openly criticise each other's work. But there's PLoS ONE that is alive and well, giving us some hope.

Michael Nielsen has a fine essay about this in his blog:

http://michaelnielsen.org/blog/the-future-of-science-2/ [michaelnielsen.org]

Bit of an arbitrary model (4, Informative)

gnutrino (1720394) | more than 4 years ago | (#33611590)

First off in case anyone is in doubt this study use a model of peer review - no experiment or observation of an actual peer review process was done. That's not to say interesting and enlightening things can't come from modeling but in this case the moldel they use seems very questionable and highly arbitrary. This part in particular is highly dubious:

Each reviewer produces a binary recommendation within the same timestep: ’accept’ or ’reject’. If a paper gets 2 ’accept’ it is accepted, if it gets 2 ’reject’, it is rejected, if there is a tie (1 ’accept’ and 1 ’reject’) it gets accepted with a probability of 0.5.

If a single 'bad' reviewer (i.e. one that gives the 'wrong' answer as determined by the 'correct' method of reviewing mentioned as a control in the paper) can cause a paper to have a 50:50 chance of acceptance or rejection it doesn't seem too suprising to me that a relatively small number of them could cause the process to become '[not] much better than by accepting papers by throwing (an unbiased) coin' - because in their model, in the case of a reviewer disagreement, that's exactly what is happening!

This is not news to scientists (3, Insightful)

Kurofuneparry (1360993) | more than 4 years ago | (#33611644)

The peer review system is great for regulation, standardization and unification. However, all scientists that I've worked with/researched with/spoken with much about this topic admit that the system can be annoyingly flawed by group think and conformity. One bad apple ruins the bunch, right?

The good news? While this part of the scientific community is not immune to problems, the slack is picked up elsewhere: As long as methods, data and results are transparent, reproducible and published we can actually have quality science.

I often speak to people about scientific research and they're shocked that it's not full proof. This is kind of like buying software (perhaps even a Microsoft product) and finding that it's not perfect. Science is done by committee and progresses slowly. "If we know what we were doing, it wouldn't be called research" ~Albert Einstein

Then again, I'm an idiot....

Doing something unprecedented (5, Interesting)

oiron (697563) | more than 4 years ago | (#33611654)

I actually read the comments on TFA, and down at , there's a particularly interesting one: [physicsworld.com]

This study overlooks not only the role of the editor, but also the process in which the authors are able to answer the referees' objections. When the referees are competent, this leads to better papers through useful suggestions. On the other hand, when they aren't, overcoming the exasperation of the authors, their objections are easily brushed away, and the paper eventually gets through. Also, when the case is particularly contentious, there's still the option of calling for an adjudicator. In summary, the peer-review process is far more complex than this simulation might suggest. On the dark side, I’ve also noticed that referees are sometimes reluctant to object papers from certain renowned authors. The human factor is hard to remove. I guess many people will agree that there’s a need to look for better approval systems, specially today, when there’s an explosion of submissions. However we must also acknowledge that the present system has served its purpose of maintaining a certain quality.

There's actually a reasonably intelligent discussion going on in there...

Re:Doing something unprecedented (1)

Shrike82 (1471633) | more than 4 years ago | (#33612020)

There's actually a reasonably intelligent discussion going on in there...

I'm not sure I'd be comfortable on that site then. I prefer the casual trolling, mis-modding and general idiocy right here on /.

Using Professors' Egos to your advantage... (4, Interesting)

happy_place (632005) | more than 4 years ago | (#33611662)

My dad (has PhD in a scientific field from Cornell) told me that when submitting a thesis to a review board of professors, it really doesn't matter how "Tough" a professor is as long as that professor in your committee has a rival. Take advantage of their ego with an equally assertive ego. You purposefully choose the rival professor to join your committee as well. Then they'll spend all the review board discussions and presentations contradicting and arguing one with another, and in the end they'll both be so incensed, that they cancel each other out, and it doesn't matter what you presented... I guess the TFA is only pointing out that this occurs at the publishing level as well.

it happens (0)

Anonymous Coward | more than 4 years ago | (#33611758)

My first anonymous coward...

My boss (a leading authority in his area, a serious person) has found around 10 errors in epidemiological modelling papers (papers that, by they way, influence WHO policies that influences public health). These are proven errors (in mathematical formulas). They vary from minor to massive.

He informed all authors of the respective problems. How many corrected or retracted the manuscripts? ZERO.
Will he write a paper documenting all the errors (as they are hard-science type of errors, in formulas)? Are you crazy? Remember that the other authors in the area peer review his papers. Therefore we will not risk their wrath.

But note that this varies a lot from area to area. I work in another area (related to genetics) and 90%+ of the people that I know are honest.

But yes, some areas need a cleaning up.

Welcome to the real world... (1)

Voltas (222666) | more than 4 years ago | (#33611780)

Welcome to the real world intellectuals....this is how things are done. I'm not saying its right, but knowing the right people is how things function. I know the University and intellectual world like to think that they are removed from such things but that is just not the case.

I remember a news show highlighting a "famous" professor that did some amazing work in his early carrier and then proceeded to build a bigger carrier while reusing fake data and bogus information. He WOULD NOT have been able to do so if he had not known the right people and had the "right amount of fame". No one doubted him...however if his papers where published by a "nobody" they would have been caught QUICKLY.

Is this news to scientists? (3, Insightful)

0123456 (636235) | more than 4 years ago | (#33611848)

Peer review only works if the reviewers can be trusted and don't form a clique to get their work in and keep other people out. Surely anyone with even basic knowledge of human psychology would understand this?

Re:Is this news to scientists? (1)

interkin3tic (1469267) | more than 4 years ago | (#33612046)

Many scientists don't spend much time thinking about the peer review process itself. Maybe they question it when a good paper of theirs gets turned down, or when a bad paper they disagree with gets published, but they don't spend a lot of time thinking about what could replace it.

Indeed, what COULD replace it? No review at all? A system where you get to strike one reviewers comments?

Anyway, for most scientists, it's just something that exists and you just deal with it. I certainly have no intention of trying to change the peer review system: I'd rather do science.

Pity the internet is full of cranks (3, Insightful)

sam_handelman (519767) | more than 4 years ago | (#33611872)

Because this is an important question for serious people, but has no bearing on why various cranks (Intelligent Design people, climate change "skeptics", Time Cube, etc.) may have trouble getting their work in print. Papers by such people generally don't end up in the peer review phase - they aren't sent out for evaluation by the journal, so peer review doesn't matter.

  That said, peer review provides substantially the same benefit as those "shoplifters will be prosecuted" signs you see in department stores.

  Shoplifters are very seldom if ever actually prosecuted - but the threat, even the vaguest menace - of public scrutiny has an impact on behavior. I'm not talking about scientific fraud (which peer review will seldom catch,) but about quality of reasoning, doing the needed controls, etc. We may have a system that rewards good research little-better than an unbiased coin, but the <b>perception</b> that it works, or that it might work for you, motivates people to do the work needed to survive peer-review.

The issue applies to more than papers (2, Interesting)

grandpa-geek (981017) | more than 4 years ago | (#33612072)

The government uses peer review to evaluate proposals for science and engineering grants. The same issues probably apply to those evaluations.

I have experienced a situation in which one reviewer recommended turning down a grant for reasons that could be considered as biased, although the bias was groupthink rather than individual. The other reviewers were enthusiastic about funding the grant and regarded it as a potential game-changer. It didn't get funded. A few years later the game-changing nature of the technology was recognized, but it was too late for the original applicant.

They are just as human (1)

phrostie (121428) | more than 4 years ago | (#33612074)

the problems of nepotism and tribalism are everywhere.
from internet message boards and professional office environments to national government and international politics.

here's a paradox for you,
someone could do a study on how to eleminate nepotism and tribalism.
then they can put it up for peer review.

Peer Review is not a blanket solution (0)

Anonymous Coward | more than 4 years ago | (#33612120)

In my humanities field, where the vast majority of publications in the last 100+ years were not peer-reviewed, we're seeing pressure from administrators to publish in "peer-reviewed" journals. And, yes, we're seeing the predictable results:

A. Any half-competent referee can figure out pretty quickly the author.
B. While there are data and interpretation in the articles, the author and referee are supposedly skilled at argumentation, and, even when they're not, "nothing new here" will sink any article the ref. doesn't like.
C. Nobody pays us to referee articles; it takes a lot of time, and does nothing for our personal careers.

We had a fun case recently, where one professor (known for shooting down his enemies with the "nothing new" response) ironically turned out to have built his career on plagiarizing other's (non-peer-reviewed, usually) material and submitting them to peer-reviewed journals, with a first page of his own that was sufficient to identify him as the author. He made professor pretty quick. In one case, he plagiarized some guy's published Ph.D. thesis, and submitted it to a journal. The journal sent it to (rumour has it) the victim's Ph.D. advisor to be refereed, and this advisor recommended it for publication without noticing that it was a wholesale rip-off.
In another case, as relatively unknown scholar complained to a peer-reviewed journal that this dude had plagiarized his thesis. From what I understand, the journal submitted the guy's complaint, the plagiarizer's response, and the evidence (which was conclusive) to blind peer review, and the referee sided with the plagiarist. After the extent of the professor's copying has been published, I suppose that journal will bypass peer review and retract the article.

In the sciences, you can argue that a "few bad apples" will cause problems, but in the humanities it's outright broken, and basing hiring and promotions purely on the prestige of the journal or the presence of peer review demonstrably leads to failure.
Load More Comments
Slashdot Login

Need an Account?

Forgot your password?