Science Blog

Science news straight from the oven

Navigation

  • Topics
    • Aerospace
    • Animals
    • Anthro and Archaeology
    • Bio and Medicine
    • Brain and Behavior
    • Business and Economy
    • Computers and Electronics
    • Education and Outreach
    • Energy and Environment
    • Geoscience
    • Internet and Communication
    • Media and Entertainment
    • Nanotech, Chem and Materials
    • Physics and Numbers
    • Security and Defense
    • Software
    • Space
    • Transportation
  • Reader Blogs
  • Commerce
  • Register/Login
Home Topics Anthro and Archaeology
  • Contact
  • Home
Google

Similar entries

  • Paradoxically widely accepted Peer Review Revealed
  • "If Peer Review Were a Drug, It Would Never Be Marketed" -- Altman
  • Open the review process to improve quality of science publications
  • Science publication is slow

Recent Comments

  • I am a 24 year old woman in
  • evolutionary artifact
  • web-based research has advantages and disadvantages
  • herbal is good
  • 1 in 100
more

Reader Blogs

  • My First Blog
  • Should we trust experiments on the web?
  • New to the good place
  • Wy do we remember so little of what we see: Results from an experiment
more

Against peer review

coglanglab's picture
  • Anthro and Archaeology
  • Bio and Medicine
  • Brain and Behavior
  • Education and Outreach
  • Energy and Environment
  • Internet and Communication
  • Nanotech, Chem and Materials
  • Physics and Numbers
  • Space
 

In response to a recent post, an anonymous commenter wrote that

It would be scientific misconduct ... to make statements based on someone else's unpublished work ... Scientific results don't exist until they have been peer reviewed and published.

Peer review has become the gold standard of the scientific community. Bring up a scientific finding, and the first thing you may be asked is, "Ah, well, is this peer reviewed?" (For those who don't know, peer review means that, before the journal will publish a paper, one or more other scientists who study similar topics). There is now even a popular blog aggregater that focuses exclusively on blogging about peer reviewed research.

In the age of the Discovery Institute there are some good reasons to focus on peer reviewed research as a way of excluding quacks. It's a way of saying that this research has been vetted.

That said, when I read comments like the one above, I think the time has come to push back, and point out that peer review is not the arbiter of truth. Truth is the arbiter of truth, and peer review is merely a flawed tool we use to help get there.

Peer reviewers don't check to make sure the results are true. Peer reviewers do not typically replicate the experiment in question. They do not check the math. Most of what they do is check that the arguments are reasonable and that the experiment(s) were well designed. Peer reviewers do not necessarily even have to agree with a paper they accept. They may simply think the data are compelling and the arguments are worth hearing, even if they may be wrong.

Thus, peer review does a reasonable job of weeding out quacks. Luckily, most scientists are not quacks, so what does it do for the rest of us? I'm not sure, but I think a partial answer is that two minds are better than one. Reviewers often notice things that the authors missed -- not because the authors weren't smart, but because research is damn complicated and you can never think of everything.

Typically what happens, at least in psychology, is that the reviewers suggest additional analyses or additional experiments that would make the paper stronger. Based on those comments, the authors may run new experiments then revise the paper and resubmit. Peer reviewers, in this sense, aren't so much vetters or fact-checkers as editors. Peer review is a way of improving -- not perfecting -- an article.

So is it scientific misconduct to refer to unpublished work? I don't think so. It is dangerous, though, because there are good reasons (above) to be more confident of something that has gone through peer review. It is a bit impolite to refer to something that has not been published, because then your audience can't go look at it themselves. And that's the main point. Peer reviewers are not the judges of truth, but all of us are on the jury.

(I should say that I primarily have experienced peer review within psychology. People from other fields may have different experiences, so comment away.)


Submitted by coglanglab on Fri, 2008-04-18 10:59.
  • coglanglab's blog
  • Printer-friendly version
  • 2184 reads


Blog away!

Submitted by coglanglab on Tue, 2008-07-08 08:56.

Anonymous: You can always publish them yourself on a website.

Another option is to try to go through the PLoS review process, which is *relatively* streamlined and painless. The disadvantage is that they charge a very steep publication fee. However, most if not all government funding agencies have obligated themselves to pay those fees for work they funded.

That's the best I can think of.

Please try my web-based experiments

  • reply

Where to publish without review?

Submitted by Anonymous on Fri, 2008-07-04 18:01.

Are there any online options for publication of papers that have not gone through the entire review process? I am retired and don't want to put any more work into 5 manuscripts that got through internal (federal government research) peer reviews, and then were never submitted (waiting for a book publisher) or stopped by one external reviewer who wanted me to do some more work that I couldn't do for a variety of reasons. I feel awful that all that work was done and it is just sitting in a drawer.

  • reply

Re: "Peer Review" is not

Submitted by Halliday on Tue, 2008-06-03 11:03.

You're right that what is all too commonly called "peer review" is not, strictly speaking, the real peer review. It's actually a shortcut version. (However, this is not to say that I don't think this pre-publication refereeing cannot serve a useful purpose.)

I most certainly agree that there are far too many instances where words are misused (even abused) to mean what they do not, thus loosing their more true meaning. While I agree that language is dynamic and should be able to "evolve",* I abhor the way words are all too often stripped of their more original, appropriate meaning by the improper application of such in place of more appropriate words (I can share many examples, and I'm sure others can think of many I haven't).

Unfortunately, it has been occurring for many decades and there's no sign of it ceasing. However, we can always hope that by raising the red flag and recommending the substitution of more appropriate words (that, perhaps, the users just haven't thought of) the tide may, at least, be slowed.

David

* I certainly don't believe in being like the French and "policing" the language to prevent "corruption" from other languages.

  • reply

"Peer Review" is not

Submitted by Anonymous on Tue, 2008-06-03 00:17.

Journal publishers and the rest of the academic publishing world only have yourselves to blame for the problems with what they call "peer review".

I write as a graduate physicist (Imperial), a specialist lawyer in and published academically on complex technical evidence, a former university lecturer in law and former member of an editorial board.

Peer review in science has always meant to me the process, if and when it occurs, of scrutiny by intellectual peers of the work of others after they have published and brought their efforts into the public domain and not before.

This is an essential aspect of healthy science. Science requires open communication to thrive. What you call "peer review" closes it off and is not.

In contrast, the publishing industry in a commercial endeavour to give credence to the papers they publish, call the pre publication vetting of articles for publication "peer review". That is not and never has been "peer review" in science and it is a corruption and bastardisation of the term.

In academic circles the term "academic refereeing" is used and might be a more appropriate term, if somewhat overstretching the point. A couple of old blokes skimming over a mate's paper in the pub on a Sunday lunchtime is not what I would grace with refereeing and is most certanly not "peer reivew".

If you claim what you call "peer review" is not done by a couple of old blokes in the pub on a Sunday, then tell me how many times it is done whilst sitting on the toilet or on a plane whilst drinking the complimentary drinks or in fact not properly done at all. And the plain fact is you cannot.

I suggest more care is taken with the words used and that we all take care to use words that mean what they are meant to and not what they are not.

We need to restore the term "peer review" to its proper usage and thereby try to keep the nonsense and non science out of science.

  • reply

What alternative is better?

Submitted by Anonymous on Mon, 2008-04-21 05:50.

I've been reading a lot of literature lately, both peer reviewed, and non-reviewed white papers.
I've found some excellent work and some real dogs in both categories. In the following discussion I am considering my recent reading of many papers on computational docking of ligands in protein active sites, which I have been reading as part of writing a book of my own (yes, my publisher sends it out to experts in the field for review).

Some of the badly done work that was published in the peer reviewed literature is particularly prevalent when the authors are selling a product which is the topic of the paper. Some of these papers make a particular method look good by comparing it to the oldest, weakest, and most flawed of it's potential customers. Others compare products by testing them on only a very small number of protein-ligand systems, which we can surmise may have been chosen to highlight the abilities of the product. There does seem to be some correlation between finding better studies in more prestigious journals and questionable studies in more questionable journals. At the very least, the peer review process seems to force authors to at least attempt to follow accepted practices.

The non-reviewed white papers also have contained
some excellent scientific work as well as some very badly done work. The badly done white papers tend to be shameless plugs for products that the companies sell thinly veiled as scientific studies.

Thus, the worst to best ordering on average tends to look like this.

worst
The worst of the non-reviewed papers.
The worst of the peer reviewed papers.
Plenty of average quality publications.
The best of the white papers.
The best of the peer reviewed papers.
best

The peer review process can be abused by the reviewers. It gives reviewers an opportunity to reject the publications and grant proposals of their competitors, which certainly opens the potential for a conflict of interest. This is magnified by the fact that most reviews are anonymous, thus making reviewers feel that they can get away with vague statements about the suitability of the work.

There have also been documented cases where work with glaring errors got published, apparently because the work was authored by a big name scientist and the reviewer rubber stamped the
acceptance without looking at it closely.

The problem is that complaining about these questionable publications and grant reviews won't fix anything unless you have an alternative process to propose. Here are some suggestions off the top of my head.

1) Reviews should not be anonymous. If the reviewer is being fair and honest, they should have nothing to fear by having the author know who they are.

2) Papers that publish side by side comparisons of products or technique should not be published by individuals who created those products/techniques or stand to make money from the sales of them.

3) People who create competing products/techniques should not be reviewers. There are plenty of people who utilize techniques, but don't invent them, who can review papers and grants.

It would also be nice to have some standard for comparison studies as far as number of cases compared, what they are being compared to, etc. However, I don't know how to phrase that requirement. I'm sure it would be different from one type of study to another, depending upon the nature of the field, data available, etc.

  • reply

Post new comment

  • Web page addresses and e-mail addresses turn into links automatically.
  • Allowed HTML tags: <a> <em> <strong> <cite> <code> <ul> <ol> <li> <dl> <dt> <dd> <img> <blockquote>
  • Lines and paragraphs break automatically.

More information about formatting options

Copyright, Science Blog.
Think. It's not illegal yet. Read our Privacy Policy.
RoopleTheme