The concept of peer review has been getting a lot of journalists excited recently. It was attacked during the climategate scandal, and is now under the microscope again, this time over stem cell research. Supposedly, an elite cabal of stem cell experts are blocking some results from publication, through their position as reviewers.

Such a serious allegation deserves closer inspection.

The peer review idea is a simple one: the editor of a journal picks a highly respected, published researcher with no biases who will (usually) anonymously review the paper, find any fundamental flaws, and give an impassioned review. They must decide whether the paper is correct and should be published, or is wrong and should be rejected.

That’s the utopia. Unfortunately, it isn’t the reality.

The reality is an imperfect system with some flaws. Two things in particular make the peer review process desperately difficult. First, most research is extremely intellectually challenging. Second, reviewers are human. For obvious reasons, we can “fix” neither of these issues. We can, however, understand how they impact the refereeing process.

Proof in science is not simple. Experimental results always have uncertainties and errors that must be accounted for. Hence, the best we can do is prove results to an acceptable degree of uncertainty, and not everyone agrees on what is acceptable.

Thus papers that aren’t obviously wrong, fall into this “might be right” category. Sometimes, there’s a clear breakthrough; most of the time, though, it’s only a small step forward. That’s when things get tricky.

This evaluation is the most subjective part of peer review, and is called the “relevance” or “standards” question. How new or original is the research? Is the result only a tiny improvement over a previous one? Does the article fit with the subject area of the journal? That last one is usually easy to answer. The others are distinctly less so.

One person’s marginal improvement is another’s revelation. This is not an exaggeration. It depends on how close an individual researcher is to a given theory or subject area. There is not universal agreement on the value of different theories, or even fields. No statistic can answer this question either.

I’ve rejected papers on relevance grounds. Each case was difficult in its own right, but I had conviction in my conclusions. I do not know how many of the authors appealed to the journal editor, but I do know I never received those papers to review again.

Then there is the human element. The journal’s specialist editors are supposed to weed out collaborators of the author as possible reviewers but they don’t always do this. I’ve been sent papers that I had to return because I had worked with the author.

The reviewer is also required to declare any conflicts of interest. At this stage the honour system is in force, as it is unlikely an editor will ever have time to verify whether someone is being honest or not. But what if a reviewer just doesn’t like someone? Maybe they had an argument at a conference or an email cut too close to the bone. Scientists are human. Indeed, I would argue that scientists are extremely passionate about what they do. The desire to discover, and in some cases compete, drives them day-in, day-out.

Take the relevance and standards question, mix in the human element, and you can see where the stem-cell scientists’ complaint comes from.

But could a group of experts actually conspire to prevent publication? I think this is very unlikely for two reasons.

First, in most fields there is a wide variety of journals to publish in. You can also appeal to the editor against a referee’s decision and ask for a second review. The editor may not grant the appeal, but most of the time they do. So unless you hold some really unsupportable ideas, sooner or later you’ll get a referee that is at least open to your concept.

Second, I believe that the vast majority of scientists (but obviously not all) are ethical. One good thing about science is that if you get caught falsifying results, you’ll never get another position. There are, therefore, compelling reasons to be truthful and unbiased.

The consequences of an “incorrect review” are less severe, and understandably so. You can’t always reproduce what is in a paper. This is a very real problem. Extensive laboratory work or large computer simulations cannot be replicated by a reviewer. In these cases, the onus is on the authors to prove their methods work. But even the best reviewers can miss things.

So as a referee, if you get something wrong or miss a critical issue, you won’t lose your job. You’ll perhaps get a reputation with an editor, but it’s unlikely this will ever be widely known – the anonymity of reviewers is protected by editors. In theory, a group of unscrupulous people could hide behind that shield, but in practice, the sheer number of reviewers and journals would make a wide conspiracy seem very, very unlikely.

Despite its flaws, peer review does have lot of good things going for it. While scientists may complain about “bad reviews,” I really think that’s far better than no review.