hould peer review be abolished? That's the position taken in an interesting new book
titled Essentials of Scientific Research: A Practical Guide, which is coming
out this August. Given the centrality of peer review to science, this is a controversial
topic, but as with many controversies, those who benefit from it are on one side and
those who do not are on the other.
It's even more relevant in light of the recent flood of bad papers on the Wuhan coronavirus. Lancet and NEJM both retracted reports of clinical trials that used fake data, and many other papers that used flawed clinical design or improper analysis got into print, doing enormous harm to science.
A pawn experiencing unfavorable peer review
Advocates of peer review point to its benefits:
- Academic scientists are required to publish a certain number of papers each year to remain employed, so they spend much of their time writing and re-writing papers of marginal significance, knowing (and often hoping) that their colleagues never read them. This means we're drowning in papers that add little to our knowledge. Peer review reduces this number.
- It encourages researchers to do elaborate control experiments and multiple repetitions to ensure the results are correct.
- It keeps flawed research out of the scientific literature.
- It forces scientists to be polite to each other because they know the other person could retaliate by rejecting their next paper.
Opponents point to its flaws:
- Peer review enforces uniformity of thought, as dissenting ideas may be considered faulty because they challenge the current dogma. Peer review thus encourages the lemming effect.
- It makes papers unreadably verbose as authors try to appease the egos of any potential reviewers.
- Because of the pressure to publish, junior scientists spend inordinate effort trying to get into top journals, often doing unethical things like inviting famous people to be co-authors and fabricating data to get past the reviewers.
- Once published, an article can't be edited or annotated to indicate new findings or to correct errors. The only recourse if an error is found is retraction, which authors resist at all costs because it is bad for their careers and because it may throw out months or years of potentially good research in a vain attempt to preserve the purity of the literature.
- It leads to reductionism, whereby laymen believe that anything in the peer-reviewed literature must be true (see below).
- Peer review is political. I've known people who reject any paper that fails to cite them. Others reject any paper written by any competitor or by anyone they dislike personally. Some reject any paper written by someone whose political views differ from their own. This leads to politicization of science, which is the worst thing that can happen to it.
- Peer review acts as a gatekeeper, depriving the public of radical new ideas.
On that last point, I was recently invited to be a co-author on a paper about detecting ghosts using ultraviolet photography. As much as I wanted to encourage the author to continue this out-of-the-box thinking, I had to decline: I knew nothing about ghosts, and yes, I was also a bit skeptical of some of the results, but mainly I declined because I hadn't really contributed.
On another paper where I agreed to be a co-author, the researcher was in dire straits and didn't have access to the equipment he'd need to do the extra experiments we both knew the peer reviewers would demand. So this person's discovery will be lost forever and he could well be forced out of science.
Neither of these two papers has much chance to survive peer review. That's the real cost of the peer-review system: as science gets more and more exclusive, the ones at the top benefit, new ideas are blocked, and science risks getting stuck in dogma. Carl Sagan used to say extraordinary findings require extraordinary evidence. That's a roundabout way of admitting you're being too dogmatic.
Quantity vs quality
Many of the flaws in peer review stem from the emphasis on quantity as a measure of research productivity. This means researchers avoid risky projects and favor those that are easier to publish; that is, those that fit the prejudices of the reviewers and editors.
If done fairly, peer review can protect genuine discoveries from the prejudices of politically motivated editors. But peer reviews are rarely fair and rarely carried out by experts in the field. Usually they're given to postdocs and students without the knowledge of the editor, and the PI signs his name to it.
One defender of peer review advocates making it more rigorous, saying: “The outcome will certainly be fewer publications in biomedicine, but their individual impact will be greater.” Not coincidentally this person is a department head at a big university as well as a journal editor, secure in her job, and so feels no need to be concerned.
Peer review supports reductionist thinking
As evidence she cites a paper by anti-science statistician J.P.A. Ioannidis titled “Persistence of Contradicted Claims in the Literature,” which claims that people are citing observational studies that have been “contradicted” by randomized control trials (RCTs), which he and his co-authors believe are always more conclusive.
Oh, if only the real world were that simple. It's an unscientific, reductionist point of view: RCTs are superior to observational studies, therefore their conclusions are by definition correct. They may be correct; they may even be more likely to be correct; but it is not always so. Statisticians worship probability, but probability plays no more role in deciding scientific truth than consensus does.
It's possible to nail a statistical fact to any level of certainty you want: p<0.001, p<0.0001, whatever, and still be wrong because you're asking the wrong question. And there are an infinite number of ways a question can be wrong.
Take the famous RCT they cite purporting to show a lack of protection against Alzheimer's disease in women by hormone replacement therapy, or HRT. It is now known that the time interval between menopause and HRT is a critical factor in determining whether HRT provides a benefit or a risk. The RCT ignored this variable, so it was the RCT, not the less rigorous observational studies, that came to the wrong conclusion because their question was wrong. Creating a more stringent gatekeeping system that inhibited discussion of the earlier studies would have stifled creativity and frozen a false, oversimplified picture in the literature.
This is why science can never be reduced to a ritual of picking hypotheses and nailing them into the ground with statistics. This is what Ioannidis would have us do, and doing it would destroy science. It's why I call his ideas anti-science. The history of science has shown us again and again that it's better to have a single case report that points us in the right direction than an RCT with a million patients that does not.
As a gatekeeping system, peer review benefits those who have passed through the gate by keeping others out. Indeed, the advice nowadays to young academics is that a paper in a low-end journal does more harm than good. Young scientists must publish in a top journal or perish in the attempt. That means stuffing papers with data from expensive high-tech instruments and adding famous collaborators. This benefits big universities but freezes out junior investigators at smaller schools.
Big Cheeses
The irony is that these big papers found in highly-ranked journals may have more color (in the Chinese sense, meaning sexier or of prurient interest), and they may have more big cheeses than the headquarters of Wisconsin Cheese Corporation, but they're also stuffed to the gills with dogmatic assumptions, and I've found their conclusions are more likely to be wrong than articles in more modest journals. By reading them we learn which topics are unprofitable to study. Thus peer review does work, but in the wrong direction.
What's the solution? Should we abolish it or de-anonymize it? (They amount to the same thing: nobody is going to risk a competitor's wrath by putting their name on an unfavorable review.) My suggestion: keep peer review but publish the paper regardless, along with all the reviewers' comments and the authors' replies. Give the authors a chance to revise their paper if they wish to avoid the humiliation of seeing it criticized in public. Scientists will do anything to avoid loss of face, and so will reviewers.
This would preserve our access to fringe results, add context to them, and maybe even stimulate new ideas. It will get this albatross off our necks that says we have to publish every stupid useless result instead of discovering things. Most importantly, it will help us get rid of all those bean-counters who decide the fate of science by counting numbers of publications as a substitute for understanding them.
jul 24 2020, 7:11 am
