Peer Review

I realize that this isn't exactly timely, but who has time to put up posts in a timely manner? Not me. (I hope to have an untimely post on Coyne's hard determinism later in the week, too.) Anyways, via Leiter, I learn of this prediction of the demise of peer-reviewed models of academic publishing. (Or is it an attack? Hard to tell.) According to Aaron J. Barlow, associate professor of English at a CUNY college, "Blind peer review is dead. It just doesn’t know it yet."

Barlow argued that the traditional system of blind peer review -- in which submissions are sent off to reviewers, whose judgments then determine whether papers are accepted, with no direct communication with authors -- had serious problems with fairness. He said that the system rewards "conformity" and allows for considerable bias.

He described a recent experience in which he was recruited by "a prestigious venue" to review a paper that related in some ways to research he had done. Barlow's work wasn't mentioned anywhere in the piece. Barlow said he realized that the journal editor figured Barlow would be annoyed by the omission. And although he was, Barlow said he didn't feel assigning the piece to him was fair to the author. "It was a set-up. The editor didn't want a positive review, so the burden of rejection was passed on to someone the author would not know."

I agree that this sort of thing shouldn't happen, and that the editor was behaving inappropriately. (The editor's behavior is also extremely strange. Correct me if I'm wrong, but editors typically have a lot of discretion concerning submissions, and if the editor thought that the paper wasn't good enough for inclusion in the journal or didn't think it would be a topical or stylistic fit, or just didn't want to devote the pages to it, she has the freedom to reject the paper all by herself. It's not as though she is obligated to ensure that every submission is refereed.)

But I don't agree that this incident is an indictment of anonymous refereeing, or that even reveals any problem of any kind with blind peer-review procedures as such. The only connection between what this editor has done and the institution of blind review is that there has to be such a thing as blind review in order for an unscrupulous editor to employ it in bad faith.

The way I see it, there are at least three important reasons to retain double- (or triple-) anonymous (peer) refereeing in some form:

1. The presence of peer reviewed journals and presses acts as a gatekeeper. There's a huge amount of material out there, and it is helpful to have other people to do the work of figuring out which stuff is worth reading. There would be a lot more material out there if we went to an all self-publish model in which there were no gatekeepers.

Also, this is a great deal of help to us unknowns. The only reason why anybody has ever read my work is because I published it somewhere. (Or because I submitted it to them in a way that caused them to have a professional obligation to read it.) Somebody once cited my work, and I am absolutely positive that this would never have happened if it weren't for the journal I published it in.

2. The work is improved by the editorial process. I've had papers accepted "as is" only a couple of times. The rest of the time the papers were conditionally accepted if I could make certain changes. Sometimes these were suggested by the referees; others they were insisted upon by the editor operating independently of the referees. In each such case, the changes were worthwhile and made the paper much better.

To be fair, Barlow acknowledges this:

"I love the editorial process" when comments result in a piece becoming better, he said, and digital publishing allows this to happen easily. But traditional peer review simply delays publication and leaves decision-making "in the dark."

But Barlow is wrong about the darkness. The darkness is not bad. It is the key to everything:

3. The anonymity of blind review procedures are extremely important. They are the only way to protect authors from all sorts of cognitive biases. Any suggestion that there should be less anonymity than there is now is hopelessly naive. There should be a lot more anonymity, not less.

There's also a strange reference to "guessing about how to handle a 'revise and resubmit' letter." I've had some R&Rs. As a referee, I've handed out some R&Rs. A lot of my friends have had R&Rs. I've never heard of an R&R where you would have to guess about what to do. You get an R&R because the editor believes in the paper and thinks that, with a little work, it would be worth publishing. It doesn't make any sense for an editor to make the author guess about what she should do. And in every case I've ever heard of, the editor has done the sensible thing and just explained what to do.

And even if the editor engaged in the bizarre practice of making the author guess about what revisions she was asking for, it still wouldn't constitute an objection to the practice of anonymous peer review as such. It would constitute an objection to this kooky editor's oddball R&Rs.

Long live anonymous peer review.

--Mr. Zero