How to be a reviewer/editor

Many articles have been written about how to be a good/responsible/fair/rigorous/timely reviewer or editor. Having now reviewed more than 400 papers and having been an editor for 100 more, I find myself developing rather strong opinions on the subject. If those opinions meshed nicely with the ones previously published, a blog wouldn’t be needed – but they don’t. Instead, I find myself holding rather different views on how to be a reviewer and editor. As time has gone on, these opinions have strengthened, not weakened, and so perhaps it is time to get them out there.

How to be a reviewer – 1 simple rule.

Don’t reject papers!!!!!!! How’s that for a minority opinion? Even before we start our reviewing careers, we are told to be very stringent and critical and to only accept the very best stuff. But – as I will explain – this does not work.

As a reviewer, your goal is to improve the scientific literature, which you can achieve by helping good papers get published, by stopping bad papers from getting published, and by improving papers before publication. The straight-up reality is that the second option is out: you simply can’t keep stuff out of the literature. Hundreds of journals exist and so rejecting a paper at one journal just means it will end up getting published in some other journal (Fig. 1), especially in this new age of pay-as-you-go open access publishing. Worse yet, if you reject a paper, the authors have no obligation to follow your suggestions for improvement. Thus, rejecting a paper actually makes the scientific literature WORSE. Instead, you want to keep whatever paper you are reviewing in play at the same journal. That way, the author will be encouraged/required to follow your suggestions for improvement. You and the authors can work together to craft the best possible paper – what a wonderful world (Fig. 2).

Fig. 1. If at first you don't succeed, try try again. Network of submission flows among journals. From Calcagno et al. 2012 (Science).

Several exceptions to this rule might seem necessary. First, some papers are just irredeemably bad in the sense that no amount of re-analysis/re-writing will make them tolerable. In these cases, you have no choice but to reject them – but remember that they will likely just pop up in some other journal in close to the form you previously rejected them. Thus, you really need to be somewhat relaxed about what you consider irredeemable. By this I mean that the paper really has to be fraudulent or completely (not just partially) incomprehensible. In practice, I think such heinousness applies to a vanishingly small subset of papers.

The second exception occurs when the paper just isn’t at all suited for a journal. By this I don’t mean that it “isn’t good enough” (but see below); I instead mean that it is a paper about behavior in a journal about morphology, or some such. Again, however, this is extremely rare as it is the job of the editors to catch this mismatch.

A third exception might occur when you think the paper is far below the quality expected for the journal. The most obvious example is Nature/Science, where we often look at papers and think “Sheesh, I had 10 papers rejected from these journals that were way better than this one.” It is very hard to resist this sentiment and so, yes, sometimes you won’t be able to avoid the temptation to suggest rejection simply because you think the paper “isn’t good enough for the journal.” In addition, papers in such journals tend to get a lot of attention – and so by rejecting them, you will certainly reduce the attention paid to them, thus in essence "keeping them out of the literature" in another sense.

I should make sure to clarify that “don’t reject” means don’t reject without the possibility of resubmission. In many cases, the paper really does require a ton of work and so it really should be rejected “in its current form” or “without prejudice” and resubmitted. Note, however, that specifying this as a reviewer is likely – given the editors different objectives (see below) – to get the paper rejected. Thus, a better option is usually “major revision” – that is, if the authors really can do what you say for improvement, then the paper should be publishable.

Fig. 2. The different types of reviewers, according to http://matt.might.net/articles/peer-fortress/ what I am suggesting is a hybrid "heavy weapons guy" - "medic" - "engineer" - but without the bad parts.

How to be an editor – no simple rule.

In contrast to your role as a reviewer, your role as an editor becomes a balance between your desire to improve the literature (Don't reject papers!) and the journal’s desire to have you improve their journal in particular. Thus, you now end up rejecting papers because your journal wants to publish the very best work and thus increase its impact factor and prestige and subscriptions (and money) and so on. Nowadays, a lot of pressure is placed on editors to reject papers (ideally without review) so that you don’t end up accepting more papers than the journal has funds to publish. (And so you don’t end up wasting everyone’s time with a review process that is likely to fail anyway.) This necessity can be quite frustrating when one tends to fall more on the “improve the literature” side of the balance. Indeed, I see many papers that could be good being rejected simply because they aren’t as good as other papers that are being submitted.

So how does one decide which papers should be rejected and which shouldn’t? In my opinion, acceptance or rejection should not in any way depend on the actual results of the study. If the study is well-motivated, interesting, well-designed, well-executed, and well-analyzed, then it should be published regardless of whether or not it confirms a specific prediction or hypothesis or theory. Perform this thought experiment: take a study that has a negative (e.g., non-significant) result and imagine a positive (e.g., significant) result instead. Would you want to publish the paper? If so, accept it even though it has a negative result. Often, people complain about the design of a study with negative results (“they should have done this, they shouldn’t have done that”) but the reality is that they would not have complained if the result had been positive.

Other reasons to NOT reject a paper (as a reviewer or editor) are: poor stats, poor writing, poor citation of the literature, poor graphics, and so on. All of these things can be fixed with revisions. Just tell the authors what they need to fix. If they can do it, great, if they can’t, fine, you can always reject it next time.

Reasons TO reject a paper from your awesome journal: samples sizes that are too small (but you can encourage the authors to collect more data), a study design that is incapable of testing the hypothesis (but perhaps the hypothesis/study can be rephrased/reframed in a way that it can be tested), and lack of replication at the level where inference is being attempted. For me, this last problem is often the most damning. For instance, if one wants to make inferences about populations in two environments (north vs. south, cold vs. warm), then at least two independent populations of each type must be studied. Of course, studying only two populations (one of each type) is still a good suggestive study; it just might not be good enough for my awesome journal.

How to be an open access editor – 1 simple rule

Accept everything!!!!! In sharp contrast to the above, your role here is to make money for the journal. Yes, I know this is a cynical perspective in a growing culture where open access is considered a paragon of virtue: “make your science freely accessible to everyone, and the world will be a better place.” Having now worked with several open access journals, however, it is very clear that the entire goal is to make money. Consider this: pay-as-you-go open access journals don’t make a cent unless they publish your paper. Stated another way, they lose money every time they reject a paper. Indeed, that is why so many publishers have started open access journals to which they “refer” papers they have rejected from their flagship subscription-based journals. Before these new ventures, every rejected author simply went and paid someone else (if all else fails, PLoS ONE!) to publish their paper instead (Fig. 3) – so the publishers said, “hey, cool, we can also get money from the papers we reject – how awesome is that.” By the way, if you like open access, check out my new super-easy, super-fast, and super-cheap open-access journal “MyScience”:  http://ecoevoevoeco.blogspot.ca/2014/09/myscience-newer-faster-cheaper-easier.html

Fig. 3. PLoS ONE publications (100,000 as of June 2014) = lost revenue to for-profit publishers. The solution: start your own open access journal. (Graphic from PLoS Blogs.)

In my opinion, a better solution is simply to pay for open access at the subscription-based journals, which is nearly always an option, or simply publish in the subscription-based journal and then put the PDF on your website. Yes, I know the publishers imply you shouldn’t do this but I have been doing it for 20 years and no one has complained yet.

Oh s**t.

Having just written the above, it now strikes me that authors reading this post will have a more prosaic inspiration – “hey, I should recommend Hendry as a reviewer/editor – he won’t reject my paper.” Go for it. These days I receive so many requests to review that I turn down most of them anyway: instead only accepting reviews for papers that are squarely in my areas of expertise, which some folks suggest are quite circumscribed. And now I am struck by another realization: editors reading this post might have a different prosaic inspiration – “hey, I better not recommend Hendry as an editor/reviewer as he won’t reject enough papers.” Go for it. I get too many requests to review anyway.