Tuesday, November 4, 2014

How to be a reviewer/editor

Many articles have been written about how to be a good/responsible/fair/rigorous/timely reviewer or editor. Having now reviewed more than 400 papers and having been an editor for 100 more, I find myself developing rather strong opinions on the subject. If those opinions meshed nicely with the ones previously published, a blog wouldn’t be needed – but they don’t. Instead, I find myself holding rather different views on how to be a reviewer and editor. As time has gone on, these opinions have strengthened, not weakened, and so perhaps it is time to get them out there.

How to be a reviewer – 1 simple rule.

Don’t reject papers!!!!!!! How’s that for a minority opinion? Even before we start our reviewing careers, we are told to be very stringent and critical and to only accept the very best stuff. But – as I will explain – this does not work.

As a reviewer, your goal is to improve the scientific literature, which you can achieve by helping good papers get published, by stopping bad papers from getting published, and by improving papers before publication. The straight-up reality is that the second option is out: you simply can’t keep stuff out of the literature. Hundreds of journals exist and so rejecting a paper at one journal just means it will end up getting published in some other journal (Fig. 1), especially in this new age of pay-as-you-go open access publishing. Worse yet, if you reject a paper, the authors have no obligation to follow your suggestions for improvement. Thus, rejecting a paper actually makes the scientific literature WORSE. Instead, you want to keep whatever paper you are reviewing in play at the same journal. That way, the author will be encouraged/required to follow your suggestions for improvement. You and the authors can work together to craft the best possible paper – what a wonderful world (Fig. 2).

Fig. 1. If at first you don't succeed, try try again. Network of submission flows among journals. From Calcagno et al. 2012 (Science).
Several exceptions to this rule might seem necessary. First, some papers are just irredeemably bad in the sense that no amount of re-analysis/re-writing will make them tolerable. In these cases, you have no choice but to reject them – but remember that they will likely just pop up in some other journal in close to the form you previously rejected them. Thus, you really need to be somewhat relaxed about what you consider irredeemable. By this I mean that the paper really has to be fraudulent or completely (not just partially) incomprehensible. In practice, I think such heinousness applies to a vanishingly small subset of papers.

The second exception occurs when the paper just isn’t at all suited for a journal. By this I don’t mean that it “isn’t good enough” (but see below); I instead mean that it is a paper about behavior in a journal about morphology, or some such. Again, however, this is extremely rare as it is the job of the editors to catch this mismatch.

A third exception might occur when you think the paper is far below the quality expected for the journal. The most obvious example is Nature/Science, where we often look at papers and think “Sheesh, I had 10 papers rejected from these journals that were way better than this one.” It is very hard to resist this sentiment and so, yes, sometimes you won’t be able to avoid the temptation to suggest rejection simply because you think the paper “isn’t good enough for the journal.” In addition, papers in such journals tend to get a lot of attention – and so by rejecting them, you will certainly reduce the attention paid to them, thus in essence "keeping them out of the literature" in another sense.

I should make sure to clarify that “don’t reject” means don’t reject without the possibility of resubmission. In many cases, the paper really does require a ton of work and so it really should be rejected “in its current form” or “without prejudice” and resubmitted. Note, however, that specifying this as a reviewer is likely – given the editors different objectives (see below) – to get the paper rejected. Thus, a better option is usually “major revision” – that is, if the authors really can do what you say for improvement, then the paper should be publishable.

Fig. 2. The different types of reviewers, according to http://matt.might.net/articles/peer-fortress/ what I am suggesting is a hybrid "heavy weapons guy" - "medic" - "engineer" - but without the bad parts.

How to be an editor – no simple rule.

In contrast to your role as a reviewer, your role as an editor becomes a balance between your desire to improve the literature (Don't reject papers!) and the journal’s desire to have you improve their journal in particular. Thus, you now end up rejecting papers because your journal wants to publish the very best work and thus increase its impact factor and prestige and subscriptions (and money) and so on. Nowadays, a lot of pressure is placed on editors to reject papers (ideally without review) so that you don’t end up accepting more papers than the journal has funds to publish. (And so you don’t end up wasting everyone’s time with a review process that is likely to fail anyway.) This necessity can be quite frustrating when one tends to fall more on the “improve the literature” side of the balance. Indeed, I see many papers that could be good being rejected simply because they aren’t as good as other papers that are being submitted.

So how does one decide which papers should be rejected and which shouldn’t? In my opinion, acceptance or rejection should not in any way depend on the actual results of the study. If the study is well-motivated, interesting, well-designed, well-executed, and well-analyzed, then it should be published regardless of whether or not it confirms a specific prediction or hypothesis or theory. Perform this thought experiment: take a study that has a negative (e.g., non-significant) result and imagine a positive (e.g., significant) result instead. Would you want to publish the paper? If so, accept it even though it has a negative result. Often, people complain about the design of a study with negative results (“they should have done this, they shouldn’t have done that”) but the reality is that they would not have complained if the result had been positive.

Other reasons to NOT reject a paper (as a reviewer or editor) are: poor stats, poor writing, poor citation of the literature, poor graphics, and so on. All of these things can be fixed with revisions. Just tell the authors what they need to fix. If they can do it, great, if they can’t, fine, you can always reject it next time.

Reasons TO reject a paper from your awesome journal: samples sizes that are too small (but you can encourage the authors to collect more data), a study design that is incapable of testing the hypothesis (but perhaps the hypothesis/study can be rephrased/reframed in a way that it can be tested), and lack of replication at the level where inference is being attempted. For me, this last problem is often the most damning. For instance, if one wants to make inferences about populations in two environments (north vs. south, cold vs. warm), then at least two independent populations of each type must be studied. Of course, studying only two populations (one of each type) is still a good suggestive study; it just might not be good enough for my awesome journal.

How to be an open access editor – 1 simple rule

Accept everything!!!!! In sharp contrast to the above, your role here is to make money for the journal. Yes, I know this is a cynical perspective in a growing culture where open access is considered a paragon of virtue: “make your science freely accessible to everyone, and the world will be a better place.” Having now worked with several open access journals, however, it is very clear that the entire goal is to make money. Consider this: pay-as-you-go open access journals don’t make a cent unless they publish your paper. Stated another way, they lose money every time they reject a paper. Indeed, that is why so many publishers have started open access journals to which they “refer” papers they have rejected from their flagship subscription-based journals. Before these new ventures, every rejected author simply went and paid someone else (if all else fails, PLoS ONE!) to publish their paper instead (Fig. 3) – so the publishers said, “hey, cool, we can also get money from the papers we reject – how awesome is that.” By the way, if you like open access, check out my new super-easy, super-fast, and super-cheap open-access journal “MyScience”:  http://ecoevoevoeco.blogspot.ca/2014/09/myscience-newer-faster-cheaper-easier.html

Fig. 3. PLoS ONE publications (100,000 as of June 2014) = lost revenue to for-profit publishers. The solution: start your own open access journal. (Graphic from PLoS Blogs.)
In my opinion, a better solution is simply to pay for open access at the subscription-based journals, which is nearly always an option, or simply publish in the subscription-based journal and then put the PDF on your website. Yes, I know the publishers imply you shouldn’t do this but I have been doing it for 20 years and no one has complained yet.

Oh s**t.

Having just written the above, it now strikes me that authors reading this post will have a more prosaic inspiration – “hey, I should recommend Hendry as a reviewer/editor – he won’t reject my paper.” Go for it. These days I receive so many requests to review that I turn down most of them anyway: instead only accepting reviews for papers that are squarely in my areas of expertise, which some folks suggest are quite circumscribed. And now I am struck by another realization: editors reading this post might have a different prosaic inspiration – “hey, I better not recommend Hendry as an editor/reviewer as he won’t reject enough papers.” Go for it. I get too many requests to review anyway. 

10 comments:

  1. My numbers would be similar to Andrew’s, although with somewhat fewer “reviewer” roles and somewhat more “editor” roles. Which may be why I’d agree wholeheartedly with Andrew’s “how to be an editor”, but have some reservations about his “how to be a reviewer”.
    I think the key point onw which we’d differ is Andrew’s suggestion that “in the literature” is a binary thing – a paper is either “in” or “out”, and (Andrew is quite right) you can’t keep it “out”. But the literature is not that simple. Instead, we have venerable journals of high quality (think American Naturalist); new journals of high quality (think Ecology Letters); and new journals of abysmal quality (think the Erudite Journal of Ecology and Environmental Research, from which I got some lovely spam the other day). (That we mostly don’t have venerable journals of abysmal quality is reassuring). If I review a paper and reject it, and that paper ends up unmodified in the Erudite Journal, I would not agree with Andrew that I’ve made the literature worse – instead, I would say that I’ve acted to preserve the accurate signalling associated with the location of a particular paper in the literature. “In” and “out” has only limited signalling power, but “Am Nat” vs. “Erudite” has quite a bit. Actually, I think Andrew is tacitly endorsing this view in his comments about journal fit and the quality of manuscripts for the very top journals.
    Having played up our disagreement, let me finish by enthusiastically seconding Andrew’s advice that the best quality papers are made by reviewers and authors working together. Wonderful reviewers have made great contributions to some of my papers – sometimes, while rejecting them!

    - Steve Heard

    ReplyDelete
  2. Steve,

    I would agree entirely if the world were divided into two types of journals: venerable journals of high quality and new journals of abysmal quality. If so, we would all know to pay attention to the former and ignore the latter. However, the world is full of everything in between, including many newish journals of mediocre quality (think ... no just kidding, I won't name any). I would expect that someone rejected from Am Nat would submit to those mediocre journals and not to ones that are known to be abysmal. Since there are dozens of journals of mediocre quality, such is where a paper rejected from venerable journals will end up – not in abysmal journals. Yet readers can’t ignore those mediocre journals since they have some great stuff. So, in the end, a paper rejected from Am Nat will get just as much exposure in YOUR MEDIOCRE JOURNAL HERE. The exception – as I noted – is Science/Nature, where publication will get you much more exposure (deservedly or not) than in Am Nat or any other venerable or mediocre journal.

    Cheers,

    andrew

    ReplyDelete
  3. Interesting read, and I mostly agree, especially with the notion that people should accept papers more frequently because they just come out somewhere else anyway. However, I strongly disagree with the notion that as a reviewer, "You and the authors can work together to craft the best possible paper – what a wonderful world (Fig. 2)." I just don't see why a reviewer should have the power to dictate how I spend my time and research dollars based on a (typically) shallow reading of our work. It is not their paper and it isn't really their place to suggest major new experiments or directions. For every useful suggestion from a reviewer, I've literally had something like 20 useless suggestions that just eat up time and patience. If the evidence doesn't support the claims, then just say that. Perhaps suggest what claims the evidence does support. If the authors want to make a stronger claim, it should be up to them to decide how best to support it.

    More thoughts here:
    http://rajlaboratory.blogspot.com/2014/04/how-to-review-paper.html

    ReplyDelete
  4. Ah, yes, That is certainly annoying. Here is the cool part though. You don't HAVE to follow the suggestions. You can either:

    1. Try to convince the reviewer/editor otherwise through revisions and a good cover letter. I have found this often works as long as you are polite and well reasoned.

    2. Submit somewhere else, which is no different from if you had been rejected. As what I am suggesting amounts to a collaboration between reviewer/author (mediated by the editor), either party can withdraw at any time they like. I am merely suggesting that the reviewer needn't so quickly be the one to withdraw (i.e., reject the paper) thus giving the author more options (withdraw = submit elsewhere, continue = revise).

    ReplyDelete
  5. If I submit somewhere else, I run the risk of having a new editor send my manuscript out to the same reviewers who rejected / made unreasonable revision suggestions at the last journal. Both editors and reviewers work* for multiple journals, often with multiple publishers, so there is no reliable way for me as an aspiring and headstrong author to choose a journal that will not send my m/s to the same jerks who just told me I suck.

    * I realize very few editors and effectively zero reviewers regard that work as their "day job", nor are most reviewers (I don't know about editors) paid for their efforts. It's still work, for an organization.

    Regardless of whether my paper is any good, its eventual publication in any particular journal is not a reliable indication of quality - good journals publish boring, trivial, and just plain bad papers and bad journals occassionally publish really good stuff. There is presumably a frequency / quality distribution for each journal that we, as readers, can use to guess at an individual paper's likely quality, but my hunch is that the variance along that axis is sufficiently large to make any such guesses rather weak.

    And "Don't Ever Reject Papers", justified by "they'll show up in the literature anyways, unimproved" sounds excessively pessimistic - everything is going to hell anyways, don't waste your time paddling against the current. Some papers, surely, need to be crushed on sight?

    ReplyDelete
  6. I couldn't agree more. Good constructive reviewing demands a lot of time many people are not willing to do, unfortunately. Rejecting is just the easy way out.

    ReplyDelete
  7. Andrew, as I previously replied to one of your blogs, I totally disagree with the way you are putting ALL Open Access journals in the same bucket. Given your stature, you may influence lots of younr researchers on this false reality. I know at least one Open Access journal that is not a fit at all with your "PlosOne" vision of all OA journals. What about several excelllent other Plos journals, or some very strong BMCs, or nature Communications. Yes of course publishers are out there for money, but this is another story that concerns essentially any non-Society journals, being Open Access or not...

    ReplyDelete
  8. Oups, sorry for not adding my name on the previous: Louis (Bernatchez)

    ReplyDelete
  9. .. and by the way... you are both an outstanding reviewer AND an outstanding AE ! LB

    ReplyDelete
  10. Evolution Letters, another damn Open Access Journal. Andrew, you think they will Accept everything? .... http://onlinelibrary.wiley.com/journal/10.1002/(ISSN)2056-3744

    ReplyDelete