On retraction

Prologue: The subject of scientific retraction has been very much on my mind in the past year, as an editor, a co-author, and a member of the scientific community watching retractions (or demands for retraction) play out at other journals. Most recently, I had a stimulating Oxford style debate with Dr. Elisabeth Bik, for the annual meeting of the Council of Science Editors, which had me thinking at length about the procedures we follow and decisions we make. This culminated over the weekend in a stimulating conversation about science, law, and ethics of retraction while on a hike with a friend who is a lawyer. Inspired by that conversation, by the past year and a half, and by my own experience retracting a paper, I feel driven to share some of my personal perspective about scientific retraction. I want to note that in this essay I am expressing my personal opinions. These opinions do not necessarily reflect my behavior or decisions as Editor: in that role I am bound by traditions and procedures that may differ from my personal preferences. I also want to emphasize that I will be speaking in broad generalities of ideas in this essay, and not about particular cases. The following text is meant to be thought-provoking, stimulate debate, and is not a definitive statement of editorial policy.

    - Dan Bolnick

Let's begin by considering what retraction is for. They say science is 'self-correcting', meaning new findings get published that may contradict and eventually eclipse old misconceptions. So why do we issue retractions at all, when we could just let new papers lead us forward? I see two distinct motives (there may be more, but I think these are the two major principal component axes, so to speak).

1.  Removing information from the public record that we now know to be false. Sure, new papers may later get published that correct our view. Self-correcting science and all that. But, like an old land mine from a forgotten war, the incorrect paper is still there for some unwitting reader to stumble across and, not knowing about the newer work, cite and believe.

2. Punishment for misdeeds. The retraction itself may be viewed as a form of public censure, a slap on the wrist.

I want to do a bit of a deep dive into each of these, because each seems simple at first glance, but has complexities and caveats once we delve into them. To start, it helps to recognize that we can have concerns about papers for many distinct reasons. Sure, there are obvious and egregious cases of fraud where the data and conclusions are fundamentally false - duplicated or altered images that are central to the conclusions of the paper, for instance. But it can be more subtle than that, so let's do something analogous to the classic table we all learn about type I and type II error in statistics (false positive, false negative). The rows of are table concern validity of the conclusions: a paper can report results that are factually correct, debatable, or incorrect. The columns are about the evidence, which can be rigorous, flawed/low quality, or unethical in part, or fraudulent at heart.

Conclusions are:	Rigorous evidence	Flawed evidence	Unethical in part	Fraudulent at heart
Valid	Great paper!	Right, but for the wrong reasons	Fruit of the poisoned tree	Falsifying to reach a true inference
Dubious	Over-reaching	Debatable & unclear	Bad scientist!	Why bother?
False	False positives happen. In good faith.	Honest mistake. Luckily, science is ‘self-correcting’	Really bad scientist!	Falsifying to reach a false inference

I won't take the time to consider every single permutation, but let's start with a case study. On my hike yesterday, my lawyer friend asked whether Charles Darwin's Origin of Species was correct on all points. Clearly not, I responded. In particular, Darwin had a deeply incorrect understanding of genetics, that led him astray on a few points. We know better now. Science being self-correcting and all that. So, my friend pressed, why don't we 'retract' Origin of Species (nevermind that it wasn't published in a journal)? It contains errors. We know that unambiguously. But it is correct in the main, and of historical value.

Let's proceed to a trickier one - Wilhelm Johanssen published a paper in The American Naturalist in 1911 expounding the genotype concept of heredity, a revolutionary paper that defined 'genotype' and (to quote a recent historical paper :"the theory must be recognized as a creation that provided the theoretical foundations or the framework for the upcoming reductionist material science of genetics, quite beyond the perception of its instigator." But, towards the end of the foundational paper Johanssen makes fun (rather harshly) of the notion that genes have anything to do with chromosomes. 

So, now we have a classic paper in a journal (hence amenable to retraction, unlike Origin of Species) that makes a false claim. Should we retract it? Well, my first instinct is that it is old and part of the historical record. We make mistakes in science, and we advance beyond those mistakes, and it is valuable to historians of science to leave our tracks in the sand as we stumble towards (the mirage of?) some truth. There's a legal idea of a statute of limitations - a time period beyond which someone cannot be charged with an old crime. The time frame varies with the severity of the crime (in the eyes of the law), and some have no such limit. So is there a statute of limitation on being wrong? In that case, how old must a paper be to be retraction-proof when proved wrong? Nested Clade Analysis was wildly popular among phylogeographers when I was a beginning graduate student, but we now know the emperor had no clothes, and nobody uses it anymore (I hope). Is that old enough to escape retraction? How old is too old to bother? The answer depends on the severity of the crime, the culture and nation. It also matters a great deal whether the action was illegal at the time it was committed. Today the standard is to judge past actions by the contemporary rules of their era.

Or  is it not about age, but intent? Do we tolerate good-faith error? Johanssen had his reasons, and reached a reasonable and honest conclusion by the standards of the day with the evidence available to him at the time, and we should not judge him harshly for not knowing what we know now. Templeton was sincere in his desire to reach phylogeographic inferences and at the time we lacked the toolkit for approximate bayesian computation (ABC) methods that can test complex biogeographic genetic models. He meant well, he did the best he could with the tools at the time, and if it didn't work as well as he thought, it was honest and well-meaning, and so no retraction is needed. As an even more obvious case, I might set out to test a null hypothesis with a statistical significance threshold of alpha = 0.05. Like, maybe I want to know whether drinking carbonated water cures COVID (hint - it won't though luckily its harmless), the null is that carbonated water has no effect on COVID. And let's say the null is genuinely true (carbonated water won't fix COVID). I do my experiment, and by sheer bad luck I am one of the 5% of such studies that find a P < 0.05, and I publish a paper rejecting the null. I'm wrong, but I followed standard procedures. Should that fictional example result in a retraction following other studies? Do we generally retract all significant results when subsequent more powerful studies reach the contrary conclusion? Or vice versa? Currently I think the standard is very much that errors made in good faith don't warrant penalties at least. If they are recent, then a correction is certainly called for. Maybe even a retraction if the core conclusion is wrong. 

I did a retraction for this reason myself - a reader noted he couldn't reproduce a result with my data. I looked back at R code I wrote in 2007 while first learning R, and found a one line error based on a misunderstanding of how tails of probability distributions were being reported. It affected the core result. So even though it was in good faith, I retracted the paper (this was a few years ago). Knowing what I know now, I might have urged my past self to a less knee-jerk extreme reaction. I was driven by emotional horror. What I ought to have done was to publish a Correction. The data are still factually valid. The question is still interesting. It's just that a positive result (morphological differences between individuals affects diet overlap between them) is now a negative result (no, it doesn't). That's still a useful result that I would cite, just for different reasons. This falls into the second column bottom row - I had published flawed evidence for something that was false, and I self-reported the flaw.

There's a related tricky issue particular to theory papers. I'm not aware of many cases (any) of fabricated algebraic solutions or model results. But there are plenty of cases where authors made honest mistakes in their math, stating somewhere that x = y when in truth x != y. These can be simple typos made during writing (not analysis). Or, they can be part of the calculations and, when changed might have little to no impact on the paper's conclusions, or could radically alter them. The former clearly merit correction to the equation, the latter teeter between correction and retraction, depending on the severity of the error. But again, intent matters here. It is easy for math to be demonstrably wrong and to prove errors, compared to data analysis and experimental designs that can be critiqued but are more a matter of judgement. Yet we don't often retract math errors because corrections suffice and errors made in good faith are typically treated more leniently. 

So on the whole, I lean towards the personal opinion that good-faith errors, whether a coding problem or a random false positive or false negative (they happen!), are best corrected as science builds on past results, rather than penalized. Positive results might be corrected into Negative results. And as a field we want that to be made public freely and openly. 

We want to encourage self-correction. When Corrections (or even Retractions) have a powerful negative connotation, scientists who find (or are told of) an error in their own work will naturally hesitate to Correct or Retract. I sure stressed about it. I shouldn't have. When I retracted and publicly explained why, I received incredible positive support from the community. We want science to be self-correcting, to more rapidly reach true inferences. If we penalize self-correction, if we put an onus on it, it will happen less. People respond to incentives. If you want something to be done, reward those who do it, not punish them.

Now let's delve into a grey area: sometimes scientists collect data and reach a conclusion that is actually valid in real life. But, the evidence that they use to support their conclusion is flawed. I don't have a specific case in mind, but hypothetically imagine a researcher sequences a population of fish to look for the presence of a particular allele. It turns out their sample was contaminated, or swapped for another sample, so they aren't sequencing what they think they are.  But by happy (?) chance, the contamination or wrong sample actually has the allele, as does the population they think they are sequencing. So they correctly conclude that the allele is present, but they do so for a wrong reason. This is maybe especially likely in ancient DNA research, where cross-contamination (and absence of DNA in focal samples) is a common problem. I like to think of this hypothetical case because it highlights the role of truth / falsehood in our judgements. As an editor or author, I would be very tempted to retract such a paper, and at a minimum a correction is needed. Their published report cannot in fact prove what they claim, because the evidence marshaled for it is fundamentally irrelevant or incorrect. Yet, this is done entirely in good faith, and the outcome is factually true.  This is the "Right but for the wrong reasons" cell in the grid above. So retraction isn't simply about being correct or incorrect.  Nor is it simply about malicious fraud / conscious misleading.

Just below this in the "Flawed evidence column" I put "debatable". This is for cases where scientists are maybe tackling a difficult subject and the data are simply not definitive. It may be hard to collect good data and the results may be ambiguous. The end result might look promising to one person (who favors the idea being tested) but sloppy and unconvincing to a skeptic. This is the stuff that scientific debates are made of. My favorite these days is the unnecessarily vitriolic debate over evolutionary neutral (or nearly neutral) theory. There are strongly held differences of opinion and each side genuinely believes the other is deeply wrong. This is a scientific work in progress,  that  may be resolved with further data or theory. It is definitely not the journal's job to retract something that is in this debatable category, even if it may eventually prove to be wrong as agreed upon by all parties in the future. We are a platform for vigorous informed debate, not partisans in it. And when the debate is eventually settled, we keep the record of that process, rather than deleting the losing side via sweeping retractions.

So to recap mid-way, I started by saying one reason to retract is "Removing information from the public record that we now know to be false." That's appealing and obvious. Except that we don't usually retract things that we now know to be false, if they are old, or if the error was arrived at in good faith. And sometimes we might retract things that are actually true, if they were arrived at for the wrong reasons. 

Let's explore this latter point some more, by considering true conclusions reached in bad faith. What if a published paper reports a true conclusion, with rigorous valid data and large sample sizes (sounds great, right?), but having obtained the true data through unethical means. Maybe they didn't have a collecting permit to sample the focal species, or do the experiment in the field. Maybe they didn't have IACUC (animal care) approval, or IRB (human subjects) permission? They smuggled permitted-collections out of the country? We are moving into the "Unethical" column of the table above. Scientists sadly are sometimes overeager to do their work and adopt unacceptable methods. What do we do in this case?  As an Editor (and in general as a member of the scientific community), I think a paper should not be accepted for publication if it didn't follow proper legal and ethical procedures. Let's go with a problematic hypothetical example. A researcher discovers a miracle cure for cancer. And in their eagerness to test it they skip IRB approval, they skip informed consent, they administer it to unknowing, non-consenting individuals... and they cure their cancer. Should that paper be published? It is tempting to say, hell yes we want a cure for cancer. But we got there through procedures we cannot condone. What to do? One might be tempted to keep the paper and result, but levy some penalty on the researcher. But ultimately, it is the duty of a journal to retract papers that are arrived at via unethical means, regardless of the validity of the outcome. That miracle cure for cancer? It'll have to wait for someone else to replicate the initial study via ethical and permissible means. That means a delay in treating people with cancer, and loss of life, which feels deeply wrong. 

(Note afterwards - one twitter reader noted that if the work was done unethically and in secret do we really trust the results to be true? A fair question, but for the purpose of exploring this ethical conundrum please suspend disbelief for a second and accept that in this fictional situation the actual scientific conclusion is documented rigorously enough that we can trust its veracity).

(Note afterwards - thanks to @SocialImpurity for pointing out this real life case that is relevant to this ethical dilemma)

There's a fascinating analogy here. In US courts, there is a legal metaphor known as "The fruit of the poisoned tree". (Note, this is derived from British Common Law, and is not a universal doctrine globally). Say there is a burglary. A cop catches an accomplice, and in order to find the ringleader of the gang, tortures the accomplice (that's not ok) into revealing the location of the getaway car. Finding the getaway car, which matches the security videos, and has fingerprints all over it and the money hidden under the seat, the police identify, capture, and prosecute the ringleader. The good evidence of guilt (the fingerprints on the car with the money) is the product of a bad process (torture, the poisoned tree). And the fruit of the poisoned tree is inadmissible in court. Here's the bit that makes many people uncomfortable: the thieves go free. Not because they were innocent. They absolutely were guilty. But because the evidence of their guilt is the result of a flawed process. So we throw it all out. Much like we would throw out all scientific results (the fruit being the miracle cure) that arise from the poisoned tree (the IRB human research violations). The motive in both cases is to deter bad behavior (by scientists, or police) in pursuit of a good outcome. The consequence is that good outcomes (cancer cure, convicting the guilty) are set aside fully knowing they are good outcomes. Now, I think many of us get uncomfortable at this result, and would argue that the guilty criminals should be convicted and the police punished. But, that's not the American & British legal tradition. I should note that the Fruit of the Poisoned Tree metaphor is strictly about police behavior, and has no legal teeth that extend into other areas like scientific publishing.

Note that the fruit of the poisoned tree doctrine will probably make you squirm. I do. I'd love to see that miracle cure made available. But what ethical boundaries are you willing to waive? Experiments on people without consent. That's bad. But do the ends justify the means for you? That's the logic (I presume) that drove the syphilis experiments on the Tuskegee Airmen, or Dr. Mengele's horrific experiments on Jewish concentration camp prisoners. Are you willing to use the results of those experiments, willing to publish them, or let them remain on the books? I'm not.

But these are extreme cases. Let's turn the dial of moral outrage down, and likewise adjust the slider bar away from "cancer miracle cure" towards something more realistic. Would you publish (or, retract if already published) a paper that used stolen archaeological artifacts? Genome sequence data taken from non-consenting individuals unaware of what their drawn blood was to be used for? Currently the expectation is that Editors like myself would reject papers that used ill-gotten samples to reach a conclusion, regardless of the validity of the conclusion. And that standard makes no reference to how important the conclusions are (e.g., an interesting intellectual advance, versus the miracle cure, both are equally subject to this standard).

So, hopefully we are in agreement that a scientific result (whether true or not) arrived at through unethical means will get rejected (and maybe the experiment redone properly and ethically, if perhaps more slowly). But let's unpack this some more and ask what the statute of limitations is here. How far back does the principle apply? Institutional Review Boards for human subjects research began in 1974 (see this article for a useful history). That means no research before 1974 (the year I was born, incidentally) was IRB approved. Today we require IRB, so does that mean all journals should retract human subjects research from before 1974? Clearly we don't do that. The key difference is that the pre-1974 papers were adhering to the ethical standards of their day.

There are other kinds of unethical behavior that get caught up in scientific publishing. What constitutes retraction-worthy unethical behavior? What if the communicating author(s) left someone off the author list who contributed significant work that would normally be sufficient to warrant co-authorship. That's not cool, its not ethical, but is it retraction worthy? What if they added an author who really did nothing at all, simply to gain credibility by association, or as a bribe for some quid pro quo? Unethical, in both cases. But retraction worthy? Probably not, partly because it can be hard for editors to effectively adjudicate who did or did not earn the right to be listed as an author. And partly because there are other simpler remedies - adding an author, or publishing a statement removing an author.

To push this line of questioning into an even more uncomfortable realm, what if the lead or senior author on a manuscript, or a published paper, was guilty of sexual assault? Does unethical behavior that is entirely decoupled from the paper in question, impact the editorial board's view of a scientific text and conclusion? Should we decline to publish, or retract, papers by individuals whose behavior the Editor finds abhorrent? Publishing their paper promotes their name and role in the scientific community, and we might not want to do that. But, the technical conclusions of the paper are (in this hypothetical case) entirely unaffected by the authors' behavior. If we open the door to editorial decisions based on abhorrent behavior, who gets to determine that criterion? Something that is abhorrent and unethical to one person might be celebrated by another (not sexual assault mind you, but other forms of sexual behavior, or political or religious belief perhaps). How much evidence must be given to the journal's Editor to reach an informed decision about whether or not the abhorrent behavior took place? This particular issue ties me in knots, and I confess I don't know what I would do in such a situation, and am simply glad that it hasn't come up in my role as Editor handling new paper submissions, to my knowledge (I wouldn't be at all surprised if some current or past authors have been guilty of such offenses). What would you do? 

Speaking of abhorrent, what about racist and eugenicist research of the past? The American Naturalist has plenty of articles by Charles and Gertrude Davenport, leaders at the Cold Springs Harbor lab for eugenics. Should we retract their papers from the 1900's through 1940's? Or, one of the first women to publish in The American Naturalist wrote a horribly racist rant that is offensive enough that, when I quoted from it in a lecture (to drive home the point forcefully and openly that AmNat published awfully racist stuff), a audience member subsequently admonished me for showing it. Should I go back and retract that paper, which I suppose I have the power to do as Editor? I wouldn't, simply because I think that would be whitewashing an ugly past that we are better off acknowledging and confronting openly rather than making it as if it never existed.

We are almost done, I swear. But I'm having fun posing these questions.

The last column in my table, above, concerns downright fraud.   Image manipulation. Data alteration or fabrication. Changes to data or misreporting results with the intent to mislead. The italicized bit can be hard to prove without mind-reading, so often we content ourselves with evaluating whether data or an image are biologically plausible. If not, the results are false and data are not trustworthy regardless of intent. In effect, we say that (not knowing an authors intentions or actions) a paper could either be in the right hand column (fraud) or the bottom row (falsehood), and either one could be retracted. In recent cases, decisions to retract reflected patterns in data that appear to be fraudulent, but we cannot with certainty discern whether there was bad intent, or simply large-scale and recurrent accidents in data management. The latter seems deeply unlikely, but from the standpoint of journal decisions (retract or not) the distinction is irrelevant.

So to recap this second part of my musings, I started by saying the second reason to retract is "Punishment for misdeeds." Certainly the 'Fruit of the Poisoned Tree" doctrine I described represents a punishment, because the scientific conclusions may be really true and valuable yet we'd still retract - as a penalty and as a deterrent to future scientists considering the same misbehavior to get data. As editors and a community, we would walk away from true knowledge if it was ill-gained. But I expressed discomfort at the thought of using retractions as a stick to punish some other forms of unethical behavior. And actual fraud we might argue that the retraction is for biological falsehood (or lack of trustworthy evidence at least), rather than as punishment per se.

Note that I'm absolutely not saying that misbehavior should go unpunished. Sexual assault, for instance, should be prosecuted in court. Sexual harassment as well, or at the very least pursued seriously and in good faith by the university or institution where the harassment took place. What I am saying though is that punishment may not be the journal's primary role. The journal serves to communicate information, and (to the best of our abilities) to check that the information is good. We can decide to stop communicating something that we no longer trust. But our capacity for punishment is very limited. At most, I could retract papers from an author who committed systematic fraud. But I'd check each paper separately rather than assume the fraud was universal. Perhaps I could refuse to consider any further submissions from that author. But what if they learned their lesson and now pursued an honest and careful path to new science? Should we forgive and forget (and verify)? Ultimately, journals only have the ability to punish using the limited tools at their disposal - the papers they publish. It is the employer (the university, institute) that has the power to fire, or reassign duties away from research to teaching, or put on unpaid leave, etc. The employer provides money, and money has a leverage the rest of us lack.

There's another reason why I generally am wary of using retraction as a form of punishment. I noted earlier that we want to encourage people to self-correct - that helps clean up our literature faster. When retractions are systematically equated with punishment, authors who find errors will be more hesitant to self-retract. One solution would be a linguistic one - if we adopted two different terms for two different kinds of retraction. Honorable self-correcting retraction, and dishonorable retraction for fraud or unethical behavior. Call them subtraction and detraction, respectively (I'm open to alternatives). But I think it is crucial that we punish, and crucial also that we reward honest self-correction. These need to be kept separate in the sociology of science.

   

So, back to my original premise: when is it appropriate to retract?

Are we removing information from the public record that we now know to be false? Are we punishing? The answer is subtle, because we don't want to conflate punishment with retraction. Because journals have the power to retract but limited power to punish, which is better done by employers (who journals can and should notify). Because we sometimes retract things that are true (fruit of the poisoned tree) but often don't retract things that are false (in good faith, or old). 

This subtlety is made even more challenging by a whole other set of questions: is the false or fraudulent information central, or peripheral, to the conclusions of the paper in question? Obviously if the primary point of a paper turns out to be fundamentally incorrect (as in my own retraction for a coding error), retraction is a fair outcome. But what if it is a minor aside? Let's say Matt Daemon describes a detailed experiment on how the proportion of feces in dirt affects plant growth on Mars. And he says in the paper that he was wearing a blue space suit when in fact it was green. That's a falsehood, but irrelevant to the biology. It should be corrected, but has no bearing on the conclusions. Or, he falsely reports that feces helped grow potatoes on Mars, then builds and obtains analytical solutions for a set of ordinary differential equations that correctly establish that nutrient addition would help. The math is true. Do you retract it? Doing so would be leveling accusations at an innocent equation by assuming guilt-by-association.

The last thing I want to comment on is the role of the journal, and those who volunteer time for it, in publicizing retractions. If the point of a retraction is to correct something we now know is false, we must reach out to people who previously read the paper and notify them of the retraction. Otherwise, they will continue to hold a false belief based on their original reading of the paper before retraction. Furtive retraction is no retraction at all. And, if the point of a retraction is punishment (which, again, is not my core belief), again the punishment has the most impact if it is public. Either way, retractions need to be clearly conveyed with the original paper on the journal website, with corresponding changes to Jstor, PubMed, Google Scholar, etc. Using social media to disseminate the fact of a retraction is entirely reasonable as a means to connect to readers who use that channel. The drawback of social media of course is the ease with which it tips into personal rather than professional. But any procedure that disseminates the announcement of one (or more) retractions will invite gossip, speculation, extrapolation, and personal judgements, especially when misconduct (or the strong suspicion thereof) is involved. 

The other interesting side of social media dissemination of Correction / Retraction decisions, is that it invites armchair editors to inveigh. I've been called a "coward" and "lazy" on social media for a decision to issue a Correction rather than Retraction, by someone who didn't know the paper, hadn't read it, didn't understand the negligible role that the data played, or the nature of the model being reported (not parameterized with or testing data). If there's one thing I hope the essay above has taught, is that retraction and correction span a diverse set of complex considerations. There are considerations of scientific fact or error. These may be central or peripheral to the paper. They may be reached by ethical or unethical means. They may be old errors or new ones, satisfactory by the standards of their day but not today. And each of these dimensions is a continuum ranging from totally-fine to absolutely-bad with grey areas in between. Sometimes, questions of fraud or retraction are unambiguous. Editors should engage fully with these and do the right thing. But challenges to papers can also fall into grey areas, or be good by one criterion and bad by another. The Editor's challenging job (supported by their editorial board and reviewers in consultation with authors) is to reach a final judgement. These are, in my experience, sometimes easy and sometimes hard. So, if an Editor simply ignores critiques, that's bad. But if they don't act exactly the way you would like, consider that they are weighing many themes and may have a justifiable (not necessarily cowardly / lazy) rationale for their decision.

Thanks to J. Shukla for inspiring conversation.

PS, added after: For some reason this site isn't allowing me to reply to comments. My apologies. So a couple of responses here.

1. I remain horrified that Google Scholar and WebofScience don't pair retractions with their targets so when a paper comes up in a search you immediately see it is retracted.

2. I haven't ever received a request to retract a paper because the author was engaged in personal misconduct (e.g., sexual harassment, etc), so the scenario mentioned there is hypothetical, meant to be thought provoking and ask, what are the boundaries between misconduct that warrant retraction, and misconduct that does not. The Committee on Publication Ethics standard is that it should be pertinent to the conclusions of the paper, but that has fuzzy boundaries.