Friday, January 27, 2017

ALTERNATIVE FACTS: Scientists got 'em too!

“WHAT”, I hear you saying, “That is precisely the point. Science has the facts and the facts are the facts – no alternatives. Anyone who makes statements not supported by the data are presenting fiction, not alternative facts.”

Seen in an Ottawa book store.
Yet maybe we scientists shouldn't be so smug about our FACTS. As the editor and reviewer of countless papers, I am struck by how often two (or more) fully qualified, insightful, and fair scientists can make opposite assertions of fact based on the same data and analysis. Most obviously, data are often interpreted in the light of preconceived expectations (we are all Bayesians) and, so, faced with a series of mixed results, scientists consciously or unconsciously emphasize the subset of results that support their expectations, largely ignoring or discounting or explaining away the rest.  “Ah”, I hear you saying “Those are interpretations, not facts. Facts belong in the results, interpretations are in the discussion.”

I am not so sure. Everyone knows that the same data can be analyzed in multiple ways, each yielding different outcomes. Hence, the statements and statistics in the results are interpretations too, not facts. “Ok, then, the DATA are the facts.” Yet, data are not collected with error, and – sometimes – are biased. Data collected in different ways at different times in different years can yield different outcomes. Different measure of central tendency (mean, median, etc.) yield different numbers. Different data transformations yield different outcomes. Given that data have no meaning without interpretation, even in the narrowest possible sense of a central tendency, data are not facts either.

Alternative facts?
“Ah, but this is what measures of uncertainty are for.” Indeed, standard errors, confidence intervals, credibility intervals and so on can – and should – be calculated but, again, they have no meaning without interpretation. Anytime you look at a result and a credibility interval and say “I therefore conclude the evidence more strongly supports option 1 than option 2.” Sure, but some other scientist can come along and say “I require a higher degree of confidence, so my conclusion is different.” Or. “I would use a different measure of uncertainty.” Or. “You did not include all sources of uncertainty.” Or “You didn’t consider option 3.” And, of course, we ultimately need to accept an outcome one way or the other or we never act on the information we have collected.


So, are there any real facts out there, any “objective truths?” Yes, of course, there are many of them, but many, perhaps most, of those are – ultimately – unknowable. Instead, they are “latent variables” about which we try to make inferences based on our imperfect surrogate measures.

Andy Gonzalez and I with our alternative facts
Alternative facts, which I have just argued are pervasive in science, depend on the level of inference. “Evolution isn’t a theory, it is a fact.” Sure, evolution as a process is a fact. However, evolution in any given case (a particular population facing a particular environmental change) might not be. Don’t believe me? Just look at the knots people studying adaptation to climate change or fisheries have tied themselves into trying to say that evolution has or has not occurred in any given instance. “Look at allele frequency changes,” you might say. But no one cares if those are changing (and they ALWAYS are), we care about the evolution of the traits. “Ok, then, measure the genes influencing those traits.” Yes, that works but what matters is the effect size: how much of the change in phenotype is the result of evolution? That is extremely hard to measure. Climate change is a fact, certainly, but stating that it is a fact is not informative or helpful. We need to know the effect size (rate of change) and its consequences – and then it depends on the place, the time, and the measure taken. Plenty of room for “alternative facts” in this context.

Alternative facts?
So, maybe us scientists shouldn’t be so smug. If we are ever going to state conclusions as facts (as opposed to facts being unknowable), then room exists for alternative “facts” in the same sense. Otherwise we wouldn’t have debates and arguments. Otherwise, we wouldn’t need to replicate our studies, and those of other scientists. Otherwise, we wouldn’t have retractions.

The difference between science and propaganda is that scientists are willing to give up our “facts” when enough evidence suggests they aren’t “facts” after all. 



--------------------------

This post was motivated by reading scientific papers where authors are testing an hypothesis, obtain results that are mixed with respect to that hypothesis, emphasize only those results supporting the hypothesis, and then conclude in favor of the hypothesis. Presumably this sequence of events usually transpires without INTENT to deceive - and, regardless, the consequences are rarely dire. The same cannot be said for the current US government.

3 comments:

  1. I'm glad that you added the paragraph at the bottom. I still find this post to be in extraordinarily poor taste at a moment when everybody with any moral compass at all is freaking out about Trump's crusade against truth. From the title of this post on down, you persistently draw a false equivalence between the "alternative facts" of Trump and his ilk, and the sometimes problematic relationship between absolute truth and science's iterative, flawed attempt at approximating that truth. But Trump's "alternative facts" are really simply lies – deliberate, bald-faced lies, told for reasons of power and profit. Science, on the other hand, has no truck with such lies; any scientist caught doing such things (deliberately falsifying data, for example) gets kicked out of the club. So there is no equivalence there at all, and using the term "alternative facts" to refer to what scientists do is deeply offensive, and potentially quite damaging as well. When people at government agencies like NASA and the EPA are in fear of being muzzled and losing their jobs because of Trump's "alternative facts", and the whole world is at risk of suffering catastrophic effects from climate change because of "alternative facts", and Muslims and Mexicans are being reviled and barred from the U.S. because of "alternative facts", this post seems more than a little tone-deaf. I feel sure that your heart is in the right place, Andrew, which is why I'm not more angry about this. But I don't think you understand how wildly inappropriate this post really is.

    ReplyDelete
  2. I totally agree that what Trump is doing has no justification or rationale. I think he could very well be the worst thing ever to happen to America - maybe even the world.

    Thanks Ben

    ReplyDelete

Predicting Speciation?

(posted by Andrew on behalf of Marius Roesti) Another year is in full swing. What will 2024 hold for us? Nostradamus, the infamous French a...