The Hot Hand, Freakonomics, Climate Denial, and Closure Temperatures – Assumptions in Models

I started listening to a few sports podcasts rather religiously about 3 years ago. I self select for shows, avoiding anything that involves someone with a nickname or who yells all the time, basically avoiding anything that takes sports too seriously. My favorite, as I’ve mentioned in an earlier post, is Slate’s Hang Up and Listen, a shining star in the otherwise unimpressive Slate universe </rant>. One of the things that has really struck me while listening is the importance of “advanced statistics” to modern sports. These aren’t things that casual or stat-uninterested fans like myself care much about or can even define (like PECOTA, which is obviously an acronym for Player Empirical Comparison and Optimization Test Algorithm, duh), but include a huge assortment of metrics and statistical analyses that are designed to reduce the value of a player or strategy down to something that can be ranked and quantified, or to help evaluate what “common knowledge” is just not true. Some of these things seem reasonable, especially the emphasis on the absurdity of some of the more traditional stats and how they don’t really reflect the worth or accomplishments of a player, or how dumb some ideas are, like punting from your own half of the field. What strikes me though is the pretense that somehow these measurements and analyses are free of subjectivity, that they somehow elevate sports discussion to some objective plain far above the knuckle dragging fans and coaches who rely on their gut feelings. Emblematic of this movement for me is the Hot Hand Fallacy, an idea that has been around for a while, but recently became the poster child for how dumb casual fans are. The basic idea is that some people believe a certain player can develop hot streaks, where they are shooting a basketball, for example, statistically better than expected. According to a variety of analyses that look at the chances of making a basket as a function of if you either made or missed the previous shot, there was no such thing as a hot hand. Players basically have the same shooting percentage, but the human love of identifying streaks and patterns imposes the idea of a hot hand on our brains when we see them make 4 or 5 in a row, even if that isn’t statistically significant. The hot hand fallacy popped up all of the time, people used it to complain about players and coaches taking too many shots, changing their game plans, blah blah. To sum up, if you believed in hot streaks in sports, you were probably an unsophisticated doofus who couldn’t do statistics.

But then this study came out. What these authors suggested is that when someone feels like they are on a hot streak, they start taking harder shots, which means they make them less often, because the shots are harder. If you control for that in a particular way, then the analysis does support that players will at times develop a hot hand, that they will in fact shoot at a statistically higher percentage than expected.

I’ll admit, when I first heard about the hot hand fallacy, I was also one of those people who, maybe because I am a snooty scientist, liked to think of how smart I was for knowing that hot hands were a myth. This was dumb on my part, because I didn’t question the assumptions and simplifications that went in to the statistical model.

I even had precedent. A few years earlier I’d read Freakonomics, the best-selling book that used economic/mathematical analyses to study a variety of phenomena, basically applying economics math to human behavior, crime statistics, and a bunch of other things. My feeling reading this book was that you could summarize each study in the following way:

You might think this one thing, but you aren’t smart, and economists know everything and will now show that you are wrongedy wrong wrong.

I don’t mind being told I am wrong, I don’t mind learning new things, and I like to think I have a very open mind (for example, I now admit that my wife was right and a fish spatula is the greatest kitchen utensil ever made). However, I am inherently suspect of statements that imply some sort of mathematical analysis is straightforward and objective, especially one that requires a host of simplifying assumptions. There isn’t an unambiguous way to leap observations across scales of human behavior, measurement, psychology, and history. To do that, you need to filter and simplify data and define rules of cause and effect. So any study that doesn’t explicitly discuss its assumptions and simplifications should be suspect. This goes beyond the normal issues of reporting science in the mainstream that blow conclusions out of proportion, and makes it sound as if there are objective methods for understanding the direct effects of actions and policies. The farther you get from an action the more the effects are predicted in a Rube-Goldberg fashion. Sometimes this works well, and there are obviously some things that matter more than others, but whenever someone says “research shows X,” and they can’t explain the assumptions and simplifications, or don’t at least acknowledge them, then we should be suspicious.

It isn’t that the people doing the work aren’t smart, but they can often fall prey to a belief that mastery of one subject means mastery of others. My complaints about Freakonomics might stem from my inherent distrust of economists. This is certainly at the core of a recent event involving Nate Silver and his blog FiveThirtyEight.com. Nate Silver is a statistician who became famous for correctly predicting the last presidential election, and used that as a platform to launch his website. The premise of the website isn’t bad, using statistics to analyze a bunch of different things. The problem was when the blog waded into phenomena well outside of its depth. Specifically climate change.

Michael Mann has detailed the multiple problems with Silver’s climate analysis in many places, but it can be summed up as a smart person believing that their background prepares them to understand everyone else’s specialty. We’d balk of course if Nate Silver announced that he was now using his expertise to perform open heart surgery, build bridges, or defuse a nuclear bomb, but when he contradicts the experts in something equally as complicated, climate modeling, somehow we are OK.

Of course a big part of becoming an expert in any field is learning how to understand the data, how to make appropriate simplifications and assumptions, and how to evaluate the quality of interpretations based on the quality of the data and the construction of the models and other interpretive framework. Most of the IPCC reports of course are highly detailed discussions of climate change minutia, just as most of the papers I read or review are discussion of these details. The IPCC of course discusses a variety of independent models, using their dispersion or agreement as evidence for their reliability. Like many climate change deniers, Silver falls into traps that are shockingly ignorant to those who study the field, but is somehow blinded by his own confidence into thinking he has found something people who’ve dedicated their lives to a field have overlooked. Silver, and other deniers, aren’t that different than the man portrayed in the This American Life story Sucker McSquared, where a smart guy decides that Einstein was wrong, even though he doesn’t understand the basic math required for physics.

So what does this have to do with thermochronology and closure temperatures you ask? As a field, thermochronology is used in two main ways. It is either interpreted in a rather straightforward way using simple graphs, or it is modeled using a variety of complicated algorithms. I first learned about the models at a 2005 short course, and specifically learned about how wrong some of the assumptions I made about how heat acts in the lithosphere, how it is advected especially, can lead to gross misinterpretations of low-temperature thermochronology data. The models that have come on the scene have largely helped address this problem, focusing on the shape and evolution of the subsurface thermal structure. As thermochronology has been used in less and less ideal settings, these lessons have become more and more important.

What has been become apparent though, is that these models, while fixing one set of assumptions, have required many other simplifications that can often be just as significant. Landscape evolution models, for example, must often be simplified from actual landscapes, just because of limitations in computing and the inherent complicated nature of nature; faults become planar, erosion becomes instantaneous, and disparate lithologies become particles that all behave the same. In the same way the concept of the closure temperature, or closure depth, of a mineral is often simplified. For example, we know now that the diffusivity of He in apatite and zircon, and likely every other phase, is a function of radiation damage, which is itself a function of both U and Th content as well as the thermal history of the grain. More importantly, each of these things, closure temperature and date, come with associated uncertainties. In many cases, the uncertainties in the diffusion kinetics themselves are significant enough to affect models, yet there are as yet no good ways to assess the uncertainties in these models. In my reading and reviewing at least, I have yet to see “predicted dates” associated with realistic uncertainties.

My point isn’t that models aren’t useful, on the contrary, I find them insightful and very interesting. When used properly they can add a great deal to a study, and help you design your field time and sampling strategy to maximize your time and effort. My point is that with all of these things, it is absolutely crucial to understand the inherent assumptions and simplifications of any model or method that you use, and that just because you are smart and have mastered a technique doesn’t mean that you can honestly evaluate the world. I am sure these things will come, but IMHO assessing uncertainties in any interpretation, including the assumptions and simplifications used to construct the interpretive framework, whether it be a simple graph or a complex algorithm, is absolutely essential. For many of us, the details of these models are out of our depth, like Nate Silver wading into climate dynamics, but that doesn’t mean we can’t be aware.