Lies... Damned Lies... and Statistics!
By Matt Richter
My inspiration for this post is Tim Harford. Tim's book, THE DATA DETECTIVE, is fantastic. The book contains many more strategies and considerations than listed below. And the introduction to the book contains the story about how smoking was initially correlated and then linked as a cause for lung cancer. His podcast, MORE OR LESS, is superb, as as is his other podcast, CAUTIONARY TALES. I’m addicted to both.
So, statistics get a bum rap.
In our daily lives we either don't understand how to use statistics or we don't know how to question what we are seeing. So much so that we too often (societally) dismiss any usage as "fake news." This is like blaming the hammer for failing to darn a sock. Or worse blaming the hammer for putting the hole in the wall.
Statistics is a tool. Tools can be used properly, as designed. Or, they can be applied badly or improperly.
When used properly, statistical analysis has helped save generations. Whether from the identification that smoking causes lung cancer to the safety of vaccinations. Statistics have even been used to identify large scale criminal endeavors, or find evidence to catch serial killers (see The Art of Statistics by David Spiegelhalter for more on those). Over and over, when used properly, statistics enables us humans to see beyond the density of our immediate surroundings and explore the bigger story- the bigger picture of what is likely.
As L&D pros, good data analytics require good statistics. When faced with data- especially analyzed and interpreted data, consider the following eight factors— in no particular order (again, if you read Tim Harford’s book, you find a bunch more:
1️⃣ THE SAMPLE MATTERS. It is collected data as a representation of a larger group. In the smoking example, that would be smokers & non-smokers over a period of time with/without lung cancer. If your sample is biased or non-representative, the conclusions cannot generalize to the greater population. And size does matters (joke intended)! Bigger is more reliable.
2️⃣ BIAS. We all tend to look for what we expect or hope to see. The researchers do, as do we readers of that research. So, we need to look for how researchers controlled for those biases. Good research methodology often handles this. But we also need our own minds to be open to see how the forest might look differently than we expected.
3️⃣ METHODOLOGY. How are the data collected? Or, the research itself designed? This can get pretty complex. This is where I often turn to the research translators- those who are trained in both research methodologies and my respective field- to provide insight. A poor design, or a poor gathering process will yield garbage. Garbage in gets you garbage out. I have mentioned many of the translators I turn to before, but here they are again… Paul Kirschner, Will Thalheimer, Clark Quinn, Julie Dirksen, Patti Shank, Karl Kapp, Ruth Clark, and more!
4️⃣ CORRELATION VERSUS CAUSATION. Lots of things are linked. For example, it's cold outside. Winter is flu season. These two instances are often correlated. It is easy then to draw the conclusion that the cold causes the flu, rather than a virus. Wearing a scarf will not prevent the flu. A vaccine, hand sanitizer, or washing hands might. A flu vaccine… perhaps. Causation is when one thing causes another. We need to differentiate and take care not to mix the two up. Many of the reasons educators still claim learning styles should be considered when designing learning is due to this confusion.
5️⃣ REPLICATION. The inherent value of large numbers is you can see patterns, consistencies, correlations, and causes. The risk, however, is that with large numbers, you can also get anomalies. For example, Tim references the Sheena S. Iyengar and Mark R. Lepper study that set up a jam-tasting stall. When more jams were available to try, fewer people bought jam. Fewer options- and more jam was bought. The conclusion- too much choice can be a bad thing. The problem is that this study has failed to replicate enough to be reliable. It's not that the researchers were trying to fool us. Not at all. But one study does not equal said reliability. In L&D, we take a study- often translated by journalists- and extrapolate the conclusions as fact. We need to search out replication in order to see if the original study and its conclusions were an anomaly.
6️⃣ PUBLICATION. Tim notes that journals are often incented to publish conclusions that are interesting. And many journals avoid publishing studies that replicate (or fail to do so) studies because they are, frankly, not so interesting. In L&D, most aren't trained researchers. We don't have the habit of asking about replication. So, we read a study (or a summary) posted in a respected journal, and we accept it as truth. But we still need to be critical thinkers and challenge.
7️⃣ PERSPECTIVE. Time, reference points, experience, and other factors influence those doing the research, reading the research, and interpreting the research. How far away is Mars from Earth? Well, it depends on perspective. Where are we at a given point in our respective orbits around the sun? At our farthest points we are ~401 million km. At our closest, ~56 million km. On average, ~225 million km. Just asking for the distance isn't enough. We need the context of the question to pin it down. All three options can be true. Or, even false. In L&D, what is the actual performance problem? What is the context? The audience, etc.?
8️⃣ THE QUESTION. Good research starts with a question, or a hypothesis. But we have to be careful of results that apply HARK-ing- Hypothesizing After Results Known. In other words, confirmation bias. We search out the data and the hypothesis to match what we hope to be true. Because combinations of choices can lead to literally a myriad of conclusions, design methodology is imperative to evaluate. Harford shares the example of Simmons, et. al.'s "proof" that listening to "When I'm Sixty-Four" by the Beatles would make you 18 months younger. How so? Buy Tim's book and read it!
I already mentioned Tim’s book, THE DATA DETECTIVE. And I mentioned the wonderful THE ART OF STATISTICS by David Spiegelhalter. I have also enjoyed greatly, Nate Silver’s THE SIGNAL AND THE NOISE. If you want to dig into a wonderful concept in probability studies, Bayes’ Rule, Silver gives a wonderful set of explanations and usages that are easy to grasp. and the brilliant Sharon Bertsch McGrayne provides a deeper and more nuanced explanation and history in her book, THE THEORY THAT WOULD NOT DIE. To see how statistics and research can be applied in large scale public health settings, the legendary Hans Rosling’s book FACTFULNESS is fascinating. And I like, even more, Steven Pinker’s ENLIGHTENMENT NOW: THE CASE FOR REASON, SCIENCE, HUMANISM, AND PROGRESS. Another fun resource is the MUNK DEBATES. In 2015, Pinker and Matt Ridley debated Alain de Botton and Malcolm Gladwell on the question of whether humankind’s best days lie ahead. You will see each of the above factors used and applied during the debate… mostly by Pinker and Ridley.😁 Enjoy!