Why we draw large conclusions from small sample sizes and how it affects our leadership.
This year, I have been listening to more audiobooks as a form of diversifying my learning and sourcing inputs of knowledge that I wouldn’t traditionally have or make space for. If you missed my most recent article on How to Learn Deeply Daily, you’ll want to open that article before you do anything else and flag it to go back and read. That article is all my methods for how I learn, going deeper into what it means to diversify learning inputs and the benefits of shifting our methods as we all are bombarded by how much content we are absorbing. I usually wouldn’t push a previous article as much as this, but I really, really believe in these practices so make sure you check it out.
Back to audiobooks — I’ve listened to a handful of books this year that have been fascinating (Educated by Tara Westover, Open by Andre Agassi, Originals by Adam Grant, or Team of Rivals by Doris Kearns Goodwin to name a few). But one book that has really stood out this year, not only for its challenging content but mainly for its storytelling was The Undoing Project by Michael Lewis. In this book, Lewis gives a detailed account of the lives and research of Israeli psychologists Daniel Kahneman and Amos Tversky, the former of which went on to win a Noble Prize in Economics in 2002.
The premise of the book explored the original studies done by the pair in the field of decision-making and the processes and biases that most human beings operate within. Their research began to prove the ways in which, during uncertain situations, our minds erred, often systematically and typically against the standard statistical expectations. This is what I want to explore more in this article today. The work of Tversky and Kahneman went on to found what we now know as Behavioral Economics. Their premises influenced so many areas of our modern thinking, including areas such as Big Data studies, evidence-based medicine, economic practices, and government regulations.
Within Lewis’s storytelling, he touches on one area of decision-making that I thought was particularly interesting, especially as it relates to leadership. When looking at how people make decisions, Tversky and Kahneman observed that extrapolation, making big conclusions from small sample sizes, is more common than we might expect. Geoffrey Kabat, a contributor to Forbes in the Science and Technology section, recaps the duo’s findings on this topic when he says…
“ they showed that people make judgments based on a very small sample drawn from a much larger population. For example, when shown two large bags filled with poker chips — one containing 75 percent white chips and 25 percent red chips and the other containing 75 percent red chips and 25 white chips — people tended to think they could tell which bag they were holding after selecting only a small number of chips. But the surprising thing was that psychologists also made the same mistake! A knowledge of statistics did not guard them against this mental pitfall.”
Feedback and Extrapolation Examples
In this article, I’d like to look more closely at how we draw large conclusions from small sample sizes and why it is affecting our leadership. We’ll look at a few practical examples within modern leadership before diving into what I believe is at the root of this risky behavior.
For anyone in leadership, you know the pro’s and con’s of feedback. We’ve all been there. On the receiving end of both positive and negative feedback. There is certainly such a thing as constructive criticism, but sometimes it feels like that phrase is just slapped on to whatever is being said to make the words more palatable. We’ve all felt the warm, deep-in-your-soul feeling that comes when someone gives us great feedback. That feeling that comes when you pull off an event that goes better than expected or you do that technical part of your job just right and someone notices and comments on it.
But like me, you’ve also probably been on the receiving end of some negative feedback. Have you felt that moment when you knew you could have done something better or you thought you crushed something only to find out that you have a few more steps to go?
The tricky thing with feedback, especially within a consumer-driven business and culture, is that it can be difficult to differentiate what should be listened to or rejected. If we’re not being careful in our leadership, this situation can easily turn into an Extrapolation Conundrum. Taking the feedback of a few small voices, we can go on to make big assumptions or choices that are largely driven from a small sampling of our total audience.
Of course, sometimes who is giving the feedback matters in determining how much gravity we give to their words. But in a lot of situations, leadership decisions are made based on the loudest voices at the table, which can often be the critics. We have a responsibility to listen to good and bad feedback, but that doesn’t necessarily mean that our decisions can always reflect the one bad thing we hear.
Let’s scale down this feedback example to a more personal and practical note — employees and supervisors. Most of us fill one of these two roles in our profession. Maybe you’re the new kid on the block or perhaps you’re the Senior VP of a Fortune 100 Company. Either way, we’ve all been in situations, maybe even daily, where our impulses push us to make bigger conclusions from relatively smaller sample sizes than we expect. Maybe it’s that one email we receive that is in a tone we don’t really appreciate, and we have to fight the urge to not jump to the conclusion that our co-worker is actively frustrated or out to get us. Or that text that doesn’t have any punctuation and is super short!
While those can be the worst, they can also lead us towards making bigger assumptions than necessary. Maybe for you, you’re great with communication, but it’s practical gritty work of your job. You’ve executed a certain task a few times with either moderate difficulty and based on those few times, it’s easy to draw the conclusion that most times going forward will look like your few experiences. When we are in the middle of difficulty, it is hard to remind ourselves that things will get easier over time.
When we make big assumptions from small sample sizes, we are not only going against much of our own history, but we are also betting against statistical truth. What is so fascinating about this is that many of us think we are being statistically accurate in our assumptions.
The Gambler’s Fallacy
As Geoffrey Kabat mentioned above, even many psychologists and statisticians missed the question about the poker chips and the backpacks. This line of thinking is what leads for many us to what Tversky and Kahneman coined as “The Gambler’s Fallacy.” A great example of this can be described using this illustration.
If I were to show you a small coin, say a quarter, and ask you to predict if after flipping it in the air it would land on heads or tails, what would be your guess?
Tails, because it never fails right?
Let’s say that the coin lands on tails. Congratulations!
Let’s say I ask you the same question four more times and you’re feeling lucky so you stick with calling “tails.” And believe it or not, the quarter lands four more times on tails. You’ve been correct on 5/5 coin flips, all on tails. Rather impressive right?
After a quick pause to revel in our perfect guesses, if I asked you a sixth time to predict what the coin would land on, heads or tails, what would be your guess? For most people, there would be a quiet small voice in our head that says something to the effect of, “If it’s tails five times in a row, surely, at some point it has to switch over and land on heads.” Maybe we convince ourselves that this time, the sixth flip, HAS to be heads because the previous five flips were ALL tails. That wouldn’t be an illogical conclusion, right?
This is The Gambler’s Fallacy. Do you see the problem there? The statistical odds of flipping heads or tails is always, every time at 50% for both. It’s an either/or, split chances guaranteed. But based on our previous five flips of all being tails, we begin to think that heads is destined to land. Internally, we begin to shift the odds, making conclusions based on wrong statistical assumptions. If we flipped a coin 10 times and they were all tails, on the 11th flip, it wouldn’t be crazy for our brains to think that the likelihood of the coin landing on heads was something like 95%. But the statistical odds are still the same. Hence, The Gambler’s Fallacy, or in other words, the Extrapolation Conundrum.
There are so many things in our lives that we experience a few times and then assign much larger conclusions to. Dating relationships, navigation routes, weather patterns, restaurant experiences, investing opportunities to name a few. Anything that involves some level of risk comes with the possibility of this Extrapolation Conundrum.
The fact that the Conundrum is there isn’t the issue. Extrapolation is never going to go away. But the fact that a lot of leaders aren’t even really aware that this Conundrum is a reality in their lives or their circles makes this topic more relevant. Our job as leaders is not to destroy the Extrapolation Conundrum, but to make it visible and to keep our hearts and minds tuned into these situations so that we can lead ourselves and the people around us towards more accurate interpretations of our daily lives.
Why? Avoiding Regret.
I believe that the practice of drawing large conclusions from small samples sizes is often rooted in our best efforts to avoid regret. I could write a whole article about this idea and likely will in the future, so I won’t go into all of the details here. But when we think about why we take small sample sizes of information and or experiences and extrapolate them towards larger conclusions, we begin to notice where regret fits into the picture.
Lewis Carroll said,
“In the end, we only regret the chances we didn’t take, the relationships we were afraid to have, and the decisions we waited too long to make.”
But in the day to day, the fear of regret often leads to avoiding opportunities that may or may not lead towards future difficulties. We see this especially in areas that address our failures or shortcomings. It is easier to assume the worst from the three difficult circumstances we’ve experienced than it is to get to the root of the issue or address the possibilities, thus acknowledging the accurate statistical implications of the situation at hand. We often trend towards the least stressful and least “losing” outcome. This desire to avoid regret perhaps does more than almost anything else in driving us towards taking small examples and making large conclusions.
A leader who is constantly trying to avoid regret will be a leader that is full of assumptions and unaware of Extrapolation Conundrum in his or her life. If we truly aim to be men and women who have clear vision in the midst of the paths we are leading people towards, we must fight to not flee, to be aware of how our minds and tendencies, if we’re not careful, will end up leading us towards places we don’t necessarily want to land.