Elude the Illusion: Understand The Illusion of Validity So You Don’t Fall Victim To This Common Decision Making Bias

The illusion of validity will often cause people to make risky bets on a roulette wheel
Have you ever placed a bet that a certain number will appear on dice or a roulette wheel? Maybe a number has come up repeatedly so you assume that there is less of a chance that this number will appear in the next roll or spin, even though every number has an equal probability of coming up. Many people fall victim to this bias and end up losing money at casinos. This phenomenon can be explained by the illusion of validity. Defined as a person’s tendency to overestimate their accuracy in making predictions given a set of data, the illusion of validity is one common source of bias in decision making (Einhorn, 1978).
So why exactly are we so bad at making accurate predictions? To answer this question, we must first discuss the role of patterns in the illusion of validity. People love to look for patterns, and when they find these patterns they want to associate a meaning to them. A classic example of this phenomenon is Pareidolia, where humans see faces in inanimate objects such as the moon, washing machines, or trees. Another relevant example how people find and make meaning of patterns is superstition. Maybe you wear a special shirt when your favorite football team plays because the last two times that you wore this shirt your team won. We all know that washing machines don’t have faces, and the shirt a sports fan wears will not affect the performance of a person’s favorite team. Nevertheless, humans often associate inaccurate meaning to patterns they observe not realizing there is often no meaning behind a pattern, it is just simply a pattern.

The illusion of validity completely disrupts our ability to make accurate predictions
In terms of the illusion of validity, when people observe data they often make predictions or conclusions about the data based on patterns that they see with the goal of making sense of their observations. However, as confident as one may be in their predictions, these conclusions often have no scientific grounding (Tversky & Kahnemen, 1974). In his recent book Thinking Fast and Slow (2011), Kahneman brings this bias to life in sharing an example from his work as a psychologist selecting leaders in the Israeli Army. They designed a drill where soldiers had to work together in groups of 8 through challenging obstacles. Kahneman was confident that those who succeeded in the drill would become the best leaders over time. But as these soldiers progressed in the Army, Kahneman learned that other things better predicted the selection of leaders. Nevertheless, he still kept using the drill for the early identification of leaders. Kahneman and his colleagues were experiencing the illusion of validity. And this experience contributed to his research on this decision making bias.
Like me, you might be wondering why, if there is evidence that disconfirms peoples’ predictions, why people still are susceptible to the illusion of validity? Unfortunately, as the example of the Israeli Army above, even when people are well informed on the illusion of validity, they still can be influenced by it. A study by Oskamp (1965) helps us understand why. In this study participants, either clinical psychologists or undergraduate students, were asked to make predictions about patient’s reasons for being in the hospital—were they there for psychiatric problems, or another medical reason? The researchers found that there was not a large difference in the accuracy of predictions made — experienced clinical psychologists were only slightly better at making predictions than untrained undergraduates. Oskamp explained how clinical psychologist participants remembered the instances in the past where their predictions had been correct while ignoring the predictions that they had made in the past that were incorrect (Oskamp, 1965). In other words, when making predictions, people recalled memories of specific events when they made successful predictions of patient type, while ignoring or deemphasizing times that they were incorrect. In essence, these people were making predictions based on selective memories because they failed to remember or address the times when their predictions were incorrect.
Although we like to think that our memory recalls all kinds of experience with similar accuracy, we all in fact have selective memories. Selective memories can help to explain the illusion of validity and provide some rationale for why people seem to be ignoring the times that their predictions were unsuccessful. One reason that people seem to be “ignoring” their incorrect predictions is because they simply do not remember that they were incorrect. Another decision making bias may come into play — retrospective bias. With retrospective bias, a person may, for example, predict that a seven will be rolled on a die. However, after the roll reveals a five, that same person may look back and think, “I knew it wasn’t going to be a seven,” even though they obviously did not. As a result, we are making inaccurate predictions because the information that we are basing our predictions on is faulty because we based it on selective or even false memories.

Standardized test scores are not a good predictor of college success
How does the illusion of validity impact you in the real world beyond the casino? Well, the decisions that are made about us and for us are often tainted by the illusion of validity. For example, being admitted to college has become increasingly competitive and unpredictable. The admissions officers are basing their decisions off a whole host of things such as high school GPA, extracurriculars, supplemental essays, and, standardized test scores. Other factors that admissions officers have recently paid increasing attention to are diversity, gender, financial need, and athletics. These “experts” in admissions try to predict who will be successful based on the data provided by each applicant. However, these experts can be overconfident and simply wrong at predicting who will be most successful in college. For example, in the past, standardized tests are often used by large universities as an important factor in admissions because it is easily comparable across candidates. However, more schools are now making reporting test scores optional, based on recent evident by Hiss (2014) that other factors are just as accurate, and even more accurate in predicting which applicants will be successful at their college. So why are so many admissions officers still placing so much confidence in the predictive power of standardized test scores? Perhaps the illusion of validity is one reason. They used it in the past and believe it will be a good predictor in the future, even though the college experience is becoming more complex with issues like diversity and helping disadvantaged students gain access to elite colleges becoming higher priorities. I suggest that admissions officers need to be aware of the illusion of validity and find ways to continue challenge conventional wisdom on the admissions process.
I hope that by reading this entry you have learned the basics of the illusion of validity. Going forward let’s make sure to question both our own and expert’s predictions so that we all make better decisions whether at the casino or in college admissions!
References:
Einhorn, H.J., & Hogarth, R.M. (1978). Confidence in judgement: persistence of the illusion of validity. Psychology Review, 85(5), 395-416. doi:10.1037/0033-295X.85.5.395
Hiss, W.C., (2014, February 5th) Defining promise: optional standardize testing policies in American college and university admissions. Retrieved from https://offices.depaul.edu/enrollment-management-marketing/test optional/Documents/HISSDefiningPromise.pdf
Oskamp, S. (1962). The relationship of clinical experience and training methods to several criteria of clinical prediction. Psychological Monographs: General and Applied, 76(28), 1-27. doi:10.1037/h0093849
Weinstein, Y. & Roediger H. (2010). Retrospective bias in test performance: providing easy items at the beginning of a test makes students believe they did better on it. Memory & Cognition, 38(3), 366-376, doi:10.3758/MC.38.3.366.
Visuals:
- https://giphy.com/gifs/southparkgifs-26uflBhaGt5lQsaCA
- https://betfaq.com/land/winreg/?a=ad666&ac=doorways&sa=mrfeathe
- http://affinitymagazine.us/2017/06/09/why-standardized-testing-doesnt-define-you/
This is a very interesting post. The Illusion of Validity reminds me a lot of the Gambler’s Fallacy; this phenomenon states that people believe that the more something is occurring in the present, the less likely it is to occur in the future. So if someone is playing the slot machine and they keep losing, they will assume that soon enough they are due for a win; this is not true, just as are people’s predictions which you have covered in your blog post. I think that confirmation bias may have something to do with people’s tendency to only focus on the times that their predictions were correct. Confirmation bias is a phenomenon in which people tend only to attend to the information which confirms their beliefs or opinions. If one believes that they ought to have been right in their predictions, as you have stated, they will look only back on the memories which go to confirm these predictions. You bring up a good point in mentioning false memories as this clearly plays a role in the Illusion of Validity. As you said, people will go back to the memories which confirm their predictions, but these memories are not always trustworthy, as phenomena such as confirmation bias affect how we relive our memories.
I really enjoyed reading this post. I recently visited a casino in the Bahamas and felt the exact way you described above. I overestimated my ability to predict what number the dice would roll onto, therefore loosing some money… Do you think this is the same reason that on multiple choice exams I find myself counting how many a’s, b’s, and c’s I have to determine the answer to a question I am unsure about. I do not know why this is the case, but maybe it is related to the illusion of validity. Furthermore, what you said about SAT scores predicting future success is intriguing to me. So much pressure is put on the score you receive on the test, but I do believe schools are slowly becoming aware of this problem. Many colleges nowadays are test optional, giving people who might be really great prospects but did not score as well as they needed to, a chance to have a spot.
I hope that now I know more about this cognitive bias that I pay closer attention to it. But, as you said, maybe there are some things we just do not pick up on, like which letter we choose on a multiple choice question, because we think there is some correlation between the amount of times we used the letter and what the correct answer is. Thank you for sharing such interesting information, I was enlightened by this post.