Human beings are incredibly good at simplifying things into patterns and models. That’s how we make sense of the world. But human beings are also notoriously awful at evaluating our own thinking. And dealing with the inefficiencies thereof.
I just finished a book called Psychology of Intelligence Analysis. If you’ve ever heard of anchoring bias, that’s kind of where this conversation leads off. Anchoring bias is the tendency of humans to start from a number/figure/model when thinking about something and adjust (often poorly and inadequately) based on new information. The whole point is, we don’t adjust nearly enough. We get anchored on a bad way of thinking to start with, and fail to compensate for it — we are weighed down by the anchor of initial thought, so to speak. It’s prevalent in all human affairs, all the time.
In one chapter, author Richards J. Heuer describes people’s use of confidence ranges, a related thinking phenomenon, which was sort of a novel concept to me:
(Note: if you have trouble visualizing what he’s talking about, don’t worry — it’ll get simpler as we talk about it.)
Anchoring provides a partial explanation of experiments showing that analysts tend to be overly sure of themselves in setting confidence ranges. A military analyst who estimates future missile or tank production is often unable to give a specific figure as a point estimate. The analyst may, therefore, set a range from high to low, and estimate that there is, say, a 75-percent chance that the actual production figure will fall within this range. If a number of such estimates are made that reflect an appropriate degree of confidence, the true figure should fall within the estimated range 75 percent of the time and outside this range 25 percent of the time. In experimental situations, however, most participants are overconfident. The true figure falls outside the estimated range a much larger percentage of the time.
And in the footnote, Heuer elaborates:
Experiments using a 98-percent confidence range found that the true value fell outside the estimated range 40 to 50 percent of the time.
What, Sir, Is The Point
Now this makes me wonder: if we are this bad at estimating probability, why even bother?
Let me simplify and then add to the above text: let's say you're unsure of the likelihood of some particular outcome. Like, for instance, whether Russia is going to invade Ukraine. So you estimate a confidence range of 30-60%.
Ok, so what's your confidence in that range itself? Let's say you're 75% confident in that estimation range.
But then I could even ask, what's your confidence in your confidence? What’s your estimation that this 75% confidence is even appropriate? You could be wrong about that. Maybe you're only 50% confident in that second-level assessment.
Now I realize that we’re getting kind of dream-within-a-dream-within-a-dream with this. And you might think that’s ridiculous. But this isn’t trivial stuff. The more you can A) simplify your thinking, B) dispense with foolishness and wasted effort, and C) put conviction behind your decision making, the better off you are. That’s, for the most part, the whole idea of my blog.
So, by the time we get three layers deep in your analyzing of your own estimates, you're actually only operating on a confidence level of, say, 35%. At this point, you're doing nothing more than guessing. You're actually not providing any meaningful information or insight to the conversation.
You see this kind of behavior from economists and politicians a lot — people who get paid to do nothing other than defend their own jobs. People whose job is literally to give their own job the illusion of utility. And if there are two groups of people on planet earth you don't want to imitate, they are academic economists and politicians.
So the question then becomes, what is the use of doing any of this at all?
Well, there is huge utility in estimating the probability of outcomes. For instance if you're analyzing a company for investment and you can legitimately say you're 85% confident their earnings will increase by next summer. Or you’re trying to figure out your chances with that pretty girl who keeps looking at you over the top of her historical fiction novel, and you think your chances of getting her number are at least 70%.
These are reasonable estimations where you can legitimately be confident. Because there are few enough moving parts and you feel well enough equipped to take a guess. A good, useful guess.
And good, useful guesses are how we get through life.
Unfortunately, the majority of situations are not this clean.
So perhaps it is best to reorganize our own thinking to make the most of our mental real estate and resources. We can sort our decision-making factors into categories, such as:
- High transparency, high-confidence factors — the center of our decision making processes
- Factors where we can reasonably estimate, in a way that makes the factors actually useful as information
- Low-transparency, low-confidence variables. "Wash factors." “Unknown unknowns.” And wherever possible, we ignore these factors completely. Or loosely cancel them out against each other. Because why fret about something that we cannot even use as an effective tool?
And there are going to be a lot of those factors. The third kind. The Too Hard Pile. And that is perfectly okay. Luck, variance, and (if you believe in it) fate are always at work, and we must let them do their jobs.
I published a piece called "Deciding, Fast and Slow" earlier this summer. And what we're talking about now are the Slow decisions. The careful ones. The ones with a lot of research and thinking required.
And so I'm not suggesting to half-ass your research. Or to discount things that are reasonably important. What I'm suggesting is to spend maximum time on the areas of your decision making where you actually feel you have control. And to not get bogged down in the details that don't even make any sense to you. Or the details that are so obscure and difficult to work with that all they do is confuse you.
I mean, imagine you spent 3 weeks analyzing a situation and gave each ridiculous, obscure variable its fair share of attention. And then you came to realize that you probably would have made the right guess with just the first 3 or 4 variables you were looking at.
I bet that would be a big sigh moment.
The Point Is, Everything Is a Bet
Life is, to a large extent, a numbers game. Everything is a bet. I know this may not be all that pleasing to hear, for people with high sensibilities. For True Romantics. But the more I have read and learned, the more I think of life in terms of bets and probabilities. The Law of Large Numbers is one of the best friends you can have.
You cannot win all bets. It's not possible. Life won't allow it.
And you cannot be good at analyzing all variables. Again, it's not possible. Your brain won't allow it. And you have limited mental resources and attention capacity.
Variance (temporary swings in fortune) will steal some of your joy and make some of your decisions “wrong” in the short-term. But in the long run, the quality of your decision making will make itself quite clear.
So what you can do is get really, really good at making bets in whatever way makes the most sense to you. Using your frameworks and your skills, and ignoring whichever variables you're not good at working with.
Now it goes without saying, I’m not telling you to stick to a perfectly-defined, rigid comfort zone. That’s juvenile. That’s the opposite of what I’d like to see you do. I’d like to see you add more skills to the “I can use these effectively” pile. To move outward and upward, so that you have more, bigger, and better frameworks for developing a world view.
If you're not good at estimating outcomes based on statistics, stick to psychology. If you're not good with abstract, poorly-defined information, defer to models and charts.
You can’t account for everything. Not even intelligence analysts can account for everything. And they get paid to account for everything. Bunch of impostors.
It's okay to say I don't know. In fact, it's one of the most useful things you can possibly say. Because it allows you to move the hell on. It allows you to focus on what you do know. Which is way, way more useful.
Drink some water, unless you don’t know how
JDR
“Improving decision quality is about increasing our chances of good outcomes, not guaranteeing them.” - Annie Duke