信心

Counterpoint Global Insights

Confidence

Methods to Assess Confidence Under Uncertainty

CONSILIENT OBSERVER | March 22, 2023

Introduction Investing is an activity that is inherently probabilistic. Nearly all investment opportunities present a range of possible outcomes with some chance of occurring. The goal is to invest in situations where the expected value, the sum of the potential outcomes times the probability that they happen, is different than the price. Coming up with thoughtful probabilities can be hard. Academics who study the intelligence community find it useful to distinguish between probability and confidence. 1 Probability is an “estimate of the chances that a statement is true” and confidence is “the degree to which an analyst believes that he or she possesses a sound basis for assessing uncertainty.” 2 The important point is that these are distinct concepts that often get combined. We believe that it is useful for investors to separate them.

One way to recognize the distinction between probability and confidence is to consider how judges instruct juries to come up with guilty verdicts in U.S. criminal cases. The prosecutors must prove that the defendant is guilty “beyond all reasonable doubt,” which means there are no other reasonable explanations for the evidence. A jury must return a verdict of “not guilty” if their confidence in the evidence is low, even if they think the probability the defendant is guilty is high.

This report discusses a framework for evaluating analytic confidence developed by Jeffrey Friedman, a professor of government, and Richard Zeckhauser, a professor of political economy. Investors should find this valuable for a couple reasons.

First, it is useful to have an operating model for measuring confidence so that it is possible to distinguish between cases where the probabilities are the same and confidence levels differ. Second, confidence can play an important role in the investment process. For example, the price of two investment opportunities may present the same discount to expected value, but confidence in the probabilities for one may exceed those of the other. That nuance may be relevant for determining the appropriate weighting of securities within a portfolio or evaluating diversification. 3

Michael J. Mauboussin [email protected] Dan Callahan, CFA [email protected]

.

Common Ways to Describe Confidence

The first challenge is to pin down what confidence means. Scholars use the term in different senses. Daniel Kahneman, a professor emeritus of psychology and the winner of the Nobel Memorial Prize in Economic Sciences, suggests that confidence is a feeling based on how clearly and easily a person processes the information they have. For example, members on one side of the political spectrum are more confident in their judgments than those on the other side.4 Kahneman notes that “declarations of high confidence mainly tell you that an individual has constructed a coherent story in his mind, not necessarily that the story is true.”5

Another sense relates to the accuracy of judgments. For example, a person is overconfident if his or her subjective probabilities exceed the objective outcomes. While decision makers can be underconfident, there is substantial evidence that forecasters, even those deemed to be experts, are overconfident.6 This is especially true when participants are certain that their answers are correct.7

Finally, confidence is sometimes described in statistical terms, reflecting the data used to come to a conclusion. For example, statisticians use a “confidence interval” to provide a range of estimates for a parameter with an unknown value. As an illustration, if you take a sample of 36 men in the U.S. and find their average height is 70 inches, and you know the standard deviation of height is 3 inches, you can say with 95 percent confidence that the average height of all men is between 69 and 71 inches.8 Confidence in this sense is most relevant for normal, or bell-shaped, distributions.

In many cases, confidence is a sense that has roots more in psychology than in statistics. Consider that the evidence shows that job interviews, as commonly done, are largely ineffective. Even so, interviewers are often confident they have selected the right candidate for the job.9 This is consistent with what Kahneman and the late Amos Tversky, a professor of psychology, called the “illusion of validity.” A good perceived fit between the predicted outcome and the input generates unjustified confidence.10

In reality, analysts often fail to use statements of probability at all and resort to words or phrases that can be interpreted with a wide range of outcomes. For example, what does it mean if an economist suggests that a recession in the next 12 months is a “real possibility?”11 Based on a survey of a large number of participants, interpretations of this phrase fall between 25 and 85 percent, a range that is neither informative nor helpful.

Forecasts should include probabilities rather than words for the sake of clarity and as the basis to provide feedback.

Before we discuss methods to assess analytic confidence, we need to spend a few moments on the ways to come up with probabilities and how they apply to this discussion.

The Forms of Probability

There are three main approaches to setting probabilities.12 The first is frequency, which is based on inference from a sample of an appropriate reference class. For example, if you rolled a die 1,000 times, you could say that the probability of landing on a 3 is 16.7 percent (1 in 6) based on the outcomes that you observe. Casinos rely on the frequency approach because they control the payoffs for various activities. While there is normal variance in the short run, the outcomes converge to the probabilities in the long run. For frequentists, probability is the long-term frequency of events occurring.

Second is the propensity approach, which is based on how a physical system generates outcomes. In this case, when a perfect cube is rolled that has the numbers 1 through 6 marked on each side, it will have the propensity

to land with the “3” face up 16.7 percent of the time. Engineers commonly use this method when they assess the likelihood of physical systems failing.13

The third is subjective, which says that probabilities are assessed based on the personal belief of the observer.

One way to pinpoint this probability is to measure an analyst’s willingness to bet.14 In this case, an analyst who is neutral to risk would be indifferent between doing nothing and betting $1 on rolling a 3 if the payoff was $6 ($1 = .167 × $6). Investment analysts deal mostly with subjective probabilities.

An analyst should revise his or her subjective probability, a measure of a degree of belief, as new information becomes available. The formal way to do this is with Bayes’ Theorem, which updates a prior probability with a new probability conditioned on novel information.15 The frequentists and Bayesians have had intellectual feuds over which approach is superior, but most aspects of investing require a Bayesian approach.16

That investors operate primarily with subjective probabilities does not mean that they cannot be set carefully or that the degree of confidence will not vary. We now turn to the heart of our discussion: how to judge confidence in a probabilistic assessment.

Assessing Analytic Confidence

Friedman and Zeckhauser describe three dimensions to analytic confidence: reliability of available evidence, range of reasonable opinion, and responsiveness to new information. They also ran experiments to affirm that each dimension contributed to how individuals made decisions. Experiment participants included members of the intelligence community as well as respondents on Amazon Mechanical Turk. We examine each source of confidence.

Reliability of available evidence. This dimension says an analyst has a sound basis for assessing uncertainty when they have a solid amount of relevant knowledge related to the case. It answers the question, “Can I defend this estimate with a substantial amount of information?”17 This dimension is based on facts rather than speculation or opinion. Facts and opinion are both important in investing, but facts should always carry more weight in assessing confidence.

This dimension considers how much information is available, the importance of particular data points, how knowledgeable the analyst is about the topic, and whether independent sources converge on similar assessments.

Information that is highly reliable can generate a wide variety of probabilities. For instance, when drawing one card from a complete deck of 52 cards, you can say with high confidence that the odds are 50 percent that the card is a particular color, 25 percent it is a specific suit, and 23 percent it is a court card.

Likewise, the same probability can reflect a range of reliable information. A judgment that a candidate has a 50 percent chance of winning may reflect either high confidence in a tight race or no information and therefore no basis to favor one candidate over another.

Psychologists distinguish between two ways of coming up with a prior probability.18 The first is to examine base rates. This considers the current situation as an instance of a larger reference class. That allows the analyst to examine the distribution of past outcomes and to calibrate the probabilities and outcomes for the case in question.

The second approach, called the “inside view,” combines the particular conditions with personal inputs. This is a natural way to think but can overweight our experience in a way that distorts our perception. Thoughtful forecasts are a blend of base rates and the inside view, but we tend to overweight the inside view. Participants in studies make better forecasts when they are exposed to, and consider, base rates.19

The major challenge in applying base rates is finding an appropriate reference class. An analyst can be justifiably more confident when a problem is clearly part of a reference class that is well defined. There can also be a trade-off between specificity and sample size. You want the base rate to be specific enough to capture what you are trying to estimate while at the same time having a sample size that is sufficient to draw a reasonable inference.

Sales growth forecasting is a good illustration of where base rates can be useful for investors. This is relevant because sales growth is the most important driver of value for most businesses.20 Analysts commonly rely on the inside view when modeling sales growth even though research demonstrates that using base rates can improve the quality of sales growth forecasts.21

Confidence in the reliability of available evidence is a mix of its strength and weight. Strength reflects the extremeness of outcomes, and weight is the predictive validity based on sample size. If you want to assess your confidence that a coin is biased to land on tails, the percentage of times the coin landed on tails captures the strength and the sample size sets the weight.

Our degree of belief in a particular hypothesis typically integrates strength and weight, and there are prescribed ways to combine them properly. But forecasters tend to overemphasize the strength of evidence at the expense of its justified weight. This leads to a predictable pattern. Analysts are overconfident when the strength is high and weight is low (high percentage of tails in a small sample of flips) and underconfident when the strength is low and the weight is high (over 50 percent tails with a very large number of flips).22

Confidence from reliability grows when the forecast is informed by a large sample of outcomes from an appropriate reference class. Confidence can also be the result of dispositive information, although that is rare in the investment industry.

Range of reasonable opinion. When you are dealing with a complex system, the inputs may not lead to outputs in a simple fashion. That means that if people tamper with the system, even if their intentions are good, they do not know what outcomes their actions may produce. This dimension answers the question, “Might reasonable people give substantially different answers to this question?”

Climate change is a classic example. There is reliable evidence for a large and sustained increase in carbon dioxide (CO2) in the atmosphere. The Earth’s CO2 levels are in a range last seen millions of years ago.23 The challenge is that it is hard to determine the impact of this rise, and scientists, who are equally serious and qualified, can come to different conclusions about the implications.24

Psychologists call the probability assigned to an outcome “first order uncertainty.” The reasonable range of probabilities for first order uncertainty is called “second order uncertainty.” It describes how uncertain you are about an uncertain outcome.

In February 2023, media reports said that the U.S. Energy Department had concluded that the Covid pandemic “most likely” arose from a laboratory leak but made its judgment with “low confidence.”25 One way to think about

this is that the Energy Department deemed the first order uncertainty to have probability of more than 50 percent (“most likely”) but with a high second order uncertainty (“low confidence”).

One case of range of reasonable opinion in economics is an open letter that a number of economists, political strategists, and investors sent to Ben Bernanke in November 2010. Bernanke was at the time the chairman of the Federal Reserve. The letter expressed concern that asset purchases associated with quantitative easing “risk currency debasement and inflation.”26 The dollar index forged higher and inflation remained muted in the following years. In 2014, all the authors stood by their initial message.27 A charitable but sensible interpretation is that reasonable people could disagree on the consequences of quantitative easing and that the concerns this group expressed simply did not come to pass.

An important takeaway is that confidence has to be, to some degree, related to the nature of the system. At one end of the spectrum are systems that are simple to understand, such as dice and cards, which is why we use them to explain probabilities. There is no second order uncertainty with these systems. At the other end are systems that are complex and non-linear, where the distributions of the outcomes are “wild” and making forecasts is inherently challenging.28 Economies and ecosystems are examples.29 Second order uncertainties are high.

Investors deal with systems near both ends of the spectrum and their confidence should vary accordingly.30 This is especially relevant when trying to integrate macroeconomic factors with the drivers of corporate value.

Responsiveness to new information. The idea is that analysts should be more confident when they expect that additional analysis will have little impact on their beliefs. This answers the question, “Is my view likely to change substantially if I study the subject further?”

The response reflects how strongly an analyst holds a prior view and the availability of useful information given the necessary time and money to access it. This introduces the constraints of time and resources.

Responsiveness to new information compels the decision maker to think about the cost and benefit of pursuing additional information. This sets a decision threshold. For an investor, the issue is whether the pursuit of new information is likely to change the overall investment thesis. An investment thesis is based on a variant perception, the difference between what the investor believes will happen and what is priced into the stock.

The idea is that there is almost always the opportunity to gather new information at a cost. Early on, new information may change the thesis so the benefit of pursuing it is worth the cost. But at some point, additional information is unlikely to change the thesis so seeking it is not worth the time and money. The added evidence may not be noise, but its payoff is poor.

Experiments have shown a noteworthy link between accessing additional information and confidence.31 Researchers measured the accuracy and confidence of participants making bets on college football games.

Accuracy is the proportion of correct predictions, and confidence is the subjective probability of being correct. A participant is well calibrated when his or her subjective probabilities match the outcomes. When subjective probabilities are higher than the outcomes, the participant is overconfident. When subjective probabilities are lower than the outcomes, the participant is underconfident.

As participants gained access to more information about the teams playing, their accuracy remained flat, but their confidence increased. This means that the gap between subjective probability and the results widened.

Additional data made the participants overconfident rather than smarter.

This can be relevant for investors because gathering additional information can delay a decision, create a cost, or foster a false sense of confidence. The relationship between the amount of information and confidence is tricky and investors should be very mindful of the trade-offs involved.

These results have a fascinating twist. The researchers developed a statistical model that became more accurate when fed the extra cues. The model was better than the humans at incorporating new information. The confidence of the participants tracked the statistical model, even though their accuracy did not improve. It is as if the participants knew that more information should lead to more accurate forecasts, but they did not have the skill to integrate the additional data.

Conclusion

Success in active investing requires finding gaps between expected value and price. Expected value, in turn, consolidates outcomes and their associated probabilities. This report draws attention to the distinction between probabilities and confidence in those probabilities. Further, we argue that assessing confidence is helpful in the investment process.

Investors generally rely on subjective probabilities, or how strongly they believe various proposals. These probabilities should be updated as new information is revealed. Bayes’ Theorem provides a proper way to update probabilities, but the key is to revise figures in the right direction and magnitude. Great forecasters refine their forecasts with greater frequency and granularity than do average forecasters.32

Scholars who study analytic confidence describe three relevant dimensions: reliability of available evidence, range of reasonable opinion, and responsiveness to new information. Reliability says that information exists to come to a sound conclusion. Range of reasonable opinion acknowledges that the outcomes in complex systems can be wide and difficult to forecast. Responsiveness to new information weighs how likely it is that a thesis will change following the introduction of additional data.

Investors can use these dimensions to complement their probability assessments and sharpen their ranking of investment opportunities. Confidence may also be valuable in assessing the potential weighting of investments within a portfolio.

Endnotes 1 Jeffrey A. Friedman and Richard Zeckhauser, “Analytic Confidence and Political Decision-Making: Theoretical Principles and Experimental Evidence From National Security Professionals,” Political Psychology, Vol. 39, No. 5, October 2018, 1069-1087 and Jeffrey A. Friedman, War and Chance: Assessing Uncertainty in International Politics Oxford, UK: Oxford University Press, 2019).

2 Friedman and Zeckhauser, “Analytic Confidence.”

3 For example see some of the quotes at Investment Master Class, “Position Sizing,” at http://mastersinvest.com/positionsizingquotes.

4 Benjamin C. Ruisch and Chadly Stern, “The Confident Conservative: Ideological Differences in Judgment and Decision-Making Confidence,” Journal of Experimental Psychology, Vol. 150, No. 3, March 2021, 527-544. 5 Daniel Kahneman, Thinking, Fast and Slow (New York: Farrar, Straus and Giroux, 2011), 212.

6 Philip E. Tetlock, Expert Political Judgment: How Good Is It? How Can We Know? (Princeton, NJ: Princeton University Press, 2005) and Don A. Moore, Perfectly Confident: How to Calibrate Your Decisions Wisely (New York: Harper Business, 2020).

7 Andrew Mauboussin, “Why Are We So Confident? Exploring a New Dataset of Confidence Calibration Assessments,” Medium, October 17, 2017.

8 The confidence interval equals X ± Z × σ/√n where X is the mean, Z is the variable for a particular confidence interval, σ is the standard deviation (a measure of variance), and n is the size of the sample. In our example, the average height was 70 inches, Z is 1.96, σ is 3, and the sample is 36. We can be 95 percent confident that the average of the population is 70 inches +/- 1 inch (1.96 × 3/6).

9 Jason Dana, Robyn Dawes, Nathanial Peterson, “Belief in the Unstructured Interview: The Persistence of an Illusion,” Judgment and Decision Making, Vol. 8, No. 5, September 2013, 512-520 and Geoff Tuff, Steve Goldbach, and Jeff Johnson, “When Hiring, Prioritize Assignments Over Interviews,” Harvard Business Review, September 27, 2022.

10 Daniel Kahneman and Amos Tversky, “On the Psychology of Prediction,” Psychological Review, Vol. 80, No. 4, July 1973, 237-251.

11 Andrew Mauboussin and Michael J. Mauboussin, “If You Say Something Is ‘Likely,’ How Likely Do People Think It Is?” Harvard Business Review Blog, July 3, 2018.

12 Gerd Gigerenzer, Calculated Risks: How to Know When Numbers Deceive You (New York: Simon & Schuster, 2002), 26-28 and Friedman, War and Chance, 52-58.

13 Gigerenzer tells the story of the Ariane rocket, produced by Daimler-Benz Aerospace, which purportedly had a “security factor” of 99.6 percent in spite of repeated accidents. The security factor was calculated based on the physical features of the rocket and ignored other variables including human error. See Gigerenzer, Calculated Risks, 28-29.

14 For a book that treats this topic and more, see Annie Duke, Thinking in Bets: Making Smarter Decisions When You Don't Have All the Facts (New York: Portfolio/Penguin, 2018).

15 For a good discussion of Bayes’ Theorem, see Nate Silver, The Signal and the Noise: Why So Many Predictions Fail—But Some Don’t (New York, The Penguin Press, 2012), 232-261. For a practical discussion of how to apply this to investment analysis, see Yiding Lu, Geoff Robinson, and Amar Hamoudi, “Fundamental Analytics—Improve Your Forecasting Accuracy: Adopt a Bayesian Mindset,” UBS Global Research and Evidence Lab, October 25, 2022.

16 Sharon Bertsch McGrayne, The Theory That Would Not Die: How Bayes’ Rule Cracked the Enigma Code, Hunted Down Russian Submarines, and Emerged Triumphant from Two Centuries of Controversy (New Haven, CT: Yale University Press, 2011).

17 Based on Friedman and Zeckhauser, “Analytic Confidence and Political Decision-Making.”

18 This discussion relies heavily on Daniel Kahneman and Amos Tversky, “On the Psychology of Prediction,”

Psychological Review, Vol. 80, No. 4, July 1973, 237-251.

19 Dan Lovallo and Daniel Kahneman, “Delusions of Success: How Optimism Undermines Executives’

Decisions,” Harvard Business Review, Vol. 81, No. 7, July 2003, 56-63.

20 Michael J. Mauboussin and Alfred Rappaport, Expectations Investing: Reading Stock Prices for Better Returns—Revised and Updated (New York: Columbia Business School Publishing, 2021), 53.

21 Etienne Theising, Dominik Wied, Daniel Ziggel, “Reference Class Selection in Similarity-Based Forecasting of Corporate Sales Growth, Journal of Forecasting, forthcoming.

22 Dale Griffin and Amos Tversky, “The Weighting of Evidence and the Determinants of Confidence,” Cognitive Psychology, Vol. 24, No. 3, July 1992, 411-435.

23 “Carbon dioxide now more than 50% higher than pre-industrial levels,” National Oceanic and Atmospheric Association, June 3, 2022.

24 Nicholas Stern, “Current Climate Models Are Grossly Misleading, Nature, Vol. 530, February 25, 2016, 407- 409.

25 Michael R. Gordon and Warren P. Strobel, “Lab Leak Most Likely Origin of Covid-19 Pandemic, Energy Department Now Says,” Wall Street Journal, February 26, 2023.

26 “Open Letter to Ben Bernanke,” November 15, 2010. See www.wsj.com/articles/BL-REB-12460.

27 Caleb Melby, Laura Marcinek, and Danielle Burger, “Fed Critics Say ’10 Letter Warning Inflation Still Right,” Bloomberg, October 2, 2014.

28 Benoit B. Mandelbrot and Nassim Nicholas Taleb, “Mild vs. Wild Randomness: Focusing on Those Risks That Matter,” in The Known, the Unknown, and the Unknowable in Financial Risk Management: Measurement and Theory Advancing Practice, Francis X. Diebold, Neil A. Doherty, and Richard J. Herring, editors, (Princeton, NJ: Princeton University Press, 2010), 47-58 and Nassim Nicholas Taleb, “On the Statistical Differences between Binary Forecasts and Real World Payoffs,” International Journal of Forecasting, Vol. 36, No. 4, October 2020, 1228-1240.

29 Garrett Hardin, best known for his work on the tragedy of the commons, describes three filters against folly: the literate, the numerate, and the ecolate. The ecolate addresses dealing with a complex system. As Hardin writes, the essential question with ecolate analysis is, “And then what?” See Garrett Hardin, Filters Against Folly: How to Survive Despite Economists, Ecologists, and the Merely Eloquent (New York: Penguin Books, 1985). 30 There has been a recent back-and-forth between Phil Tetlock, a professor of psychology, and Nassim Taleb, the author. Tetlock and his colleagues suggest that certain types of problems with binary outcomes are useful to study, with the goal of improving subjective probabilities. Taleb and his colleagues call Tetlock’s research agenda a “masquerade” that fails to consider consequential outcomes of systems with fat tails. Tetlock acknowledges Taleb’s viewpoint, but the reverse does not appear to be true. See Philip E. Tetlock, Yunzi Lu, and Barbara A. Mellers, “False Dichotomy Alert: Improving Subjective-Probability Estimates vs. Raising Awareness of Systemic Risk,” International Journal of Forecasting, Vol. 39, No. 2, April-June 2023, 1021-1025 and Nassim Nicholas Taleb, Ronald Richman, Marcos Carreira, and James Sharpe, “The Probability Conflation: A Reply to Tetlock et al.” International Journal of Forecasting, Vol. 39, No. 2, April-June 2023, 1026-1029. 31 Claire I. Tsai, Joshua Klayman, and Reid Hastie, “Effects of Amount of Information on Judgment Accuracy and Confidence,” Organizational Behavior and Human Decision Processes, Vol. 107, No. 2, November 2008, 97-105. For an early paper on this work, see Paul Slovic, “Behavioral Problems of Adhering to a Decision Policy,” Working Paper, May 1, 1973.

32 Jeffrey A. Friedman, Joshua D. Baker, Barbara A. Mellers, Philip E. Tetlock, and Richard Zeckhauser, “The Value of Precision in Probability Assessment: Evidence from a Large-Scale Geopolitical Forecasting Tournament,” International Studies Quarterly, Vol. 62, No. 2, March 2018, 410-422.