反馈

Counterpoint Global Insights

Feedback Information as a Basis for Improvement

CONSILIENT OBSERVER | March 16, 2022

Introduction

All outcomes are the result of a process. The outcome alone is an Michael J. Mauboussin indication of the quality of the process in situations when luck plays no role. But an outcome can be an unreliable indicator of the quality Dan Callahan, CFA of the process in the short term when a favorable result follows a good process only with some probability.

Improving process is crucial in all cases if the goal is to achieve an outstanding long-term outcome. Feedback, defined as information used as a basis for improvement, is central to making the process more effective. In this report, we discuss multiple facets of process improvement, including getting the right people and helping them thrive, the role of organizational structure in fostering good decision making, and specific mechanisms to create feedback that sharpens execution (see exhibit 1). While we draw from principles in other fields, our primary focus is on the investment management industry.

Exhibit 1: Facets of Process Improvement

Source: Counterpoint Global.

People

If you want to build an excellent, repeatable process you need to start with having the right people. You then need to train them to be effective, figure out how you should deploy their skills, and institute a policy of practice.

Finding individuals that are likely to add value to an organization is an essential activity that is rarely done as well as it could be. How do you find the right people? The first step is to identify the skills that can lead to success.

A recent discussion with an executive at a National Football League team made this point. The team first assessed which skills, typically around a half dozen per position, it deemed necessary to succeed. Identifying these skills helps in drafting new players, evaluating the current roster, and acquiring players through trades or free agency.

Another important aspect of identification is determining which skills can be improved through coaching and organizational support and which cannot. Players generally follow a performance curve, which includes gains until the age of peak performance and declines following it. But some abilities, including speed and throwing accuracy, are difficult to modify at any point. The team’s front office tried to isolate which skills it could strengthen and sought to build processes to capture the opportunity.1

What skills are valuable for investing? We believe that two areas of research in psychology, one on superforecasters and the other on the rationality quotient (RQ), identify characteristics that are associated with successful long-term investment results. There is substantial overlap in the qualities of superforecasters and individuals who have a high RQ.

Superforecasters. In the fall of 2011, Intelligence Advanced Research Projects Activity (IARPA), an agency within the U.S. intelligence community, launched a tournament to see if there might be a way to improve forecasts of social, political, and economic outcomes.

A team of psychologists, including Philip Tetlock, Barbara Mellers, and Don Moore, created the Good Judgment Project (GJP), one of a handful of teams seeking to answer these questions accurately. The GJP was the best of the teams and purportedly outperformed members of the intelligence community.2

The researchers found that about two percent of the thousands of volunteers that the GJP recruited delivered a lot of good projections. The accuracy of their forecasts was beyond what chance alone would dictate.3 Tetlock and his colleagues called these people “superforecasters.”

Because this was a research project, the GJP team also sought to understand what led to good forecasts. They examined the traits of the forecasters, measured the impact of training, and assessed the value of working in teams versus individually.

Superforecasters and successful investors share a lot of the same qualities. They seek to anticipate outcomes in a probabilistic world by understanding what has happened in the past, properly updating their views to reflect new information, and working hard. These skills do not come naturally to many, and the research suggests that how we think is more important than what we think.

Beyond a commitment to learning and engagement, superforecasters share some common characteristics (see exhibit 2). It takes little effort to see how these concepts apply to investors. The first is philosophical approach and outlook. This requires being comfortable with the sense of doubt and recognizing that degrees of doubt are a better way to approach problems than settling on certainty. Humility stems from a recognition that the world is

complex, not from self-doubt. Finally, superforecasters don’t believe what happened had to happen and are open to considering counterfactuals.

The second set of characteristics has to do with ability and thinking style. This starts with being actively open-minded, a willingness to consider a conclusion different from the one you currently hold.4 People who score well on the actively open-minded scale tend to be less susceptible to cognitive biases.5 Tetlock and Dan Gardner, his co-author of the book Superforecasting, write, “Beliefs are hypotheses to be tested, not treasures to be protected.”6 Intellectual curiosity and numeracy are also essential elements of thinking style.

Methods of forecasting are the third suite of characteristics. Here, the big ideas are to think in probabilities and to update views as new information is revealed. This is difficult in the face of confirmation bias, which is one of the most potent cognitive challenges. Confirmation bias is present when we process information to favor opinions that we hold in high esteem and with great conviction.7 Unlike other biases, confirmation bias does not lessen with high intelligence.8

The ability to communicate well is another important characteristic of an investor. You don’t really understand an idea unless you can communicate it clearly in speech or the written word.

Exhibit 2: Characteristics of Good Forecasters

Philosophical Approach and Outlook Cautious Understand few things are certain Humble Appreciate their limits Nondeterministic Don't assume that what happens is meant to be Abilities and Thinking Styles Open-Minded See beliefs as hypotheses to be tested Inquiring Are intellectually curious and enjoy mental challenges Reflective Are introspective and self-critical Numerate Are comfortable with numbers Methods of Forecasting Pragmatic Not wedded to any one idea or agenda Analytical Consider other views Synthesizing Blend diverse views into their own Judge the probability of events not as certain or uncertain but as more or less Probability-Focused likely Thoughtful Updaters Change their minds when new facts warrant it Intuitive Shrinks Are aware of their cognitive and emotional biases Work Ethic Improvement-Minded Strive to get better Tenacious Stick with a problem for as long as needed

Source: Paul J.H. Schoemaker and Philip E. Tetlock, “Superforecasting: How to Upgrade Your Company’s Judgment,” Harvard Business Review, Vol. 94, No. 5, May 2016, 72-78.

Rationality Quotient. Research has shown that there is a positive correlation between intelligence quotient (IQ) and stock market participation and results.9 The second area of research is on the “rationality quotient,” or RQ, a measure of rational decision making.10 This idea was developed by the psychologists Keith Stanovich, Richard West, and Maggie Toplak. While intelligence quotient, or IQ, is correlated with RQ, “[t]he magnitude of the correlation leaves plenty of room for disassociations between intelligence and rationality.”11 As with the characteristics of superforecasters, how people think is more important than raw measures of intelligence.

IQ is an estimate of where a person falls on a scale of general intelligence (g). Standardized tests such as the SAT, historically used as part of an assessment for college admissions, are a good proxy for g.12 In turn, g is commonly characterized with fluid and crystalized components. Fluid intelligence measures reasoning in novel domains. Crystalized intelligence is declarative knowledge. Fluid intelligence peaks on average at an age in the early 20s, while crystalized intelligence tends to rise through most of one’s life. Fluid intelligence measures a process while crystalized intelligence reflects stored knowledge.

Cognitive scientists use the term “rationality” in a normative sense, which means how you ought to think and act given your goals and the state of the world.13 Rationality is generally broken into two parts: instrumental and epistemic. Instrumental rationality means that you behave so you achieve your objectives subject to your physical and mental constraints (“what to do”). Epistemic rationality measures the degree to which your beliefs map accurately to the world (“what is true”).14

Daniel Kahneman and Amos Tversky developed lines of research in psychology, including prospect theory and heuristics and biases, that documented how people depart from normative standards. Prospect theory describes how people behave when faced with uncertainty and shows that they suffer losses more than equivalent-sized gains. This observation defies the tenets of expected utility theory which serve as the foundation for instrumental rationality.15

Research on heuristics and biases shows that people often rely on heuristics, or rules of thumb, that are fast and simple but that can lead to a biased assessment. For example, when considering the probabilities of a specific outcome, we commonly appeal to the heuristics of what is available in our minds. Nearly one-third of deaths in the United States are related to heart disease while this cause receives only 2-3 percent of media coverage and Google searches. Death from terrorism is relatively rare, but terrorism has a nearly one-third share of media accounts and Google searches.16

Exhibit 3 shows the elements of rational thinking and in particular notes the inhibitors to crystalized rationality. For instance, some beliefs are testable, which means they are tied to the world and can be confirmed by observation. Other beliefs are distal, which are not testable or supported by experts or scientific knowledge.17 Distal beliefs can be detrimental to rational thinking.

Exhibit 3: Rational Thinking Skills

Absence of irrelevant context effects in Practical numeracy

thinking beliefs Open-minded/objective reasoning styles Prudent attitude toward the future Sensitivity to emotions Source: Keith E. Stanovich and Richard F. West, “The Assessment of Rational Thinking: IQ ≠ RQ,” Teaching of Psychology, Vol. 41, No. 3, July 2014, 265-271.

Noting that tests of IQ do a poor job of measuring RQ, Stanovich, West, and Toplak created the comprehensive assessment of rational thinking, or “CART.” Exhibit 4 shows the CART’s 20 subtests, which are weighted to calculate an aggregate score. Take a moment to examine the subtests and you will see that they capture many of the characteristics of superforecasters and the qualities that successful long-term investors share. Scores on assessments of actively open-minded thinking have a stronger correlation with RQ than IQ.18

Exhibit 4: Subtests of the Comprehensive Assessment of Rational Thinking

CART Subtest CART Points Scientific reasoning 20 Probabilistic and statistical reasoning 18 Reflection versus intuition 10 Financial literacy and economic knowledge 10 Rejection of conspiracy beliefs 10 Probabilistic numeracy 9 Other subtests (14 others) 71 Total CART Points 148 Source: Keith E. Stanovich, Richard F. West, and Maggie E. Toplak, The Rationality Quotient: Toward a Test of Rational Thinking (Cambridge, MA: MIT Press, 2016), 72.

Cognitive capabilities change through life and follow an arc resembling an inverted U. But IQ, which measures performance relative to a cohort, remains relatively stable. That’s the bad news. The good news is that researchers believe that RQ, and the thinking process of superforecasters, can be improved to some degree.

This improvement can happen for individuals and collectively as a group. We start by considering training that can help individuals improve their decision-making process.

Training. We discuss three aspects of training. The first has to do with improving forecasting skills and is a direct result of the research done by the team at the GJP. Second are proper techniques for investment analysis, including making sure that investors use shorthands properly. The final one relates to grasping which aspects of a decision-making process can be systematic versus which require judgment.

There are multiple forms of training to improve forecasting skills and they are not equally effective.19 The didactic approach teaches participants about judgment errors. For example, an instructor might share the research on biases associated with overconfidence, framing, anchoring, and availability. This method has the benefit of being relatively inexpensive to implement and the drawback of being mostly ineffective.

The next training tactic is to explain specific techniques to check for bias.20 The use of base rates is among the most effective of these. In order to make a forecast, most of us gather relevant information, combine it with our own views and experiences, and project. Base rates consider the problem as an instance of a larger reference class and therefore start with a relevant distribution of past outcomes. For instance, instead of forecasting a company’s revenue growth by doing a bottom-up analysis, an approach using base rates would examine the historical distribution of growth rates for similar businesses.

Base rates are underutilized in general because most forecasters place substantial weight on their own analysis and experience and base rates are not readily available in many instances. Good statistical predictions combine evidence for the specific case and base rates in proper measure.21 The leaders of the GJP found that teaching participants about base rates improved the accuracy of their forecasts relative to those who did not receive the training.

A third training approach is to structure problems differently so as to provide insight into the solution. Here’s an illustration: imagine a test for a certain type of cancer where the prevalence is 0.3 percent, 50 percent of those who have the cancer test positive, and the false positive rate is 3 percent. If someone tests positive, what is the probability that they have the cancer? See what you come up with.

Doctors who were asked this question answered between 1 and 99 percent, and roughly half said 50 percent, the test’s sensitivity, or 47 percent, the test’s sensitivity minus the false positive rate. The need to combine probabilities makes this question confusing, and even professionals who should know how to interpret the data struggled.

The answer is around five percent. The way to structure the question to make it easier to grasp is to translate the probabilities into natural numbers.22 For example, assume a population of 2,000 people. We know that 6 of them (.003 x 2,000) will have the cancer. We know that half of them, 3, will test positive. The false positive rate says that about 60 of the people without the cancer will test positive (.03 x 1,994). The probability that someone who tests positive has the cancer is therefore 3 divided by 62.8, or 4.8 percent.

The final training approach, which addresses the core theme of this report, is timely and accurate feedback about probabilistic forecasts. Think of weather forecasters. They predict the weather with probabilities (“there’s a 30 percent chance it will rain tomorrow”). The weather outcome is a form of feedback. There is evidence that feedback provides lessons that help increase the accuracy of future forecasts.23

There are two qualities in particular that are worth cultivating. The first is calibration, which measures how closely percentage forecasts match the outcomes. A weather forecaster is well calibrated if it rains 30 percent of the time that rain was predicted with a 30 percent probability.

The second is called discrimination, or resolution, and considers how different a particular outcome may be from the base rate. For instance, it rains about one in three days in London, England.24 You can predict a 33 percent chance of rain each day and appear well-calibrated at the end of the year. But those forecasts would not be helpful in planning picnics or walks in the park. Discrimination means it rains when you say rain and it’s dry when you call for sunshine.

We will discuss specific methods to improve feedback for investment professionals in the last section of this report.

The second aspect of training for fundamental investors focuses on the correct methods for investment analysis.

The process for seeking excess returns varies by investment organization, so it is important to match the tools with the task. But aspects of this training are particularly relevant. One is that there are right and wrong ways to do particular calculations. The other is to make sure that all investors on the team understand the economic implications of the assumptions they make.

For example, measures, including return on invested capital (ROIC), free cash flow, and customer lifetime value, should be defined properly and calculated consistently throughout an investment organization. Investors who feel the need to depart from standard definitions can be encouraged to calculate the measure in the agreed-upon way and then introduce the alternative approach with a clear explanation of why it adds insight. There also ought to be a consistent way to estimate the opportunity cost of capital.

Another essential element of this training is to make sure that investors who use shorthands, predominately multiples to assess valuation, understand the economic implications of those shorthands. For instance, a price-earnings ratio of 30 or an enterprise value-to-earnings before interest, depreciation, and amortization (EBITDA) ratio of 18 embed assumptions about incremental ROIC, growth, and the cost of capital. An investor seeking to discern value should make those metrics explicit so that they can be discussed and debated. Shorthands should have grounding in proper drivers of economic value.

Effective communication skills are also an important aspect of investment analysis and can be improved via training. Steven Pinker, a cognitive scientist with an expertise in linguistics, states flatly that “writing is an unnatural act.”25 He makes the case for prose written in classic style, which entails showing the reader something concrete in the world in a way that is conversational. He notes that with this style “the writer has worked hard to find something worth showing and the perfect vantage point from which to see it.”26

Proper data visualization, using graphics to display data, is also important. When done well, data visualization conveys useful information to the viewer.27 Charts and graphs that are used more to make a point than share relevant material are common in the investment industry. Indeed, research shows that how data are presented can lead to different judgments about the same data.28

The final aspect of training has to do with process development. Investors who rely on fundamentals need to consider carefully the potential sources of market inefficiency that lead to market mispricing. Ed Thorp, a trained mathematician and highly successful hedge fund manager, said, “There is a market inefficiency if there is a participant who can generate excess risk-adjusted returns that can be logically explained in a way that is difficult to rebut.”29

There has been a lot of interest in recent years in blending quantitative and qualitive approaches to create a process that is more robust than one based on either approach alone. The idea is that man plus machine beats either man or machine. The key is assessing which aspects of investment process can be addressed

systematically and which must be in the realm of judgment. For instance, Lewis Goldberg, a psychologist, found that models of decision makers outperformed the decision makers that were modeled. This reveals an essential observation: decision makers often know what to do in principle but fail to execute in reality.30

Practice versus Play. Practice is common for participants in a particular domain who want to get better. Athletes are an obvious example. Even elite professionals allocate a substantial amount of time to honing their skills. The ratio of practice to play varies by sport, but there are no cases where practice is completely absent. Practice entails applying an idea, belief, or method versus just understanding theories related to results in an activity.

One of the reasons that investment management is difficult is that there is an enormous amount of information to absorb. As a result, it is easy to get drawn into a pattern of all playing and no practicing. But no playing and all practicing is also obviously bad. The goal is to strike an appropriate balance between doing the job and practicing so as to improve the investment process.

Studies of practice reveal a number of relevant points. First, practice in most activities allows us to pass through stages until we reach the autonomous stage, a point where skill becomes habitual and fluid. Performance is acceptable and repeating the activity does not improve results. The classic example is driving. Most of us can get around reasonably safely but certainly lack the skills of expert drivers. We get good enough and stop. And that is fine for many things we do.31

Second, there’s a useful distinction between regular, or even purposeful, practice and deliberate practice.32 Deliberate practice requires full attention, occurs at the edge of a person’s ability, and requires timely and accurate feedback. A good teacher or a coach can structure the practice and make appropriate corrections to performance. Deliberate practice is more applicable to domains that are well understood, such as music or athletics, than it is in activities that are less objectively measured.32

Third, it is interesting to consider what practice looks like in the investment industry. The answer to a large degree depends on how an investment manager seeks to generate excess returns. What skills a quantitative manager needs to succeed are very different than what a long-term oriented fundamental investor requires.

Quantitative managers rely more on a statistical approach and fundamental investors appeal more to judgment, but the common denominator is the ability to make good decisions in the face of uncertainty.

Fourth is the issue of skill transfer. Effective practice isolates the skills that are necessary to produce results. Researchers like to distinguish between drills and scrimmages. Drills distort the game so as to work on a specific skill under maximum concentration. Scrimmages mimic the game to get a feel for the cadence and circumstances.33

Skill transfer is the degree to which working on a skill in one setting translates into another.34 For example, employees at Susquehanna International Group, a trading and technology firm, play poker, chess, and strategic board games to hone their skills and “apply them to complex financial markets.”35 The main question is how, and to what degree, the skills you practice help improve your investment process.

Finally, it helps to start by considering what we usually do in order to think about how we can do it better.

Investors, just as people in many walks of life, usually think by analogy.36 They compare two things with the sense that one idea will help explain or clarify the other.

More formally, we identify a source analog that becomes the basis of comparison with the target. The source commonly comes from memory. We then compare the source to the target to make inferences, often in the form of similarities. These inferences are then modified to capture differences between the source and the target. We then consider the lessons from the analogy’s success or failure.37

We typically tap our memory to come up with the source analogy. If we remember the right one, we learn about a new situation quickly. But more often than not we run into two potential problems. The first is the lack of breadth. We are simply unaware of other analogies that may be more appropriate than the one that is available to our minds. The second is a lack of depth. We make inappropriate inferences because the analogy relies too much on superficial similarities and fails to recognize causality.

The use of base rates is the remedy for the problems of breadth and depth. Base rates are particularly useful in certain cases where the reference class is well understood and has a distribution of outcomes that exhibit “mild randomness” and hence can be readily modeled statistically. Base rates can be more challenging when there is no ready reference class or when the distribution of outcomes show “wild randomness” and hence are difficult to measure.38

In domains such as sports and music, coaches or teachers oversee practice and provide valuable real-time feedback to improve results. The distinction between critique and correction is useful. A critique points out that something was done wrong and explains how to do it better. A correction requires doing it over, and properly, at the earliest possible opportunity.39 This means that errors in reasoning or technique should be identified and corrected on the spot.

Atul Gawande, a surgeon at Brigham and Women's Hospital in Boston, Massachusetts as well as a writer and public health researcher, wondered about the widespread use of coaches in sports and the dearth of coaches in medicine. He asked a surgeon under whom he’d trained to be his “coach” for one of his operations. He suggests “the case went beautifully,” but his coach had lots of observations about what he could have done better. There is a lot of opportunity to improve coaching in the investment business. Gawande suggests, “Coaching done well may be the most effective intervention designed for human performance.”40 But it starts by professionals acknowledging that there is room to improve.

Organizations

It is essential to find the right people and help them flourish, but it’s also important to recognize that most people make decisions from within an organization. Research shows that individual beliefs and productivity are shaped in part by organizations.41 For example, star investment professionals who switch firms tend to see a deterioration of their performance. The main reason is that the star’s skills were a better fit for the prior firm than the new one.42

Culture can play a significant role in mediating beliefs and behaviors. Researchers often describe culture as reflecting values and norms. Values are the standards of behavior that employees seek to realize, and norms are the degree to which they behave according to those values.43 Values and norms are influenced by governing objectives, incentives, teamwork, and the overall organizational environment. We want to examine these in the context of feedback.

Governing Objective. An organization’s governing objective is a clear statement of what it is fundamentally trying to accomplish. It molds an organization’s culture, communications, and choices about how it allocates resources. A clear governing objective gives decision makers a basis for dealing with the trade-offs that naturally arise in a dynamic setting. It also provides insiders and outsiders with a benchmark for assessing results.44

The most vivid example of dueling governing objectives for corporations is a focus on creating shareholder value versus building stakeholder value. We argue that creating long-term value considers the interests of all stakeholders while providing a basis for choosing between strategic and financial options. Stakeholder value seeks to maximize the interests of all stakeholders, which is impossible to do simultaneously.45 As a result, stakeholder value ends up being a matter of management discretion with no clear way to assess outcomes.

Investment management organizations also have tension between two potential objectives.46 Charles Ellis, the founder of Greenwich Associates and an investment industry luminary, distinguishes between the profession and business of investment management. The profession is focused on generating attractive long-term returns for investors. The business is focused on producing sales and profits for the investment firm. Ellis correctly observes that a healthy business is necessary to support the profession, if only to attract and retain talent, but that the pendulum can swing away from the profession toward the business. He emphasizes that the optimal balance must always favor the profession over the business because it is the best path to attract and develop talented investors.

Incentives. Feedback that encourages individuals to stay true to the governing objective can come from sources inside or outside the organization. Incentives can also play a large role in encouraging actions consistent with the organization’s broader goal. Proper incentives make pursuing the governing objective “an act of enlightened self-interest” for an organization’s members.47

Effective incentives are very difficult to design, and incentives by themselves rarely promote the proper behaviors. But every effort should be made to align incentives with the governing objective. Investors committed to the profession should have enough skin in the game to feel the organization’s ups and downs, but not so much as to impair judgment. Research shows that funds with high portfolio manager ownership do better than those with low ownership.48 That said, managers who own large stakes in the funds tend to take fewer risks, especially after a spell of poor results or investor outflows.49

Teams. One of the most profound changes in the investment management industry has been the shift from single- to team-managed funds. About 65 percent of funds had a single manager 30 years ago and about 20 percent do today (see exhibit 5).50 Team-run funds have generated higher excess returns than have single-managed funds, and they have done so with a lower standard deviation.51

Exhibit 5: Mutual Funds Shift from One to Multiple Managers

80 Team-Managed

70

Source: Campbell R. Harvey, Yan Liu, Eric K.K. Tan, and Min Zhu, “Crowding: Evidence from Fund Managerial Structure,” Working Paper, April 9, 2021.

We have already seen that trained forecasters do better than those who receive no training. The researchers at GJP also found that teams forecast more accurately than the trained individuals. Teams do better than individuals because they have additional knowledge. The forecasts of teams of elite forecasters are nearly twice as accurate as those of untrained individuals (see exhibit 6). A group can add value with complex problems that require a forecast.52

Exhibit 6: Training and Teaming Can Improve Prediction Accuracy

No Training 36%

Training 41%

Teams 44%

Elite 66% Teams

Improvement in Accuracy Versus Random Guesses

Source: Paul J. H. Schoemaker and Philip E. Tetlock, “Superforecasting: How to Upgrade Your Company’s Judgment,”

Harvard Business Review, May 2016, 72-78.

Whereas it is possible to measure the general intelligence (g) of an individual, researchers suggest that groups have a similar measure, which they call collective intelligence (“the c factor”).53 One study that included more than 5,000 individuals and 1,300 teams found that the variables that best explained collective intelligence included measures of “group collaboration process” and individual member skill. Less explanatory factors included group size, social perceptiveness, and group composition. The ranking of these factors varies by task.54

Studies of teams have also found that psychological safety is essential to success. Psychological safety exists when people feel comfortable saying what they think without concern that they will be punished if they make a mistake. Leaders can foster this openness by emphasizing collaboration over conflict and curiosity over blame.55 The virtue of psychological safety is that all points of view get a voice.

But the evidence of the value of group discussion is mixed. It can be the case that someone’s insight gets drowned out by the opinion of another, which is especially likely when there are differences in seniority. Collective calibration is a group quality that overcomes this. The idea is that individual members who are better calibrated, meaning that their confidence matches their accuracy, make the group’s discussion more fruitful.56 This underscores that training in forecasting techniques can benefit individuals as well as teams.

Successful teams have members who are capable. Feedback should focus on how the team is managed.

Essential elements include the ideal team size and makeup, having a leader or moderator who makes sure that all information and alternative views are brought to the surface, and proper voting techniques.57

Environment. Researchers distinguish between algorithmic and heuristic tasks within an organization.58 Algorithmic tasks deal with situations where cause and effect are clear and the goal can be achieved by following a recipe. Heuristic tasks arise in evolving and novel environments where there are no set steps to success. For instance, creating policy is essentially a prediction and policies often don’t succeed.59

Essentially all companies include algorithmic and heuristic elements. But in general, algorithmic tasks are more associated with businesses with tangible assets, and heuristic tasks accompany intangible assets. Visualize a steel mill in 1900. You can imagine a lot of workers carrying out tasks such as moving ore, feeding furnaces, and shaping steel but only a few fashioning new instructions. Algorithmic and heuristic tasks were both important, but most of the manpower was dedicated to implementing instructions.

Now visualize a research and development (R&D) lab where scientists seek to create a vaccine to provide immunity against a novel pathogen. Creating an efficacious vaccine is the hard part. The manufacturing must be done to exacting standards but the process is relatively straightforward.60 Tasks within organizations are migrating from algorithmic to heuristic, following the evolution from tangible to intangible assets.61

Leaders of organizations that partake primarily in heuristic work should strive to create an environment conducive to success. First, it should be committed to ongoing learning. In investment management, this means constant reading, welcoming guests who have done useful or provocative work, and sharing ideas across the organization efficiently.

Good organizations also welcome different points of view. This starts in the hiring process with an emphasis on cognitive diversity. Cognitive diversity measures functional knowledge, personality, representations, mental models, and categorizations. Most research shows that cognitive diversity is one of the key ingredients in groups that make good decisions.62

Frameworks for surfacing and selecting good ideas are also important. Ideally, these have been established and reinforced via training and practice. These mechanisms include the use of base rates and red teams, a method to create structured disagreement. Auditing the decision-making process is also a vital mechanism for feedback.

Finally, leaders should attempt to create an environment of equanimity. All organizations go through ups and downs, so removing peaks and valleys is essential. A focus on proper process with a long-term view contributes to this attitude.

Feedback During Process Execution

Providing feedback to fundamental investors is inherently difficult because the process to make decisions is often poorly defined and the outcome, the rise and fall of asset prices, is noisy in the short run. One way to solve this problem is to break down decisions into measurable components. But it all starts with documenting decisions and how they are made. Without a record of what you expect to happen, it is difficult to measure the accuracy of your predictions.

An investor who buys or sells a security in the hope of generating an excess return must have a variant perception, a well-founded view that the market has not priced properly. The variant perception, in turn, can be broken into subcomponents. Those subcomponents can be evaluated using a mechanism to score the accuracy of the predictions. This provides a solid foundation for intermediate feedback.

The Brier score, developed by the meteorologist Glenn Brier in the 1950s, is a popular method for tracking the accuracy of probabilistic forecasts.63 One version adds the square of the forecast error for the event, (forecast of event − outcome)2, to the square of error for the non-event, (forecast of non-event − outcome)2. For binary events, the value of the outcome is 1 if the event occurs and 0 if it does not. For instance, if you predict rain tomorrow with an 80 percent probability and it does rain, your Brier score is 0.08 [(0.80 – 1)2 + (0.20 – 0)2]. This version has a scale from 0 to 2 and, as in golf, a lower score is better.

You can calculate the Brier score for a prediction when you state a measurable outcome by a specified date with an associated probability. The straightforward step is to translate the essential elements of the variant perception into forecasts that can receive a Brier score. This allows for timely feedback on the aspects of the investment case that matter.

You might imagine a case where the market has priced in 100 new value-creating subscribers for a business during the next quarter and your variant perception is that the number of new subscribers will meaningfully exceed that level. Your prediction might be “the company will have more than 110 new subscribers by the end of the quarter with an 80 percent probability.” You can calculate your Brier score once the company has disclosed the subscriber number.

The best forecasters don’t just set a probability and leave it at that. They constantly make small refinements to their probability estimates as new information arrives. Those who make poor forecasts tend to either stick to their initial forecast or make rare revisions.64

The intelligence community has developed a technique called “linchpin analysis’’ that can be helpful in isolating the foundation for a variant perception. Roger George, who worked at the Central Intelligence Agency and has taught intelligence studies, summarizes the approach in the following steps:65

 Identify the main uncertain factors or key variables (drivers) that will determine an outcome.

 Identify working assumptions (linchpin premises) about how the key drivers will operate.

 Advance convincing evidence and reasoning to support the linchpin premises.

 Address any indicators or signposts that would render linchpin premises unreliable.

 Ask what dramatic events or triggers could reverse the expected outcomes.

You can use signposts as periodic checks on the thesis. Should you develop an investment thesis and find that you are missing widely on your forecasts about outcomes, the signposts present an opportunity to revisit your thesis and, if appropriate, to revise your view.

Most investment organizations have the raw material to keep Brier scores but fail to do so. Part of it is that keeping track of decisions requires some discipline. But the bigger issue is that when we are wrong, we generally cope by crafting a story to explain what happened in a way that avoids making us look bad. We are poor at predicting but great at explaining the past.66

But feedback can provide a payoff. First, just knowing that a component of an investment thesis will be evaluated encourages more thoughtful analysis and input. The relevance of base rates is immediately clear when individuals are asked to attach probabilities to events.

There is also evidence that this type of feedback improves calibration.67 Researchers in one study asked weather forecasters to make predictions on outcomes such as wind speed, visibility, and precipitation. They were generally overconfident, which means their subjective probabilities exceeded the observed outcomes (see light blue line in exhibit 7). The forecasters were then given extensive feedback and asked to repeat similar forecasts. The results were substantially more accurate the second time (see dark blue line).

Exhibit 7: Feedback Improves Calibration 1.0 Perfect Calibration

Source: Allan H. Murphy and Harald Daan, “Impacts of Feedback and Experience on the Quality of Subjective Probability Forecasts: Comparison of Results from the First and Second Years of the Zierikzee Experiment,” Monthly Weather Review, Vol. 112, No. 3, 1984, 413-423.

Other initiatives may be helpful as well and can actually be considered a part of practice. One is to periodically audit the process behind major decisions. Importantly, this should include decisions that turned out fine as well as those that turned out poorly. The goal is to honestly appraise the quality of the process, as good outcomes can be the result of a bad process and good luck, and bad outcomes can be the result of a good process and bad luck.

Another procedure is to keep track of decisions that led to no action. For example, an organization may contemplate the purchase of a specific security and pass on it. That decision should remain in the database and, like the decision to act, be evaluated periodically.

Finally, we can collect feedback from machines. Chess is an interesting example. Chess programs were deemed to have achieved superhuman levels of performance when Deep Blue beat world champion Garry Kasparov in 1997. The Elo rating system is a standard way of measuring relative skill in zero-sum games. The best chess player in the world today, Magnus Carlsen, has an Elo rating of 2865. The best chess program, Stockfish 14, has a rating of around 3540. (AlphaZero Chess, which beat Stockfish years ago, does not have an official rating but it is assumed to be higher than that of Stockfish.) The rating difference suggests that Carlsen would have less than a two percent chance of beating Stockfish.

Carlsen and other elite players now use the chess programs as teachers. They can consider moves and then consult the program to get feedback on their strength. This is a machine giving feedback in order to improve a human’s game.68

You might imagine a similar process in investment management. Position sizing within a portfolio, for example, can be expressed mathematically when the goals, constraints, and inputs are well specified. In concept, portfolio managers could have run a portfolio management program to suggest optimal position sizes given the assumptions. The investor could then compare their own intuitions against the machine’s calculations to see if and why there are differences.

Conclusion

Feedback is information that is applied to improve results. In this report, we focused on how feedback can improve the long-term outcomes for investment organizations.

We started by discussing the qualities of good thinking, including the characteristics of superforecasters, a group that made accurate forecasts above what chance would dictate, and the rationality quotient, a measure of rational decision making. The important insight is that intelligence quotient and rationality quotient measure different things and have only partial overlap. Superforecasters would likely score high on the test of rationality.

In particular, a significant attribute is being actively open-minded.

Getting the right people is great, and their skills can be enhanced through proper training. Learning how to apply base rates properly is a particularly effective tool for decision makers facing probabilistic outcomes. Beyond training, it is crucial to consider an appropriate ratio between practice and play. Investing is difficult in that there is always some information out there that we don’t know. As a result, the temptation is to play constantly. But evidence in other fields suggests that allocating some time to practice, assuming the possibility of skill transfer, may lead to improved outcomes.

Investors rarely operate fully autonomously and are generally part of an organization. That organization can shape beliefs and productivity. We discussed a number of key considerations, including the organization’s governing objective, the degree to which incentives align with that objective, the function of teams within organizations, and what type of environment is most conducive to success in a field dominated by heuristic tasks.

Generating timely and accurate feedback for fundamental investors can be difficult because the process is rarely uniform and the outcomes are noisy. One approach to dealing with this problem is to break an investment thesis into subcomponents and to track the accuracy of each of the probabilistic forecasts.

Buying or selling a security with an expectation of excess returns implies a view of what is going to happen that the market has yet to reflect in a price. The objective is to translate that variant perception into a cluster of measurable results and to keep score of how well the outcomes match the subjective inputs.

The encouraging news is that there is evidence that receiving timely and accurate feedback makes you a better forecaster. The apparatus to provide this feedback exists, but implementation requires some focus and discipline.

Endnotes 1 For a discussion along the same lines, see Cade Massey, “How to Identify Talent: Five Lessons from the NFL Draft,” Behavioral Scientist, April 23, 2018.

2 David Ignatius, “More Chatter Than Needed,” Washington Post, November 1, 2013.

3 Philip E. Tetlock and Dan Gardner, Superforecasting: The Art and Science of Prediction (New York: Crown Publishers, 2015) and Paul J.H. Schoemaker and Philip E. Tetlock, “Superforecasting: How to Upgrade Your Company’s Judgment,” Harvard Business Review, Vol. 94, No. 5, May 2016, 72-78.

4 For a good resource on this topic, see the website of Jonathan Baron, a professor of psychology at the University of Pennsylvania: www.sjdm.org/dmidi/Actively_Open-Minded_Thinking_Beliefs.

5 Keith E. Stanovich, The Bias That Divides Us: The Science and Politics of Myside Thinking (Cambridge, MA: MIT Press, 2021), 55.

6 Tetlock and Gardner, Superforecasting, 191.

7 Stanovich, The Bias That Divides Us, 9.

8 Ibid., 58.

9 See Mark Grinblatt, Matti Keloharju, and Juhani Linnainmaa, “IQ and Stock Market Participation,” Journal of Finance, Vol. 66, No. 6, December 2011, 2121-2164; Judith Chevalier and Glenn Ellison, “Are Some Mutual Fund Managers Better Than Others? Cross-Sectional Patterns in Behavior and Performance,” Journal of Finance, Vol. 54, No. 3, June 1999, 875-899; Aron A. Gottesman and Matthew R. Morey, “Manager Education and Mutual Fund Performance,” Journal of Empirical Finance, Vol. 13, No. 2, March 2006, 145-182; Mark Grinblatt, Matti Keloharju; Juhani T. Linnainmaa, “IQ, Trading Behavior, and Performance,” Journal of Financial Economics, Vol. 104, No. 2, May 2012, 339-362; Andreas Hefti, Steve Heinke, and Frédéric Schneider, “Mental Capabilities, Trading Styles, and Asset Market Bubbles: Theory and Experiment,” University of Zurich Working Paper No. 234, August 30, 2016; and Wändi Bruine de Bruin, Andrew M. Parker, and Baruch Fischhoff, “Decision-Making Competence: More Than Intelligence?” Current Directions in Psychological Science, Vol. 29, No. 2, April 2020, 186-192.

There also appears to be a genetic component to investing. See Amir Barnea, Henrik Cronqvist, and Stephan Siegel, “Nature or Nurture: What Determines Investor Behavior?” Journal of Financial Economics, Vol. 98, No. 3, December 2010, 583-604 and David Cesarini, Magnus Johannesson, Paul Lichtenstein, Örjan Sandewall and Björn Wallace, “Genetic Variation in Financial Decision-Making,” Journal of Finance, Vol. 65, No. 5, October 2010, 1725-1754.

10 Keith E. Stanovich and Richard F. West, “The Assessment of Rational Thinking: IQ ≠ RQ,” Teaching of Psychology, Vol. 41, No. 3, July 2014, 265-271.

11 Keith E. Stanovich, Richard F. West, and Maggie E. Toplak, The Rationality Quotient: Toward a Test of Rational Thinking (Cambridge, MA: MIT Press, 2016), 315.

12 Meredith C. Frey and Douglas K. Detterman, “Scholastic Assessment or g? The Relationship Between the Scholastic Assessment Test and General Cognitive Ability,” Psychological Science, Vol. 15, No. 6, June 2004, 373-378.

13 Steven Pinker, Rationality: What It Is, Why It Seems Scarce, Why It Matters (New York: Viking, 2021), 7. Also Maggie E. Toplak, Richard F. West, and Keith E. Stanovich, “Education for Rational Thought,” in John R. Kirby and Michael J. Lawson, eds., Enhancing the Quality of Learning Dispositions, Instruction, and Learning Processes (Cambridge: Cambridge University Press, 2012), 51-92.

14 Stanovich, West, and Toplak, The Rationality Quotient, 6-7.

15 Daniel Kahneman and Amos Tversky, “Prospect Theory: An Analysis of Decision under Risk,” Econometrica, Vol. 47, No. 2, March 1979, 263-292.

16 Hannah Ritchie, “Does the news reflect what we die from?” ourworldindata.org, May 29, 2019.

17 Robert P. Abelson, “Beliefs Are Like Possessions,” Journal for the Theory of Social Behaviour, Vol. 16, No. 3, October 1986, 223-250 and Stanovich, The Bias That Divides Us, 8.

18 Stanovich, West, and Toplak, The Rationality Quotient, 314. For more on the CART, see Keith E. Stanovich, “The Comprehensive Assessment of Rational Thinking,” Educational Psychologist, Vol. 51, No. 1, 2016, 23-34.

19 Welton Chang, Eva Chen, Barbara Mellers, and Philip Tetlock, “Developing Expert Political Judgment: The Impact of Training and Practice on Judgmental Accuracy in Geopolitical Forecasting Tournaments,” Judgment and Decision Making, Vol. 11, No. 5, September 2016, 509-526.

20 Daniel Kahneman and Amos Tversky, “Intuitive Prediction: Biases and Corrective Procedures,” Decision Research Technical Report PTR-1042-77-6, 1977.

21 Daniel Kahneman and Amos Tversky, “On the Psychology of Prediction,” Psychological Review, Vol. 80, No. 4, July 1973, 237-251.

22 Gerd Gigerenzer and Adrian Edwards, “Simple Tools for Understanding Risks: From Innumeracy to Insight,” BMJ, Vol. 327, September 27, 2003, 741-744.

23 Allan H. Murphy and Harald Daan, “Impacts of Feedback and Experience on the Quality of Subjective Probability Forecasts. Comparison of Results from the First and Second Years of the Zierikzee Experiment,” Monthly Weather Review, Vol. 112, No. 3, March 1984, 413-423.

24 See www.metoffice.gov.uk/research/climate/maps-and-data/uk-climate-averages.

25 Steven Pinker, The Sense of Style: The Thinking Person’s Guide to Writing in the 21st Century (New York: Viking, 2014), 27.

26 Ibid., 30.

27 The classic text is Edward R. Tufte, The Visual Display of Quantitative Information, 2nd Ed. (Cheshire, CT: Graphics Press, 2001).

28 Stephen A. Spiller, Nicholas Reinholtz, Sam J. Maglio, “Judgments Based on Stocks and Flows: Different Presentations of the Same Data Can Lead to Opposing Inferences,” Management Science, Vol. 66, No. 5, May 2020, 2213-2231.

29 Ed Thorp interview in Jack D. Schwager, Hedge Fund Market Wizards: How Winning Traders Win (Hoboken, NJ: John Wiley & Sons, 2012), 217.

30 Lewis Goldberg, “Man Versus Model of Man: A Rationale, Plus Some Evidence, For a Method of Improving Clinical Inferences,” Psychological Bulletin, Vol. 73, No. 6, June 1970, 422-432. Also, see Daniel Kahneman, Olivier Sibony, and Cass R. Sunstein, Noise: A Flaw in Human Judgment (New York: Little, Brown Spark, 2021), 117-122.

31 Stan Browne, Deb Clarke, Peter Henson, Frida Hristofski, Vicki Jeffreys, Peter Kovacs, Karen Lambert, Danielle Simpson, with the assistance of the Australian Institute of Sport, PDHPE Application & Inquiry, Second Edition (Melbourne, Australia: Oxford University Press, 2009), 150-151.

32 Anders Ericsson and Robert Pool, Peak: Secrets from the New Science of Expertise (Boston, MA: Houghton Mifflin Harcourt, 2016). Also, Daniel Coyle, The Talent Code: Greatness Isn’t Born. It’s Grown. Here’s How. (New York: Bantam, 2009).

33 Doug Lemov, Erica Woolway, and Katie Yezzi, Practice Perfect: 42 Rules for Getting Better at Getting Better (San Francisco, CA: Jossey-Bass, 2012), 48-51.

34 Robert R. Hoffman, Paul Ward, Paul J. Feltovich, Lia DiBello, Stephen M. Fiore, and Dee H. Andrews, Accelerated Expertise: Training for High Proficiency in a Complex World (Expertise: Research and Applications Series) 1st Edition (New York: Psychology Press, 2014), 49-65.

35 See https://sig.com/quantitative-trading/game-theory/.

36 Dan Lovallo, Carmina Clarke, and Colin Camerer, “Robust Analogizing and the Outside View: Two Empirical Tests of Case-Based Decision Making,” Strategic Management Journal, Vol. 33, No. 5, May 2012, 496-512. 37 Keith J. Holyoak and Paul Thagard, Mental Leaps: Analogy in Creative Thought (Cambridge, MA: MIT Press, 1995), 15.

38 Benoit B. Mandelbrot and Nassim Nicholas Taleb, “Mild vs. Wild Randomness: Focusing on those Risks that Matter,” in The Known, the Unknown, and the Unknowable in Financial Institutions, Frank Diebold, Neil Doherty, and Richard Herring, eds. (Princeton, NJ: Princeton University Press, 2010), 47-58.

39 Lemov, Woolway, and Yezzi, Practice Perfect, 52.

40 Atul Gawande, “Personal Best: Top Athletes and Singers Have Coaches. Should You?” New Yorker, October 3, 2011.

41 Mirta Galesic, Henrik Olsson, Jonas Dalege, Tamara van der Does, and Daniel L. Stein, “Integrating Social and Cognitive Aspects of Belief Dynamics: Towards a Unifying Framework,” Journal of the Royal Society Interface, Vol. 18, No. 176, March 2021.

42 Boris Groysberg, Chasing Stars: The Myth of Talent and the Portability of Performance (Princeton, NJ: Princeton University Press, 2010).

43 John R. Graham, Jillian Grennan, Campbell R. Harvey, and Shivaram Rajgopal, “Corporate Culture: Evidence from the Field,” Working Paper, January 28, 2022.

44 Michael J. Mauboussin and Alfred Rappaport, “Transparent Corporate Objectives—A Win-Win for Investors and the Companies They Invest In,” Journal of Applied Corporate Finance, Vol. 27, No. 2, Spring 2015, 28-33. 45 Michael C. Jensen, “Value Maximization, Stakeholder Theory, and the Corporate Objective Function,”

Business Ethics Quarterly, Vol. 12, No. 2, April 2002, 235-256.

46 Charles D. Ellis, “Will Business Success Spoil the Investment Management Profession?” Journal of Portfolio Management, Vol. 27, No. 3, Spring 2001, 11-15.

47 Mauboussin and Rappaport, “Transparent Corporate Objectives,” 31.

48 Ajay Khorana, Henri Servaes, and Lei Wedge, “Portfolio Manager Ownership and Fund Performance,” Journal of Financial Economics, Vol, 85. No. 1, July 2007, 179-204.

49 Linlin Ma and Yuehua Tang, “Portfolio Manager Ownership and Mutual Fund Risk Taking,” Management Science, Vol. 65, No. 12, December 2019, 5518-5534.

50 Saurin Patel and Sergei Sarkissian, “To Group or Not to Group? Evidence from Mutual Fund Databases,”

Journal of Financial and Quantitative Analysis, Vol. 52, No. 5, October 2017, 1989-2021 and Campbell R.

Harvey, Yan Liu, Eric K.K. Tan, and Min Zhu, “Crowding: Evidence from Fund Managerial Structure,” Working Paper, April 9, 2021.

51 Saurin Patel and Sergei Sarkissian, “Portfolio Pumping and Managerial Structure,” Review of Financial Studies, Vol. 34, No. 1, February 2020, 194-226.

52 Schoemaker and Tetlock, “Superforecasting: How to Upgrade Your Company’s Judgment.”

53 Anita Williams Woolley, Christopher F. Chabris, Alex Pentland, Nada Hashmi, and Thomas W. Malone, “Evidence for a Collective Intelligence Factor in the Performance of Human Groups,” Science, Vol. 330, No. 6004, October 29, 2010, 686-688.

54 Christoph Riedl, Young Ji Kim, Pranav Gupta, Thomas W. Malone, and Anita Williams Woolley, “Quantifying Collective Intelligence in Human Groups,” PNAS, Vol. 118, No. 18, May 25, 2021.

55 Laura Delizonna, “High-Performing Teams Need Psychological Safety. Here’s How to Create It,” Harvard Business Review, August 24, 2017; Julia Rozovsky, “The Five Keys to a Successful Google Team,” November 17, 2015; Amy Edmondson, “Psychological Safety and Learning Behavior in Work Teams,” Administrative Science Quarterly, Vol. 44, No. 2, June 1999, 350-383; and Amy C. Edmondson and Zhike Lei, “Psychological Safety: The History, Renaissance, and Future of an Interpersonal Construct,” Annual Review of Organizational Psychology and Organizational Behavior, Vol. 1, 2014, 23-43. See https://rework.withgoogle.com/blog/five-keys-to-a-successful-google-team/.

56 Ike Silver, Barbara A. Mellers, and Philip E. Tetlock, “Wise Teamwork: Collective Confidence Calibration Predicts the Effectiveness of Group Discussion,” Journal of Experimental Social Psychology, Vol. 96, September 2021, article 104157.

57 James R. Larson, Jr., Pennie G. Foster-Fishman, and Timothy M. Franz, “Leadership Style and the Discussion of Shared and Unshared Information in Decision Making,” Leadership and Information, Vol. 24, No. 5, May 1998, 482-495 and Ricardo Barahona, Stefano Cassella, and Kristy A.E. Jansen, “Do Teams Alleviate or Exacerbate Cognitive Biases: Evidence from Extrapolation Bias in Mutual Funds,” Working Paper, February 25, 2022. 58 Daniel H. Pink, Drive: The Surprising Truth About What Motivates Us (New York: Riverhead Books, 2009), 29-32.

59 J. Peter Scoblic and Philip E. Tetlock, “A Better Crystal Ball: The Right Way to Think About the Future,” Foreign Affairs, Vol. 99, No. 6, November-December 2020, 10-18.

60 This discussion is inspired by Paul Romer in “The Soft Revolution: Achieving Growth by Managing Intangibles,” Journal of Applied Corporate Finance, Vol. 11, No. 2, Summer 1998, 8-27.

61 Bradford C. Johnson, James M. Manyika, and Lareina A. Yee, “The Next Revolution in Interactions,” McKinsey Quarterly, No. 4, 2005, 21-33.

62 Jie Wang, Grand H.-L. Cheng, Tingting Chen, and Kwok Leung, “Team Creativity/Innovation in Culturally Diverse Teams: A Meta-Analysis,” Journal of Organizational Behavior, Vol. 40, No. 6, July 2019, 693-708 and Matthew Corritore, Amir Goldberg, and Sameer B. Srivastava, “Duality in Diversity: How Intrapersonal and

Interpersonal Cultural Heterogeneity Relate to Firm Performance,” Administrative Science Quarterly, Vol. 65, No. 2, 359-394, 2020.

63 Glenn W. Brier, “Verification of Forecasts Expressed in Terms of Probability,” Monthly Weather Review, Vol. 78, No. 1, January 1950, 1-3.

64 Pavel Atanasov, Jens Witkowski, Lyle Ungar, Barbara Mellers, and Philip Tetlock, “Small Steps to Accuracy: Incremental Belief Updaters Are Better Forecasters,” Organizational Behavior and Human Decision Processes, Vol. 160, September 2020, 19-35 and Christopher W. Karvetski, “Forecasters Who Think Again Are More Accurate,” Technical White Paper, February 2022.

65 Roger Z. George, “Fixing the Problem of Analytical Mind-Sets: Alternative Analysis,” International Journal of Intelligence and Counterintelligence, Vol. 17, No. 3, 2004, 391.

66 “Want Better Forecasting Skills? Silence the Noise,” Knowledge@Wharton, November 26, 2019.

67 Murphy and Daan, “Impacts of Feedback and Experience on the Quality of Subjective Probability Forecasts” and Fergus Bolger and Dilek Önkal-Atay, “The Effects of Feedback on Judgmental Interval Predictions,”

International Journal of Forecasting, Vol. 20, No. 1, January-March 2004, 29-39.

68 Oliver Roeder, Seven Games: A Human History (New York: W.W. Norton & Company, 2022), 91-92.