2015思想领袖论坛：注意——现在支付还是以后支付

Proceedings June 25, 2015 Tarrytown, NY

2015

Thought Leader

Forum

Table of Contents

Michael Mauboussin, Credit Suisse Introduction and Welcome .................................................................................................................................3

Paul Dolan, Professor of Behavioral Science, London School of Economics Attending to Happiness .....................................................................................................................................9

Iris Bohnet, Professor of Public Policy, Harvard Kennedy School Paying Attention to What You See: How To Design Gender Equality..................................................................29

Max Bazerman, Professor of Business Administration, Harvard Business School The Power of Noticing ....................................................................................................................................49

Christopher Chabris, Associate Professor of Psychology, Union College The Illusion of Attention...................................................................................................................................65

Michael Mauboussin Introduction

We all face a similar challenge: how to pay attention to what matters so as to make sure that we make good decisions. There are lots of bids for our attention, and the introduction of technology in recent years means there are more beeps and buzzes to divert our focus than ever before. Failure to allocate attention properly in the short run can add up to problems in the long run. Hence, the theme for the Thought Leader Forum in 2015—“Attention:

Pay Now or Pay Later.”

This year’s forum featured economists, psychologists, and even a world-class pickpocket. Each explained how focusing the spotlight of attention left a great deal to the shadows. We heard that how you allocate your attention can affect your happiness, how organizational mindsets can perpetuate implicit biases, how a failure to pay attention to small problems can snowball into predictable surprises, and how the notion of multitasking is a myth that should be delegated to the trash heap. And you know that focus can be manipulated when a pickpocket takes an audience member on stage and proceeds to swipe all of his valuables in front of a delighted audience.

You can think of attention on three levels—individual, organizational, and financial markets. In our personal lives, we have to figure out how to filter the signal from the noise. In most cases, the answer is not to try to drink from a fire hose of information, but rather to prioritize attention. Further, one useful approach is to shape your environment in order to shift some of your cognitive load. Finally, we know that creativity is generally associated with a mind-wandering state, but constant focus on screens limits that time. Carving out time for mind-wandering may be the path to doing your best thinking.

Attention is also very important for your organization. Companies, similar to individuals, often have mindsets—a lens through which to see the world. Mindsets can have implicit biases that diminish performance. So, you need to think carefully about whether you can include “nudges” in your organization to steer behavior away from the biases.

Motivation is also a crucial topic, and evidence suggests that the conditions of intrinsic motivation—autonomy, mastery, and purpose—can be fragile. Effective leaders shape the organization’s attention in such a way to promote motivation.

Attention is important for markets, too. There is always a topic, or worry, of the day. Understanding how to weight the relevance of that topic is critical. It is also important to pay attention to the person on the other side of your trade. What does he or she know that you don’t? Research suggests that we tend to be overconfident in our beliefs. Allocating attention toward the motivation of others can help check that overconfidence.

We all feel that the pace of change in the world is speeding up. This means that there are more demands on our attention than ever before. And there’s plenty of evidence that our attention wasn’t so good to begin with. So, we need to be aware of the challenges of attention and, more importantly, to adopt methods to allocate attention wisely.

The following transcripts not only document the proceedings, they also provide insights into how you can improve your attention management—as an individual and as a leader. One theme that ran throughout the day was how limited our attention truly is, and how much is going on outside our awareness. Once you recognize this reality, you can take concrete steps to address it in your own life and take advantage of it in business.

Michael Mauboussin Credit Suisse

Michael Mauboussin is a Managing Director of Credit Suisse in the Investment Banking division, based in New York. He is the Head of Global Financial Strategies, providing thought leadership and strategy guidance to external clients and internally to Credit Suisse professionals based on his expertise, research and writing in the areas of valuation and portfolio positioning, capital markets theory, competitive strategy analysis, and decision making.

Prior to rejoining Credit Suisse in 2013, he was Chief Investment Strategist at Legg Mason Capital Management.

Michael originally joined Credit Suisse in 1992 as a packaged food industry analyst and was named Chief U.S.

Investment Strategist in 1999. He is a former president of the Consumer Analyst Group of New York and was repeatedly named to Institutional Investor’s All-America Research Team and The Wall Street Journal All-Star survey in the food industry group.

Michael is the author of The Success Equation: Untangling Skill and Luck in Business, Sports, and Investing, Think Twice: Harnessing the Power of Counterintuition, and More Than You Know: Finding Financial Wisdom in Unconventional Places. He is also co-author, with Alfred Rappaport, of Expectations Investing: Reading Stock Prices for Better Returns.

Michael has been an adjunct professor of finance at Columbia Business School since 1993 and is on the faculty of the Heilbrunn Center for Graham and Dodd Investing. He is also chairman of the board of trustees of the Santa Fe Institute, a leading center for multi-disciplinary research in complex systems theory. Michael earned an AB from Georgetown University.

Michael Mauboussin Credit Suisse

Good morning. For those of you whom I haven’t met, my name is Michael Mauboussin, and I am head of Global Financial Strategies at Credit Suisse. On behalf of all of my colleagues at Credit Suisse, I want to wish you a warm welcome to the 2015 Thought Leader Forum. For those who joined us last night, I hope you had a wonderful evening. We are very excited about our lineup for today.

I’d like to do couple of things this morning before I hand it off to our speakers. First I want to highlight the levels at which you might consider today’s discussion of the role of attention. I then want to discuss the forum itself, including what you can do to contribute to its success.

You might listen to today’s discussion of attention at three different levels. Some of the points will span multiple levels, but these are some of the ideas that we’ll hear about throughout the day. So here they are.

The first relates to attention for us as individuals. How do I learn to pay attention more effectively? How can I improve my happiness or the happiness of those around me? How do I balance deep focus and mind-wandering?

How do I handle the information deluge?

The second is at the level of your organization. This room is full of leaders. What biases are embedded in the organization? How do you motivate employees? How do you pay attention so as to avoid business discontinuities and, in an even more extreme case, ethical breaches?

The final level is that of markets. What information do we pay attention to? And how do we weight it? How do we pay attention to what others are doing without mindlessly falling in line with what they do?

So, let’s start with the individual. One of the big challenges most of us face is lots of demands on our time and attention. Our natural inclination is to want to do many things at once—to multitask. Well, the scientific community has studied multitasking, and the verdict is in: it doesn’t work. When we think we’re multitasking, we are really doing sequential tasking—and each individual task is done poorly. Further, switching tasks is cognitively costly.

So, the prescription is pretty straightforward: don’t do it. And don’t pretend that you can. You may gain some emotional gratification from multitasking, but the evidence is clear that it degrades your cognitive functioning.

Simply speaking, your mind can operate in two states: focused attention or decoupled attention, also known as mind-wandering. Focused attention is required to complete specific tasks, such as writing a crucial email or reading a legal document. Mind-wandering allows us to consolidate thoughts and provides easier access to associative thinking. Indeed, we often do our best thinking in the mind-wandering state.

I was once at a presentation where the speaker asked people when they got their best ideas. The most popular answers were: in the shower, while working out—and, more specifically, running—and on plane rides. This was before the days of WiFi and handheld devices.

So, the antidote to too much focus is to carve out time to allow for mind-wandering. Walking, especially in a natural setting, is a very effective method to encourage creativity.

Finally, most of us have the sense of being overwhelmed by information. And that sense is not misplaced. One estimate suggests that we are exposed to five times more information daily today than we were 30 years ago.

That’s the equivalent of reading 175 newspapers!

The antidote for this is to reshape your environment. One idea is called “brain extenders”—offloading your cognitive load on to the environment. Let me give you two trivial examples. Say you check the weather forecast before you go to bed and see there’s a good chance for rain tomorrow. Instead of trying to remember that, you can simply place your umbrella by your door, or by your car keys. That way, the environment itself ensures that you remember.

Another case is the ideas that pop into your mind as you are trying to focus. Rather than try to remember them,

Michael Mauboussin Credit Suisse

simply keep a stack of index cards nearby and jot down what you’re thinking. Again, that will free your mind and allow you to focus.

The next level is that of the organization. We have a room full of leaders, and you’re all trying to build great cultures that will last. Here’s a question—and you’ll hear Max Bazerman talk about this later this morning: are there problems in your organization that will become bigger if you don’t address them now? If so, are you doing anything about it?

Now, it may seem obvious that you would do something about it. But what if it relates to a person who is highly productive or a business that is very profitable? What then? In his great book, Creativity, Inc., Ed Catmull, the president of Pixar Animation Studios and Walt Disney Animation Studios, describes numerous instances when he had to change the direction at Pixar to preserve the firm’s culture. The firm wasn’t in trouble at these junctures. They were just vulnerable. He noticed, and took action.

One key idea in attention is mindset—the lens, defined by a set of beliefs, through which an individual or organization sees the world. One famous example is Carol Dweck’s work on growth versus fixed mindsets in learning. Part of paying attention is carefully considering your organizational mindsets. What you may find is that there are biases in those mindsets—biases that may impede performance.

How might you deal with those biases? One approach is to use nudges—a structure in the environment to encourage a certain, and presumably better, behavior. Let me give you one example of a nudge that’s saved my bacon lots of times—the signs written on the streets of London that tell you to look right. For Americans or Europeans used to cars coming from the left, this nudge is a godsend. And I’m sure the British drivers appreciate it as well.

The final issue I’ll mention for the organizational level is the idea of intrinsic motivation. We all want happy and motivated employees. And there’s a fairly well developed literature on intrinsic and extrinsic motivation. For the kind of work that the people here do, intrinsic motivation is the key. What’s important to understand is that intrinsic motivation is fostered by certain conditions, including autonomy, mastery, and a sense of purpose, and that these conditions can be fragile.

Attention is relevant in a couple of ways. First, it is important to be overt about these conditions and to do what’s possible to strengthen them. But you have to pay attention to do so. At the same time, organizations commonly make changes that make sense on one level but that undermine intrinsic motivation. Pay attention to unintended consequences.

The final level at which you might consider attention is that of markets. One example is the basic idea of paying attention to the person on the other side of the trade. Why is he or she selling? Or buying? What do you know that the person on the other side of the trade doesn’t know? Research in psychology shows that we can be overconfident in our judgments.

Here you see the results of about 2,000 people who answered 50 true-false questions and were asked to assign a level of confidence to their responses. On average, people were 70% confident but correct only 60% of the time.

Paying attention to the motivations of others can be very useful in understanding the source of edge.

Markets are funny in that there always seems to be a topic of the day—concerns about Greece, China’s growth, deficits, interest rate changes, and so forth. It is also funny that there seems to be a rotation in the concern of the moment.

Michael Mauboussin Credit Suisse

Attention can be helpful if you can properly weight the importance of the information in the context of what you are trying to achieve. The key idea is that not all information should be given the same weight in arriving at a proper decision. The ability to weight information is analogous to the ability to properly weight positions within a portfolio— you can get very different outputs depending on how you consider the inputs. Naturally, the key is to place the proper amount of weight on the topic of the day and to maintain perspective.

Finally, it’s important to avoid mindlessly falling in with the consensus. There’s a great line from Seth Klarman that I love to quote: “Value investing is at its core the marriage of a contrarian streak and a calculator.” The “contrarian streak” keeps you vigilant about considering the view that’s not popular.

But the calculator part may be more important—it suggests that the popular view has driven a wedge between price and value, which allows for attractive returns. A lot of investors think they treat fundamentals and expectations separately, but few actually do. Paying attention to that distinction is crucial. So, as you listen today, you might consider attention at all three levels.

Before I speak about the goal of the forum, I want to mention the talented folks from Ink Factory. Dusty and Ryan will be graphically recording all of our presenters today. This means they will be synthesizing the words of our speakers into images and text to capture the key concepts. Their slogan is “you talk. we draw. it's awesome.” And we think you will agree.

Please feel free to take pictures of the artwork and to tweet the images. And we encourage you to ask them questions—after they are done drawing of course!

Let me end on what our goals are for the day. First, we want to provide you access to speakers whom you may not encounter in your day-to-day interactions but who are nonetheless capable of provoking thought and dialogue.

Second, we want to encourage a free exchange of ideas. Note that our speaking slots are longer than normal, in large part because we want to leave time for back-and-forth. We purposefully call this a “forum” instead of a “conference” precisely for this reason. We want to encourage an environment of inquiry, challenge, and exchange.

Finally, we want this to be a wonderful experience for you, so please don’t hesitate to ask anyone on the CS team for anything. We will do our best to accommodate you.

With that, I’m pleased to introduce our first speaker of the day.

Paul Dolan London School of Economics

Paul Dolan is an internationally renowned expert on happiness, behavior, and public policy. He is currently a Professor of Behavioral Science in the Department of Social Policy and Political Science, and is Director of the Executive MSc in Behavioral Science, both at the London School of Economics.

Dolan earned his degree in Economics from Swansea University in 1989. His masters and doctorate on “Issues in the valuation of health outcomes” came from University of York in 1991 and 1997, respectively. Dolan has over 100 peer-reviewed publications on topics that include behavioral science, subjective well-being, equity in health, and health valuation. He currently holds the position of Chief Academic Advisor on Economic Appraisal for the U.K. Government’s Economic Service. He is also a member of the National Academy of Sciences Panel on Well-Being and of the Measuring National Well-Being Advisory Forum. In addition, he is a Visiting Professor at Imperial College London and an Associate of the Institute for Government.

Dolan is currently researching the happiness hit from the 2012 Olympic Games. He is an author of the “Mindspace” report published by the U.K. Cabinet Office that seeks to apply lessons from the psychological and behavioral sciences to social policy. Last year Dolan appeared at the Hay Festival in Cartagena, Colombia and discussed the role of modern technology and happiness, as well as his work on experiences of purpose, attention, and happiness.

Dolan also gave the Queen’s Lecture on “Happiness by Design” at Technische Universität Berlin in November 2013.

In August 2014, Dolan published his book Happiness by Design, with foreword by Nobel Prize-winner Daniel Kahneman. In 2002, Dolan won the Philip Leverhulme Prize in economics for his contribution to health economics and for his work on quality adjusted life years in particular.

Paul Dolan London School of Economics

Michael Mauboussin: I’m very pleased to introduce our first speaker this morning, Paul Dolan. Paul is an economist by training and is now a professor of behavioral science at the London School of Economics (LSE).

Paul’s book, Happiness by Design, came out last year and I read a review of it by Cass Sunstein in the New York Review of Books. What struck me was the book’s central thesis: that happiness—or subjective well-being—is to some degree a function of what you choose to pay attention to and the environment you create for yourself. Both of those themes are central to our discussion today.

I mention Paul’s background as an economist because he has made important contributions to the world of measuring and valuing healthcare policy. He’s used to thinking about costs and benefits. So, he is one of the very few people who makes the explicit link between happiness and behavior. And the behavioral part is where attention fits in.

I’ll just note quickly that you’ll hear about the balance between pleasure and purpose, and that purpose is a key ingredient in intrinsic motivation—very relevant to the organizations that many of you lead.

Please join me in welcoming Professor Paul Dolan.

Paul Dolan: I’m your warm-up act for the day. Good morning.

Audience: Good morning.

Paul: Good morning!

Audience: Good morning!

Paul: Good. That was like the LSE on a Tuesday morning at nine o’clock. So, I’ve got some slides today, which I never use, so we’ll see how that goes. Are you happy? Would you say you were happy?

All right, I’ll see what I can do about that then. Just to be clear, to manage expectations, I always try and talk for half the time that I have. So I shall do that and leave half the time for questions and comments.

So, is my friend happy? Well, the answer is, it depends. That’s a very academic answer to most things, actually.

But it depends on what we ask him, and what he pays attention to when he answers the question. And when we’ve been measuring happiness for the best part of the last couple of decades, we’ve done it usually in ways that ask people to evaluate their lives. And we ask questions like, “Overall, how satisfied are you with your life these days?”

That’s a question that’s been widely used. It’s been asked in the U.K., for 10,000 people for 20 years; in Germany, similar size sample, similar periods of time. It’s asked on an international basis. And so when you hear comments made about: Does money make people happy? Does marriage make people happy? It’s nearly always in relation to a question that is an evaluation of how well life is going: how satisfied are you with your life these days?

I have some problems with that question, for reasons that I’ll elaborate on in a second. But alternatively, we could actually just ask someone how they’re feeling: “Michael, how happy are you right now?” And, “Max, how angry are you? How sad are you? How stressed are you? How worried are you?” Actually, we’ve got many more negative emotions than we’ve got positive ones, adjectives for them. I could ask you how you feel. And when we do that, we actually get different answers.

Let me tell you a story. We were talking last night about how bands manage to play the same songs for 20 years and make it sound like it’s the first time they’ve ever played the tune. But I’m going to tell you a story that I’ve told many, many times, and I’m going to try to pretend it’s the first time that I’m telling you. And so I’m very excited about telling you this new insight from the book.

Paul Dolan London School of Economics

I went for dinner with a friend of ours, a friend of my wife, and she works for “MediaLand,” or she worked for MediaLand. She changed her job actually, it’s worth noting. You could probably guess what MediaLand is.

We’re not being taped, are we? Are we being taped? We are being taped. MediaLand is fine. It’s a three-letter broadcasting corporation in the U.K. And she complained the whole of the evening about her boss, her colleagues, her commute.

Everything about her day-to-day experiences was miserable. It was very clear. The whole of the evening she just spent moaning. And then without a hint of irony, at the end of dinner she says, “Of course I love working at MediaLand.”

Now, actually, that’s not such an odd thing to say, and it’s actually how a lot of people live their lives. She worked for a great organization, somewhere she’d always wanted to work. Her parents were proud. Her friends were jealous. When she evaluated her life at MediaLand, she was happy.

But her experiences told her something quite different. Day to day, moment to moment, her experiencing self, as Kahneman calls it, was having a pretty miserable time.

So, is she happy? Well, it does depend. But I’d like to say actually she’s not. Because she’s only happy when she pays attention to the great story that she tells about the great job that she has. And actually, most of the time she’s not paying attention to the great story that she tells about the great job that she has.

She’s paying attention to the commute, to the colleagues, to the boss. Day to day, moment to moment, the experiences she pays attention to, much more readily and much more often, are telling her that she’s not happy.

And it’s actually one of the reasons why on the U.K. paperback, one of the quotes from the New Statesman is, “The book that will make you quit your job.” Actually, this is not a book about job quits at all. But I have to say, I have had a few emails from people who have quit their jobs, and also from people who have dumped their boyfriends or girlfriends, husbands or wives as a result as well, so I’m not sure it should be called Happiness by Design. More like Chaos by Design or something.

But again, a lot of people will be living in narratives about the partners or the jobs or the houses or the things that they should be doing, because they’re told either by historical accident, evolutionary processes, social constructions, by their parents, that these are the things that they should be doing in order to be happy. And actually that can often sit in sharp contrast to the things they actually do day to day.

It is worth saying at that point, though, that it’s possible for the evaluation and the experience to also be different in the opposite direction. That is, you could be telling a really bad story about how you’ve got a crap job, you’ve got an awful partner.

But actually, day to day, moment to moment, you’re having a great time. So it’s not always the case that evaluations are good and experiences are bad. It could actually be the other way around.

But anyway, this is actually quite important because … this is why I was going to use slides, for a change, because I have some data. These are data from the U.S. These are 13,000 American citizens in the American Time Use Survey (ATUS). And they’re asked to evaluate their life on a ladder between zero and ten. That detail doesn’t matter. It’s essentially an evaluation of how good life is.

And when we ask that question, in whatever way it’s asked, however it’s worded, we always find, in every single dataset, a U-shape in age. Always. In fact, there’s a U-shape in age for monkeys. Well, monkeys don’t actually report how happy they are. The zookeepers report how happy they think the monkeys are. And monkeys have a midlife crisis as well. They live to about 40. But about 20 is when they’re having this midlife crisis. Many of you in the room will be at this nadir of happiness. Insofar as you don’t die … that’s obviously part of the reason why

Paul Dolan London School of Economics

people get happy as they get older, because you kill off the miserable ones.

But insofar as you don’t die, you can look forward to getting happier, probably until the last six months of your life, which are pretty miserable. But if you can know when they’re going to be it would be a lot easier. But, massive increases past late 40s, early 50s. And it’s worth saying also that men in their 40s are the least happy group of all. So, I’ve got a tough audience to try to satisfy. And we saw that, as I said, in all evaluation data.

But let’s look in the same dataset, with the same 13,000 people, at what experiences tell us. Well, something quite different. And let me, if you can just ignore the purpose line for a second and just focus on the pleasure one, which is the thick line. It’s kind of flat, might go up a little bit as people get older. But emotions change through the life course.

Many of you, as you get older, will realize that you experience less excitement, but probably more contentment, maybe less stress, maybe more worry. I mean things change as you go through the life course. But it’s much messier. So, it does actually matter how we ask the question.

Now, let me deal then, separately, with something that Michael touched on in his introduction, which is experiences of purpose. For two and a half thousand years, when people have thought about purpose, they’ve thought about it in an evaluative sense. When I reflect on my life, does it have meaning? Does it have purpose?

Well, because of an attentional process to actually think, you know, I think it’s the fortune cookie maxim Kahneman uses, “Nothing in life is as important as you think it is whilst you’re thinking about it.”

But when you’re paying attention to the story that you tell about whether your life has meaning, it really matters. But most of the time you’re not paying attention to the story about whether your life has meaning, you’re going about your daily business engaged in activities that sometimes feel like they have a point, other times feel worthless.

And so I am interested in the meaning of moments, and not in the meaning of life. I want to know whether the things, the activities that people engage in, feel purposeful, worthwhile, meaningful, fulfilling, as people engage in them, not as they reflect on the narrative and the story that they tell about whether their life has meaning.

So, these data are actually the best dataset and about the only one really, on a large scale, where we’ve got people to report how meaningful the activities that they engage in are. In the American Time Use Survey, three activities from a diary that people fill out are chosen randomly, and they report essentially how happy they felt during that activity and how purposeful it was, or how meaningful it was, I think is the language that’s used in the ATUS.

Now, when you do that, you find something very, very interesting about purpose by age. And that is, young people, up to their mid-20s, experience very low levels of meaning in the activities that they engage in.

And that would seem to me to be particularly interesting for policymakers thinking about high school educations or university educations, and thinking about kind of trying to engage people in learning and education and it may be on-the-job training or whatever. Because if you think about it, in your own lives, there’s nothing worse than doing something that feels pointless.

Forget whether you tell a story about whether it’s meaningful or not. Actually, in the experience, in the moment, doing something that feels like a waste of time, feels like a waste of time. It hurts. You want to quickly stop doing things that feel like they’re a waste of time.

So we must be capturing, I think, we should be capturing much more the experiences that people have as they go about their daily lives, moment to moment, day to day, and not in these big grand stories that people tell.

Paul Dolan London School of Economics

When we do that, and we look at the American Time Use Survey, we have some interesting patterns that emerge. And I guess some of this is not particularly surprising. Watching television is slightly more pleasurable than time spent with children. Probably watching anything on television is more pleasurable than time spent with children.

But time spent with children is pretty purposeful. And actually that’s what … a lot of our academic interests are informed by personal intuition and insight, and I’ve considered myself for a very long time to be a pretty good happiness maximizer.

I’m pretty good at happiness. I do a pretty good job at it. And when I was thinking about whether to have children or not, the evidence suggested that I probably shouldn’t bother. At best, children are neutral in their effect on happiness. Moments of joy, long periods of worry, stress, misery, and anxiety.

But when you add in experiences of purpose to sit alongside those of pleasure, then … it’s arguable whether you’re happier overall having children, but you might be differently happy.

The balance shifts between pleasure and purpose. So, the times that you spend reading to your kids, listening to the kids read to you, tying their shoelaces, teaching the times tables, they feel like they’re worthwhile even if they’re not the most fun activities that you can engage in.

The argument in the book is that happy lives are ones that contain a good balance between experiences of pleasure on the one hand, and purpose on the other. And it’s for you to work out what that balance should look like. It’s not going to be the same for everybody, and it’s not going to be an equal measure.

But I think I can confidently make the claim that if you’re consuming, experiencing lots and lots of pleasure and hardly any purpose, you can be happier if you had a bit more of a purpose and a bit less of pleasure, and equally the other way around.

Again, our academic insight is informed by personal experience and intuition. Coming from a bog-standard, working class background, I was around people who had lots of pleasure in their lives. We had some good parties. They lived for the weekend. Not a great deal of purpose.

Then you work at institutions like Princeton and the LSE, and it’s the opposite. People having lots and lots of experiences of purpose, but hardly any fun. And so if I could sort of take the best bits of each of those worlds and put them together, I’ve kind of got the perfect happiness maximizing machine.

I wanted to speak a little bit about purpose before I move onto attention more directly. It’s important. Again, just to restate this again, important in the experience, in the experience as you live and breathe the activities that you engage in, that you pay attention to, and not as you reflect upon your life.

And there are some interesting studies showing the effects of purpose or lack of it, on outcomes that we might be interested in, like graduating from high school or staying married and being happy in your marriage.

I think perhaps more importantly is what we can do about it, and particularly in workplaces. And we had a brief discussion about this over dinner last night. I’ll talk a bit about this in the conclusion, but what behavioral science teaches us is that you can have very big effects for small changes. I spent two decades around policymakers, and I think one of the biases around policymakers is if they’ve got a big problem, they think the solution has to be big too, otherwise you’re not doing the problem justice.

The solution needs to be proportional to the problem. But actually, very small changes can have big effects. You can increase purpose in workplaces just by giving timely and salient feedback about the activities that people engage in, particularly when they’ve done something that might feel like a waste of time.

Paul Dolan London School of Economics

You started working on a project which has been terminated or cancelled or shifted and you feel like all the time that you spent working on it, all those late evenings or early mornings over the last month, have been a waste of time, and no one tells you otherwise. You get no feedback that this has been useful, that it will contribute towards future projects, that it might not be useful now but we’re going to find ways of using it later.

All these things that are, if it’s done sincerely, of course, you give people timely and salient feedback—it’s absolutely critical to improving their experiences of purpose. In work, as in, actually, our personal lives, what you want is tasks and challenges that are challenging but achievable. And that are best broken down—the evidence on this is very, very clear—broken down into bite-sized chunks, achievable little targets. And what you can do then is you psychologically, if not literally, take off, having achieved the next step.

That makes you feel good. It motivates you to then move onto the next one. You don’t set someone a target that, in a year’s time, they should achieve something or other. I mean that’s completely unmanageable in terms of people’s psychology and motivation.

And interestingly, I think, in workplaces often people will get stuck in ruts. You basically find a skill that you’re good at and you stick at it and you keep on doing it, but in time that gets a little boring. And so one thing we need to do is to think about how you can shake up attention by using different skills in different ways at different times and kind of give people the opportunity to try new things.

And so, in conclusion to this bit, it’s all about—and this is the subtitle to the U.K. version of the book—it’s all about finding pleasure and purpose in everyday life, not in the stories that we tell about the things that we think should make us happy.

Let me now move on then to attention. We are what we attend to. There’s nothing new in that. It’s been around since William James, I think, about 18-something. We are what we attend to. But as an economist, I think about, if that’s the case, then how do we best allocate attention? It’s clearly a scarce resource.

It’s no accident that we use the term “pay attention.” Of course, by paying attention to one thing you’re necessarily not paying attention to something else—either you’re doing that consciously or automatically. So, it’s a scarce resource that we need to think about using wisely.

And we need to think about using it wisely in a whole range of contexts, particularly in relation to happiness.

Because, when I said at the beginning: “Does money make people happy? Does marriage make people happy?”

The answer to that question is, “It depends.” And it depends on how much attention is paid to those things. But economists have almost just gone straight from the input to the output. What’s the direct relation between income and happiness?

But Econ 101 teaches you, if you’re trying to make a widget, you have a production process that takes the inputs and converts it into the output. You don’t just have widgets from land, labor, and capital. There’s a production process. And the production process for our happiness is the attention that we pay to the various stimuli.

So the answer to the question, “Does money make people happy?” is, “It depends on how much attention they’re paying.” And, of course, if you got a pay rise and spent all of your time paying attention to how happy you were with the pay rise, the pay rise would make you very happy for very long. And actually, that’s what you think about when you’re thinking about whether something will affect you, is you’re thinking about thinking about it.

So, when you think about whether the pay rise will make you happy, you’re thinking about how happy the pay rise will make you when you’re thinking about the pay rise.

But once you get the pay rise, you quickly stop thinking about it. In fact, once you open that paycheck you think,

Paul Dolan London School of Economics

“Actually, that’s not that much, and those other bastards are earning much more than me still. Actually that’s now making me miserable.” And pizza still tastes like pizza, the X Factor is still crap, the kids are still arguing and fighting; all the things that then draw attention to themselves in the experience of your life that have nothing to do with whether you’ve got a pay rise or not. And so that’s it. And so it’s about what we pay attention to in our daily experiences.

I have to do a shameless plug for some of our own academic work. We’ve used attention and its allocation to explain a whole range of different phenomena, particularly, I think, adaptation processes. You know, the human condition is really good at getting over stuff. Most people get over many things, not everything, in time, and sometimes relatively quickly.

We looked at very nice data in the U.K. on happiness reports annually, in the British Household Panel Surveys, now called Understanding Society. And those people are interviewed typically in September, but some of the interviews go through into October, November, and December. And as luck would have it, when the terrorists struck in New York, they did it in September.

And so we had a nice natural experiment where we could look at our interviewees who were interviewed in 2000, 2001, and 2002. And when they’re interviewed in September, October, and November, is random; the interviewers kind of turn up when they turn up.

So we could look at the impact of September 11 on well-being in the U.K. without asking people to think about how much the September 11 attacks affected their well-being when they’re thinking about it. We can do it because it’s all about attention. We can get the impact of September 11 on people’s happiness without them having to think about the impact.

And what we find, very intuitively, is that September 11 has an effect on people’s happiness on September 12, and through September. It starts disappearing in October. By November or December it’s gone.

Now, of course, if you ask people how affected they were by September 11, they would say they were affected because they’re paying attention to it. But that’s not what I’m picking up. I’m picking up their overall happiness, conditional upon when those attacks took place, and showing that there’s actually relatively quick adaptation to that event.

Of course, that’s in the U.K., a few thousand miles away, so I wouldn’t lay claim that that would be the same in the U.S. But it’s an example of many examples in the academic literature of stuff that we get used to relatively quickly, unless we’re reminded to think about it again.

Weight gain is, I think, really important for public policy purposes, of course. And what we show in a paper with a theoretical model and then some empirical data, is that basically, what you can do when you get fat is you can do one of two things: You can expend energy and effort to lose weight, but that takes energy and effort. And/or you can expend energy and effort to pay less attention to the weight gain in your overall happiness.

And we show that as people gain weight, their health satisfaction, for example, becomes a less important contributor to their overall happiness. So, we reallocate attention. And it explains the fact that as people gain weight they don’t get very much less happy.

A nice example of a focusing effect, by the way, because when we look at someone who is obese, we look at an obese person thinking they must be thinking about being obese, and, therefore, they must be less happy. But they’re not thinking about how overweight they are all of the time, in fact, hardly at all. And, of course also, weight gain doesn’t just happen overnight, it gradually grows on people, literally, and so there’s a kind of nice adaptation process anyway, even without that attentional shift.

Paul Dolan London School of Economics

I won’t speak about the unemployment and health state evaluation papers for the purposes of time because I want to get to the discussion. But one thing that I wanted to pick up on—because I just thought about this with Michael’s intro—is, absolutely right, we can’t multitask. Women can’t as well, by the way. It depends on what you mean by “multi.” I’m actually standing and talking and breathing, so I guess you could say that’s kind of multitasking.

But in an attentional point of view, you can’t pay attention to two stimuli at any one time. What we do is we single-task quickly, that’s what the brain science tells us. And, absolutely, it makes us less efficient and less happy, because all the time that your brain is switching between tasks, it’s using attentional resources. It’s tiring itself out.

So in one study, a nice study I think, people were asked to do a Sudoku puzzle and a crossword puzzle, and they could do them in any order—they could flip backwards or forwards between them, or they could have the order prescribed to them. Now, in that third condition, people finished those tasks much more quickly than in the previous two, interestingly, even when they could choose the order in which they did them. Because you’re still requiring attentional resources to decide, yeah, I’m better at Sudoku than I am at crosswords, do I do the one that I’m good at first?

All that time that you’re using attentional resources, you could just be solving the problems. So, sometimes prescription about the order in which people can do tasks might actually be more efficient. But one thing that is definitely inefficient is to be distracted. You should avoid distraction at all costs. We are nearly always happier and more efficient when we’re paying attention to what we’re doing and who we’re doing it with in the moment.

It’s essentially mindfulness without having to go on an eight-week mindfulness program and retreat for three weeks and not speak to anybody, which is my idea of a living hell, but it’s about being mindful in the moment without having to do the mindfulness training. And the way you do that … a bit more about that in a second … is to design your environment in ways that make it easier for you to pay attention to what you’re doing.

So, for example, turn your phone off. That’s a really basic insight. Really significantly, your unconscious attention is drawn to your mobile phone even if you’re not consciously paying attention to it. Hands up if you’ve ever had your phone in your pocket, felt it vibrate, taken it out, and you’ve still got no friends. Right? Right. Because unconsciously you’re thinking, “My wife should text me at some point. The kids should be in touch after three years.” And they’re not. So, you’re draining yourself of energy even when you don’t know that you’re doing so.

So, pay attention to the moment. But that’s to be distinguished, and this is really important, in workplaces particularly, from taking a break. Taking a break is an organized distraction, if you like. It’s planned. So, as Michael said, absolutely right, no one … if I said to you, have you ever had your best thought at work when you’ve been thinking, “Now I’m going to think of a really good idea”? No one has had a good idea when they sit down to have a good idea. They have a good idea when they’re in the shower, on the bus, on the plane.

And what we think is happening is, that when you consciously think about something, and then move away from it, your unconscious mind is still processing the challenge or the task or the activity. And what’ll happen is, it’s as if the answer pops into your head, randomly, by mistake or accident, but you’re actually thinking about it without knowing that you’re thinking about it.

So, in one study students were asked to choose a poster for their dorm room, and they had to just briefly look at the posters, go away for nine minutes and do something else, and then come back and choose a poster. They were much happier with their choices than students who spent nine minutes consciously and effortfully thinking about which poster to choose.

Now, this is not uncontroversial evidence, but there is a suggestion, at least, that the unconscious mind is

Paul Dolan London School of Economics

processing the decision whilst you’re not thinking about it. So, pay attention to what you’re doing, who you’re doing it with in the moment, and plan breaks to allow your mind to wander and freely think about the activities when you’re not doing so.

Very interesting … I said the subtitle to the U.K. book is Finding Pleasure and Purpose in Everyday Life. The subtitle to the U.S. version—paperback out next month by the way that’ll be cheaper than the hard back—is Change What You Do, Not How You Think.

Really interestingly, just as we were about to go to press, my U.S. publisher said, “You know what? Americans won’t get that vague, waffle-y, Finding Pleasure and Purpose in Everyday Life. They need to be told what to do.

It needs to be more directive. Change What You Do, Not How You Think. That’s it, that’s what we need. We need to tell these Americans, because they’re not as clever.”

So, that’s what you need to do. Because attention is allocated unconsciously, as well as consciously, you can’t think yourself happier. Well, you can try, but it won’t work. And if you buy any self-help book, it’ll tell you things like, “Be positive.” Yeah, no sh**, I worked that out. How do I actually do that?

And you can’t do it, so you buy another self-help book. That’s why they sell so many. The single biggest factor predicted in buying a self-help book is having bought a self-help book in the last 12 months. So, clearly, they don’t work, because they are all about changing minds. And it’s really hard to change minds.

In a whole range of different environments, particularly in public policy challenges and in corporate environments, it’s really hard to change the way people think. You have a belief that you hold, for whatever reason, and then you search out evidence to support the belief. And when you find evidence to support the belief, you’re really clever and you were right to believe it.

When you find evidence that doesn’t support the belief, then you’re even cleverer. You find a counterfactual that explains away the evidence and why it doesn’t apply to you, and why it doesn’t apply to what you know to be true, and you’re even more right to believe what you did in the first place. Really hard to change minds.

Thankfully, it’s much easier to change context, to change environments, to change situations within which people act. Because, as many of you may know if you’ve read or come anywhere close to the book, Thinking, Fast or Slow, most of what we do simply comes about rather than being thought about.

We are driven by System One, as Kahneman calls it, the old, evolved reptilian brain is there. It’s there basically to help make your life easier. You’re making up to 10,000 decisions every day. You don’t think about most of those. Your head would explode if you did. It would be effortful and take a long time.

Your automatic brain, your unconscious system, is just processing and processing and processing and always working, and always active, always present, always engaged, and trying to help make life easier for you. So, if you’re going to change your behavior—it doesn’t have to be just to be happier, it can be to be more efficient at work—you need to think about how that unconscious reptile is processing.

Because that’s the bit that’s doing the work, not your conscious mind. You need to get over yourself if you think you think about what you do, because you don’t. You might think you do, because that’s the only bit that you’ve got conscious access to. You’re really good at telling a nice story about why you’re just about to do something, and an even better story about why you just did it, that has no relation to actually why you did it. So, lots of studies around priming. Many of you will know some of this evidence.

We’ve been doing some work with hospitals in the U.K., in London, where we’re having significant increases in hand-washing rates by pumping citrus lemon smells around the wards. Citrus lemon is a clean smell. The brain associates clean smell with then wanting clean hands. But really importantly, when you ask people why they

Paul Dolan London School of Economics

wash their hands, they don’t say, “Because the citrus lemon made me.” They say, “Because it’s the right thing to do . . . It’s important . . . Hand hygiene.” All the nice stories that your conscious mind can construct to explain the behavior, because we like stories.

The human condition likes narratives and stories. It likes coherence. It likes structure and organization. And so it’s very good at telling a very coherent story about why you do what you do. You’re not lying. You’re just a poor witness to your behavior. You’re just a really poor witness to your behavior.

Priming of smells, of colors, of light . . . one of my favorite studies recently gave students an extra dollar in an experiment—overpaid them. And they looked at whether the students returned the money. How many students do you think would return the dollar, by the way? Ninety percent? Zero? Somewhere between zero and 90, okay. That’s probably going to be right, isn’t it, somewhere between zero and 90.

Well, the answer is, guess what? It depends. It depends on whether the students are under bright light or under dim light conditions. Under bright light conditions, the person who said 90 percent was quite close. It was 85.

Under dim light conditions, 50 percent. Now that’s a 35 percentage point difference in honesty primed only by light. And primed in ways that the participants in this study had no idea was influencing their behavior.

They all told a story afterwards about why they kept it or why they handed it back in, unaware that it was hugely affected by the environment, by the situation, by the context within which those behaviors were placed. So if you want honest people in your workplace, you just have to turn the lights up. But, interestingly, maybe if you want creative workers, turn the lights down.

In another study where people were asked to draw an alien . . . I know, all these crazy things . . . and they looked at how far away from the human form the alien looked. And the further away from looking like a human the alien looked, the more creative was the alien. And aliens further away from the human form were drawn under darker light conditions than under brighter light conditions. So if you want to allow your mind to run free, you might want to turn the lights down.

Context matters. That’s the really, really significant . . . those are the two words that come out of my mouth at LSE more than any other words: “context matters,” “context matters.” In fact, students have T-shirts made up with “context matters” because context rather than cognition drives most of what you do.

In the book, I talk about how you can use these insights for yourself in order to be happier. The great thing about some of these effects is you can do them even though you know you’re doing them to yourself. When people take placebos and they’re told they’re taking placebos, they still work. So, we can fool ourselves.

With the priming, my editor at Penguin in the U.K. used to come into work very stressed every day and had so much to do, she changed her password—it’s probably case sensitive, so I’m not giving it away—to “hour by hour,” so that she just more calmly took her day. And she said it worked. But even though she was changing her password, knowing that that’s what it was intended to, just the mere fact of typing that in cued her to behave differently, or to feel calmer.

So, defaults, we know this, I mean you know this much more than I do . . . pension contributions—much more likely to save if they’re opt-out than if they’re opt-in. Organ donation. We’re basically lazy. We go with the flow of whatever the preset options are. So set yourself some preset options that make it more likely you’re going to do it.

Commitments are really interesting. Make public promises. You’re more likely to act on them. People who posted their weight loss targets on Twitter were more likely to lose weight than those who just listened to a podcast of weight management. Pretty interesting. Norms. We spoke about that a bit at breakfast. If you really want to do something, find people that do it. We’re social animals. This is something that you don’t get from

Paul Dolan London School of Economics

people when you do market research, again, customer insight, asking them why they do what they do. They don’t say, “Because everyone else does.” But you do.

You want to be like other people. You want to be like people like you. We’ve done some work, some of you might have seen this, with tax collection in the U.K., of late payment of taxes. If you tell people—and in fact, this is the best intervention that we’ve done through randomized trials—94 percent of people in Exeter, if that’s where you live, pay their taxes on time, and you are in a small minority of people who do not.

So, first of all, 94 percent of people are doing something that you’re not. It’s in Exeter. Exeter is somewhere where you live and you have more association with people in Exeter than you do generally in the U.K. And then, draw special attention to the fact that you’re even more different than I just pointed out to you, from the 94 percent of people in Exeter. £700 million paid sooner in three years.

Now I would say that that is a pretty good example of a big effect for a small change. You can still do all the stuff about trying to convince people that paying their taxes on time is the right thing to do. You can do all this big solution. But you can also just tap into norms and make it much more likely that people are going to do what you want them to.

We’ve been working with energy suppliers. One of my ex-PhD students who’s now a postdoc at Chicago with John List and Steve Levitt, we’ve been doing some work with energy suppliers, and we’ve been showing sustainable reductions in energy consumption of six percent … pretty big changes … in the U.K., by feeding back information about how your energy use compares to neighbors in similar size houses.

Conclusion. Lovely, Michael. Thanks for drawing attention to the behavior-happiness loop. When I first came to these happiness data, I thought there must be something good in misery. And actually there’s not really, because when you add experiences of purpose into the mix, even if you’re angry, if that anger has a purpose and a point to it, it can actually be quite productive.

People that protest on the streets. They’re very angry people, want social change maybe. But they’re also very optimistic about being able to do something because they wouldn’t be on the streets in the first place. So they have a purpose to what they’re engaging in.

All of the things that you would want in a corporate environment, most of the things most of the time, I can’t over-claim, most of the things most of the time, come from happier workers. They’re more productive. They’re more pro-social. They’re healthier. They take less time off sick. They have less presenteeism and absenteeism.

And they’re actually nice people to hang around with, because misery loves company and so does happiness. It’s contagious. It is a contagious disease, and you want to get rid of the misery contagion and create more of the happiness contagion.

I’ve done the small changes, big effects. I’ll leave the MINDSPACE thing. I don’t want to go into that in detail, but just to say that some of the elements of that work that we did for the Cabinet Office in the U.K. are sort of captured in what I’ve just mentioned, with the “S” being salience, which is essentially attention. Your attention is drawn to things that are novel and new, and things that you can understand. That’s really important.

I just want to finish with the suck-it-and-see approach, or you might otherwise call that test-and-learn. I spent a lot of time in public policy-making and, increasingly now, in the corporate sector. And I thought public policy-making was different and they were less good at doing things than the corporate sector where market values would be quickly driven out, but that actually isn’t the case at all.

The public and private sectors suffer from very much the same problem. Whenever launching a new product or a service or an intervention, there’ll be an enormous amount of effort and work and money and resources and time going into developing, building that supertanker. They’ll do market research and customer insight, ask people

Paul Dolan London School of Economics

conscious questions about what they’re about to do or why they just did something, build that into the launch of the supertanker, set it out to sail, and watch it sink. Or actually, maybe randomly it might work.

It certainly has nothing to do with the market research. Or, it will start sailing in the wrong direction, and then we’ll chuck some more resources at it to show that it’s working in some other way, or that it really does work because we’ve spent so much time building the supertanker. Instead, why not launch ten dinghies and watch nine of them sink? And it’s okay that they’ve sunk, for two very important reasons.

One, they didn’t cost very much. And two, we can learn lessons from why they sunk. Then we can start building ballast on the one that’s sailing in the right direction until in the end we’ve got a supertanker built for an iterative test-and-learn approach.

Now, that requires a couple of important things. It requires us to learn lessons from our mistakes. And I actually think that that’s probably one of the most significant hurdles to effective decision making in any sector—is the institutional and individual memories of things that we’ve got wrong. Everyone wants to take success for the things that they did well, but you know from your own personal experiences of life, you haven’t learned very much from things you’ve done well. Insofar as you’ve learned anything about yourselves, about life, it’s when you’ve messed up. So, learn from the mistakes.

And following from that, that requires us to have an environment where we are more readily accepting of—I’m not going to use the word failure because they’re not failures if we learn lessons from them—more readily accepting of learning lessons from things that didn’t work. And that ability to take the risk and take the chance is probably the most pervasive challenge, I think, in both corporate and public sectors.

That seems a rather negative place on which to … but it’s all about happiness anyway, at the end of the day.

Thank you.

Michael: Thank you, Paul. So Paul, we’re going to open it up to questions or discussion.

Question: Can you talk a little bit about vacation and your thoughts on how to understand if it’s making you happy and what you should be perceptive about?

Paul: Vacation? Going on holiday?

Question: Yeah, going on a vacation, with and without kids. We talked about this …

Paul: You probably shouldn’t bother if you have children. There’s a very nice narrative and a very nice story about holidays being enjoyable. But actually in the moments, the experiences of them are largely miserable.

Especially when you have children in tow and you’re trying to bring everyone together and coalesce or whatever, to have a nice time, and no one really does.

But what we know about memories of the vacation, and of a whole range of other experiences, is that we don’t encode duration. We don’t remember how long something lasted. But we do remember the peak moment of pleasure or pain that we experienced during that activity.

And so when you go on vacation, you have seven . . . you know, six and a half days of hell, and you have one half day where the whole family comes together at the top of the Eiffel Tower or something and the sun’s shining and it’s beautiful. And then you come back from holiday and your memory encodes that peak moment and you say, “We had a great time.” And you want to go back again next year.

Substantively, the scientific answer of whether it’s worth it or not, is to be able to account for the six and a half days of misery and all of the memories that you draw down about the half day that was good. And sometimes it’ll be worth it and sometimes it won’t. And that’s not a particularly helpful answer, but it does depend. But I would

Paul Dolan London School of Economics

probably try to pay less attention to the story that we tell ourselves about vacations being good, and much more attention to actually what we do when we’re on holiday. And given that we neglect duration, probably have a lot more frequent, shorter breaks.

Question: Thanks. It’s all really interesting. One of the things that strikes me is that a lot of this work seems to rest on the definition of a subjective term, or subjective terms. What’s happiness? What’s purpose? And individuals might have very different definitions and we can’t sort of objectively map them to each other. How much do you think that individuality and subjective definitions affect the conclusions that you could draw from this work?

Paul: I love that question because . . . almost in the tone of that question is there’s something pejorative about the subjective nature of these things. And actually, it is subjective. The whole life is subjective. All of the things that you experience, that you enjoy, that you get pleasure and purpose and pain and futility from, are subjective experiences. All of how you feel about your financial products and services are subjective experiences. It’s about how you feel. It’s about embracing subjectivity.

When you go to the doctor and you say you’ve got a pain in your leg, one of the first things he or she will ask you is how much it hurts. That’s a subjective report of pain, which is subject to a whole range of individual differences: how much pain I’ve had in the past, how much pain I expect to have, how much pain people around me have. But it hurts as much as it hurts.

And I think it’s really important in financial services, as well, to bring some of that subjectivity into play because some decisions that people make that might appear to be stupid from a financial rationality perspective, might make a lot of sense from a subjective well-being perspective.

Take a very simple example. You’re going to go to work five days a week and you’re going to have two coffees each day. You’re going to have one on the way in and one at lunchtime from your favorite coffee shop. And you’re given the choice: do you want to pay for those coffees up front, or do you want to wait until Friday afternoon?

Well, anyone trained in Econ 101 knows to bring forward benefits and defer costs: I’m going to pay on a Friday evening, or Friday lunchtime. Well, that’s pretty horrible, because every coffee you have you know you’ve got this bill coming at the end of the week. Psychologically, let’s get the cost out of the way. Every coffee thereafter is free. So I’m happy every time I go into Starbucks or wherever I go and I’m having that coffee for nothing.

So, look through the lens of financial well-being and we might reach one conclusion. Look through the lens of psychological well-being and we might reach a different one. Also, we are contemporaneous creatures. Whilst we might tell stories about caring about our pensions and our old age and our futures, we actually don’t very much.

And so one way that you’re going to sell pension contributions is not on how it’s going to make you feel when you’re 70, but the security that you have about how it makes you feel today. So you’re getting contemporaneous feedback about how products and services are making you feel in the moment, and we need to make more salient some of those costs and benefits.

Question: Purpose. The chart you had up there showed that well into the late 20s, people were relatively low on the purpose scale. We all have experiences with talented younger people, in our groups or organizations, who may click in, they may not click in. Any particular thoughts about how you either help people inject purpose into their lives or somehow support them developing some of that purpose earlier so that they can figure out what they’re going to do and be helpful or not.

Paul: Yeah. I mean it’s a really challenging question and I don’t have a very good answer to it, frankly. One of

Paul Dolan London School of Economics

the things that—and of course these are correlational data—if you look at happiness in particular occupations, you nearly always find that hairdressers and florists are very near the top, and bankers and lawyers are very near the bottom.

And it wouldn’t be surprising if that was the case, given the environment within which those people work. If you’re a hairdresser or a florist, you’re typically engaged in social contact that people with you want to have. You’re seeing the fruits of your labor, and you’re seeing the fruits of your labor quite quickly and in ways that people generally like.

That isn’t to say that banking and being a lawyer are miserable-making jobs, just that the environments are set up in ways that make them misery making. You need to get, I think, more direct feedback on the fruits of your labor, just like doing a haircut, just like making some flowers. You need to feel like this is good and that it’s making people feel good from what I’m doing. I think that’s how you try to create purpose in the environments that might not otherwise feel like there is any.

Question: I’ve seen the U-curve that you’ve shown before, as people get more happy as they exit their 40s and 50s. And I’ve heard one of the reasons being a shrinking of the realm of possibility, and that as the realm of possibility shrinks, people say, “Okay, this is how my life is going to be, I accept it,” and kind of become more happy. And so I’m wondering what you make of that. And then the related question is, when you’re in that range, and you’re successful, often the realm of possibility expands quite a bit, and how do you deal with the unhappiness that comes from the worry associated with a wide range of possibility?

Paul: So, the simple answer, the easy answer to that question is I don’t really care about the question, because I don’t care about life satisfaction data in the same way as I care about experiential data for the reasons that I said, because they are snapshots in their constructions, in what attention is paid to when you answer the question.

So that would be where I could stop, but I won’t. And one of the reasons why . . . and as you say, really, I think one of the reasons why you get that U-shape is as you get older . . . you’re young, you have all these dreams and hopes and expectations.

And one of two things happens when you get into your 40s: either you’re in some sense lucky and you achieve all of those things and you quickly realize that they don’t make you happy, or you don’t achieve them and you’re pissed off that you haven’t.

And then you get into your 50s and you get over yourself and your future is behind you and there’s nothing much you can do about it now. You’re not going to play football for England at 55, so you should probably give that up.

And you start doing things . . . and this, I think, is the important bit, whatever the horizon’s expanding or shrinking is, you start doing things on a daily basis that actually do make you happy. You start spending more time with people that you like being with, you might start helping others a bit more, you might go outdoors, you might listen to music, all the things that you forget to do when you’re chasing the ambition and the success and the achievement. And I think that’s maybe part of the reason why we’re not as happy in our 40s as we might otherwise be. Thank you.

Question: So how have you changed what you do based on all these learnings?

Paul: I hate when people ask me that question. The question I normally get is, “How has writing this book or whatever changed me?” And my answer is . . . people expect me to be happy all the time, which is quite a challenge. But I don’t know. I mean, I think I’m like a good doctor, give advice to other people but don’t listen to it yourself. I think I’m very lucky to be temperamentally inclined to being happy.

Paul Dolan London School of Economics

I think I’ve been lucky to . . . luck is just . . . we talked a bit about luck last night. Luck is hugely significant in explaining most things. Which, by the way, the human condition doesn’t like as an explanation because we like narratives and stories and coherence. Luck doesn’t cohere. Randomness kind of is a mess. And people will say things like, “You make your own luck.” Well, that’s completely undermined the basis of . . . you can’t make randomness. So we don’t like luck as an explanation.

But I think it’s a really significant explanation for how I am and have been. I think I’ve probably fallen into some of the traps that I talk about in the book—actually, more so I think—rather than overcoming them.

Keynes, back in the ’30s, said that with growth we’d be taking more leisure time. We should all now be working three-day weeks. But actually, the complete opposite has happened. As we got richer, we’ve worked more.

We all know about inequalities in income having widened over the last decade or two or three. But inequalities in leisure time have also widened, in the opposite direction, such that richer people are now taking fewer holidays.

And what’s happened is, it’s about attention being paid to the opportunity cost of not working. And so as I get paid more money, I’m actually working more than I did when I was paid less because the opportunity costs of not working are now greater.

And I think I’ve fallen into that trap. And that reminds me not to. So, the one thing that I’m going to do is to turn down many more offers like this. But not this one. Because this is brilliant. And spend more time listening to music, with my family, going outdoors, helping other people, that I’m not doing when I’m coming to New York, or coming here.

Question: That actually gets to the question I wanted to ask, which is . . . it’s sort of two-part. And that is that one of the real challenges, it seems to me, when we think about the way organizations work, particularly in the United States, and particularly corporations, is that the environments that most corporations set up are not actually, it seems to me, well designed to maximize happiness or attention or focus or any of those things.

I mean, the time thing you’re talking about isn’t just a question of the choices people make. Now, you can make those choices because a big part of your income is freelance, and I can make those choices too. But most people, whether or not they’re well paid or not, don’t have much choice. They’re expected—if you’re a banker, you’re expected to work, whatever it is, 14, 15 . . . you know, however many hours it is. I guess I’m wondering, do you have any thoughts about how to change that?

It seems to me, in particular, this question of environment and time and priming, all of these things, have actually been going in the opposite direction. One of the reasons why Pixar—Michael talked about Ed Catmull—is so amazing, is precisely because it’s so unusual. It’s such an unusual environment for actually making people creative.

So I’m just wondering, all these things make a lot of sense, but if you look at the way most corporations are run, they’re incredibly short-term focused, they’re incredibly multitask driven, they’re incredibly bad for actually getting people to focus. I’m just wondering if you have any thoughts on how to change that?

Paul: Well, again, that’s a really big question. I think you’re generally right in terms of the way that things have gone. And I think maybe my answer is to come back to this big effects from small changes point. There’s a saying about don’t sweat the small stuff. I actually think we should sweat the small stuff.

No matter how time-poor you are, I reckon you could probably find five minutes a day to listen to your favorite tune. I reckon you could probably find another five minutes to phone your best mate. I reckon you might find another five minutes to go outdoors. You could probably find another five minutes to help an old lady across the road. You might find another five minutes to . . . whatever, 20 minutes. I’ve got four things there.

Paul Dolan London School of Economics

If you did one of those things every day, you’d be happier. If the corporations and organizations could embed in the design of their architecture ways in which they could make it easier for people to do these things, then we’d all be happier.

And I think maybe that’s what we forget. I think we forget how easy . . . maybe how simple it could be. Really simple behavioral science insight, anyone else in this room will tell you this, is the first basic insight is, if you want to do something, make it easy. I just think we don’t make it easy for ourselves or the workers that we work with, to be able to do the very small things.

Question: Do you have any concerns along the lines of, you know, William Davies’s book, The Happiness Industry, that corporations are just going to do those small things to try to sort of make people happy on the surface without actually making any substantive changes that could really make a difference in people’s experience?

Paul: Again, that’s a big question. And I’ve spent all of my academic life, essentially, focusing on structure. And this book is about agency. And it’s going to be a bit of both. I’m not going to sell many books making it about structure. I’ll sell more books making it about agency and about what you could do for yourself. But I completely and readily accept that there’s a whole systemic and structural and cultural shift that we might need to engage in, in order to get the agency bit to work better.

Question: So, the way you’ve organized the data is by age. I wonder if you organized it by men, women, extrovert, introvert? Do you have any view on that, how it might come out differently, on any of those pleasure/purpose . . . ?

Paul: The gender effects are typically very small. Women generally report slightly higher levels of life satisfaction than men do. In experience data, they tend to report—in data that we’ve got from the U.K.—both being slightly happier, but also slightly more anxious. But the effects of gender are very small.

Personality . . . that’s one of the things that there are very few data on in many of the big datasets. Where there are data, we find extroverts are happier than introverts. We find that’s the single biggest personality characteristic associated with happiness. Conscientiousness is the next. Neuroticism, of course you’d expect neurotic people to be less happy.

That reminds me, when I first went into Number 10, and was in what the press called the “Nudge Unit,” David Cameron’s senior policy advisors wanted us to look at relationships as our first behavioral insight. Could we make it better and easier for people to choose partners more effectively, because of all the problems?

It’s a serious question about social breakdown and marital breakdown and the impact on children and stuff. I came to the conclusion that it was largely random and chaotic, and that you intervened at your peril.

Sometimes you can intervene and make things worse rather than better. The only conclusion from the personality researchers was, don’t marry someone neurotic, which is pretty crap if you’re neurotic. And it would probably just make you even more neurotic to know that, but there seems to be no other matching that works.

But yeah, extroverts. A really interesting point that also goes sparking through my mind, is that I’m talking about data based exclusively on self-reports, that is, asking people questions. And it’s the subjective bit. And at the moment, the best way of finding out how happy you are is to ask you how happy you are or how worried or stressed or angry you are. But as we have big data, there are other ways and opportunities to do that.

We’re sparking off some research on these lines—we’re going to look at facial expressions, conversations, sentiment analysis of texts. Because a recent study . . . it’s always been shown in the U.S. that conservatives are happier than liberals. If you believe that the world is a just, fair place and all this inequality is acceptable, then

Paul Dolan London School of Economics

you’re probably happier than if you’re going around angst-ridden by all the inequality and injustices and about how terrible the world is. And so that’s pretty well established.

But interestingly, when these researchers looked at facial expressions and conversations, liberals were happier.

So there’s a very interesting question about whether some of what we’re picking up is due to a self-report effect, and only a self-report effect.

Question: You draw a couple dichotomies. One, for happiness, between purpose and pleasure, because they can be very different and people place different values maybe on each. There’s also this concept which you alluded to on the difference between the experiential self and the remembering self or the reflective self. And I guess your view here is on the primacy of the experiential self because a reflective self can tell stories and you’re really only experiencing it when you’re reflecting.

The question is, is there a place for the reflective self? If you can just automatically get a bio-feed of what my pleasure experience is, so you get like the good data, you know, as you were talking about, or the facial expressions, so you could say, yeah, these people in these activities are happier? Is that really the answer or, again, is there a place for being seen, of course with some reflection, because while narratives can be skewed there’s also a replacement, in most places, for that?

Paul: Because the pendulum has swung so far in favor of evaluation to the way that we think about happiness and well-being, I’m trying to make almost a more extreme case than probably is justified for experiences in order to get that pendulum to swing back to somewhere there might be more sense. The more nuanced response, therefore, to your question, would be, of course the stories and evaluations and narratives matter.

But what I’d like to distinguish between, I think, is the stories and narratives and evaluations that we tell ourselves in order to make ourselves happier, and the ones that are told by society as constructions about things that we ought to be doing.

So in the former case, one of the reasons that people of faith are happier is probably because they have a very good story about why things are bad. Often people will find God when an adverse event happens. We don’t like the fact that it was random or bad luck. We need a story and an explanation for why this adverse event happened. A higher order being is a really good explanation. And so, that ameliorates some of the pain.

But that’s distinct from, I think, the story that I told about my friend that worked at MediaLand. Because that’s a story about what a job and an occupation and status and success should look like. And it’s those kind of narratives and stories that I think make us less happy. They’re hurdles and obstacles to us actually experiencing life better.

Question: I think there’s a lot of data out there that says that most people have some baseline of happiness or unhappiness that they tend to return to. You have soldiers who go and they lose a limb and they come back and somehow snap back to whatever their normal level of happiness is.

Paul: That’s not true by the way.

Question: Is that not true?

Paul: No. Rich Lucas is a researcher at Michigan State who’s looked a lot at this, in the life satisfaction data. And again, that’s where most of the data are. He shows greater variability than would be expected. There are things that happen to us that we don’t revert to baseline to. Unemployment, for example, stays scarring even when you have a job again.

What’s interesting is that we need to be looking less at the differences in the numbers, because a 7.1 to 7.3 is only a 0.2 difference, but in terms of where you move in the distribution of people, you might be going from the

Paul Dolan London School of Economics

30th percentile to the 70th percentile. And so the absolute numbers don’t reflect very much in the actual changes that take place in people.

And if you have mental health problems, they’re permanently scarring. You don’t ever go back to baseline from those. Some things you do. And it’s about attention. It’s all about attention. Things that you withdraw attention from, you’ll revert back to baseline. Things that are consciously attention-seeking you won’t.

Question: Let me frame it slightly differently. You talked about people in London after September 11, kind of for a period being bummed out by that but bouncing back pretty quickly.

Paul: By and large we bounce back from things quite quickly.

Question: If you simplified the world and said there are some people who are lucky to have a happy temperament and some people who are not so lucky and don’t have such a happy temperament, is one group or the other more likely to benefit more from making these contextual changes or taking these steps?

Paul: That’s a very good question and . . . I mean for policy purposes, of course we’d care much more about the group that start off less happy, for reasons of inequality. For reasons of efficiency, I don’t know. That’s a very interesting question. I say this, almost glibly, that if you found five minutes each day to do those three or four things you’d all be made happier—I don’t know whether that would be to the same degree, whether people that are less happy would benefit more from it. I think that’s still a very open issue.

Michael: We have time for one more.

Paul: This is the last one so it’s always a good one.

Question: It’s the best yet. Trial and error could be a very expensive way to figure out where you should be on the balance between purpose and pleasure. So if I have kids in their 20s and they’re making career choices or family choices and they decide, “I want purpose. I’ll have six kids.” That’s kind of irreversible as a trial. How do you help people, how do you help yourself figure out where your balance, your unique balance should be? And maybe that changes over your lifespan as well.

Paul: It probably does change over your life course. Certainly, I wouldn’t have been a very good father much younger. And I guess my kind of desire for purposeful experiences has probably increased and that for pleasure has diminished. Dan Gilbert, in Stumbling on Happiness, makes a big case of drawing on the experiences of other people. If you want to know what a cruise is going to be like ask someone that’s just got off the boat.

That’s a challenge because you want to make sure that they’re people that aren’t living in the stories that we might tell. So you don’t want to go to your parents and say, “Should I take this job at MediaLand?” Because they think that you should have that job at MediaLand because they have the story that says that that’s what would make you happy.

So you need to, I think, find people that you can get suggestions and insights from that are more likely to be living in experiences than you might otherwise be. That’s probably as helpful as I can be. The other thing to remember is, because of adaptation processes and shifts, to think about what being in the condition would be like, not what becoming will be like.

So, for example, you think, would you be happier moving to San Francisco? What you’re asking is, “Would I be happier in the first week or two when I’m in San Francisco?” And you would be. But what you really want to know is, “Would I be happier in San Francisco after ten years?” And so you want to pay attention, if you can, to what happens once you’ve adjusted and shifted attention into the more steady state, rather than into the change.

And I’ll just wrap up with something that we talked a bit about last night at dinner. And that is that often too

Paul Dolan London School of Economics

much of the time we’re living in a joint evaluation between two alternatives, like this job versus that job, this partner versus that partner, San Francisco versus New York.

And what we do is we draw out what makes those two alternatives different from one another. But actually, once we’ve made the choice, we only live in one experience, and very rarely will we then be paying attention to the contrast that we made.

So actually, most things in life—and this is a nice place to finish—don’t actually matter that much, because once we experience what we’ve chosen, we quickly get on with it and we don’t really worry too much about what otherwise might have been.

Michael: With that, thank you very much Paul.

Iris Bohnet Harvard Kennedy School

Iris Bohnet is a Professor of Public Policy at the Harvard Kennedy School (HKS) and Director of its Women and Public Policy Program. She is also Co-Chair of the Behavioral Insights Group at the Center for Public Leadership at HKS, an Associate Director of the Harvard Decision Science Laboratory, and the Faculty Chair of the executive program “Global Leadership and Public Policy for the 21st Century” for the World Economic Forum’s Young Global Leaders. From 2011 to 2014 she served as the Kennedy School’s Academic Dean. Currently, she serves on the boards of directors of Credit Suisse and University of Lucerne, as well as the Advisory Board of the Vienna University of Economics and Business, and numerous academic journals. She is a member of the Global Agenda Council on Behavior of the World Economic Forum.

Bohnet teaches decision making, negotiation, and gender in public policy and leadership in both degree and executive programs, and has been engaged in the teaching, training, and consulting of private and public sector leaders in the US, Europe, India, and the Middle East.

A behavioral economist who combines insights from economics and psychology, her research focuses on questions of trust and decision making, often with a gender or cross-cultural perspective. Her most recent research examines “gender equality nudges,” interventions that decrease the gender gaps in organizations, politics, and society.

Bohnet’s academic work has been published in the best journals of her profession, including the American Economic Review, the American Political Science Review, and The Quarterly Journal of Economics. Leading media outlets around the world have profiled her work.

A Swiss citizen, Bohnet received her PhD in Economics from the University of Zurich in 1997 and spent a year as a Research Fellow at the Haas School of Business at the University of California, Berkeley from 1997 to 1998. She joined the Harvard Kennedy School as an Assistant Professor in 1998 and was made Full Professor in 2006.

Iris Bohnet Harvard Kennedy School

Michael Mauboussin: Welcome back. It’s my pleasure to introduce our next speaker, Iris Bohnet. Iris is a professor of public policy at the Harvard Kennedy School and Director of its Women and Public Policy Program.

From 2011 to 2014 she served as the Kennedy School’s academic dean.

I have long been a fan of Iris, having had the opportunity to see her in action at Richard Zeckhauser’s behavioral finance conference at the Kennedy School. She is a behavioral economist who thinks deeply, and teaches effectively, in realms including decision making, negotiation, gender issues, and leadership.

Today she will cover some of the implicit biases that we have as individuals and as organizations. But her particular emphasis will be on methods to implement changes in mindsets in order to improve outcomes. This is a key point: it’s one thing to know about biases, but it’s another thing altogether to effectively cope with them.

I also love my conversations with Iris. She has vast knowledge and a wide range of interests but is also pragmatic and grounded. Let me also plug book her forthcoming book on how to design gender equality—I’m sure we’ll hear a bit about that today.

I will also mention that Iris is on the board of Credit Suisse, but we have asked her to join us today in her academic capacity and not in her governance capacity.

Please join me in welcoming Professor Iris Bohnet.

Iris Bohnet: Thank you, Michael. Good morning. It’s a great pleasure to be with you here this morning and talk a bit about attention and that seeing really is believing, that what we see affects what we believe is possible, affects our attitudes, affects our behaviors. And Michael and I also thought that it was particularly appropriate to talk about gender equality in this room, which isn’t completely gender equal, but nevertheless you all care about gender equality, at least for three reasons.

One is that you probably believe in the business case, and you probably believe that it’s better to draw on 100 percent of the talent pool than just 50 percent of the talent pool. You also probably have heard about some of the research suggesting that diversity in teams is a good thing, and you’ll hear a bit more about that from Chris this afternoon. So that’s the business case.

Secondly, many of you, I hope, also believe in gender equality because it is a human right. Because it’s just the right thing to do.

And thirdly, many of you are fathers of daughters. In fact, could I see the hands of fathers of daughters? Yes, that is a majority. So it turns out that fathers of daughters are among the biggest proponents of gender equality. There’s been a lot of very interesting research looking at judges, male judges, as well as male politicians and male CEOs, and all of those who have daughters care much more about gender equality. So I’m going to speak to all of you, but particularly to the fathers of daughters.

This is what I want to talk about. I want to spend a little bit of time talking about gender bias. I’m going to very much build on where Paul left us off . . . and on the unconscious mind, talking about how hard it is to change mindsets, and that often it is easier to change environments. I’ll talk about some of the things that organizations can do to equalize the playing field. For example, hire and promote based on performance and merit rather than demographic characteristics.

Here’s an image of a woman whom you may or may not want to hire. The question is, what do you see when you see someone like her? This happens to be Ellen Johnson Sirleaf, who later on became the President of Liberia, a graduate of Harvard, and a recipient of the Nobel Peace Prize. When we saw her earlier it probably was hard to predict what would turn out to be the course of her life. Now she does look a bit like a leader, even in her younger years, but that’s kind of the point, that it’s hard to predict just based on someone’s appearance.

Iris Bohnet Harvard Kennedy School

Here’s another person. Her name is Heidi Roizen. Now, Heidi Roizen is a real person. Some of you might know

her. She’s a venture capitalist in Silicon Valley, a successful entrepreneur, in fact, so successful that a colleague of mine, Kathleen McGinn, at the Harvard Business School, wrote a case study about Heidi. The case study is now being taught at many business schools, including at Harvard, including at the Kennedy School, about how Heidi created these different businesses in Silicon Valley.

And here’s a bit of the case study. That’s how it starts: “Heidi Roizen, a venture capitalist at SOFTBANK Venture Capital and a former entrepreneur, maintains an extensive personal and professional network. She leverages this network to benefit both herself and others. The case considers the steps she's taken to build and cultivate a network that is both broad and deep.”

Now, we have been teaching this case with both Heidi as being the protagonist, as well as by replacing Heidi’s name with Howard. Now, Heidi is the real person, but to understand whether there’s a gender bias in the perception of our students, we had half of the students look at the case with Heidi as a protagonist and the other half with Howard as a protagonist. Everything else is completely identical. You can already see where I’m going with this.

Of course, it turns out that people don’t quite like Heidi. Heidi is too power hungry, self-promoting, and disingenuous compared to Howard. What’s worse, and particularly bad and depressing for me as an educator, is that people, and that includes women, female students, learn less from Heidi.

So when you ask students afterwards, how good was this case? How much did you learn? They learn more from Howard than from Heidi. So, seeing is believing. Heidi doesn’t fit our mindsets, doesn’t fit the stereotypical images that we have about a typical venture capitalist in Silicon Valley.

Now, a lot of studies have been done to understand how biased people are. Some of them you’re probably familiar with. This is another famous one that some colleagues of ours have done. They responded to job advertisements and sent applications with different names on them, but the application as such was completely identical, independent of whether the name was Lakisha or Emily.

So these were equally qualified people either called Jamal or Greg or Emily or Lakisha, and the question was, “Who would be invited to an interview?” And of course, what the researchers found was that if your name was Lakisha or Jamal, the likelihood that you will be called back is about 50 percent lower than when your name is Emily or Greg. Greg gets most calls, Emily second most, and then there’s not a big difference between Jamal and Lakisha.

In fact, when you go and talk to people and ask them—these are all experienced people in the banking sector, in law firms, consultancies—what they look for when they interview someone, these are the kinds of things they would say.

So here’s Denise, a white attorney who says, “I really do think it’s about finding something in common with your interviewee.”

Carlos, a Hispanic attorney says, “You use yourself to measure fit, because that’s all you have to go on.”

Now, my favorite statement is this one here that will make those of us in banking particularly proud. Ariel says about the interviewee, “She and I both ran the New York marathon. We talked about that and hit it off. We started talking about how we both love stalking celebrities in New York. We had this instant connection. I loved her.”

So these might be somewhat extreme examples. Hopefully, none of you interviews this way. But it is certainly very tempting to use ourselves as implicit standards of what we’re looking for in a job applicant.

Iris Bohnet Harvard Kennedy School

Let me show you this picture, and some of you might have seen this before. But if you haven’t, then try to compare square A with square B. I presume that most of you see square A as being darker than square B. It turns out that this is an illusion, and when I cover the surroundings, I presume most of you now see them as having the same color. Some of you look at me . . . not trusting . . . much of my work has been on trust, so I believe in proof. So I’m going back. This is what the pattern looks like. And here’s how this is working.

We all think in terms of patterns, in terms of categories. And your mind immediately makes sense of the pattern that you see here. This is a checkerboard, and you also see there’s a bit of a shadow here. And your mind accounts for the shadow but makes sense of the pattern that you see.

So, the question that you should ask yourself is, “What are the patterns that you see in your jobs, in your environments, in your families, in the world?” Do you see these two squares as being surrounded by a checkerboard, or do you see them in isolation? And if you see them in isolation, your interpretation of them is likely quite different.

Let me do another little game with you. And this, of course, is going to convince you that being here is really worth your time. Mainly, what I’m going to ask you to do is, to read the colors that you see, as a group, to me, as quickly as you possibly can. So I’m going to time you and you will read green, red, blue, orange, pink, as fast as you possibly can. Okay. Three, two, one, go . . .

That was very fast, although not completely coordinated. But very fast, as one would expect from a group of leaders. So that’s about ten seconds. That’s among the fastest times I’ve ever seen.

So let me make it slightly more challenging for you. Now, you still read the colors that you see to me. So, the first one would be orange, the second one would be green, the third one would be pink. And again, I’m going to time how long it takes you to do that. Three, two one, go . . .

I think we are giving up, although you are trying to save the day. I thank you very much. I notice you were very quick in the first round and you’re still with me on this second round. So this is pretty crazy.

You all are obviously very smart and accomplished people. And still, you are having a harder time doing this than my son did when he was four years old. My son had no problem recognizing the colors, of course, but he couldn’t read when he was four, so he was very quick. So, this is strange.

There’s a little bit of complexity here that I’m adding to the decision process. I am matching two things that don’t belong to each other. It says “green,” it should be green, or at least some neutral color such as black. But if “green” comes at you in a pink color, it is hard for you to say “pink” rather than read “green.”

This really is the basis of a test that was developed at Harvard, in collaboration with some other places, on implicit biases. That is how the researchers measure how biased we all are, by trying to see how long it takes us to make associations, for example, between women and office, and men and children. You can apply the bias to everything, and it has been applied to lots of different dimensions.

Of course, it plays a role in race, it plays a role in nationality. You can do religions, you can do how people look. You can do size, you can do weight, you can do height, you can do eye color, hair color, everything. So all of these things really matter in how quickly we are to make associations between different items.

And if ever you log onto this web page, thousands, millions of people have taken the test so far, you will find out that, I would predict, 98 percent of you are sexist, and racist, and nationalist, and many of the things that you don’t want to be.

How you interpret whether he is the leader, she is the leader, who is attractive, who is competent, who is warm, depends on what you typically see. And what we typically see depends, of course, on the roles that you’ll play.

Iris Bohnet Harvard Kennedy School

When you are first confronted with a male nurse, most of us, including I, would feel slightly uncomfortable, unclear whether this person is really qualified to do the job. In fact, the story I’m telling in my book is a true story that when I took our older son to the Harvard day care center for the first time, I had to hand him over to a male caregiver, and I thought this was the most horrible thing that ever happened to me.

I got over it relatively quickly, but in the moment I couldn’t help but expect a caregiver to be female. And he just didn’t look the part and didn’t fit my expectations. Nor are we used to women board members, women professors, women CEOs for that matter.

So, seeing is believing. And this is true despite evidence suggesting that gender diversity on corporate boards or gender diversity in senior management—that’s a newer study that Credit Suisse did—is correlated with company performance. So, even though I started by reminding you of the business case, the business case is not going to do it for us. It is a helpful motivator for us to care, but it’s not going to translate into action. In order for action to happen, we have to do much more, and that’s really where I want to go.

Here are some of the things that we have been trying so far. Let me tell you a bit about the evidence about whether or not they’re working. So, for the last 20 years the most likely program that organizations would have introduced is a diversity training program. Frank Dobbin, a colleague of mine at Harvard, a sociologist, has looked at a large sample of companies, about 3,000 companies, and he’s done a very simple correlational exercise.

Namely, he’s looked at different measures of diversity as a dependent variable. One of them, just a very simple one, is how diverse your workforce is. And then he looked at what these companies are doing. Do they have a diversity training program? Do they have a chief diversity officer? Do they do mentoring? Do they do other things?

The question was, “Are there any correlations between what we do and the outcomes that we’d like to achieve?”

And it turns out that on diversity training programs, the verdict is quite devastating. The correlation is basically zero. So even though it has been one of the favorite tools to be used to increase diversity in companies, the evidence suggests that there’s probably not much happening. This is just correlational, but if a correlation is basically zero, we can imagine that causation doesn’t do much better either.

So diversity programs don’t work, and that’s pretty much very directly related to what Paul left us off with: it is very hard to change mindsets. By definition, these biases are unconscious, and even though you might agree today, or during the program, that you don’t want to discriminate and you will change how you perceive people, how you evaluate people, how you promote people, it’s very, very hard to follow through.

It’s a bit like dieting. You can now, you know, think that you should do more exercise, eat more healthfully, but then to follow through is actually very hard. So diversity training programs haven’t helped much.

Other things that we’ve tried are fixing the people who face challenges. These include leadership training programs that many companies have introduced for women, and also include—this is a newer tendency— behavioral training programs for male students.

Many of you might have seen the new OECD report that came out earlier this year showing that the gender gap in reading and writing is now about a year for 15-year-old boys and girls. So boys lag girls by about a year in reading and writing when they’re 15 years old, and they never quite catch up.

This is in stark contrast to girls catching up in math. So, the gender gap in math, where boys typically have outperformed girls, has been closing relatively quickly in recent years, and has reversed in some countries—in Scandinavian countries, for example, it has reversed. So, schooling and boys is a huge issue. And increasingly, schools offer these behavioral training programs for boys, to fix the boys, to make the boys able to succeed in

Iris Bohnet Harvard Kennedy School

the contexts that they’re in. And the same is true for women. So, we’re trying to help women succeed in the systems that they’re in.

The evidence in those programs is a bit mixed, but I can’t draw on huge amounts of data here because, sadly enough, also understandably enough, people are rarely randomly assigned to participate in these training programs.

So, many of your organizations, of course, I’m sure, run these training programs for traditionally discriminated-against groups, and you pre-select the people who then participate in the leadership training program. If you do a great job selecting the right kinds of people, and we don’t screw it up in the leadership training programs, they will come out beautifully. So, that’s why it’s not a really good experiment.

We have a few experiments, for example, in financial literacy training, some leadership training in developing countries where people have been randomly assigned. And it looks like the evidence goes a bit in all different places. Sometimes it helps, sometimes it doesn’t help. It helps for some people but not for others. Typically, the evidence is that for those groups for whom the leadership skill that we’re trying to teach is, in fact, the binding constraint, it can help. But it only takes us so far.

Other initiatives that we have been employing in the past years are networks. Turns out networks are generally a good thing. Peers do matter. It is true that there are fewer peers for women to learn from—in our sector, for example.

And it also is true, as Boris Groysberg of the Harvard Business School has shown, that women tend to have vaster networks outside of the organization, because there are fewer peers of the same sex to learn from within the organization, which of course makes their skills also more transportable.

The most successful program to date probably has been mentoring and sponsorship where some supervisors have taken a keen interest in, in fact, mentoring particular individuals and sponsoring these particular individuals.

But what are the issues with programs that I have just been showing you? There are many, but let me just point my finger at two. One is the problem of the hammer. And that is that if the only tool you have is a hammer, everything looks like a nail. So we have been using these programs a bit on . . . not really understanding what the underlying problem really is.

Here’s a study that we did in academia just to understand, to uncover what’s really going on. And that might also be interesting and relevant for some of the professions that you’re in.

So, many of us continuously claim that there’s a pipeline issue, that we don’t, for example, have enough qualified women to fill the jobs. And that is certainly true for some professions. It is true for engineering. Fewer than 10 percent of the students are female. It’s been very hard to change those numbers. Been pretty persistent over the last 20 years. There’s a real pipeline issue in engineering.

Turns out, in science the numbers have started to change. There are increasingly more women going into science. We’re now at about 35 percent women in science. But the issue there seems to be something that a relatively famous MIT study has coined “performance support bias.” What that means is that in the ’90s, MIT realized that they weren’t treating their female professors or their female scientists the same way they treat their male scientists.

And they have, as would be appropriate for MIT, collected lots of data on whether this was true. They found a bit of a compensation gap, but that actually wasn’t the big driver. They found huge differences between men and women in terms of support. And that meant: size of the laboratory, resources, research assistant. So the budgets that they were endowed with were much smaller for female scientists than male scientists, and they’ve

Iris Bohnet Harvard Kennedy School

rectified that since. So, that’s support bias.

Thirdly, my own discipline, economics, we found something quite surprising. And that is the blue bar here on the right. And what it shows you is that men are about 21 percentage points more likely to be promoted to tenure, to full professorships, than women. Now, this is not controlling for anything because it’s just raw data. It’s not controlling for output or expertise or performance, but it still made us wonder whether something was going on.

You also see some other differences. You see some pro-female biases, for example, in engineering or life sciences. But these are small numbers, these are small deviations from a gender neutral performance process.

This allowed us to introduce what I think is one of the only experiments on mentoring, leadership, and network programs at the American Economic Association meetings.

So we now do training programs for female assistant professors every year, and the beauty, from a research point of view, of course, is that we literally do pick the applicants randomly out of a hat. So these aren’t pre-selected people, but people apply, about 120 apply, we pick 50 every year, and then we give them additional training. And we also follow them through their careers. And they get to know their peers, they’re in a network, there are lots of different things that I just talked about in the previous slide.

We introduced it in 2006, and we can show that over the years it does matter whether someone has received additional training or not. The beauty in academia is that output is measurable relatively objectively. You can look at people’s publication record, their citations, at universities they get tenure, et cetera. So it can help. It can help to try to enable the traditionally discriminated-against groups to navigate the playing field more successfully. But still, it only takes us so far.

The second concern I have in addition to the hammer finding a lot of nails is social backlash. And that’s actually quite serious and I think a topic that we might not have talked about enough. So certainly Sheryl Sandberg’s book, Lean In, has had an important impact, has made a very important point.

And it also talks about the research on negotiation, suggesting that for women, in fact, negotiating like a man, or for Heidi, leading like Howard, is socially risky. Heidi experiences what we call social backlash. She cannot afford to behave the same way that a man can. Neither can women negotiate as assertively as men can, because they violate stereotypes about what the typical woman should look like.

So, just offering leadership training to women and saying, “Look, here’s how we’ve been doing this so far, and here’s how career trajectories are in our firm, and here are the kinds of things you need to do,” likely won’t work because of the norms that we have, these implicit associations that we have in our minds.

So what do we do? Is there some magic algorithm, some magic method that we might be able to employ to change, as Paul discussed, the context. Basically, make it easier for our biased minds to get things right. So, rather than fixing how we think, making it easier for our minds to do the right thing. And that’s where, as Michael talked about, my book is going to go. My book talks all about what organizations can do to equalize the playing field, for example, by promoting or hiring based on performance.

Let me give you kind of three ideas. The first one is a relatively simple one: why was the Obama campaign so successful in getting the vote out? Lots of motivated people, lots of passion. But one of the key reasons was that they used data. They used data, they ran experiments, they really tried to understand what it takes to get someone to the voting booth.

These include randomized, controlled trials where, for example, researchers have studied whether people, in fact, do what Paul described people doing in tax compliance. So does it matter how many other people are voting? If I call you and remind you that an election is coming up, does it matter whether I remind you how many other people are voting? It turns out, which of course is painful for an economist, people are super irrational. They are

Iris Bohnet Harvard Kennedy School

more likely to vote when everyone else is voting. When, in fact, in economics you would assume you’re more likely to vote when you are a decisive voter. But no, we are herding. We go with the herd and we’re more likely to vote when most other people are. So you can remind people of what the thing to do is.

Another study that the Obama campaign ran went as follows. They did the call and, by the way, they didn’t just do the call, they always had a control group which got the control condition, and then they had a treatment group which got the treatment, like in the natural sciences.

The control group, you would just get the regular call: “Do you plan to vote?” “It’s important to vote.” “Please vote.” The treatment group got a slight addition: “It’s important to vote, “Do you plan to vote?” And, in addition, they asked people whether they have made a plan.

Now, it seems pretty silly to ask people to make a plan about something as relatively simple and straightforward as voting, but it turns out that when you get people to think it through, including: “When are you going to vote? Who is going to take the kids to school? Are you going to go alone? Do you have transportation?” People are more likely to vote. It increases the likelihood that single households vote by ten percentage points. And if you cohabit with someone else, it increases the likelihood that you will vote by four percentage points. So it’s pretty crazy.

This follow through between intention and action is really important. Think of some of the discussions that I’m sure you and all the firms are having these days around culture. Most of us want to be honest people, ethical people, good people. But the follow through is a difficult part. It can’t just be that we have these statements and we read the statements and we train people on understanding what the moral code is in our companies. We also have to think about how we could help people to get from A to B. And plan-making is one of those interventions which turns out to be pretty successful.

Now, the Obama campaign benefited from data and, thankfully enough, data is starting to play a more and more important role in the private sector as well. And, like Paul, I have to say I’ve been similarly surprised working with the private sector, how little data, in many ways, we use in some parts of our businesses.

So we’re very good using data in our economics departments. We’re very good using data in our marketing departments. But, for example, in human resource management we haven’t been very data driven, and that’s kind of part of the mission of the book also, to kind of transform how we think about people, and make thinking about the people equally based on real evidence, the same way we think about financial decisions, the same way we think about marketing strategies.

So, I have a number of different initiatives here in my slide. One is the MIT initiative that I’ve just described. It really did help MIT to understand the root causes of the issues—not just use a hammer, not just do some diversity training program, but really understand where the gender gap comes from. There’s also an interesting study by Janice Madden that some of you might have heard about, on stockbrokers, finding the very same support bias that was found at MIT, where female stockbrokers were assigned worse performing portfolios than their male counterparts.

There’s also technology now that helps companies understand what’s going on. The two examples I have here, one is Logib, from my country, Switzerland, where the government, in fact, created a website where companies can just download a tool which helps them plug in their own data—it’s basically a bit of a more advanced Excel spreadsheet—and analyze where issues might be.

And there’s also private companies, such as EDGE, for example, that some of you might have been using, which help companies self-evaluate how well they’re doing in terms of treating their employees equally. So if you haven’t used any of those, I suggest to you that you might want to look into those—understanding where the

Iris Bohnet Harvard Kennedy School

issues are, starting with the problem, is probably a more effective and a more efficient strategy than just using a hammer relatively broadly.

Now, let me talk about another issue that we all face in our organizations that’s now more forward looking.

Forecasting—and, of course, many of you are in the business of forecasting—is tough business. In fact, Phil Tetlock has studied this question of forecasting for a long time—a new book of his is coming out this fall— looking at how well we do when we make these kinds of predictions.

And he’s studied, literally, thousands of expert decisions trying to understand whether experts are relatively good in forecasting the economic future, political events, lots of different financial markets, lots of different types of dependent variables. And his verdict is pretty damning.

Namely, his verdict at the end of the book is that we experts aren’t much better—he talks about dart-throwing chimps—a little better than chance, but not much better than chance.

So, prediction is difficult business. And of course, when we evaluate job candidates, in effect, what you’re trying to do is to predict future performance. Is this the kind of person who will flourish at our organization and do well in our organization? And there are lots of effects in psychology and behavioral science suggesting that we are very bad at predicting how well someone is going to do based on our instincts.

In fact, many of us, including I, would say that interviews are one of the most overrated instruments that we use in evaluating people. We tend to trust our gut instinct, and we will tend to behave like Denise or Carlos and look for people who look the part, either using ourselves as a standard of reference, or using the majority of people in a particular profession as a reference.

Let me just talk about one of these: the beauty premium. The beauty premium is actually quite an interesting topic. A colleague of ours, an economist, Daniel Hamermesh, has been studying beauty for about 20 years now and finds some really interesting effects. So, beauty does matter, and there is a beauty premium.

Now, in America, it turns out, the beauty premium is more pronounced for men than for women. So, men who look better than the average man, on average, make five percent more, and men who look worse than an average-looking man make about 13 percent less, on average. This is controlling for everything that you can control for.

It’s also true for women that there’s a beauty premium, but it’s less pronounced than for men. And that’s generally the pattern in the U.K., in Western countries. It’s not necessarily generalizable. So, for example, in China, it’s quite different. In China, it turns out that the beauty premium is much bigger for women.

A recent study that has just come out, I think a month or two ago in the American Economic Review, looked at job advertisements in China. And, as some of you might know, in China you can still advertise in sex-segregated columns: so these are jobs for men, and these are jobs for women.

And the simple exercise that they did was to try to understand which descriptions are most likely to be associated with male jobs and which descriptions are most likely to be associated with female jobs.

And it turns out that the adjective that was most likely to be used for male jobs was “age” or “experience.” And in female jobs, the adjective that was likely to be used was “attractive.” So, there is a premium, but the question that you might ask yourself is, “Is it predictive of anything?” Maybe that’s actually a good indicator.

Maybe beautiful people are better performers. You’re laughing. It turns out not to be true. And it’s actually quite interesting. It’s even a bit more complex than you think. Not only are they not better performers, but they run into the following problem: they are expected to be better performers. They are taken to a higher bar, and then they don’t live up to my high expectation of the beautiful people, and then I punish them even worse than I would

Iris Bohnet Harvard Kennedy School

have if they hadn’t been beautiful. So, beauty is not a good predictor of performance. So, what is?

Generally, looking at people there are very few examples. I mean there are some smart people who are able to read faces, but that’s a very difficult business. Generally, looking at people is not a good predictor of performance. So here’s one study that I think resonated with many of us in the field, which really brought home the point.

The major orchestras in this country, and increasingly around the world, have introduced something quite surprising. They now have their musicians audition behind a curtain. In fact, this was introduced in 1970 when some of the major orchestras realized that they only had five percent female musicians, when in fact many, many women studied music, and it seemed a bit surprising to not have more female players.

Having women and men audition behind a curtain increased the likelihood that female musicians would be hired by 30 percentage points. That’s blind evaluation. And that’s really what would be ideal for all of us.

It would be great if we could just base our judgments on someone’s performance and not what they look like.

But, of course, in most of your jobs and in my job as academic dean—my job was to help select professors— you don’t have the luxury of blindness.

There are some interesting tools which will come on the market this fall or next spring, which will help us keep applications blind for a very long time, maybe until the very last stage, and some of you might want to use some of these tools. So, blindness would be a good thing, but hard to implement until the end.

Max and I, with a doctoral student of ours, Alexandra van Geen were trying to find an alternative to curtains.

What we are trying to do, basically, is to find an intervention that is practicable, that organizations like yours could be using, that would debunk this little stereotype, this little reference here, that says “female nurse” or “male banker,” and help us actually focus on performance rather than demographic characteristics.

So, we ran a number of studies on comparative evaluations, where what we did was actually relatively simple.

We tried to mimic, in a very simple way, first in a laboratory, a simple evaluation situation that is involved in most hiring decisions or promotion decisions, for that matter.

Some promotion decisions that you are involved in, somebody comes up for promotion because, for example, the person has an outside offer. Certainly, also true in academia, someone gets an offer from the LSE, we have to respond. So then it’s about this one person. So then I have to evaluate this one person. Is he or she fit for the job? Or do we let him or her go?

It turns out that when we focus on one person we naturally compare him or her with the referent in our head. In fact, comparison is a very natural thing for us to do. Behavioral scientists would argue that it’s very hard to make absolute judgments. In fact, almost anything that you experience is relative: whether you like the coffee, whether you think the temperature is hot or cold, whether you think the room is empty or full. It has something to do with the experiences that you typically have, with the reference points that you have in your minds.

The same is true when we evaluate people. There are reference points, implicit reference points, which we’re not necessarily conscious of, that help us calibrate whether the person fits or not.

But we can counteract the stereotype in our heads by forcing you to evaluate at least two people at the same time. Two is enough. You can also do more. You don’t want to create cognitive overload, you don’t want to do too many. But two is already enough.

When we have evaluators look at two candidates at the same time and evaluate them comparatively, they’re much more likely to focus on performance. Now, why is that? Turns out that comparison generally is a very successful tool.

Iris Bohnet Harvard Kennedy School

Paul alluded to a study that Opower did in many different countries, where they compared people’s energy consumption to that of their neighbors. And a typical energy bill looks something like this. So, your energy consumption is compared to the efficient neighbors, all neighbors, and then you. And it turns out that you use more energy than your neighbors. In some versions there are rankings, so you’re compared to your hundred neighbors and you’re number two or number 65, just lots of different versions.

So it turns out my family was a subject in this experiment, involuntarily, as it should be, of course. And I wouldn’t show you my energy bill if it wasn’t great. But here is, literally, our energy bill. So this is us, the blue ones, and we do compare favorably with our neighbors. They also tell us a bit more about our specific energy consumption.

So, this is literally our energy consumption. We’re doing great in the cold months. And this is June, when we turn on the heat in the pool, when we do worse than our neighbors. And then by July and August we don’t have to heat the pool anymore and so we’re doing okay again. We’re doing well in winter because we got this first comparison—next thing my husband did was to get our roof insulated and install solar panels.

I am showing it as an example because it actually works. This is now a randomized control trial with millions of households around the world. And as Paul said, it’s a super simple, super cheap intervention that costs these energy companies basically nothing, provides a bit more information that does two things.

One is it helps households calibrate. How do you know whether you use a lot or a little? You see the bill, you see how much you have to pay, but is $1,000 a lot or is it a little? Compared to whom? It’s very hard to calibrate.

Calibration is a very helpful mechanism in helping our minds get things right.

And secondly, it also creates some sort of crazy competition. My husband and I, we’re now ranked number two in our environment, among our neighbors. So, we’re walking around looking at these families who spend winters in Florida and summers on the Cape and don’t use any energy at all.

But anyway, comparisons really do matter. And I have introduced them at Harvard. You can actually change how you do things. So the fun part of helping lead an institution and not just studying how organizations work, was that I got to use some of my own research and implement it tomorrow.

So, we did change how we look at faculty, how we promote faculty. And one of the simple things we did is we tried to bundle our decisions. So not only do they help us compare, but as soon as we make more than one decision—for example, now we’re trying to make three promotions at the same time, faculty promotions, and then we have nine finalists, for example, and we make three appointments—diversity emerges much more naturally.

And you’re, of course, all in the business of portfolios, so it’s very intuitive that when you look at a portfolio of choices, you’re much more likely to go for variety than for the same thing all the time.

In fact, some of this work was inspired by an early study in psychology that was a very, very simple experiment, but it’s going to make my point here. It went something like this. They went into a high school, and they offered half of the high school kids snacks for the next 30 days, and all of you on this side of the room would have to decide today what your snacks would be for the next 30 days.

And these were like 200 snacks, a lot of different choices, you make 30 choices today and that’s what you’re going to have for the next month. This part, this other half of the high school, or this part of the room, was asked every morning to choose a snack for today.

And it turns out that the group choosing for today would go for much less variety than the group choosing for the month, because even if a Mars bar is your favorite sweet, you cannot imagine that you would want to have the same piece of chocolate 30 times in a row. But on this side of the room you’re like, every morning: apple,

Iris Bohnet Harvard Kennedy School

chocolate bar, apple, chocolate bar . . . you go for the chocolate bar today and you’ll go for the apple tomorrow.

So, of course there are other things going on in that particular study with time and planning, et cetera, but it still makes the point that bundling choices can lead to very different results than doing choices sequentially.

So, in the study that I did with Max, we had people evaluate, either separately—one candidate at a time—or jointly—comparing at least two candidates at a time—and this is what we found. This is what you kind of want to see, namely, a huge performance gap in joint evaluation.

What that means is that you chose the better performer, or, evaluators chose the better performer, and basically there was no gender gap, because gender was not predictive of future performance. When in separate evaluation you see a bigger gender gap and a performance gap.

Okay, so, second take-home is: evaluate comparatively. Whenever you can, evaluate comparatively, because you do it anyway. You always evaluate comparatively, but you’re not aware of it. You always compare with this implicit referent in your mind. So make comparisons explicit and they will help you and your teams evaluate more accurately.

Now let me turn back to Ellen Johnson Sirleaf, where I started out. So, about six years ago at the Kennedy School, and I’ve been there for 17 years, so to my credit I noticed it in year 11—that’s a really steep learning curve—I noticed that we have only male portraits on our walls. Only. So, that’s quite amazing. I mean many boardrooms, of course, only have male portraits on the wall because former chairmen are on the walls.

And, so, we started to commission portraits of female leaders from around the world. Here is Ellen Johnson Sirleaf. This is a photo where we had this portrait of Ellen Johnson Sirleaf, she’s in the foreground there, inaugurated. So why are these portraits relevant?

Turns out there’s something in psychology that is called stereotype threat. And what stereotype threat suggests is that there are triggers in the environment that will prime certain parts of your identity. Maybe the triggers are male, maybe the triggers are American, maybe the triggers are white, maybe the triggers are banking. Whatever there might be, there are these environmental factors which will affect how you behave.

And one of the, I thought, more impressing studies that I’ve seen on stereotype threat, which literally has been studied in many, many different areas, was one where girls were either primed for their Asian identity, or for their female identity. So the same girl would either see a picture that would remind her of Asia, these are Asian-American girls, fourth graders, ten years old. Or, they were reminded of being female.

It turns out that when they are reminded of being Asian, they do much better in math than when they are reminded of being a woman. Same is true for boys. When you prime gender identity in terms of reading and writing, boys do worse than when they’re not reminded of their gender identity.

In fact, this is a super easy fix in the tests that we construct. A very simple fix is don’t ever, ever collect demographic characteristics in any questionnaire survey that you’ll ever conduct in your life before you ask people to fill out the survey. Because if you ask people beforehand, that will prime what they think of as they are completing the survey.

Surveys are, by the way, an interesting—this is a bit of a side comment while I have time—are a very interesting problem for us. So I’m happy to say that one of our bigger successes in terms of designing gender equality has been the redesign of the SAT. Some of you might know that in 2016 there will be a new SAT test coming out.

It’s still going to be multiple choice, heavily multiple choice. Will have other questions as well. Lots of different changes, about ten different changes. But one of the changes is influenced by our work on gender equality.

Here’s one interesting fact. In these multiple choice questions, it turns out that it has been rational to guess. And

Iris Bohnet Harvard Kennedy School

many of you, I’m sure, have been studying for the SAT, either yourself or with your kids, it’s rational to guess when you can exclude at least one alternative. So there are five alternatives, you can exclude one, you know Napoleon wasn’t a Roman emperor, then you can just guess among the remaining four. So that’s the rational strategy. There was a small penalty associated with guessing wrongly, but if you can exclude one, it’s rational to guess.

Turns out women don’t guess. Women are much less likely to guess—the same dynamic that keeps women from speaking up in public. You’ve all been in meetings. It’s typically not the women who ask the first questions, if they ask a question at all. Women need a higher confidence level to be able to speak up, and they’re more risk averse. So, female test takers haven’t been willing to guess. We’ve studied this. It turns out it costs women 70 points on the SAT, controlling for ability and performance. 70 points.

The new SAT, there are two things we could have done, and we studied both. One, which in many ways would have been my favorite one had it not led to other biases, would have been to force choices. We could have said you can’t skip anymore. So, now on the SAT you can skip. If you don’t want to answer the question you can skip. So you could have said, just force people to answer every question so nobody can skip.

Turns out that doesn’t quite work either, and it created biases among class, not so much gender biases but created a lot of stress, particularly for students with a socioeconomic background that is not necessarily represented in this group. So, we couldn’t do that either. We couldn’t force because the anxiety that was created didn’t help.

So what the new SAT is going to do is do away with all penalties. It blatantly, openly invites guessing. It now says the smart strategy is: study this, have the right answer, but if you don’t have the right answer, guess. So, of course, criticism has come back—“That’s really crazy, you want to evaluate performance and now you’re inviting guessing . . .” And the counterargument is we couldn’t force people to answer because that would have biased results against people who come from different backgrounds. So what we did was to just make it open so everyone could be guessing.

So, test taking is important, and we really have to think about these stereotypes and these kinds of triggers that our environments create. Last night at dinner I actually shared one more study that we could never do anymore these days but that really impressed me. So, in the ’60s a number of psychologists, from Harvard in fact, went to California and ran the following study.

They went into a school and administered intelligence tests to first grade, second grade, third grade, fourth grade students—four grades of students between six and ten years old. And they were interested in whether this predicts future performance. And in fact, they collected the data and they determined the 20 percent best performers. And they informed these performers of their performance and that they are in the elite group. They also informed the teachers and then tracked how well they performed over the course of their studies.

Turns out the test is very predictive. But here is the important message: these students were chosen randomly.

The test actually didn’t matter at all. You couldn’t do that anymore today. No human subjects committee would allow this. If I were a parent in this study I wouldn’t be very happy. But it is important evidence and that is that these students are chosen randomly, but they were told that they are among the best performers as were their teachers. That became a self-fulfilling prophecy.

So we really do have to worry about the messages that we all send in our organizations, purposely and often unconsciously, most often unconsciously, about whether we trigger images of economists or psychologists, or Americans or Europeans or Asians, and what they might do to people’s performance.

Here’s another interesting study on how important seeing is believing really is. So, it’s portraits on the walls, it is

Iris Bohnet Harvard Kennedy School

checking boxes with the demographic characteristics, but of course the most effective intervention likely is actually seeing people in particular roles, people in counter-stereotypical roles. Some of you, and I’m happy to talk about that in the conversation, might wonder about the effect of quotas.

The best quota experiment today is actually not Norway, but it has been run in India. India, in 1993, amended its constitution with the provision that a third of its village heads, a third of the mayors, had to be female. And a group of researchers has been able to follow the experiment. It was a real experiment because a third was literally drawn out of a hat. So a third of the villages had to have a female mayor, but two thirds didn’t.

So, you could see what difference it made. And it turns out that the first woman wasn’t very effective. Even though she already provided more public goods for her community, she didn’t think she was effective, nor did her village think she was effective. She didn’t want to be reelected, she wasn’t reelected. It was a second woman who made a difference. So, after exposure to at least two counter-stereotypical role models, mindsets started to change.

In fact, researchers have been administering the implicit association test—that I’ve just shown you before—with these villagers over many years now, and can show that after having been exposed to a female mayor for at least eight years, doesn’t have to be in a row, but eight years in the last 20 years, mindsets are starting to change and people are starting to believe that women could be political leaders.

In fact, the last paper that has just come out in Science shows that one of the key career aspirations of parents in the villages which have seen female leaders is for their daughters to become politicians. So this is, of course, not based on irrational assessment. These jobs are really rare. But it is about changing mindsets, about helping people imagine what might be possible.

Let me end on a kind of important topic. Typically, my talks just remind us that behavioral interventions are small but can have big impacts, including on big problems. But this problem is really big. This is bigger than not having enough women on boards or not having enough female professors or female CEOs.

The Economist coined it “gendercide,” a few years ago. The UN now estimates that about 158 million women and girls are missing in Asia because of sex-selective abortion or neglect during the first five years. One hundred and fifty-eight million women is about the number of women living in the United States, so that is like zero women living in this country.

That’s a huge problem. And people have tried many huge solutions, as Paul said, many huge . . . and of course, we’ve also been very nervous because of ethnocentricity concerns and coming in as Westerners and telling people in India and China that it really wasn’t appropriate to kill your daughters. So, what can we do?

So, my favorite study is a study by Rob Jensen, a former colleague of mine who’s now at Wharton, who exploited the fact that call centers preferably moved into India in the ’90s. And they also overwhelmingly hired women, so these were new job opportunities created for women. But in order to understand whether that does anything at all, of course you have to run an experiment.

So what he did was, he selected 200 villages to receive the treatment and two other control villages. The treatment in this experiment meant that he hired a firm which provided training and information for women to go and work in those call centers. The other villages didn’t receive that kind of training, and he was interested, of course, in whether more women then would go and work in the call centers. But that really wasn’t his primary motivation.

More importantly, he cared about whether this affects how parents treat their zero-to-five-year-olds—this creating options, role models, even though the changes per village were like two percent more women having work. These are small changes. But did it affect the survival chances of these girls? And that’s what he studied,

Iris Bohnet Harvard Kennedy School

that was his main interest.

So he measured survival rates, he measured health, BMI—body mass index, survival, education. And he can show that, in fact, it does matter, that in the treatment villages where economic opportunities were created for women, parents started to treat their daughters better. And the beauty of that is, it really was an efficiency argument. It didn’t affect the sons, so parents didn’t just treat their sons worse, but they were able to increase the size of the pie and treat their daughters better.

So the discussion that we have today is neither hypothetical, just about our minds, nor just about your organizations or my organization, but it really is about the world. Role models can matter. Seeing is believing, and if you can create role models either by having actual counter-stereotypical people in the jobs, or even portraits, or if you can’t do that, gender-neutral tests, that will take us a long way. With that, let me end.

These are just three ideas on how you can promote change.

Data, it’s a very trivial point, but I am making it repeatedly because many organizations just don’t collect enough data, in particular on people decisions. And I’m very excited about a new field, and those of you in human resource management call it people analytics, where we’re applying the same kinds of analytical tools that we apply to economic forecasting and financial decisions to people decisions. So, people analytics, if you ever want to check it out, is a very promising field, based on data, based on real evidence, big data, trying to help us make better decisions in our people management.

Secondly, compare. Everything you can, compare. Compare to calibrate, because you will compare anyway, typically, unconsciously.

And thirdly, think about the triggers in your environment, the threats, the opportunities in your environments, the images that you portray, and the kinds of opportunities they might create. And with that, happy nudging.

Question: Back to the beginning. Business school case, very similar to the one you mentioned, blind, male protagonist, female protagonist, and it was about a tough boss, was sort of the basis of the case. And the most disturbing thing to me about it was that the reaction was far more negative to the female boss. But peeling the onion back one step further, the reaction among the female students was far worse about the female boss than the male students’ reaction to the female boss. I don’t know if any further peeling of the onion was done on that as to why that bias existed. That was surprising to me and kind of really more disturbing than the overall conclusion, frankly.

Iris: It’s a very good question, a question that many people often ask which is, “What is the gender bias in the eyes of the observers?” And, “Are women more or less biased when they evaluate women, in this instance?” So it’s a bit of a longer answer to your question because it is an important question.

So, I think the most important message is that it’s much more important what you see than who sees. Most of the variance is eaten up by what you see. If you don’t see male kindergarten teachers, you don’t associate kindergarten teachers with men. If you don’t see female CEOs, you don’t associate women with CEOs. That’s the most important driver.

Then, we sometimes find that observers also differ. Sometimes we find some of the differences that you just talked about. And it is quite interesting that, if I may make a comparison, which is not completely fair, but which has been striking to me as I’ve been working on my book, there is a very interesting study on refereeing showing that there is a bias among white referees judging African American players, and the reverse is also true, but it’s mainly among white referees, more pronounced among white referees judging African-American players, which actually has changed dynamics quite substantially since the study has come out.

Iris Bohnet Harvard Kennedy School

We typically refer to this as in-group bias: you like your own people who look like you—homophily—kind of a general tendency. I’m telling you that because that’s typically the wisdom that we’ve had, that there’s a bit of a tendency to like people who look like ourselves. So, in-group biases have been less pronounced for women, generally less pronounced, so your sense of that study is actually right.

And there’s been a really good experiment in Spain where evaluation committees in academia are randomly created. So, from all academics in Spain you’re randomly allocated to an evaluation committee of five. And sometimes there are four men and one woman, sometimes it’s four women, one man, et cetera. So they were able to look at if it matters whether we have more women than men on the evaluation team.

And it turns out that women often are harsher when evaluating women than men are when evaluating women.

And the best explanation to date is that there’s some sort of gender-specific competition in our minds that women tend to think that there are slots for women—say, 30 percent slots reserved for women and 70 percent for men, and that the more women I have in my field, the harsher the competition is going to be.

This is supported by the fact that when these same committees evaluate people for tenure, at which point the competition is kind of over because of job for life, the gender bias goes away. So, it’s particularly pronounced when junior people evaluate other junior people who are still competing for these rare slots. Then it appears as if women have a sense of gender-separate kind of tracks for men and women and are more harshly evaluating women.

Question: A related question to kind of specific cohorts and how they evaluate themselves . . . the demographic profile, particularly in the West but also in China, means that we’re going to have to do an awful lot more, particularly for high value jobs with older people. But some of the cohorts most biased against themselves are older people—ageism among the older people. I think some of the studies show that old people are biased against old people, possibly because a lot of people don’t consider themselves to be old. And so I’d be interested in your thoughts on this, ageism and that particular aspect of bias.

Iris: I think, just to link your question to the earlier question, we generally also find that groups which in our minds either have traditionally been discriminated against or we associate with fewer leadership roles, such as older people or women or people with disabilities or African Americans, just to spell out the biases that we have, that if you are a member of this group, it doesn’t necessarily pay for you to associate with your own group. Often you want to distance yourself from the group because you know that these stereotypes go with the group.

These are the stories. That’s not actual research. But these are the stories, that women start to dress like men and start to behave like men and have a different pitch of voice and wear suits and slacks and things of that sort.

So the same might be true—now I’m not a specialist on ageism—might also apply to ageism, that you want to distance yourself from your age group, appear younger, be younger, don’t hang out with your peers, in order to be associated with the group that stereotypically is associated with having the highest profile in our world.

Now, ageism, as you rightly say, if you do the implicit-association test (IAT), ageism is a huge problem, particularly in the Western world. It’s quite interesting that the implicit association tests that have been administered in Asia haven’t quite been able to track that.

So, again, I don’t want to generalize too much, but the studies that have been done are quite interesting on the IAT and age, just a bonus for being young—young are the new generation, the entrepreneurs, really fun. And there’s a bonus for being relatively senior, much older than in Western countries because that goes with experience and respect and authority and it’s the unhappy middle again, which they found.

But generally, I think you’re right. Ageism is another bias, very prevalent in the U.S. and Europe. And, older people don’t necessarily want to associate with other older people because that might tell you something about

Iris Bohnet Harvard Kennedy School

who they are. I think we had a question here.

Question: Thank you. This is specifically a question about the SAT test, but it’s something that kind of bothers me about all of this or . . . I shouldn’t say all of it, a number of things you mentioned. Let’s say you answer 100 percent of the questions and you get 85 percent right, and I answer 80 percent but I get all of them right, it seems like it’s silly for the SAT committee to decide which one of us is a better thinker.

Why not, did anybody consider ranking with a heavy penalty for wrong answers and no penalty for wrong answers, reporting the score of whichever group you scored higher relative to, and also a check box for which kind of thinker you seem to be? More generally, it seems to me, that some of this is about getting a more diverse group of people to fit in existing slots rather than exploiting the diversity of people that kind of open up slots, or people think slots should be.

Iris: I’m not sure I completely understood your question. Are you saying maybe the whole multiple choice framework is a bad framework, or the SAT more generally is a bad framework?

Audience Member: Single number score.

Iris: I mean, generally, I completely agree with you. Thankfully, the new SAT is going to have many more sections. So the multiple choice part, for example, is going to be a much smaller fraction of the overall test. But I think maybe your implicit question, which is not the question you may have asked, I don’t know, is if the SAT is a good predictor of future performance? Right? Are we actually using a test that tells us anything about anything?

The evidence is completely mixed. I’m not defending here the SAT as a measure of future performance. The evidence, completely mixed, depends very much on demographic groups, on where the test was administered.

It’s really crazy.

If you take the test—I think , Michael, you talk about this in a book of yours, in fact—if you take the test some place and you’re surrounded by 99 other people, like in this room, you will perform worse than when you are in a group of only nine others, which of course shouldn’t matter at all because you’re competing with the rest of America. It doesn’t matter how you take the test, but even such small things in the environment matter.

So, I’m not enough an expert on the SAT to say I shouldn’t completely throw it out, but certainly some colleges have. Some colleges have said this is a super bad predictor of future performance. It has biases that we’re trying to do a better job of correcting, but we’re not really getting there. So, I think the final verdict is going to be out whether the new SAT is going to do a better job. But it does try, I think, to address some of the questions that you have.

Question: Let me ask a slightly politically incorrect question, which is, what is the imperative for diversity? Is it social? Is it moral? Is it economic? And once we answer that question, would there be some level of gradation of the diversity drivers? That is, once we define what the imperative is, would we be better served by focusing on a particular aspect of diversity, whether it is gender, whether it is class, whether it is something where the outcome is significantly better for whatever the imperative is?

Iris: Thank you. Chris [Chabris], are you here? Are you going to talk about your gender diversity study this afternoon, or no? Should I talk about it? A little bit, so I don’t want to . . . I’m not going to steal the thunder . . . Chris is going to answer the business case question but the short answer is, it’s all three of those. I think, honestly, I don’t think we can separate the human rights case from the business case. The human rights case has been there for a long time. And so I think the moral case is everyone’s individual decision: do you believe in equality or don’t you believe in equality? And, so, that’s really up to you and humanity to decide. But the business case, there’s more and more evidence suggesting that there’s a business case also, and some of it is the talent pool, which is a very straightforward one, in particular, as we have increasingly more women graduate

Iris Bohnet Harvard Kennedy School

from college than men.

The second one is gender diversity in teams, which has been shown, in Chris’s work and in the work of others as well, that cognitive diversity in particular is very predictive of a team’s success. Now I should also add—and that’s actually not just a footnote but an important comment for those of you who try to create diverse teams— there’s another piece of the evidence which is important.

There’s some research suggesting that when you ask people in a diverse team, particularly in a gender diverse team, how well their team performed, after they’ve just done the task, people will report that their team probably didn’t do so well and that they didn’t like the task much, because it is hard work.

So diversity is not simple. I’m not suggesting here to go home, diversify your environments, and kumbaya, everything’s going to be great. Diversity is hard work. It’s very hard to make diversity work, but generally we do find these positive impacts.

Now, to your question. I actually don’t think that the tools that we use depend so much on the motivation of why we’re doing it. For me, a nudge, or behavior insights, is just another tool in our big tool box that we can use to motivate certain kinds of behaviors. You could use incentives instead, we could use regulation instead, we could do lots of other things, moral appeals, other interventions.

Nudges are another tool in our tool box that I suggest we should use because they are smart and efficient and cheap. But whether you want to get from A to B, or from A to C is not really in the realm of behavioral scientists. That’s a moral decision or a business case decision that we as individuals or we as organizations have to make.

Chris, did you want to jump in?

Chris: There’s nothing else I can add right now. I’ll say a few relevant things this afternoon towards the end of my talk so we’ll come back to the topic then.

Michael: I think, by the way, I’ll mention James Surowiecki is here, and he wrote the book on this, a book on this, The Wisdom of Crowds, so if you want to talk about wisdom of crowds there’s the guy that wrote the actual book on that. One more quick one, since we’re running a little long. Steve, go ahead.

Question: Not sure this is quick, but, how important is diversity of thought in the whole scheme of diversity?

And I come at this from the angle of academia, where there’s a huge and very prescriptive thought both among faculty and students for socioeconomic diversity, gender diversity, ethnicity diversity, et cetera, but there’s not any thought, as far as I can tell, put into diversity of thought relative to the faculty, in particular.

Iris: In fact, I think the evidence suggests that the driving factor is cognitive diversity. So, I’m not sure I’m disagreeing with what you’re saying. My sense is that the literature has heavily focused on comparing the impact of cognitive diversity compared to sociodemographic diversity. And I think the strongest evidence we have, the biggest bulk of research, is that cognitive diversity, unequivocally, is a good thing.

Now, it depends also a bit on the task that we’re looking at. If we all have to run out of this room because there’s a fire alarm, we should probably all run after the leader and not cognitively have a discussion of what’s the best strategy, et cetera.

So, sometimes if you have to execute—running is probably a good thing. But creativity, brainstorming, lots of other tasks—cognitive diversity can really help. And then the question is, when is cognitive diversity correlated with sociodemographic diversity? And are there other aspects of sociodemographic diversity, in particular now and of interest in our discussion here, of gender diversity? And that’s, I think, literally, where Chris is going to take us this afternoon. But there are, in fact, other aspects that gender diverse teams bring to the task that cognitive diversity doesn’t do. Is that a fair, I think, a fair ending?

Iris Bohnet Harvard Kennedy School

Michael: We’re going to call it there. And, Chris will talk about this later this afternoon, but I’ll also mention, we wrote a piece about this last year, called “Building an Effective Team,” where we actually talk a fair bit about cognitive versus social category diversity versus values diversity and that kind of stuff, and there’s a lot of references on that. But I think you’re right. You’re talking about academia, it’s a different topic, but for our businesses I think it is cognitive.

So with that, please join me in thanking Iris for this presentation.

Max Bazerman Harvard Business School

Max Bazerman is the Jesse Isidor Straus Professor of Business Administration at the Harvard Business School and the Co-Director of the Center for Public Leadership at the Harvard Kennedy School.

Bazerman’s research focuses on decision making, negotiation, and ethics. He is the author, co-author, or co-editor of 20 books including The Power of Noticing (Simon and Schuster, 2014) and Blind Spots, with Ann Tenbrunsel, (Princeton University Press, 2011) and over 200 research articles and chapters.

In 2006, Bazerman received an honorary doctorate from the University of London (London Business School), the Kulp-Wright Book Award from the American Risk and Insurance Association for Predictable Surprises (with Michael Watkins), and the Life Achievement Award from The Aspen Institute’s Business and Society Program. In 2008, Bazerman received the International Institute for Conflict Prevention and Resolution (CPR) Outstanding Book Award for Negotiation Genius, and received the Distinguished Educator Award from the Academy of Management.

Bazerman’s consulting, teaching, and lecturing includes work in 30 countries. Details are available at www.people.hbs.edu/mbazerman.

Max Bazerman Harvard Business School

Michael Mauboussin: Welcome back. I met our next speaker, Max Bazerman, more than 20 years ago and am delighted that we are able to have him join our forum today. Max is a professor of business administration at the Harvard Business School and truly one of the great business educators in the world today.

Max’s research focuses on three areas, all of which tie intimately with the theme of attention. The first is decision making. When I am asked for the most practical book on heuristics and biases, I still point people to Max’s book, now co-authored with Don Moore, called Judgment in Managerial Decision Making. The book is now in its eighth edition and is going strong.

The second area of research is in negotiation—and I think there would be little dispute if I suggested that Max is the best teacher of negotiation in the world. By the way, I got a taste of it when we negotiated his appearance today!

Finally, he has done some very important work on ethics. One of my favorite papers is about goals: not only considering their upside, but also acknowledging their downside.

He is the author, co-author, or co-editor of 20 books, including The Power of Noticing, which directly addresses our theme today.

Max is an engaging instructor, as you will see shortly, and someone from whom I have learned a great deal.

Please join me in welcoming Professor Max Bazerman.

Max Bazerman: Thank you, Michael. And here I thought he did so well in the negotiation about being here [Laughter]. I always find the beginning of a talk the most awkward part. So what I do is I always have a hundred-dollar bill on me. Actually I have a few of them because I use these, because I run auctions, and what I auction are hundred-dollar bills, and that’s what I’m about to do.

So in a moment or two, if you like, you can participate in an auction. Some of you already do participate in auctions, and you know that the rules are important. So, let me tell you about the rules before the auction starts.

First one is that silence is in effect starting now. After the auction is over, if you want to yell and scream, interrupt, that’ll be just fine but not during the next ten minutes.

Second, this is for real money. If you win money from me, I pay you. If you lose money to me, I fully expect to collect. So, please do not enter the auction under the assumption that we’ll call off the money part.

Rule number three. If there’s a first bid, it must be five dollars. Bids go up five dollars at a time. So don’t try to start the bidding at $12 because I won’t accept that. That would violate rule number three.

And rule number four says you can’t bid twice in a row. You can’t be the five-dollar bidder and then also be the ten-dollar bidder. Some of you are wondering why would you want to. That’s a topic for an advanced lecture. For now it’s simply a rule of the auction. If you bid five and someone else bids ten, you could potentially be the 15- dollar bidder, but you can’t be five and ten.

Rule number five, standard rule of most auctions. Highest bidder gets the hundred for his or her bid, but the second highest bidder, under rule number six, has to pay his or her bid, and he or she will get absolutely nothing in return.

So if the bidding went five, ten, 15, 20, and nobody bid 25, the 20-dollar bidder would get the hundred for 20 dollars, earning a nice 80-dollar profit, but under that story the second highest bidder would owe me 15 dollars and get nothing in return and suffer a 15-dollar loss.

So I remind you, silence is already in effect. Real money. Bids start at five dollars and go up five dollars at a time.

Max Bazerman Harvard Business School

Can’t bid twice in a row. Highest bidder gets the hundred for what he or she bids. Second highest bidder owes me his or her bid and gets nothing in return.

So I remind you that silence is already in effect. Real money. Bids start at five dollars and go up five dollars at a time. Can’t bid twice in a row. Highest bidder gets a hundred for what he or she bids. Second highest bidder owes me his or her bid and gets nothing in return.

Do I hear five for a hundred? Got five. Do I hear ten? Got ten. Do I hear 15? Looking for 15. I’ve got 15. Do I hear 20? Got 15. Do I hear 20? I got 20. Do I hear 25? I got 20. Do I hear 25? Twenty-five. Do I hear 30?

Looking for 30. I got 30. Do I hear 35? Looking for 35. Thirty-five. Do I hear 40? Do I hear 40? I got 35.

Thirty-five going once. Do I hear 40 anywhere? Forty. Do I hear 45? Do I hear 45? Forty-five. Do I hear 50? Do I hear 50? Fifty in the corner. Do I hear 55? I have 50. Do I hear 55? Fifty going once on my right-hand side. Do I hear 55 anywhere? Fifty-five. Do I hear 60? Do I hear 60? I got 55. Do I hear 60? Fifty-five once. Sixty. Do I hear 65? Do I hear 65? Sixty-five. Do I hear 70? [laughter] Do I hear 70? Do I hear 70? Sixty-five once.

Seventy anywhere. Seventy. Welcome to the auction. Do I hear 75? [laughter] Do I hear 75? Do I hear 75? I got 70. Do I hear 75? Seventy-five. Do I hear 80? Do I hear 80? I got 75. Seventy-five once. Do I hear 80?

Seventy-five twice. Eighty. Do I hear 85? [laughter] Do I hear 85? I got 80. Do I hear 85? Eighty going once.

Eighty going twice. Do I hear 85? Eighty-five. Do I hear 90? [laughter] I got 90. Welcome to the auction. Do I hear 95? I got 90. Do I hear 95? Ninety going once. Ninety going twice. Do I hear 95? [laughter] Last chance for a profitable bid. You’re on the hook for 85 bucks. Do I hear 95 anywhere? Going. Ninety-five. [laughter] Hundred. I’ve got 100. Do I hear 105? [laughter] Hundred going once. Do I hear 105? A hundred and five. [laughter] Do I hear 110? Do I hear 110? Do I hear 110? One ten. Do I hear 115? One ten going once. One ten going twice. Do I hear 115? Sold for 110. And I just want you to know that I got $105 from an investor who outperformed the market 15 years in a row.

Question: Has it ever gone for less than a 100?

Max: We’ll come back. The answer’s no. Never. I thought it was going to happen today, but no. I’m going to come back to all these questions, and then if I don’t answer your question, we’ll come back to it, but first I want to tell you about this problem.

You’re company A. I’m company C. I am listed on a major stock exchange. My market capitalization is a billion dollars. I’ve announced that I am for sale. I’d be happy to be acquired by another firm at some price in excess of a billion dollars.

And the analysts have already quickly analyzed that there is synergy for both A and B because A and B are currently co-leaders in my industry and being the leader would be a strategic advantage. The estimated value of that synergy is $200 million. That is, while C is worth a billion as a standalone, C is worth 1.2 billion if owned by you, company A, or if owned by company B.

The problem is that if A or B were to acquire C, the company being left out of the deal would suffer a strategic disadvantage by no longer being the co-leader and by becoming the number two player in the industry.

You are company A, and the question is, what would you do? Now in the interest of time I’m going to short-circuit this because I want to get back to the questions that were asked about the earlier slide, but I will tell you that when I solicit bids, the most common thing I hear is, “I’ll offer you $1.1 billion.” And all I want you to think about now is, “What will be due?” And the quick answer is, they’ll bid 1.2. You’ll bid 1.3. And now we’re going up to 1.7, where you’ll both suffer a half a billion dollar loss, and you’d both be better off not bidding.

Now, I ran a 20-dollar auction in dollar increments for many years. I moved to hundreds because you’re a special audience. [laughter] I ran 20-dollar bill auctions back in the ’80s. But it was in November 1995 when I wrote this

Max Bazerman Harvard Business School

slide. And the story was US Air, who was worth a billion dollars with rounding, did announce that they were for sale, and the business press did speculate that United and American were the obvious bidders.

And I don’t know the $200 million number nor the $500 million number, but all I need is for synergy to exist for a small number of players, and for the loss to a party being left out of the deal to be larger than that synergy to create a dynamic where we can predict that if bidding does occur, the winning bidder is going to pay more than it’s worth to them. And their own stock price will go down on the acquisition. So, we can identify the competitive environment in which this is likely to occur.

Now, Robert Crandall, the CEO of American, decided to send a memo to his 118,000 employees that basically said American is not going to be the first to bid on US Air, but if United made the mistake of doing so, we would be forced to protect our position in the industry. There’s some chance that one out of those 118,000 memos made its way to United. [laughter]

We’ll skip the antitrust politics here, but it’s certainly argumentative whether or not that should be allowed. But there was good reason to know that it would be allowed, and for multiple years there was no bid on US Air. I won’t go through the history in 2000, as well as the more recent history on bidding for US Air, but back in 1995 Crandall came up with a means to communicate: “Not only am I not going to bid, but you shouldn’t either.”

Now, I ran about 400 auctions for a 20-dollar bill, I’ve run over 100 auctions for a 100-dollar bill, and the quick answer is my auctions have never ended below 100. I don’t know whether Michael was rescuing me by bidding in the auction [laughter], but things were slow, but I wasn’t nervous. Once I have five and ten, I pretty much know I’m going to 95 and 100. And that’s actually something that I want to talk about in some detail because normally in a group this size, and my guess is a lot of you . . . so how many of you have seen this auction before in some version?

And I think that that may have slowed down the auction a little bit, but I often have more enthusiasm to bid in the 5 to 60 dollar range, and often people don’t do what I think of as appropriate analysis, and it has to do with noticing something that never happened. What I’m referring to is if the bidding went 5, 10, 15, and you’re about to bid 20, what I would argue is that before you bid 20, you should be thinking about the question of if nobody new bids, what’s the 15-dollar bidder going to do?

Is she going to quit or bid 25? And if you go through that mental thought, you’ll quickly conclude that she’ll bid

25. The 20-dollar bidder will bid 30. The 25-dollar bidder will bid 35, and I think I’m going to get up to 95 and 100 hundred every time I do this for the rest of my life.

Now, perhaps a secret of investment success is to know when to quit. Because in many cases, my 95-dollar bidder and my 100-dollar bidder will continually invest ten dollars more to avoid coming in second. And if I get that going, we can go higher to 115, 120, 125, 130, and if we can clear 200, what I can tell you is my bidders get mad at each other. [laughter] And so far in my 100-dollar bill era, I’ve had four auctions that have gone to a thousand where I’ve offered those eight people out for 800 apiece, and so far they’ve all taken it. That’s a very good deal.

But what I want you to notice, that in this story, as well as in the 100-dollar bill auction, you need to notice, you need to and you can notice, information that hasn’t happened. Can I anticipate the decisions of other parties?

Michael in his introduction this morning talked about thinking about the fact that if you’re buying, someone must be selling. Do I know something that they don’t know? So often we fail to think about what other parties know.

We fail to notice that there’s information available by putting ourselves in the shoes of others. And I think that that’s something that we want to do more often.

I’m up here talking about noticing. I wrote a book called The Power of Noticing, and a lot of people find that

Max Bazerman Harvard Business School

funny, particularly a woman by the name of Marla. Now I’ve been married to Marla for 36 years, [laughter] and she would be happy to clarify that I am terrible at noticing. A lot of people write books about what they are really good at. For me this is a self-help effort. [laughter] I’ve been working on this for a quite a while, and I still have some work to do.

But I do want to convey to you what I think noticing problems look like. And I’m going to start by telling you about what I honestly view as my most important noticing failure.

I’m predicting some of you are going to treat me with benevolence and suggest that it was nothing terrible. It bothers me to this day. So, a little bit of background. I’ll try not to bore you with too much legal history. But in 1999, the Department of Justice filed a lawsuit against the entire tobacco industry. And it was filed under something called RICO statutes.

RICO are a set of laws having to do with racketeering. They were created to go after the mafia. And we probably have lawyers in the room. By the way, I’m not a lawyer. So anything that I say that sounds legal, don’t take it seriously. So, RICO was created to go after the mafia, and under RICO laws you need to prove both fraud and a conspiracy across organizations to commit fraud. The reason that the Department of Justice might file something under RICO rather than just a fraud case is that the powers for punitive remedies under RICO are potentially far more serious than under a standard fraud case.

And in this case, in 1999, here were the alleged frauds. And what you’ll generally notice is they take the form of doing something bad while publicly denying that you’re doing so. That creates a fraud. And there’s also an allegation that this was done through a conspiracy across the entire tobacco industry. Again, I’m not a lawyer. As of 1999 I knew little about that.

2004, the trial actually starts. February 2005, the DOJ litigation team thinks that they’re winning the case. And an appeals court ruled that the trial could continue, but the DOJ could not pursue its main remedy. And the main remedy that it was seeking, which at least the litigation team was serious about, was $280 billion to pay for all the costs incurred by the U.S. government as a result of these frauds. Basically, healthcare costs.

But a court ruled that RICO is a future-oriented set of laws, not a backward, punitive set of laws. So, that was an inappropriate remedy to pursue. So, the DOJ found itself thinking it was winning the case but being told you can’t have the penalty that you wanted.

So, on March 10, I was contacted by the Department of Justice. I’m an anti-tobacco guy but no expertise at this point. I was hired having to do with some work that I had done on audit firms and other organizations. I was hired to basically assess the likely psychological decisions of tobacco executives in the future and to offer an alternative remedy to the court given that they couldn’t go after the $280 billion.

And my testimony concluded—by testimony, I mean my expert report as well as the written direct examination of the DOJ to me—my testimony concluded with the argument that absent significant court intervention, misconduct would continue into the future.

And I recommended, based on the assumption that the DOJ had proven liability—I said that line like a zillion times because I was repeatedly reminded it was not my job to prove the case, only to recommend remedies— that the court should appoint monitors to the tobacco companies to consider appropriate structural changes not limited to but including removal of senior management. This is not a good recommendation if you wanted to be popular among the tobacco senior management. [laughter]

So, this is in writing. Here’s a quick history of what was going on. I was hired on March 10. Busiest year of my life before I added this work. Filed an expert report on March 21. I was deposed on April 10th. For those of you who don’t know deposed: ten people line up across from you and they take turns asking you questions, and you

Max Bazerman Harvard Business School

have to answer them all. Mid-April, submitted written direct testimony. I’m going to skip number five for a second and tell you that I was scheduled on May 4 to show up in a court where the tobacco lawyers would be cross-examining me—some of the best legal minds in the world.

And I showed up on April 30th to prepare with the DOJ lawyer who was in charge of my testimony. He was a young, earnest, brilliant guy. And when he met me on March 10 he called me Professor Bazerman. And I said, “I’d rather you call me Max,” and he said, “I’ll try, Professor Bazerman.” So, it took him a couple weeks, but after a couple weeks, by late March he called me Max and continued with Max for the next month.

But I show up on April 30 at the Department of Justice to prepare for my May 4 testimony, and he sits down across from me and he says, “Professor Bazerman,” and my ears noticed this. He says, “Professor Bazerman, the Department of Justice requests that you amend your testimony to note that it would not be relevant if any of the following four legal conditions existed.”

He then read me a bunch of legalese that I wasn’t even close to understanding, and then said, “Do you so amend your testimony?” and I said, “Now you know me well enough to know I didn’t even understand what you just said, and that if I did understand what you just said, I’d say no. So why would I possibly say yes?” And he said, “Because if you don’t say yes, there’s a good chance my superiors will be removing you from this case before you testify on Wednesday.”

So at this point I had spent 166 of the eventual 214 hours on this case, and having to do with a marital dispute having to do with how much time I spend working, I had already publicly committed that I was donating 100 percent of my earnings as an expert witness to charity. So I had spent 166 hours of time on this case. I cared about this case. And I said, “Okay, I don’t amend my testimony,” and he smiled and said, “Good. Let’s prepare in case you’re still in court on Wednesday.”

Now some of you are confused at this point in the story, and I want to clarify that I was at least that confused [laughter] as I heard this request from a Department of Justice attorney who I like and respected. I also mentioned life was busy, and I was not taken out of the trial. So, on May 4, I did appear in court. I showed up in court and, by the way, I do a moderate amount of external work. Not in this capacity. This was the third expert witness case of my life. I don’t really like this kind of work, but I was doing it.

I show up on May 4. The judge comes in. People stand up, et cetera, and the first thing I hear from the tobacco attorneys is, “Before the witness takes the stand, we would like to submit a Daubert challenge.” Now, again, for the non-lawyers, don’t get your legal advice from me, but as I understand it Daubert are a set of laws to keep junk science out of the courtroom.

So, you could file a Daubert challenge because someone said, “I read the stars last night, and I can tell you he’s guilty.” A Daubert challenge could be based on the fact that the person’s field of expertise is not a legitimate science, or because they’re not an expert.

On hearing that, the DOJ team immediately jumps up and says, “Are you questioning his field or his expertise?” and the answer to that is both. [laughter] So the judge says, “Okay, life is a little bit busy here. I’m not going to rule on the Daubert challenge. We’re going to get busy. Since he’s here we’re going to hear his testimony, but by Friday you each need to submit five-page briefs to me on whether or not the Daubert challenge should be upheld. Welcome to court, Professor Bazerman.”

So that’s my welcome to court. I’m cross-examined all day, and as soon as court ends I see the DOJ attorneys, and they basically said, “Okay, let’s get busy. We need to file a brief on Friday.”

The next two days I’m working nonstop to basically say my field of expertise is a legitimate science and I’m not a complete idiot. The judge rules I’m not a complete idiot and it’s a legitimate field. All right. And then I get busy

Max Bazerman Harvard Business School

preparing to help the DOJ cross-examine the expert witness who’s going to show up to say I am an idiot.

So this is kind of life, and I need to get back to my Harvard job. The trial ends in early June, and I move forward with life. Now I want you all to think about that April 30 episode and stew on that in the back of your mind for just a little bit. I’m going to come back to it. And instead I want to give you an investment problem because I can really use some investment advice, and this seems like the right crowd to use. You said I could get free investment advice while I’m here. Right, Michael?

Now, what you’ll see in blue is the S&P 500 over the last nine years. And what I want to know is, which of these four investments should I put my money in. I’m going to put my money in one of these four funds. So, if you have other plans, I doubt you’re an index person, but if you’re an index person, don’t tell me about indexes. I want your help on picking between these four funds.

One is the Tobacco Trade Investments fund, for those of you who are bold and brave despite the story I was telling you. Another is this rusty orange Alpha fund. Then we got that spiky Power Trade fund, and finally we have Fortitude. And despite the fact that you might not want to, I’m going to strongly request that everyone in this room pick one of those funds, and we’ll find out which one you pick by hand-raising. Everybody ready?

How many of you are going to be bold and go Tobacco Trade? I’ve never seen so much interest in Tobacco Trade. Alpha? Power Trade? Fortitude? Okay, Fortitude won. Everybody saw that? Next question. What’s wrong with one of these funds? Yes, sir.

Audience Member: Fortitude’s a fraud.

Max: Fortitude is a fraud. I showed you the last nine years of Madoff’s returns. And this is the most elite group I’ve ever showed this to, but I’ve shown this problem and lots of different variations to many audiences, and what I want to highlight is the question: “What should I invest in?”

And I understand you’re not given enough information, enough time, et cetera, et cetera. But the question, what should I invest in, leads to a focus of return and low volatility. And my guess is that if I gave you each a 15- minute assignment to think independently about what’s wrong with one of these funds, the vast majority of you would tell me Fortitude’s a problem.

You cannot dramatically outperform the market with absolutely no volatility over a nine-year period of time. We all know that that’s not possible, yet that’s exactly what Madoff provided. And there were hundreds of people who had access to this data who just didn’t notice.

My colleagues, Ann Tenbrunsel and David Messick, have a term called ethical fading, where ethical fading refers to the fact that in so many business decisions there are ethical aspects to it. But because we’re focused on other dimensions, we’re not thinking about the ethical dimension involved. And perfectly good, nice, honest investment advisors could have been recommending Madoff despite the fact that they had the knowledge and the clarity to know that that wasn’t possible.

A simulation that I use in the classroom all the time, I’m going to give you the four-minute version of a three-hour learning experience. This is a chart from a case called “Carter Racing,” written by Sim Sitkin and Jack Brittain. I think it’s one of the great pieces of business school pedagogy that exists.

The basic problem is: you’re racing a race car, it’s a half hour before race time, and it’s very cold. It’s going to be 40 degrees at race time. And of the 24 races this season, you’ve had engine problems in seven of them. And here are the temperatures for the seven races with problems. You’ll notice that at 53 degrees we blew three gaskets, not just one, and students are asked, “Are you going to race or not?”

Now, the way this case is presented, is if you race and you come in the top five, great things are going to

Max Bazerman Harvard Business School

happen. If you blow an engine on national TV, you’re out of business. So, a high stakes race today. And as students are reading the multipage version of this, I repeat three times: “If you need any additional information, please let me know.”

Very few people ask for additional information. After they make their individual decisions, if I want to kill some more time, I send them off in groups to make the decision, and there’s often discussion about whether or not temperature is related to engine failure.

It’s rare, but about one out of a hundred people asks a pretty obvious question, because if you wanted to know whether temperature was related to engine failure or not, would you want to look at the failures, the successes, or both? That was an easy question. The answer is both.

And if you asked for the temperatures for the other races, here they are, and I have a sheet of paper waiting to give them. And if you go from the fact that these are on kind of different maps and you combine the 24 races and look at which ones have failures and which ones don’t, now a very different picture emerges where it looks like cold is a problem, and it’s 40 degrees today.

For those of you who like this kind of presentation, this makes it pretty clear as well. Now, the disturbing part of “Carter Racing” is that Sim Sitkin and Jack Brittain wrote this simulation to be as close as possible while hiding the fact that this is about the Challenger disaster, where Morton Thiokol and NASA argued the night before about whether O-ring failure was related to temperature, and they only looked at the seven failures, or seven launches where they had some problem. And, despite having very smart people in the room, nobody bothered to ask for the other 17 data points.

Now, this is getting a little technical, but for me it raises a fascinating question of: have you ever been in an important group meeting where someone presented a PowerPoint presentation that guided the information that the group used, and you never even asked for the data that you would actually want in order to make the most appropriate decision? I’m guilty of that. I think that there are people who can manipulate what information we pay attention to and what information we don’t pay attention to.

This afternoon our last speaker may be the world’s expert at that very topic of how to get you to pay attention to some information and not pay attention to other information. So I’ll wait for him to talk about this in more detail, but what I want to suggest to you is every time you’re in a meeting for the rest of your life and someone’s organizing the data in front of you, have in the back of your mind: if this is an important decision, what data would I actually want to have rather than what information did someone choose to put in front of me?

Now, I’ve delayed you a little bit from that April 30 moment where I was asked to amend my testimony, and I was confused, and I didn’t really know what was going on. If there was something going on, I didn’t know exactly what to do about it or who to call, and I think I’m guilty of what many of you are guilty of, and that is when something’s off but you can’t figure out what it is, you go back to solving problems that you know how to solve.

So what happened next was life moved on. The trial ended in early June. On June 17 I was sleeping in the early morning in a hotel in London, England. I was there working with a corporate client. I woke up at five a.m. with that kind of, “I can’t believe I’m so alert and it’s five a.m.” kind of feeling that you’ve all had. And I opened up my computer and I turned to The New York Times web page. And on the front page of The New York Times was an article about a guy named Matthew Myers.

Now at the time of this story I knew about Matthew Myers. I had never met him. I now know him. At that time and still today, Matthew Myers was and is the president of Tobacco-Free Kids, one of the leading anti-tobacco organizations in the country. And he had also been an expert witness for the Department of Justice in this case

Max Bazerman Harvard Business School

against the tobacco industry.

And what the article said . . . and you can find this, June 17, 2005 . . . was that Myers came forward with an accusation that the Department of Justice, going up to the number two official in the Department of Justice, had attempted to corrupt his testimony. And specifically he was asked to amend his testimony to note that it would not be relevant if any of four legal conditions existed.

And I’m reading this, thinking, boy did I blow it on April 30. It still wasn’t clear to me what I was supposed to do, but nothing seemed like the wrong answer at this point. There was something wrong going on at the very highest levels of the U.S. Department of Justice, but it’s 5:00 a.m., midnight in D.C. and Boston.

So I called Marla. Marla knew more about D.C. at that point than I did, and I said, “Wake up. This is important.” I told her what I just kind of told you. And I said, “I think I’m supposed to do something. I have no idea what it is, and long before Washington wakes up I need to go work with my corporate client. So can you figure this out for me? I’ll call you when I get back to the hotel this evening.”

Now she was probably still in a groggy state and said okay. So, I go off and work. Come back to the hotel 12 hours later, five p.m. I call Marla, noon Boston/D.C. time, and I say, “Hi Mar. What’s up?” She says, “We can’t talk long. Louis Clark, the president of the Government Accountability Project—an organization that represents whistleblowers—he’s your attorney. Here’s his number. Call him. As soon as you’re done talking to him, write down this number too, Max,” and she gave me a second number. She said, “That’s the telephone number for The Washington Post reporter who will cover your story,” and all those things occurred.

I called Louis Clark and he represented me in this matter. And then I called The Washington Post, and they covered the story. But many of you would have been pretty comfortable with the notion that I said no to the request to amend my testimony. What else was my responsibility?

And my reaction is when something this strange has happened, it was my responsibility as a professional to go figure it out. And I didn’t because I was overwhelmed and I was busy, and what Marla had demonstrated between nine a.m. and noon on June 17th is it wasn’t that hard to figure out how to find the right people to talk

to. She made a few phone calls to her lawyer buddies in D.C. and basically outlined a course of action.

I think that that was my job to do on May 2. April 30 is a little tough, it was a Saturday. But by May 2 it was my job to have this figured out and to have taken action to identify a situation where corrupt behavior was occurring at a fairly high level of action. I’d like to be a little bit more like Matthew Myers in the future.

There are lots of opportunities to notice that I think we should do a better job of. So, Iris shared with you our study, which is one of the most interesting studies that I’ve ever worked on. Now I want to share with you the most conceptually trivial result of my career. So here it is. I’m going to try to present it in an interesting way, but it really is conceptually trivial.

All right, so as background, there are two kinds of promises that we often do. Before we take the stand in a court of law: I’m about to tell “the truth, the whole truth, and nothing but the truth.” And the other way we often make promises is we fill out a form, and at the end we sign it.

There are two differences here. One is that one’s oral and one’s written. But the other difference is one occurs before you provide the information and one occurs after you provide the information. Now, I don’t want this kind of confound, so I want to change it to both are in writing. You’re about to fill out a form.

Let’s imagine that it’s a form for something that you wouldn’t lie about in massive proportion, but maybe you would exaggerate how big that office in your home is to increase the tax deduction. So we’re talking about things where you probably know people who might commit these exaggerations. And I want you to imagine that

Max Bazerman Harvard Business School

you are a government, and you would like people to tell the truth rather than engage in behaviors that would reduce their taxes below the true amount that they owe.

Here’s the mental question for you. If you wanted people to tell the truth, does it make more sense to have people fill out a form and then sign saying I told you the truth? Or does it make more sense to have people start the process by signing a form saying I’m about to tell you the truth? So, to maximize honesty should you sign first or sign last? On the count of three, yell out “first” or “last.” Everybody with me? One, two, three.

Audience: First.

Max: There we go. That was a demonstration of how conceptually trivial this is. Most of you don’t have PhDs in social science, and you had no problem getting that question exactly right. And you had devoted a total of four minutes to think about this problem. So my colleagues and I had people solve math puzzles, and the problems come without the red circles—that’s to keep you paying attention to me rather than trying to solve the puzzles.

The question is, in four minutes, in how many of the puzzles can you find the two numbers within the grid that add up to ten? We pay you a dollar a piece to do that. So we have people try to solve these puzzles, and after they solve the puzzles we have them report how many puzzles they solve. And we have them think that we’ll have no way of knowing how many puzzles that they solve, but we have some sneaky ways that are moderately honest to [laughter] actually find out.

Specifically, if you want to know how we would do that, one irrelevant number is changed so that we can identify the person’s form, but then they throw it into a bin with lots of other people’s puzzles. So we can go in and get it later and find out the true answer. [laughter]

But they fill out a form, and they tell us how many puzzles they solved, and we pay them based on what they tell

us. This is going to disappoint you, but some people lie. [laughter] Interestingly, they tend to not lie by an enormous amount. They lie by a little bit. It’s kind of like they included the puzzle that they got wrong, but they think it was just a typo. Or they include the puzzle they were about to finish solving, but they didn’t quite get to it.

A lot of people lie by one to three puzzles. What’s interesting is that when we move the signature from the bottom to the top, we reduce cheating by 50 percent. Fifty percent reduction in cheating by moving the signature line. And we now have knowledge that it works on reporting your odometer reading with insurance companies, another place where people lie by a little bit.

And it works on Guatemalan taxpayers—not by 50 percent, by one to two percent. But if we can change the signature on the form and get an extra one or two percent on tax collection, that’s a pretty good deal in terms of the trouble. What I want you to notice about this is that we’ve been getting it wrong for decades. There’s stuff to notice. And if we can seize on the opportunity to notice what’s wrong, we can make changes that can make us more effective.

So what can you do to be a better noticer? The first thing I want to encourage all of you to do is to put it on your agenda to become a first-class noticer. Warren Bennis. Some of you have read his books on leadership. Warren passed away last year, but among the many very interesting things that he wrote, he talked about a Saul Bellow fictional character by the name of Harry Trellman, who he described as a first-class noticer. And he argued it’s a job of leaders to be first-class noticers, and too often leaders don’t notice.

So, the leaders of the United States did not notice that there were an increasing number of people who hated America, who were willing to become martyrs for their cause, who had plans to turn an airplane into a missile aimed at the Eiffel Tower in 1994, who had plans to hijack 12 U.S. commercial airplanes in Asia on the same day in 1994, and that you could board airplanes with small items that could be turned into weapons. And when you put those together, I would argue we would want our leaders to know that we should have fixed airline

Max Bazerman Harvard Business School

security long before 9/11 occurred.

Similarly, when we create an accounting system where we require firms to have external auditors but then we create incentives for the auditors to want to be rehired, to want to sell consulting services, and to want to take jobs with the firms that they audit, we know enough about psychology that when people don’t want to see data, they’re likely to not notice it. And we’ve created an industry in this country called independent auditing where the one thing that I’m very confident they can’t provide is independence. Because auditors want you, the client, to be happy.

So we want to put it on the agenda of leaders to become first-class noticers. We want people to realize that when you’re in the midst of the crisis, that’s not the best time to notice information on the periphery. So we want to plan in advance. We want to be able to take an outsider’s perspective. We want you to audit your organization and ask: are you creating the conditions under which you can expect your employees to notice critical information? From the prior study, this is number three, and this is the other number three. I’m noticing I’m doing a better job than I did before I proofed it.

We want you to realize that when something’s wrong, it’s your job to go figure it out as opposed to moving back to things that you know how to solve. This is my fundamental issue that I think I got wrong in the tobacco story that I told you. You want to identify predictable surprises.

And Michael alluded to this before. I would encourage all of you to think about whether or not there are problems in your organization that if left unattended will only get worse, yet you’re ignoring those problems. And if the answer is yes, what are the barriers, and how can you get busy on these?

I think 9/11 was a predictable surprise. I think that Enron was a predictable surprise. I can’t tell you that I wrote about airline security before 9/11. I can tell you that I wrote about the impossibility of auditor independence under current legislation long before Enron exploded. We can identify the conditions where bad things are going to occur, and I think that executives need to do a better job of doing that on a regular basis. Okay, now I short-circuited all that intentionally.

I also want to show you this picture of a young man. A few things I want to show you about this picture. One, notice that hair, that head of hair. [laughter] It’s a full head of hair. Very thick glasses. This photo’s a few years

old. This is a fabric. Only the older members of the audience would know it. It was called Qiana. It had a slippery feel to it, so if you spilled anything on it, it would slide right off. [laughter]

And the last thing I want to tell you about this photo is that this is me. I’m guessing age 17. I’m playing in a bridge tournament. By the age of 20 I was a very, very, very good card player, both a highly-ranked tournament bridge player but also a gambler. And this picture is part of my defense of why I’m such a lousy noticer because I spent a lot of my time between 15 and 22 playing cards.

And when you’re playing cards, either for significant stakes or in a competitive tournament environment where you want to win, you basically want to pay attention to 52 cards and the behavior of the other players. Everything else is a distraction. Everything in the environment, what’s happening in the room, if you’re paying attention to that, you’re using cognitive energy that you could be placing on coming up with the right strategy with the 52 cards and the other players. I think that focusing rather than noticing is a really good strategy for being a good card player.

Now the next thing I want to highlight is the fact that leadership looks very little like playing cards. It is the job of leaders to identify information in the periphery that nobody told you is part of the problem. Your problems do not come perfectly specified. You do not know who all the players are. You do not know what all the tools are. And so often we make analogies from games to corporate life, and I think we’re often missing a critical aspect of

Max Bazerman Harvard Business School

noticing information that’s out there on the periphery.

All right, so this is Max as a 17-year-old. Could be 18 but I think 17. And it’s really been the last dozen years that I’ve been trying to go through this noticing reform effort to become a reasonable noticer. It’s a hard problem. I can report some success, but the success is limited. I want to highlight the limitations of my efforts by showing you a video. But before we start it I want to describe to you what’s occurring.

First of all, this video was not made by Chris, who you’ll be hearing from this afternoon. On the other hand, it’s very clear that his work is the very clear flowchart of what you’re about to see. So Chris deserves the intellectual credit with Dan Simons on what you’re about to see. What you’re going to see is the reception desk at a conference that was hosted by the Behavioral Insights Group that Iris and I directed at Harvard.

And I’m simply showing up this day. I think I’m the second character you’ll see show up at the registration desk.

Keep your eye on the woman who’s registering the participants, the folks showing up. There are a number of well-known scholars who are showing up who are experts in social science. We’re going to see them show up and get their folder for the day’s events. I think that that’ll give you enough to follow the video. Can we start that video, please?

[Video clip]

Max: That was Cass Sunstein by the way. Francesca Gino.

[Video clip]

Max: So in the defense of my colleagues and of myself, this might be in the category of things that we did not need to notice. But on the other hand, it does bother me. So I don’t think I mind not noticing the switch. Other than, if I was paying enough attention to the person who was nice enough to be registering me, perhaps I would have been more aware, and I would have been more in the moment.

And I think a lot of us who are made fun of in that video would like to think that we notice more important things. We just don’t quite know if that’s right. And with that I want to open up the floor to your comments and your ideas. Yes, please.

Question: One question actually just about that. Have you tried anything like that with non-academics? [laughter]

Max: Chris, you want to answer now or later?

Question: I mean, I know the gorilla study . . . But actually, that specific experiment with humans. Because I wonder if on the spectrum, academics are actually more inclined not to actually connect with other people and notice them.

Michael: Max, Apollo [Robbins] did it for his show too. Let’s get the mic in front of Apollo Robbins. He can . . .

Max: So, I’m more used to hearing this question about college sophomores, and I often highlight that they’re people too, which is controversial. Apollo.

Apollo: Chris’s partner, Dan Simons, we did a show that was about just over an hour for National Geographic, and it was specifically a focus on paying attention, and we did it in Las Vegas. That’s not to say that everybody was drinking there which would be the other equation. People say, “Does that only work on people that are in Vegas drinking?”

So, it’s always contextual to whatever that subgroup of those people are. But generally it’s exactly the point of what you were saying with noticing—across demographics, across cultures. I think one question that might be

Max Bazerman Harvard Business School

interesting to tag on to what yours would be is, I’ve heard some studies that sometimes there’s cultural differences of foreground and background, in priorities, does a foreground draw less with Asian cultures, that they appreciate everything on the same spectrum. There’s a lot of questions in those areas. But generally people pretty much behave the same. It’s more about what they’re thinking about versus what they’re actually seeing.

Question: Well, the real question I had was actually about when you were talking about noticing and you talked about 9/11. The obvious example for people in this room is the financial crisis. And one of the things you sort of alluded to but didn’t really talk that much about in there is incentives, and I was wondering if you could speak some—I mean, with the auditing obviously that is all about incentives—but when you think about the decisions that banks made in the run-up, clearly a lot of that had to do with the fact that there was not an incentive in some degree to notice. In fact, there was arguably an incentive not to notice. So I’m just wondering if you could speak just about the role that incentives play inside organizations in terms of maximizing noticing.

Max: Okay, I think I heard about four questions, and I’m going to try to unpack and answer at least kind of a few of them. First of all, you’re going to hear from Chris later on about particularly the visual illusions, and Chris and Dan talk about change blindness.

They also have some wonderful work on inattentional blindness, and Ulric Neisser I believe came up with the term inattentional blindness. Both of those terms have the word “blindness” in them. My colleagues and I who are very inspired by that work, we tend to use the word “bounded awareness.” And the reason to create a new term—it’s because academics need new terms of course.

But the more serious reason to create a new term is that so much of what we don’t notice isn’t just with our eyes. It’s awareness that we need to look through the data to see things. Just a quick comment on the audit piece. And I just want to highlight that I think there are many, many honest auditors.

But if I ask you the question . . . I know that we have a lot of parents here because Iris had all the fathers of daughters raise your hands. If I asked you, “How smart is your oldest child?” I don’t expect that you’re going to be a great source. And the problem isn’t integrity. The problem is when people want to see data in a particular direction, they’re incapable of objectivity.

So I think that the leaders of the audit firms, I think that they are lobbying in ways that I find to be dysfunctional for the public good. I think a lot of the auditors are completely honest people who do want to do the right thing.

But when you are rewarded for not doing the right thing, it affects your judgment in ways that you’re not even aware of.

In terms of the financial crisis, there’s a book about the people who did notice. It’s Michael Lewis’s book, The Big Short, and he identifies people who did notice. Unfortunately they did not work for the government. They basically became part of a very active short selling business. And they became very, very rich people.

But it provides some documentation that noticing is important. I think that you bring up a critical issue, and that is that so many people in the distribution chain of the investments having to do with mortgage-backed securities were doing just fine moving them on to the next layer. So that they were personally doing just fine. And even retrospectively a lot of people made a lot of money and did just fine.

That means that it becomes the obligation of senior executives to think about the systems that were created.

Obviously it becomes sort of the role of government to intervene in a system where there are too many incentives for businesses to do the wrong thing. So I think that the intervention level is really senior executives of major corporations and government to fix these kinds of problems. And I think we could do a pretty similar analysis in terms of LIBOR. Yes, please.

Question: Missing in this case is all the instances where people do notice. So, it’s a lot more enjoyable and

Max Bazerman Harvard Business School

instructive I guess to learn where people don’t notice, but what can we learn from where people do notice? And in particular can we learn things about individuals who don’t notice sometimes and do notice other times? What’s the temperature difference?

Max: Great question. It’s a hard one to have a good response to. Iris and I are both laboratory experimentalists at heart, and yet we think that these messages have relevance to very real world problems that are often noisy. I can do noticing in the lab. We are studying a much more precise version of the Madoff investment task to look at a lot of those kinds of questions. We could do it in the lab.

It’s tough to do it in the field for exactly the reasons that you specified. It’s hard to document all the uneventful situations where executives notice that they should do something. They did something and therefore nothing happened. And that was a success. But you really don’t get credit for keeping things from happening.

The most visible example of a problem that was identified—it was solved, at least according to many people— would be Y2K. A lot of people feared that Y2K was going to be an enormous problem. Money was invested.

Some people would say we wasted our money. And I don’t have the technical expertise to sort out those two different stories. So, I think that your comment is exactly on target. I’m not sure I have a great answer for you.

Question: Was that the full dataset, or were there a number of people who actually did notice? A hundred percent of the people did not notice the change? This is a Carter Racing question. Did you only show us the negative responses and there were positive responses as well?

Max: All I did was lift a video that other people had created but, Chris, do you want to answer now? What percent of people notice that kind of story?

Chris: 25 percent of the people noticed and three-fourths missed it.

Question: And then the question would be, was there a commonality in the people that did notice?

Max: I’m sure that the people who made the video I showed you did not study those people carefully. Whether Chris has or not, we’ll delay that for his talk this afternoon.

Question: Why wouldn’t they?

Max: Because they were trying to do something funny to make fun of the organizers [laughter] and show it as a closing element of that conference. So that was the main motivation. Social scientists, when they’re running studies, are typically not going to do it in that kind of noisy environment. So it’s not just that the people change, but you saw the male and then the tall person switch from being dark hair to blond hair. So, it’s not the right condition to do a scientific study. Yes, please.

Question: If you’re within an organization and you’re trying to get a team to notice things better—and it could be something as simple as, are there errors in data to something much more complex—where should your focus be? I heard you say some things—the environment, people’s habits, people’s incentives. How would you guide a manager of a team to increase noticing?

Max: So, you did a good job of asking a good question, and then you started to answer that. All of the things you just listed are exactly right. What is your suspicion that something wrong is occurring? Go learn more about that, and identify what’s at the root of it, but often at the root of it is, “we institutionalized a bad practice 13 years ago and nobody’s ever rethought whether or not it’s the right system.”

So, I’m arguing that society has put the signature at the wrong place on forms for decades, and it just became institutionalized, and that’s the way we do things. Lawyers tell me we can move the signature to the top, that there is not a massive legal reason why we don’t.

Max Bazerman Harvard Business School

So, we want to identify the cause, and sometimes there’s a very easy structural change that can make all the difference. We want to grab that opportunity if we can. We want to think about incentives. And I think incentives have two parts to them. Incentives affect what people want to do in order to get their reward. But we also want to think about the aspects of how they will affect people’s judgment in ways that aren’t measurable.

In kind of any conflict of interest story where people are doing the wrong thing because of a conflict of interest, it’s both bad people who are doing the wrong thing to collect more money, but it’s also people’s judgments are tilted in ways that they’re not even aware of. So, we want to diagnose the specifics of the problem to figure out what a fairly easy intervention would be to find out how to get them to notice more.

The other thing I would say is start with the low-hanging fruit of very valuable fruit. I don’t think that you need to become obsessed about noticing everything. When Marla continues to make fun of me for not noticing things that she notices despite the fact that I wrote a book called The Power of Noticing, I try to see the humor in it.

On the other hand, I’m actually okay with the fact that there are lots of unimportant things that I’m not going to notice. And what I want to do is have special effort for those things that are important to notice that could still easily escape my vision of the problem. I don’t know if I’m answering your question or not. I’m doing my best.

Going once. Going twice.

Question: I got one. Max, I want to ask you about goals.

Max: Okay, go for it.

Question: A lot of organizations, a lot of companies have specific goals. There are a lot of good things about goals. What are the limitations of goals?

Max: Some of you know the work of Ed Locke and Gary Latham and others who have written a lot about goals in the organizational behavior literature. I was at a conference a number of years ago, and somebody offered their work as a sort of testimony of a kind of great social science. I blurted out something that I think wasn’t all that nice, and I basically argued that viewing the goal-setting literature in organizational behavior as a positive literature is what happens when you focus on an inappropriately narrow set of objectives.

So, we know that if you’re doing a boring task in a laboratory and you give people moderately difficult, specific goals, you can do a pretty good job of increasing their performance. But Barry Staw also produced research that shows if you give people an incentive to increase the number of units that they produce, you’ll also get lower quality units. So people don’t perform all that well on the aspect that you don’t reward.

The other thing that Lisa Ordóñez and Maurice Schweitzer and Bambi Douma find is that if you give people moderately difficult goals that they have trouble achieving, there’s another way to get there. It’s called cheating.

So, moderately difficult, specific goals increase cheating, and for me I’m a pretty goal-driven guy. I work hard. I want to accomplish things, but I think that moderately difficult, specific goals, if we can’t capture the full array of what we want from the employee, are potentially dangerous because they tend to lead to underperforming on things that we forgot to put in the goals. And we also have the risk of an increase in cheating. And both of those effects turn out to be pretty well substantiated.

The paper we wrote is called Goals Gone Wild [laughter], and it got some visibility because goal-setting is a little bit like mom and apple pie. Who’s against goals? And the answer is I think that we should be very concerned about inappropriately narrow goals. If we go back to some of the sort of ethical issues in the investment world, you can easily imagine how bad things can accrue over time if we focus on some narrowly-defined goals. Thank you all very much for your input.

Michael: Thank you, Max.

Christopher Chabris Union College

Christopher Chabris is co-author (with Daniel Simons) of The Invisible Gorilla: and Other Ways Our Intuitions Deceive Us, published by Crown/Random House in May 2010. This book explains six “everyday illusions,” which are intuitive but mistaken beliefs that we all hold about how our minds work. It shows how these illusions work, how we can spot and avoid them in our lives, and how relying on our intuition is a perilous decision-making strategy in law, medicine, business, politics, and other areas. Chabris and Simons are the creators of the world-famous “gorilla experiment” that shows how much we are missing in the world around us. This experiment is one of the most widely-discussed and demonstrated in all of psychology, and was covered by The New York Times, The New Yorker, Dateline NBC, CBS Early Show, Scientific American, NPR, and the BBC, among other media outlets. It appears in textbooks and museum exhibits, and was even discussed by characters on an episode of CSI: Crime Scene Investigation.

Chabris is an Associate Professor in the Department of Psychology at Union College in Schenectady, New York.

He is also an Adjunct Assistant Professor of Neurology at Albany Medical College, a Research Economist at the National Bureau of Economic Research, and a Visiting Scholar at the MIT Center for Collective Intelligence in Cambridge, Massachusetts. Previously he was a Lecturer and Research Associate at Harvard University, and a Postdoctoral Fellow in Radiology at the Massachusetts General Hospital and Harvard Medical School. He received his PhD in Psychology and his AB in Computer Science, both from Harvard University. He also holds the title of National Master from the U.S. Chess Federation.

Chabris conducts research on a wide range of topics, focusing on two general themes: how and why people differ from one another, and how cognitive illusions affect our decisions. His scientific articles have appeared in leading journals such as Nature, Nature Neuroscience, Proceedings of the National Academy of Sciences, Psychological Science, Neuron, Cognitive Science, Perception, Journal of Risk and Uncertainty, and Social Neuroscience. He has also written for The Wall Street Journal and other publications.

Christopher Chabris Union College

Michael Mauboussin: I hope you all enjoyed lunch. It’s my pleasure to introduce our next speaker, Chris Chabris.

Even if you didn’t know Chris before today, it’s very likely that you have encountered his work through the now-famous “gorilla video.” This is probably the best known study demonstrating inattentional blindness—cases where you miss an obvious stimulus under certain conditions, absent any other cognitive impairment.

Chris works on a wide range of topics, but generally the work focuses on two themes: how people differ from one another and cognitive illusions. Over the years, I have had lots of great conversations with Chris, but I’ll mention three areas that I have found particularly interesting.

First is the work on attention. It is astounding how much we miss in our day to day lives, as the gorilla video shows, and recognition of our cognitive abilities and limitations is fascinating and somewhat scary.

Second, Chris has collaborated with other researchers on some very cool work on teams and team effectiveness.

Most of us work in teams, and you’ll hear today what makes for a smart team.

Finally, Chris is a master chess player, and the lessons from chess—everywhere from how the game has evolved to the lessons from freestyle chess—are relevant for the world of business and investing.

Please join me in welcoming Professor Chris Chabris.

Chris Chabris: Okay. Well, thank you for that introduction. Thanks all for being here and coming back from lunch. I'm a cognitive psychologist and neuroscientist so I'm acutely aware that the blood's in your stomach and not in your brain right now. So, let's start trying to get it back with a brief memory test. Do you remember the name of the famous movie star that Iris showed a picture of this morning in her talk?

Audience Member: It was George Clooney.

Chris: George Clooney, right. Do you remember what he was wearing? Anybody?

Audience Member: A beard.

Chris: Facial hair, a grey jacket, and a white shirt open at the neck. So, please generalize from that example to me, apply the halo as you see fit. I'm going to talk today about the illusion of attention and a little bit at the end about how to see what we're missing and maybe overcome the illusion of attention a little bit. Let's switch over to the first slide.

I thought I would start with a picture of the environment for attention in parts of the finance world. As I found this with a Google image search, I take it that this is a pretty standard depiction of a trading floor. Look at all the people talking. There are so many screens, there's so much information to take in.

Each person has four screens. They've got a few telephones. They've got a cell phone, they've got a speaker in the wall, they've got monitors hanging from the ceilings, they've got all kinds of information.

All of this is competing for our attention, obviously. And information by itself seems to be rewarding for human beings and even for other primates. There are some very clever studies done with monkeys where they record from neurons in the monkey's brain, record electrical activity from the neurons.

They find that the neurons that respond normally to pleasurable stimuli, like water when you're thirsty or food when you're hungry, also respond to merely learning small bits of information about what's going to happen next.

For example, if the monkey is going to receive either water or juice in a very short period of time, and they're going to get that water or juice no matter what happens or what they do, they will still exert effort to find out which one it's going to be, even though that gives them no control over what it's going to be. So information by itself, even for primates, can be very rewarding. And that's part of the problem for attention in this environment.

Christopher Chabris Union College

Now, of course, I would submit that this environment is designed under the assumption that multitasking works and that people can actually multitask and correctly select the information they want to pay attention to without significantly reducing their performance. And of course it's not just finance, right?

In the modern world of medicine there are probably about as many screens if not more here, all kinds of devices, things making all kinds of noises. There are people talking, maybe five, ten people in there at once. And then of course there's the person on the table himself or herself who needs some attention paid to them every once in a while during the process.

So, multitasking is everywhere. I don't think it's quite gotten this far yet. I think this was an unmanufactured prototype, I believe, maybe from the 1980s judging by the vintage of the electronics. But technology has caught up with our needs. I won't ask you to raise your hand if you've ever done this. So, what's the problem here?

Well, the problem I think is not of course that we don't have enough hands to multitask, right? The octopus solution wouldn’t work for human beings because the problem is we don't have enough brain. In essence, we don't have enough mental resources in our brain to handle multiple tasks at the same time or, even as was mentioned earlier, to rapidly switch between tasks without significantly reducing our performance.

And there are all kinds of reasons for that. Every time you switch tasks, there's a cost involved to reestablishing the mental state that's associated with the task you're switching to, and you have to switch back, and so on.

Many experiments show this.

You may have heard about a line of research that suggests that about 2.5 percent of people are so-called supertaskers, which means people who can actually switch between two tasks rapidly without performing any worse than they would if they did the two tasks sequentially. I fear, however, that about 50 or 80 percent of people think they're supertaskers when in fact it's only 2.5 percent, if even that many.

Now, we don't actually need to do experiments like that to see the difficulty in multitasking. We can see it in the world all around us. For example, this is a security camera video from a shopping mall. Watch the woman in the middle as she walks through the mall looking down at her phone. Okay.

So, one lesson of this video is that pratfalls are always good for waking people up after lunch. You'll see that the people in the control room now show us the reverse angle replay, kind of like it's ESPN or something like that.

So we can see what the other camera picked up of this action.

And you will see that she comes in from the right side on this next clip. Yep, she's going to come in from the right, helpfully indicated by the people filming this with their cell phone cameras. And she's going to still fall into the fountain. What's interesting about this view is that you can see what happens next. She quickly gets up out of the fountain, as we'll see—and it's extra poignant for having such a slow frame rate, it seems like inevitably time ticking forward—now she gets out and she walks into the store on the left.

And I used to wonder what store she immediately would walk into after having fallen into the fountain in the mall.

And then someone suggested to me maybe she's going to buy a new phone. Because obviously the phone was so important to her that she couldn’t be without it during that time, so she needs to get it replaced right away.

This is an example of the difficulty of multitasking. We have of course done research in cognitive psychology on this, as was mentioned or alluded to earlier. Dan Simons and I did an experiment about 15 years ago at Harvard University. Many of you may have seen this experiment or been a participant in it in some setting before. It was based on a study that had been done in the 1970s by Ulric Neisser, one of the founders of the entire field of cognitive psychology.

So I thought I would show you a version of this experiment. And what I want you to do is, even if you've done

Christopher Chabris Union College

this before, I want you to try to do it again to see if you can see the effect that I'm trying to point to. As you may be expecting, you're going to see two teams of people passing basketballs back and forth on the screen. One of them is wearing white shirts and one of them is wearing black shirts. Stop me if you've heard this joke before.

You're supposed to focus on the people passing the ball who are wearing white shirts and ignore the people wearing black shirts. And just count silently in your head—as Max said earlier, silence is enforced. Count silently in your head the number of times that the people in white shirts pass their ball. And then I will ask you afterward. So what this is trying to measure is how well you are able to focus attention on a task you have been given, in this case by me. It's not a very long video, but keep your attention sustained throughout, and count the number of people, the number of times that people wearing white pass the ball. Here we go.

[Video clip]

Okay. Can I have a volunteer? How many did you count?

Audience Member: Sixteen.

Chris: Okay. Raise your hand if you got 16. Okay. Anybody get a different number? It's okay to admit it.

Audience Member: Fourteen.

Chris: Fourteen? Okay. Anybody get 15? Fifteen? Okay. Seventeen? Okay. So it's about 14 to 16 depending on your count, right? But the real question is did you see anything else besides the people passing the balls around?

Audience Member: A gorilla.

Chris: Somebody said a gorilla. Okay. Raise your hand if you saw a gorilla come across. Okay. Now be honest.

Put your hands down. Raise your hand if you didn't see a gorilla come across. Be honest. Okay. So still, a few of you did not see the gorilla come across.

Now of those who raised your hands before because they saw the gorilla, all you people raise your hands again, the people who saw the gorilla. And now, keep them up if you knew to expect a gorilla.

Okay. There we go, look at that. So nobody really lowered their hand because everybody thought the guy's going to have a gorilla in the video, maybe even have seen that video before, and so on. So you knew to expect a gorilla.

For those of you who didn't see the gorilla by the way, you don't believe me, I'll show the video to you again in a second. But before that, did anybody notice anything else unusual happen besides a gorilla? Anybody? Yes?

Audience Member: Did somebody leave the stage?

Chris: Did somebody leave the stage? Yes. What kind of person left?

Audience Member: I think a person wearing black.

Chris: A person wearing black left. Okay. So, raise your hand if you honestly noticed a person wearing black left the stage. Okay. So if you look around you'll see maybe it's like five or ten percent of people noticed that. Anybody notice anything else?

Audience Member: The curtain changed color.

Chris: Yes. So, now, honestly raise your hand if you noticed that the curtain changed color halfway through.

Okay, so again, like five of you or something like that. For those of you who saw the gorilla and knew to expect the gorilla, I congratulate you because you successfully divided your attention. So, it is possible to sometimes

Christopher Chabris Union College

divide attention between two different tasks. Those two tasks were doing the counting thing that I told you to do and observing to see if a gorilla shows up.

But the fact that you were expecting something to happen didn't mean that you were generally able to expect the unexpected and see all the other very salient changes that we made. This video is a follow-on made by Dan Simons, my research collaborator, for a video that we made 15 years ago at Harvard where there was no curtain change and nobody left. It was just the gorilla. And that's the video we used for our original experiments.

For those of you who didn't see the gorilla or the curtain change or the person leaving, watch it again. You have to trust me that this is the same video that you saw. I can't actually show you what keys I pressed and so on. It's the same video.

Don’t count by the way just to make sure you see everything. When this has been done for TV shows like Dateline, they've often gotten people that accuse them of switching tapes and so on when they try to play it back and see what they missed. You can't just kind of expect the unexpected and think that you're going to see everything important that happens.

Logically speaking, you can really only expect what you expect. So, expecting the unexpected is a logically impossible thing to do. You can't just sort of be alert to anything that might happen. There's always some kind of filter and something that draws your attention more than others.

Interestingly, the slogan at the bottom of the notepads here says “Attention: Pay Now or Pay Later.” So if you counted accurately and missed the gorilla, you paid both times, right? You paid attention the first time and you succeeded at a task that was temporarily important to you, at least, going along with my instructions. But you paid later because you missed something.

If you were one of those people or if you ever have been caught out by this test, it's completely normal and okay. It's not an IQ test. We actually studied that. There's no correlation between IQ and noticing the gorilla. Although, everybody who notices likes to believe that there is. I won't tell on you if you go and use it on somebody else and say it's an IQ test or something like that. I'd say wait until you see how they do before you tell them whether it's an IQ test or not.

This video and many other experiments like it illustrate a concept called “inattentional blindness,” which was built upon the work of Ulric Neisser in the ’70s. But the term was actually coined by Arien Mack and Irvin Rock in the ’90s. And it's a very robust phenomenon.

For example, people have studied tasks similar to this video using eye trackers, which measure every place your eyes look on a screen while you're doing a task. And they have found that people who don't see the gorilla actually look at it for an average of one second. So, looking is not the same as seeing. You can look at something and not actually see it, which most people find to be a very sort of counterintuitive and paradoxical fact about how vision works.

Another thing that I think is interesting about this is that it does not appear like this phenomenon is an artifact of our sort of Western, technologically-oriented culture. We watch video from an early age. We have devices. We learn to use computers. There are a lot of stimuli competing for our attention.

So, perhaps this ability to pay attention and therefore have inattentional blindness for things you're not paying attention to, could be some kind of learned or acquired cognitive tic or something like that.

Well, I don't think that's actually the case. And one reason why I don't think so is that the publisher of the Korean edition of our book actually ran this experiment during halftime of an actual basketball game in Seoul, and had the video shown up on the jumbotron, and had everybody count and then raise their hands if they saw

Christopher Chabris Union College

the gorilla and so on. These are the people actually watching the video. You of course see that those two people in the middle there are not going to notice any gorillas or anything else because they're distracted by something else. But in fact, about 50 percent of people failed to see the gorilla in the original video in Korea, just the same results we had with Harvard undergraduates who were our original subjects.

We've taken this even further in more recent research. Here in the middle in the white cap is my collaborator, Corinne Apachelo, who's in the psychology department at Penn. And she goes periodically every couple of years to Tanzania and visits the Hadza people, who are a hunter-gatherer society thought to be the closest extant human population to human ancestral populations of Africa. And they don't have formal education. They don't even have permanent homes. They don't have much technology at all, although that's changing over time, of course.

But we got the exact same results with them. About 50 percent of them don't notice the gorilla. And also, more of them missed the gorilla when they are paying attention to the white-shirted people than when they are paying attention to the black-shirted people.

And the reason for that is that what you pay attention to influences what you notice. If I had asked you to pay attention to the people wearing black shirts, you would have been much more likely to see the gorilla because the gorilla shares the predominant color with the people wearing black shirts.

We have now tested this with the Shuar people in Ecuador, the Hadza people in Tanzania, villagers in India and Uganda, Zulu farmers in South Africa, and we're working to test other populations around the world.

And, so far, we think this is a universal aspect of human cognitive architecture. It's not some kind of artifact of Western technology. It seems to have fairly deep roots. So, using attention for one task or type of visual stimulation just takes it away from other things. That's just the brain we're stuck with.

So, that's inattentional blindness. The “illusion of attention” that the talk is titled after relates to this, but it's not the same. The illusion of attention is the mistaken belief we intuitively have that we pay attention to and notice much more of what's going on around us than we actually do. Inattentional blindness wouldn't be so bad if everyone were completely aware of it and acted accordingly and weren't surprised when they missed things like the gorilla and so on.

But one reason why this experiment has become popular is because people are surprised that they missed that thing. And that in fact reveals sort of the deeper problem—that this is a counterintuitive thing we're not aware of, and have difficulty taking account of, in our everyday behavior.

Every time, for example, you use your phone while you're driving, you are subject to the illusion of attention because you are implicitly acting on the belief that using your phone does not subtract significant cognitive resources that you might want to use for the primary driving task, which after all ought to be the primary task.

Nothing you can do on your phone is probably going to bring you as much risk, or risk to other people, as what you can do behind the wheel. Well, maybe not all the time, but generally speaking, the car is more dangerous than the phone.

In fact, colleagues of ours have found that when they surveyed people who don't know about this gorilla experiment and tell them about it and say, "Imagine an experiment like this, there are these people and this gorilla and so on. Would you notice the gorilla?" Ninety percent or more of people say yes they would notice it, whereas in fact only 50 percent, depending on how you run the experiment, actually do.

That's the illusion of attention. And I call it an illusion to make an analogy with visual illusions like the checkerboard illusion that Iris showed you this morning. That's a classic visual illusion. The world doesn't look to us the way it really is in the sense that the light coming off of those two squares is the same color light, but it

Christopher Chabris Union College

looks different to us. Our brain corrects because we know that it's a checkerboard. So, it actually makes the light look different. That's a true visual illusion. And even once you know how it works, it still looks like a checkerboard to you. You can't go back into the low-level visual areas of your brain and say, "Wait a minute.

Don't see that as white anymore. See it as gray." It just doesn't work that way.

Likewise, this illusion—the famous monsters in the tunnel illusion—which monster looks bigger? You said the one in the back, right? Okay. First of all there is no back. That's a flat screen up there so the back itself is an illusion. I tricked you. There are two illusions going on here. That's an illusion of depth. This is what the artists discovered in the Renaissance. That if you make a bunch of lines that all converge on one point, then we will see that as being farther away. And that creates the illusion of depth. So depth is an illusion here too.

But, also, the size of the monsters is sort of an illusion because they're actually the same physical size. I drew parallel lines connecting the same points on the two. And the objects on the screen are the same size, but our brain corrects for that and assumes that the one that looks like it's in the back is actually bigger because it's filling the entire height of the tunnel, unlike the first one.

So you can't really make yourself not see that. You can do things like sort of squint and try to focus on different parts of it and so on, but as soon as it flashes up again it looks like one of them is big and one of them is small because that's the best interpretation of the world you're looking at.

Likewise, illusions like the illusion of attention I don't think go away just from knowing about them. It's not like we've broken the spell or something like that. You may even get in your car on the way home today, pull out your phone and start texting or looking at Facebook or whatever, whatever you guys like to do. And I want to reinforce again the point that these illusions are not born of ignorance or stupidity.

So, people who are subject to illusions like the illusion of attention are not like this guy who just didn't know what surge protectors were for. This is pure lack of knowledge. Actually, he's very surprised that it turns out that it's electrical current that surge protectors deal with. And you can find this of course on the internet because this was the first question, and he went away with $0. So he's sadly been immortalized for that.

Cognitive illusions like the illusion of attention are not like this. They're not things you can correct merely with knowledge of facts. The illusion of attention I think is also very related to the power of expectations. The way we think about expectations is often we think that things that we don't expect will grab our attention.

And in fact when we did a survey, a telephone survey of a representative sample of the U.S. population using a polling firm, we actually asked that question. And over 75 percent of people said, "Yes, if unexpected things happen in my field of view, it will grab my attention, and I will notice it."

But that's actually kind of backwards because expectations actually tend to determine what we see and what we miss, not what will grab our attention. So my favorite example of this comes from the following video. Just watch what the team in red does in this high school football play.

[Video clip]

An interesting thing about this video is of course nobody touched him until it was his own players mobbing him in the end zone. I noticed some of you applauding and laughing and so on. This, of course, the technical term for what you just saw is a trick play. So a trick play is a play that's not illegal. It technically conforms to all the rules or else it would just be an illegal play, but what it violates is all the expectations of the other team.

So, the expectations that were in the heads of the defending team determined how they interpreted what they saw or even what they paid attention to. Perhaps they didn't even notice the quarterback receiving the ball from the side of the center and just strolling through the line because they weren't paying attention for that kind of

Christopher Chabris Union College

thing.

Now, it's more than high school football where this can be a problem. Expectations can determine what we see in much more serious situations. For example, this lengthy title says, “Profiles in patient safety: Misplaced femoral line guidewire and multiple failures to detect the foreign body on chest radiography.” What that means is a patient came into the emergency room in Rochester, and the patient had to have a central line put in because they couldn’t get an IV in. And they needed a central line. And then they were admitted to the hospital.

And the central line guidewire was left inside their body by accident and was observed on several consecutive CT scans and X-ray scans . . . or maybe I should correct myself. It wasn't observed. It was visible on several CT scans and X-ray scans that were read by numerous radiologists but not actually seen until the patient had been there for several days.

Why was that? Well, when the radiologists were tasked with looking at these scans, they were not told look for anything you see that might be unusual. They were given a potential diagnosis. There was a reason why the woman was in the hospital. She was in the hospital because she was having problems with her lungs, and they were looking for issues with the lungs—pulmonary embolism and so on—and did not see the guidewire. This process of sort of stopping the search once you've found what you're looking for is called “satisfaction of search” in the cognitive psychology literature.

Satisfaction of search is when you have an attention-demanding search task, and you complete it, and then you stop without looking to see if there's anything else that would satisfy your search. So, if there are three things that satisfy the criteria you're looking for, you will often stop when you found one and not even realize that there were more going on, or that there were other unexpected things happening that you maybe should have paid attention to as well.

Now, Max showed you earlier the video of the registration counter. As he said, that was based on an experiment that Dan Simons and I did with a couple of colleagues at Harvard earlier. And I'm going to show you the predecessor of that experiment now. You can just watch and see how it works.

One thing I'd like to add to what Max said—the question was raised afterwards about whether it’s just academics who make these kinds of mistakes and so on—well no, we've done them with lots of different people over the years, old and young, students and non-students.

Dateline NBC has run that experiment, and it works perfectly well. Dateline NBC actually ran it twice. They ran it once in a coffee shop in New York City where someone ducked behind the desk at the coffee shop and said let me get you your sheets and so on. It works fine.

The most interesting result I think was that inadvertently when we ran the experiment at Harvard one time, several of the subjects had come directly from a lecture on inattentional blindness and change blindness and saw a sign saying be in an experiment, go up to the eighth floor. And they went up to the eighth floor, and they made exactly that mistake of not realizing that the person talking to them had changed into someone else.

So, here's the original version of that experiment. This was done by Dan Simons and Dan Levin. The guy in the center is the subject, although he doesn't know that yet. He's just been approached in the quad at Cornell and was asked for directions to a building, and watch what happens. He starts giving directions. The guy at the left is the experimenter.

[Video clip]

And you might call this now “the moving counter.” But this was the way it started, with a door being carried by people. Now it's a completely different person talking to him. Incidentally, that's Dan Simons now talking, my co-

Christopher Chabris Union College

author and the co-author of this paper. And the interaction goes on just as it would have otherwise. And in fact, this gentleman did not notice at all that he was talking to a completely different person after the door went through. And about 50 percent of people in this experiment didn't notice that.

I don't have time to show you the video, but it's not an artifact of this gentleman being older. They ran another very, even more clever experiment where they approached students on the quad to do the same thing, but the experimenters disguised themselves as construction workers. So, they were still the same two people. They looked like different people—they had different heights, different sounding voices, and so on, but they were wearing construction worker-appropriate clothing.

The idea is that in everyday situations like this, you may only really be storing a few bits of information about the world around you. You have a rich perceptual experience of the world around you. You probably feel like you're processing a lot of details of my face and my voice and you're hearing me say a lot of words, and you feel like you're processing all those words. Sad to say, you're not retaining a whole lot of the information that comes in from the outside world and probably even less than you think.

So it may be rational in some sense to not pay attention to the exact details of the particular random student you're talking to or giving directions to. Maybe all that's important is are they male or female, young or old, and so on. And it’s interesting that in Max's video they managed to change a man into a woman because I don't think that's ever been done in an actual academic study.

So, there could be some blindnesses that academics are especially prone to. Maybe it's changing men into women or something like that. That's why I asked him afterwards if he has data because I would really like to see what actually happened with that.

This phenomenon, as Max mentioned, is called change blindness. And I think this happens a lot in the world around us also. We just don't realize it. Here's an example that I found in the news. Missing woman unwittingly joins search party looking for herself.

Now, how could this happen? We can find the answer in the third paragraph: “One of the women on the bus left to change her clothes and freshen up” when the bus stopped for a break. “When she came back, her busmates didn't recognize her” because she was wearing different clothes. So, soon there was talk of a missing passenger, and she didn't realize that they were talking about her. So, she started joining the search for herself.

If you think this is an isolated event, the related stories on the very same webpage include a missing toddler found in her bed. No, the toddler was not out on the search party, but the toddler was apparently right where the toddler should have been and yet people didn't notice that.

Now, we're gradually going to get into questions of change blindness and inattentional blindness that I think sort of focus more on the environments that may be of more direct relevance for you. And I'm going to try to get there a little bit by now showing you some other examples of change blindness that happen even when you know that something is changing.

So, in the counter example in the prank that Max's friends played on him, none of those people knew that something was going to change. But I'm going to tell you right now that something's going to change and hopefully show you how hard it is to see what's changing.

Here's a scene of soldiers boarding a plane, and what you're going to see happen—and this is done by Ron Rensink, by the way, who's a professor at University of British Columbia—what you're going to see happen is the screen is going to start to flash. And each time the screen flashes, the picture is going to change. And, in fact, it's going to change back and forth between two versions of the same picture.

Christopher Chabris Union College

All you have to do is tell me what's changing during the flashes. And I don't want you to shout it out. Just raise your hand when you're sure you've figured it out. And we'll see how long it takes you all to raise your hands.

Here comes the flickering.

[Video clip]

We've got three so far. Again, silence is enforced. Six, seven, eight, nine, ten, eleven, twelve, thirteen. I'm not keeping time but 15 seconds have elapsed already and still less than half of you have seen what's changing.

Okay. We're getting closer to half. Some of you still have not noticed it. I'm not going to use up all the rest of my time waiting for everybody to raise their hand.

So, please pay attention to where the engine on the wing of the plane is. Look at that. As soon as you paid attention to the right thing, it immediately became apparent. Here's a case of what's sometimes called “explicit change detection” in the cognitive psychology literature. You know something is changing and yet you still have great trouble seeing it.

Why is that? Well, of course, in the real world the world isn't flashing at you all the time with things changing back and forth. What does the flash do? The flash in essence sort of destroys your short-term representation of what you were looking at and you've got to start all over again. Basically your attention is sort of re-diverted every half second or so to something new. And it's very hard to systematically deploy attention over a stimulus like that that's constantly changing.

Now, you don't often see data like this, but I would submit to you that it’s a little bit like a situation of multitasking where you, for your own reasons, may switch your attention away from something important and then switch it back. And it's as though you're starting all over again often when you keep switching back and forth between multiple tasks.

Here's another example. Something is going to change now and I want you to raise your hand when you see it.

Whoa, suddenly a whole bunch of hands went up. Okay, so about half of you have noticed it. What changed?

Okay. Two buildings disappeared. This is what happened. Except, very slowly. So, Photoshop—we morph them so that they change very slowly and in this case disappear. Let's try one more example. What's changing? Raise your hand when you can tell. Okay, about a third of you, maybe half of you this time. So here's what it was.

These are fairly salient changes. These are not like a few pixels here or there or something up in the corner that's irrelevant. Two buildings in the Chicago skyline disappeared, three paintings on Van Gogh's wall disappeared. I should say credit for these videos is to Jackie Mandart, my former research assistant who made them.

These are examples of gradual changes. And what the literature on change blindness shows us is it's especially hard to notice slow changes while they're in progress. And here's where I think the issue becomes sort of more clear for decision making and being smart about attention. In the real world, when you have to see something change, you usually have to watch it pretty closely.

That's the case in the case of vision, and I think it's also the case in a lot of other more abstract conceptual domains. So, if your attention is elsewhere, you won't see what's changing.

For example, in this article that I got from the New York Times from a few years ago—lots of quotes. You may not know that this is a bull market “from financial headlines that have often focused on weakness in the economy and on the myriad crises in Europe, the Middle East and Washington.” So we saw from Max's graph that from the trough there was sort of a very steady run-up as run-ups go. Not a headline-grabbing kind of thing, but where there were a lot of other things going on that drew people's attention.

Christopher Chabris Union College

I think this relates also to the “topic of the day” issue that Michael brought up at the beginning. The daily topic draws our attention, right? A good rule of thumb to follow, I think, is that if someone ever says, “Today is all about X,” there's a pretty good chance you should be paying attention to something other than X. Or if someone says, “A all comes down to B,” well then there's some chance that A's related to B but you better not deploy all your attention looking for the relationship between A and B because you're going to be missing a lot of other important things that are going on besides how A relates to B. That kind of language I think mostly distracts our attention.

Another example. People used to say things like this. Nobody says it anymore, but people used to say that they sort of suddenly woke up to the idea that China was a powerful economic force. For example, in 2010 they were already the second largest economy ranked by GDP, according to the IMF. So I actually looked at this and I looked back at the chart—I just got it all off of Wikipedia, it's very easy to find—of where China ranked in previous years at five-year intervals.

It turns out it's about the smoothest rise through the rank tables that you can imagine, but people sort of act as though it was like that. Like as if all of a sudden Mexico became the second largest economy or something like that overnight, and we just suddenly realized it. Well, this I would submit is a form of change blindness but at a much more long-term and abstract level. I think this is one reason why we don't notice things like slow changes in financial indicators and other important facts about the world.

Now, I want to get a little bit philosophical here briefly. Well, I want to talk about something that in artificial intelligence used to be called “the frame problem.” Suppose you're a robot. Temporarily imagine you're a robot.

Suppose you walk into a room and you look around. And what you see with your vision system in your robot brain is that there's a chess set on the table. I've got to work in chess somewhere there. There's a chess set on the table. So, being a smart robot with good AI algorithms, you enter the fact into your database that says I can play chess in this room. Now you're aware that you can play chess in that room, and you can use that in your reasoning process going forward.

Now, suppose a human walks in, and perhaps out of fear that the computer will crush it in chess and the desire to protect its fragile human ego, it picks up the chess set and leaves. Well now, if you're the robot, how do you know that the fact I can play chess here is no longer a valid statement about the world? That used to be considered a very difficult problem for artificial intelligence. Basically to get computers to represent the relationship between things they believe and the information that causes them to believe those things. And how do I know when to delete beliefs when the information no longer supports them?

It used to be thought that this was a significant problem for AI in part because it was assumed that humans did this very well. But I think change blindness and inattentional blindness actually show us that often humans don't even notice the information that's changing that should affect their beliefs. So, any conclusions drawn from those facts might not any longer be true after time passes or something happens. And the problem is how to update your database of things you think are true about the world when some of those things change.

I think the relevance of this is that we often form beliefs that can animate a lot of actions we take. It might be a view on a particular company or industry or economy or currency or whatever. How often do we write down or formalize in some way the reasons for our beliefs or what would have to change in order for those beliefs to change? How often is that formally represented?

I don't think it actually is very often. At least I never do, but I'm not a professional in this field. But I have not heard too many stories of this happening. I think if it's something that does happen I would love to hear more about it, but it seems to me like something that should happen more.

Christopher Chabris Union College

There's another further problem in our knowledge base, and it’s that we think it's more complete than it really is.

So, we think we know more about how the world works than we really do. I can't prove to you anything about your knowledge about markets or currencies or economies or anything like that, but I can show you that most people don't understand things that are vastly simpler than financial markets and complex organizations.

For example, almost nobody knows how a toilet works. Everybody knows how to work a toilet but very few people know how a toilet actually works. I won't challenge you to explain how a toilet works. There's probably someone here who could explain that very well. But what about this? How well do you understand how a bicycle works?

So, Rebecca Lawson, a British psychologist did an experiment where she asked subjects this question and she had them rate their own knowledge on a scale of one to seven. So, one meant I have no idea how a bicycle works; seven meant I have a very good understanding of how bicycles work.

And then she surprised them with the follow-up question: draw a bicycle being sure to show the wheels, pedals, and chain. And she actually gave them a head start by drawing the frame of the bicycle. So all they had to do was put in the chain, the pedals, and the wheels. And here's some of what she found. These are actual drawings by her subjects as reproduced in her journal article.

I would like to observe a couple of things. First of all, in all four of these cases if you look closely, you will see that the chain connects the two wheels. If you can visualize yourself trying to ride a bike where the chain connects the two wheels, as soon as you turn left, the chain is going to crumple and the whole thing is going to fall over and I'd be down there right on the floor.

Also interestingly, the hammock pedal up there where the pedal is not connected to anything that could actually drive anything. It's just kind of like a little spinning child's toy in the middle of the bike or something like that. There are various interesting features. The pedal on the front wheel is especially good. That'll break your shin very quickly. Interestingly, drawing C was made by someone who rode her bike to and from work every day.

So, I think the problem with our sort of attention to knowledge in the world around us is that we assume our knowledge is actually greater than it is. This relates to the well-known phenomenon, “unskilled and unaware of it,” which was coined by Justin Kruger and David Dunning. In fact, this is called the “Dunning-Kruger effect” sometimes. You may have heard of it under that name.

Dan Simons and I, and our collaborator, Dan Benjamin, did an experiment on this. And one thing we were worried about in demonstrations about people's overconfidence about their own knowledge is that often the knowledge is random trivia facts in the experiments that are used, sort of “how long is the Nile River” or something like that.

Or in the one that Michael mentioned earlier, a whole bunch of questions about factual knowledge. Maybe people don't really have such a great idea of what their factual knowledge is. But here is a domain where people do really get very precise information about how good they are at something. Even in the field of finance, it's hard to know how good someone really is as an investor.

But in chess, everybody does know exactly how good they are. Chess has a rating system. The rating system is actually the most accurate rating system that I'm aware of for any kind of game or sport. It's designed by a guy named Arpad Elo, and it basically calibrates very well to people's skill. And everybody knows their rating and they follow their rating and so on.

So we went to a couple of big chess tournaments, and we asked two simple questions of the players. The first question was, “What's your rating right now?” And it turned out when we looked at this and we looked up their ratings after they told us—because they gave us their ID numbers and everything too—that everybody pretty

Christopher Chabris Union College

much knew their rating. It within one or two points out of 3,000, which is awfully close.

Then we said, “What should your rating be to reflect how good you really are at chess?” Now, the correct answer, sad to say, is “Exactly what my current rating is,” or at least that's the most rational answer.

However what we got looked like this. On the x-axis is their actual chess rating plotted against, on the y-axis, what they think their rating should be to reflect how good they actually are. And you'll notice that almost every dot is above the 45-degree line.

So, almost everyone in the sample thinks they're better at chess then they actually are. And this is in a domain where they get perfect feedback or near perfect feedback about how good they are. That poor guy over there, the outlier to the right, I don't know what's wrong with that guy, severe lack of confidence or something like that.

But otherwise, another interesting thing to note is that if you go up and to the right on the actual chess rating scale—this is up to Grandmaster level over here—those people up there are not too far above the line. But if you go over here to the weaker players—I mean and these are very competent tournament players, but they're still not as strong as the Masters and Grandmasters—their dots are much higher above the line.

So, in fact, the weaker you are as a player, the more you overestimate your skill. I can obviously sense some analogs in the world of stock market investing and so on here. But it's hard to measure actual skill there. Here we have an actual measure of skill.

Okay. So, moving along, I have to talk briefly about Oprah Winfrey because everybody has to have a celebrity in their slides apparently. We're often told that the solution to all this complexity of the world, and what to pay attention to and what grabs our attention and whether our knowledge is adequate and so on, is just not to worry about it. Instead, rely on your intuition.

And unfortunately, we really do believe that this works for ourselves, but even more so we often tend to believe it works for other people. We have this sort of belief that there are special people who really do have special intuition who can do things that would seem supernatural to the rest of us.

So Oprah advised, in her magazine: “Learning to trust your instincts, using your intuitive sense of what's best for you, is paramount for any lasting success. I've trusted the still, small voice of intuition my entire life. And the only time I've made mistakes is when I didn't listen.” So, don't take Oprah's word for it. On her website, she published a list of great moments in intuition. A timeline. I'm going to share two of them with you right now.

The first one was in 1940 when Winston Churchill was dining at 10 Downing Street and a German bomb hit nearby. He orders his staff to leave the kitchen. Moments later another bomb falls obliterating, you guessed it, the kitchen. Okay. Brilliant intuition, went with his instincts, saved the staff of the kitchen of 10 Downing Street.

Or, what about this one, also from Oprah's timeline. In 1961, Ray Kroc’s funny bone instinct—and this is much more relevant for investing—becomes the stuff of business legend when he ignores his lawyer's advice and borrows $2.7 million to buy out the modest fast food franchise he helped build. Now, more than 47 million people a day eat at McDonald's, whenever this was done.

So I think you will find the problem with this line of argument pretty obvious. It's a little bit of a Carter Racing problem. It's very obvious that you may wonder how it's possible to even miss this, yet there's business book after business book written exactly on this premise of telling us about great success stories, picking out great success stories, and retrospectively looking at what they have in common.

In this case, it's they ignored advice and went with their instinct. And there are no books, unfortunately, or pages on Oprah's website about all the people who rejected the legal and financial advice they got and borrowed $2.7 million and went into business and lost it all. None of those people are memorialized with pages on Oprah's

Christopher Chabris Union College

website or stocks in the Dow Jones Industrial Average or wherever McDonald's is these days.

In fact, I think that happens a lot more often, actually. I have a feeling there are more people who reject a lawyer's advice, borrow a large sum of money, and lose it than people who reject a lawyer's advice, borrow a large sum of money, and become billionaires. I don't have the data on that, but I'm guessing that that's the case.

This is the same error that happens when you're thinking about an old friend or something like that and then suddenly an email comes right from that person. Wow, what a coincidence. People attach great significance to those salient coincidences, but we never think backwards about all the times we thought about our friends and they didn't email us or call us right then. Or all the times that our friends surprised us with an email and we weren't thinking about them. Or, of course, all the times we weren't thinking about our friends and they didn't email us, which is 99 percent of life.

These are not great moments in intuition. They're great moments in what cognitive psychologists call “illusory correlation,” and they're another example of how we can pay attention to the wrong things often to our detriment. Very similar to the Carter Racing case of not thinking about the information you don't have and asking for it even after Max reminds you three times that you can go ahead and ask for that information. Nobody went to Oprah's website probably and typed in, “What about all the people who lost money?”

Again, I feel like everything in my talk has been alluded to earlier, but I've got to mention now Mr. Madoff because I think it's very relevant to how believing in the power of intuition can get us into big trouble. We saw that the returns were very smooth and upward-trending and so on for a long period of time. And you all know all about this so I'm not telling you anything you don't know. But a lot of people think to this day that the SEC never looked into Madoff. But in fact, as we all know, they looked into him on multiple occasions.

And the Inspector General's report after the fact tried to answer the question: How did they not see what was going on? And of course, there were many reasons. But the most interesting reason to me is described in the following quote from the Inspector General. He wrote in one of the examinations that the SEC conducted that they asked Madoff how he was able to achieve these returns. Madoff's returns were extraordinary and always consistent. He never had a down period no matter how bad the market, as we saw in the graph.

And they asked him: “How are you able to do this?” Bernie Madoff said to them that he had a "gut feel" of how to time the market. They asked him what the basis was for this gut feel, and he said it was his observations of the trading room. He said he could actually sit on the trading room floor and “feel the market” and know exactly when to buy and exactly when to sell. And Madoff always bought at the right price and always sold at the right price day after day, year after year.

The most important sentence in this quote that I'm reading you is the next one: “Inexplicably, the SEC just accepted these answers and went on.” I would submit to you that this sort of false belief in the power of intuition, or the idea that there are special people who can—even if I can't do it, there are special people who can— access these feelings and this information in an unconscious way and achieve otherwise mathematically impossible levels of success, can do us real harm. It really distracts us from what we should be paying attention to.

Now, there have been no cartoons up until now except for those little stick figures in the first talk, so I'm going to give you The Simpsons to help me drive this home.

[Video clip]

Now, again, I won't try to explicitly highlight the relevance of this to what goes on in business. But I think most of us most of the time are like Homer Simpson. We kind of grope around in a complex world. We see these cause and effect relationships and draw simple inferences about how things work. And based on what we pay attention

Christopher Chabris Union College

to, namely, we see the bear patrol going around. We don't see any bears. Of course, some people then have the insight that they can monetize that kind of thinking, and there we see it right there in Lisa. We see this kind of thinking all the time. “It is inarguable, though, that Geithner’s stabilization plan has proved more effective than many observers expected, this one included.”

Well, of course, there's no proof that anything was effective. All we know is the way things were beforehand and the way things were afterhand. And we like the way things were after better than we like the way they were before. Now it could be true that the stabilization plan had a lot to do with it. You guys would be the experts on that, not me. I would never say that expert opinion isn't relevant.

But in many cases we can't run the right experiments or analysis to really answer these questions. And at least we should be aware of that. We should be aware of what our knowledge is based on when it comes from intuition and expertise and not from actual data. It's inarguable that Homer's bear patrol has proved more effective. It wouldn’t really make any sense in that way.

So I'm going to skip the next couple of slides. I've got to tell you of course it seems like from what I've been discussing you've already all read Danny Kahneman's book, but if you haven't you should really get it and read it: Thinking, Fast and Slow. Danny Kahneman is in fact not just a Nobel Prize winner in economics, but he's the second most influential psychologist ever after Freud. And Kahneman's influence has been positive.

So there's a lot to him. And one of his best sayings from that book is that the human mind operates on the principle—especially when we're thinking intuitively—that he calls “WYSIATI.”

WYSIATI stands for what you see is all there is. So, the assumption we often make is that what we see before us, in other words, what our attention has been drawn to, is all that's relevant. I mean, think of the Carter Racing example again. You're given this chart of numbers. You really focus on that chart of numbers. You do a lot of analysis on that chart of numbers, not thinking about the information that you weren't given. If you operate on the assumption of what you see is all there is, you can make significant mistakes.

So, trying to gain some immortality myself, I decided to coin WYMIJAI. Try to use it in your everyday conversation as much as you can: what you miss is just as important. I'm going to finish up the talk now, thankfully, with a few tips on seeing what you're missing.

So, in fact, Dan Simons and I wrote a book called The Invisible Gorilla, which we published, and now we're thinking about a sequel called Seeing What You're Missing. So, this is maybe a little bit of a preview of what we've been thinking about.

One, I alluded to earlier—there might be value in formalizing and recording the web of assumptions behind your thinking. This is not often done but I think it can actually have value. One thing you might do is create a checklist of things you might want to pay attention to in the future to see if they've changed.

Checklists are a great way to externalize processes of memory and cognitive control that are really hard to keep going in your brain at all times, especially with all those distractions we've been going through the entire talk so

far. So, keeping a checklist that could be derived maybe from some kind of formal explicit reasoning about the reasons for your thoughts may also help people from, in the future, changing the reasons once the thoughts turn out to be unpleasant or incorrect. Then you can just go back and change the reasons. Better not to do that.

Second, as you saw in the gorilla analogy, it's kind of hard to divide your attention between, on the one hand, a focused task like counting the number of passes as I asked you to do, and on the other hand, general awareness of things that might be changing. Change detection is hard enough even when we're looking for it, but just being generally aware of what might be changing, like the curtain in the background or the person walking off the stage, is hard.

Christopher Chabris Union College

So, a good strategy to do this is to divide attention between multiple people actually. Use two brains or more. One person or group can focus on a specific task. Another can focus on the broader picture. But not both.

Incidentally, police officers have often adopted this tactic when they go to an unknown situation and there's a subject that they're supposed to be investigating.

Often, what they will do is one officer will approach the subject while the other one stays farther back, because he knows that if they both get close to the subject they're both going to focus on him and what's going on right there. And if there are other people around or other things happening, they're not going to notice. So that's a good practice, I think, to adopt more broadly.

The third one is to “red team” important decisions or projects. If you have the resources—since groups tend to focus so explicitly and doggedly on the tasks they're given— it's a good idea to give different tasks to different groups and have some people try to do the opposite of what the first group is doing, or try to disprove the conjecture of the first group.

My favorite example of this is the Macintosh computer. The Macintosh computer was in essence a red team that was trying to show that the Lisa and the rest of Apple's product strategies sucked and they could do better. And now they're the basis for the entire company going forward since then.

And then not quite finally, but finally for this slide: include women on any decision-making team because teams with more women are smarter. And I say that deliberately. I deliberately use the word smarter because of another line of my research that Michael alluded to, and Iris also—we've been essentially trying to develop an IQ test for groups.

Often nowadays, major decisions, even minor decisions, are made by groups. And when groups of people work together, we can think of them in some ways like a single person—just a mind composed of multiple different minds. And it turns out that groups vary quite a bit in how intelligent they are.

So we did develop an IQ test for groups. This means the group works together to answer the questions—we’re not giving IQ tests to the individual people in the group. And one of the things we found is it's good to have smart people in your group. But it's also good to see how the group works together. So, groups that are smarter, meaning they make better decisions and solve more problems when working together, tended to have more women.

Here's a graph to show you that. The groups that we were using were between two and five people. So, the percentages on the x-axis are only the possible percentages that you can divide two, three, four, or five people into. The pattern is a little bit messy as you can see, but the only significant fit to this is a linear increase with the number of women.

Some people like to say, “Wait a minute, this point is lower over here when you've got 100 percent women.”

True, but not significant in our data so far. So, it could be that exclusive, all-female teams maybe are not the best, but certainly it seems like up until that point, adding women makes teams smarter. That's of course all else being equal. So this does not mean ignore expertise, ignore intelligence, ignore other issues of fit and background, and so on. But it does mean that it seems like teams with more women make better decisions.

And by the way, we replicated this finding many times, including teams that work exclusively online. So, we have tested people where they're only communicating through chat. The effect of women is still significant. And also in Germany and Japan we also got significant effect too.

Now, one reason why it's good to have more women on your team is that women tend to score higher on tests of social intelligence or social perception, and we used that also. We found that it's good to have people on your team who have greater social intelligence, probably because they are better at thinking about what the other

Christopher Chabris Union College

people on the team are thinking about and keeping track of what the other people know and don't know and how they think. And also perhaps slightly better at decoding non-verbal information. So, more information can be exchanged in a group faster.

If I think Michael just said something stupid, and we're working in a group together, I can just raise my eyebrows at you and eye roll and then we know that what he said is nonsense. I don't have to say, "Hey, Michael just said a really stupid thing, didn't he?" So, that saves a lot of time and feelings and so on if we can communicate better non-verbally in a group. If you want to read more about this research, the easy way is we wrote an op-ed in the New York Times. And then you can link from there to the other papers that we wrote.

I want to cram a couple more little facts in here. One is that cognitive diversity is also important, as was mentioned earlier, and we're still working on that in our research group. And finally, two bonus ways to see what you're missing.

Remove attention-grabbing stimuli. So what does that mean? Of course, get in the shower. So when you're in the shower or doing something else, you do not have stimuli competing with your attention. Does anybody ever survey people about how happy they are in the shower? Because I would put that at almost the highest level for

me. Literally. Being in the shower. And maybe it's tied to the extra creativity that comes from being in the shower.

It turns out there are products that can help you. If you feel like you get your best ideas in the shower, you can write them down in the shower now with waterproof notepads. In fact, a student of mine actually did a survey of this. We found that 33 percent of people do believe they get their very best ideas in the shower.

And, for example, one of them typed into our survey two years ago: “I decided to give my husband of 37 years the divorce he wanted. I didn't want to get a divorce, but while showering it occurred to me that it was futile to try to force him to stay.” So it takes all kinds of creativity I guess.

And then finally, direct attention to information that breaks illusory correlations. Illusory correlations are all around

us. Often, paying attention to failure I think is one way to get at that. And my favorite example of that is the Bessemer anti-portfolio. How many of you are familiar with the anti-portfolio? Well, if you go to their website, they have a page which lists all the investments they turned down that would have been extremely profitable.

For example, eBay: "Stamps? Coins? Comic books? You've GOT to be kidding…No-brainer pass." FedEx:

“BVP passed on Federal Express seven times.” Of course, this is a little bit of marketing. This makes them look cool and so on, but they are in a way memorializing things that might have been mistakes so they don't forget about those things and only focus on successes. They still have 115.5 IPOs, apparently, despite all that bad decision making.

So I guess I'll conclude here. Woody Allen in his famous standup comedy routine said, “I would like to end with an affirmative message to share with you, but I don't have one.” So, would you settle for two negative messages? So, there's a lot of negativity here. It's an illusion, attention is an illusion, intuition is a myth, we don't know as much as we think we do, and so on. But I think there are some ways we can try to work on that. And I'd really love to talk about that or anything else with all of you now. Thanks a lot.

Question: Chris, in that study showing that a group with more women does better, it didn't look monotonic. It looked like it maxed around 60, 70. So, granted, maybe women are slightly better at this. Was that more of the diversification effect? One hundred wasn't the right number or is that not significant?

Chris: Well, if you try a curvilinear fit, like if you hypothesize that it's best at the midpoint and lowest at the endpoints, that does not explain as much variance as a simple linear fit. Now, maybe there's something much more complicated going on but we don't really have the data to say.

Christopher Chabris Union College

So, I don't think I'm ready to say we have proved that the best thing to do is to fire all your men and put all women on your team or something like that. Obviously, that would be a premature conclusion, not to mention the fact that this is holding all else constant and this is in a laboratory experiment. Those caveats aside, I just don't think we have the data to tell apart those two hypotheses.

I do agree that it seems logical that you wouldn’t want to max out either way. However, I also do believe that cognitive diversity and other kinds of diversity are probably more important for effective problem solving than demographic diversity. I don't have any data on that. Iris would probably know better than I would.

It's kind of hard to measure cognitive diversity. There are very simple measures of it. We're trying to do more sophisticated things like measure thinking styles. Do people like to think verbally or visually or spatially and so on?

And it turns out, it seems in the data so far, but this is very preliminary, if you put people together in a team who all have a similar thinking style, that's not very good because you sort of lack for paying attention to different kinds of things that nobody's paying attention to.

But if you have too much diversity, that's not so good either because it's harder for people to exchange information. Like if I'm a very visual thinker and you're a very verbal thinker, we might not communicate as well as we would have if we were more mixed and our team were more in the midpoint of that. But this is, I think, very early days for that kind of research.

Question: Have you tried the gorilla test on children?

Chris: I haven't, but it has been done. And it turns out what happens is little kids miss the gorilla a lot, like eight-year-olds and ten-year-olds. But as they get older they tend to increase their noticing. And I think that's because the main task becomes a little easier. So when you're eight, just focusing on that ball passing around and so on consumes more cognitive control, more mental resources, and therefore you are less likely to notice unexpected things. I think it goes down with aging as well, where again it becomes kind of harder to focus on the main task as well. So, noticing then starts going back down—noticing the unexpected thing.

Question: A question on intuition, and I'll overlook the fact that you dissed Oprah on that one.

Chris: Sorry.

Question: Yeah. Do you discount it entirely? I've always assumed that intuition can . . . not always, but if someone has cumulative experience in a field, and they can't articulate why they feel a certain way, but it's hitting them . . . In our businesses, I certainly look to people with experience—and especially if they can articulate their reasoning—but even if they can't, it's just their intuition. Do you discount that entirely, or do you think there's something to it?

Chris: I would not discount that at all. For example, if there were a chess Grandmaster who was very bad at explaining verbally what he's doing, but he told me this is the right move, I would go with him 100 percent of the time, even if he couldn't explain it, and I knew that he was that good and he had that experience. That to me is expert pattern recognition or expertise, and that's a real thing. That's a very important thing, and all of you have it to some extent in different fields.

The problem I have is with this generalized concept of intuition. Oprah's telling us to trust your instincts. Well, unfortunately, I have no good instincts about mortgage backed securities, or I have no good instincts really even about which house to buy or how big a mortgage to take or so on. I need to do something other than just trust my instincts in that case. So, for any low frequency decisions, novel situations, and so on, intuition is way overvalued in those cases. In a highly structured environment like playing chess or bridge or whatever, then yes.

Christopher Chabris Union College

And we know cognitive psychologists have essentially figured out how that works. We have a very good idea of how that actually works.

Michael: I have one thing to add, Chris. I think it's very domain-specific. And actually Danny Kahneman wrote a paper with Gary Klein talking about where intuition works and where it doesn't. And I think it's got to be fairly stable environments, fairly linear environments where it's going to work well. The psychologist Greg Northcraft has a line which I love. He says, "You have to distinguish between experience and expertise." We often think they are the same thing but they're different, right?

Chris: Yep.

Michael: And the key to an expert is he or she has a predictive model that works. So you could do something for a long time, have a lot of experience, but no predictive model.

Chris: Surviving in a field could be an indication of expertise because you might have washed out at some point or something like that, but of course not necessarily. That's probably, that's one reason why I like to study chess players because we know if a 12-year-old is rated 2500, that means he's exactly as good at chess as a 50- year-old rated 2500. They may be vastly different in what they can articulate and explain, but at least we know their skill is the same. I don't think there's any other field where we really know to that great of an extent what that skill is.

So, the further away we get from a field like that, where there's all this constant feedback and a really perfect rating system, the more we should be suspicious of instinct and so on, and at least go in with our eyes open about what we're doing. If we're going to go interview Madoff, and Madoff's going to say I do it all by gut feel and so on, well, we need to keep in the back of our mind the possibility that that's bullsh**. And not just say, “Okay, good, thank you very much,” and then go on like, “Have another five years.”

Question: I want to ask you a question about the chess players and their perception of being better than they

are. That seems like a very rational optimism bias to me. They seem to be pretty sensible in thinking that they were slightly better than they are. So there are very good reasons why you might be optimistic, so you don't really want them to be having the same ranking perception as they actually have.

Chris: I’m not sure that's . . . so I'll divide that into two answers. One is, we sort of tested for a form of optimism bias by looking at whether they were accurately forecasting their own future improvement—you could actually sort of know in some sense, you could know that you're better than your rating, and your rating still hasn't caught up. Or you know that it's going to catch up soon or something like that because you're really improving.

So we looked at that. We actually extended out five years and looked at their rating one, two, three, five years afterwards. And they did catch up a little bit but not nearly . . . they caught up maybe one-third of the amount that they were overestimating.

The second question then is, is it adaptive to have this rosy view of your own expertise? Is there some value in it? And I'll agree that there might be if it motivates you to practice more and play more. But I think it also may motivate you to not work hard enough. If you think you're actually better than you are but that somehow the world is not recognizing that, you may not actually do the work. So, there's very little research on this that I know

of. I think there should be more.

The one study I can point to is children playing chess—I think they were maybe 10 to 15 years old, I don't know the exact age range. They were asked similar questions to the ones we used. And then they were followed afterwards to see who improved and who didn't.

Christopher Chabris Union College

And the ones who improved more were the ones who had a better idea of exactly how good they were to start with, kind of suggesting that thinking you're better than you are is not a recipe for improvement in the future. It might be some kind of recipe for motivation in the present, but maybe not for improvement in the future. But I'd love to talk more about that and see if there's better research on that that we can think about.

Question: Is there any research on the differences between men and women?

Chris: Oh, in the overconfidence of chess players? Well, sadly no, because there are almost no female chess players who go to chess tournaments—maybe five percent or something. I’m not sure we had a single female player in our sample just because of sampling. But that would be interesting to look at. And there are more women playing nowadays.

One thing we have found, by the way, is that although women on average are rated much lower in chess than men, it's probably somewhat due to the fact that there are a lot more boys than girls in most locales where kids start playing chess. So, there could be some sort of stereotype threat things going on, and competition things going on.

We looked at some locations where there are as many girls as boys starting, like in elementary school or something like that, and in that case, they actually start with equal ratings. And overall, they don't drop out at different rates or anything like that. So, it could be something about drawing girls in, something that's perceived as an all-boy activity, maybe, that changes the pool of girls who get into it or something. Obviously an area where a lot more needs to be learned.

Question: Have you ever done the gorilla experiment and stopped it right after the gorilla came on stage and then left?

Chris: To see if they forget the gorilla?

Question: Yeah. Is it a retention problem or is it an actual attention problem?

Chris: We haven't done that. But some other collaborators of ours by themselves have done that and other versions of that, looking basically for the inattentional amnesia hypothesis, which is sort of like we have this rapid amnesia, we forget about it. And so far there's no support for that.

We also tried a kind of priming study, as Paul alluded to earlier, where we looked to see if people who didn't see the gorilla could still pick the gorilla out of a lineup. Like, what did you see? And we did this a long time ago, and we didn't find anything. It was as though it never happened to them.

But I think that research could be done again maybe more effectively. But it's a very strange thing that you can look at it for a second, I agree. It's very counterintuitive, and yet it doesn't seem to form an accessible memory trace of any kind. It's weird.

Question: So, the second thing is in terms of the workplace itself. You show the trading floor, and I assume you have some thoughts about how you could rearrange or redo the workplace maybe to do that. But along those lines, is there research on the impact of say, fatigue, on this kind of stuff? Because you would think that that would make it even harder to actually pay attention.

But maybe not. Maybe it actually makes you less good at picking the number of passes. I'm trying to think about the modern workplace, which is so much about, in a sense, burdening people with so much information, with asking them to work when they're tired, et cetera. And I'm wondering, it seems like all that would make it even harder to recognize what you're not actually seeing.

Chris: Well, I think that's probably right. Fatigue in general is a big downer for cognitive performance of all

Christopher Chabris Union College

kinds. And there's a lot of literature on that. Although, it's kind of hard to measure people's cognitive ability moment to moment because things like IQ tests are designed to tell me how I'm different from you rather than how you're different now than you were three hours ago.

I'm actually working a little bit on that too, hoping to make a map of how it changes. I think clearly it changes a lot from fatigue and coffee and things like that. Now, as far as noticing unexpected things, the main thing that guarantees that you're going to notice the gorilla is if you stop doing the counting task. So, everyone who stops the counting task but still looks at the video sees the gorilla. It does grab your attention then.

Maybe there's some kind of interaction with fatigue where, as you suggest, fatigue or other distractibility makes you just give up on the main task sooner. I'm not sure whether that's a good thing or not because, as we know, switching from a main task and then switching back to it has its own costs. I'm not sure I have any really deep thoughts about how to fix things. There's probably some clever technological design that we could do to do this.

One of my favorite examples is—I read about this in some book somewhere—a guy has two computers at home. One of them has no internet connection, has no Wi-Fi or anything like that. And when he wants to write, he goes into that room—it's in another room—and turns on that computer. He can't lock himself in, obviously, but he does a lot of things basically to make it a little bit harder for him—just like Iris said earlier, “make it easier to do things”—he makes it harder to distract himself.

But I'm not sure you can redesign a trading floor so that everybody's got an isolation booth that they could go into when they really need to think, but maybe something along those lines with a better technological solution to it.

Question: There's been a lot of popularizing, I would think, over the past 10 years of some of these concepts, with a lot of books and so forth. Where have you seen the applications of this in industry? I mean it’s obviously very applicable to investing. I could rattle off a number of other verticals. But are you seeing real attention in the government or in other places, financial services, healthcare?

Chris: Well, certainly, one area of regulation that I think has been influenced by this, at least conceptually, is the regulation around cell phone use while driving and so on. That's an obvious sort of direct example. But they don't quite get it right. The first movement was hands free. So, in fact, New York would give you a bonus if you bought a hands free headset or something like that.

And that's a little baby step in the right direction, but it's really not the hands that are the problem. It's the mental distraction. But I think that's becoming more and more known, and in fact I should give kudos to Oprah because she has been trying to popularize that. So, it's not like Oprah's the worst person in the world or anything. She just got it wrong on one thing. So that's one obvious example. I haven't really compiled a list in my head, but I think it's really interesting to think about—where has this kind of thing had greater influence?

Question: Is there awareness of how to make groups perform better?

Chris: I think there's some awareness. But, that's what we're working on for our next project, is to figure out what more detailed advice we can give on how to solve this problem. So, I'm thinking about it. And I think maybe part of it is really recognizing that it's in groups that we do a lot of important work. And getting the group to structure better might be easier than getting individuals to structure their own mental life better.

Paul has good advice, but it's hard to get people to change the way they behave. But groups can, in a way, have more structure imposed on them externally and by the tools they use. That might be the way to go.

Michael: On that note, we'll call it. Thank you very much, Chris.

Chris: Thank you very much.

Apollo Robbins

Apollo Robbins is a performer, speaker, consultant and one of the world’s leading experts on pickpockets, confidence crimes and deception. A pioneer in the application of deception to operational environments, Robbins uses pick-pocketing and sleight-of- hand to demonstrate perception management, diversion techniques and self-deception.

Known as “The Gentleman Thief,” Robbins first made national news as the man who pick-pocketed the Secret Service while entertaining former U.S. President Jimmy Carter. He has picked the pockets of more than 250,000 men and women. Forbes has called him “an artful manipulator of awareness,” and Wired Magazine has written that “he could steal the wallet of a man who knew he was going to have his pocket picked.”

In 2006, Robbins and his partner Ava Do, co-founded Red Handed Media, which quickly became a go-to expert for the science community, law enforcement and government agencies studying human behavior. Drawing from his 20- plus years of experience as an entertainer and trainer, Robbins and Red Handed Media use transmedia and live event production to develop immersive training modules with a focus on experiential learning.

Robbins has been featured in The New Yorker and in numerous publications such as The New York Times, Los Angeles Times, and The Wall Street Journal. He produced and co-hosted the National Geographic program Brain Games, which was nominated for an Emmy as an Outstanding Informational Series. His popular appearance on The Today Show is a YouTube favorite at more than seven million views. The TED editors described Robbins’s talk at TED Global 2013 as a revelation in the flaws of human perception. His latest project, a Warner Brothers film starring Will Smith, opened in international theatres in February 2015.

Note: No transcript available.

This document was produced by and the opinions expressed are those of Credit Suisse as of the date of writing and are subject to change. It has been prepared solely for information purposes and for the use of the recipient. It does not constitute an offer or an invitation by or on behalf of Credit Suisse to any person to buy or sell any security. Nothing in this material constitutes investment, legal, accounting or tax advice, or a representation that any investment or strategy is suitable or appropriate to your individual circumstances, or otherwise constitutes a personal recommendation to you. The price and value of investments mentioned and any income that might accrue may fluctuate and may fall or rise. Any reference to past performance is not a guide to the future.

The information and analysis contained in this publication have been compiled or arrived at from sources believed to be reliable but Credit Suisse does not make any representation as to their accuracy or completeness and does not accept liability for any loss arising from the use hereof. A Credit Suisse Group company may have acted upon the information and analysis contained in this publication before being made available to clients of Credit Suisse. Investments in emerging markets are speculative and considerably more volatile than investments in established markets. Some of the main risks are political risks, economic risks, credit risks, currency risks and market risks. Investments in foreign currencies are subject to exchange rate fluctuations. Before entering into any transaction, you should consider the suitability of the transaction to your particular circumstances and independently review (with your professional advisers as necessary) the specific financial risks as well as legal, regulatory, credit, tax and accounting consequences. This document is issued and distributed in the United States by Credit Suisse Securities (USA) LLC, a U.S. registered broker-dealer; in Canada by Credit Suisse Securities (Canada), Inc.; and in Brazil by Banco de Investimentos Credit Suisse (Brasil) S.A.

This document is distributed in Switzerland by Credit Suisse AG, a Swiss bank. Credit Suisse is authorized and regulated by the Swiss Financial Market Supervisory Authority (FINMA). This document is issued and distributed in Europe (except Switzerland) by Credit Suisse (UK) Limited and Credit Suisse Securities (Europe) Limited, London. Credit Suisse Securities (Europe) Limited, London and Credit Suisse (UK) Limited, authorised by the Prudential Regulation Authority (PRA) and regulated by the Financial Conduct Authority (FCA) and PRA, are associated but independent legal and regulated entities within Credit Suisse. The protections made available by the UK‘s Financial Services Authority for private customers do not apply to investments or services provided by a person outside the UK, nor will the Financial Services Compensation Scheme be available if the issuer of the investment fails to meet its obligations. This document is distributed in Guernsey by Credit Suisse (Guernsey) Limited, an independent legal entity registered in Guernsey under 15197, with its registered address at Helvetia Court, Les Echelons, South Esplanade, St Peter Port, Guernsey. Credit Suisse (Guernsey) Limited is wholly owned by Credit Suisse and is regulated by the Guernsey Financial Services Commission. Copies of the latest audited accounts are available on request. This document is distributed in Jersey by Credit Suisse (Guernsey) Limited, Jersey Branch, which is regulated by the Jersey Financial Services Commission. The business address of Credit Suisse (Guernsey) Limited, Jersey Branch, in Jersey is: TradeWind House, 22 Esplanade, St Helier, Jersey JE2 3QA. This document has been issued in Asia-Pacific by whichever of the following is the appropriately authorised entity of the relevant jurisdiction: in Hong Kong by Credit Suisse (Hong Kong) Limited, a corporation licensed with the Hong Kong Securities and Futures Commission or Credit Suisse Hong Kong branch, an Authorized Institution regulated by the Hong Kong Monetary Authority and a Registered Institution regulated by the Securities and Futures Ordinance (Chapter 571 of the Laws of Hong Kong); in Japan by Credit Suisse Securities (Japan) Limited; elsewhere in Asia/Pacific by whichever of the following is the appropriately authorized entity in the relevant jurisdiction: Credit Suisse Equities (Australia) Limited, Credit Suisse Securities (Thailand) Limited, Credit Suisse Securities (Malaysia) Sdn Bhd, Credit Suisse AG,Singapore Branch,and elsewhere in the world by the relevant authorized affiliate of the above.

This document may not be reproduced either in whole, or in part, without the written permission of the authors and CREDIT SUISSE.