[00:03:06] Annie Duke: I’d done my major area exams, I was out on the job market, I had my thesis research finished. And I got sick. So I had sort of something that was kind of chronic and it turned acute and I actually ended up in the hospital for a couple of weeks with it and it was just very clear like I needed to take a little bit of time off.

[00:03:27] Annie Duke: And so I had to cancel my job talks and fully intending to become an academic and get a tenure track position in academics as one does. so I took off and then I was going to come back the next year and finish up the next year and go back out on the market. It was during that time when I was taking time off after being there for five years that I started playing poker and I knew a little bit about poker before that because my brother was already a professional poker player and I’ve watched him play quite a bit and he had brought me out to Las Vegas a few times on just vacations during graduate school vacations.

[00:04:01] Annie Duke: I could not afford on a fellowship. And so I played like a little bit, but not seriously. And he suggested that maybe I do that because. I just honestly, like I really needed money. I didn’t have my fellowship anymore. I didn’t come from a family that had money. So I just need to be able to support myself.

[00:04:18] Annie Duke: So I started playing poker, which was really kind of ideal for me because I didn’t know how I was going to feel from day to day. I was still recovering from this illness. And because I was going to go back to academics, I didn’t want to start a career or something like that. So poker seemed like a good thing to do in the meantime.

[00:04:34] Annie Duke: My brother gave me some tips. I already knew some about it from watching him and talking to him about it. And when I started playing, I just started doing really well, really quickly. I was up actually in Montana, a very weird place to be playing poker, but they had legalized poker. And so I would, go to downtown Billings and go play poker, which was kind of strange.

[00:04:57] Annie Duke: Even stranger, when you realize that at the time that I did this, poker was not on television, there was no internet poker, so I think it’s hard for people to kind of understand what it was like in the olden days in the sense, but pre 2002, that poker players weren’t cool, they weren’t on television, people didn’t know who they were, they didn’t even understand the most basic thing, which was that you could make your money playing poker.

[00:05:23] Annie Duke: When I told people that I was playing poker, that it would generally end up somewhere in the, Oh, your husband must make a lot of money, or are you going to Gambler’s Anonymous, where it’s if you flash forward a decade from there, people were like, Oh, my God, that’s so cool. So it was, kind of a strange thing to do, but I loved it.

[00:05:42] Annie Duke: I just, I loved It, it seemed to me to be just as very practical, real time, high stakes application of the things that I was learning in psychology. which is how do you actually make good decisions in a kind of environment where there’s so much uncertainty? And that’s actually a lot of what I was studying in graduate school.

[00:06:03] Annie Duke: And the uncertainty in poker is coming from two places. One is there’s just a lot of luck, which obviously people who are in markets know, and poker is just a market. And then there’s also a lot of hidden information, which should also sound very familiar. So it’s just a very high vol in relatively low information settings.

[00:06:21] Annie Duke: it’s those types of environments where our decision making can go really bad in the form of predictable human error that might occur. So for anybody who’s familiar with, for example, Daniel Kahneman, that’s really what his life’s work is about things like confirmation bias and overconfidence and in those types of environments that’s where those things get the worst.

[00:06:40] Annie Duke: So I was really interested in just like really learning the game and trying to solve for these issues. And for eight years, I didn’t go back to graduate school. I was as ABD as you could get my dissertation work actually even got published, but for eight years, I just dove headlong into poker. Then in 2002, when poker sort of got on television, I got asked by hedge fund to come speak to their options traders about how poker might inform their thinking about risk.

[00:07:09] Annie Duke: And I took it in a little bit of a different direction because I’m kind of a cognitive scientist at heart, and I talked about how the track that you’re on, whether you’ve been winning or losing recently, really distorts your risk attitudes on your next decision, which is a really big problem that poker players have as well.

[00:07:25] Annie Duke: So I really kind of talked about that issue. Broadly in poker, we might put that under a category called tilt. Which is when your brain sort of stops working because you’re emotional about things that have happened in the past. So I gave that talk. It was really fun. And that person, the person, the founder, the managing director of that hedge fund ended up recommending, starting to recommend me to other people.

[00:07:49] Annie Duke: And, and then obviously as poker got on television and I was kind of one of the OGs. I started getting asked to do like events for businesses and mostly it started off with people just wanting me to come in, like play poker for their like retreats. And I started really pushing my manager to offer talks because the thing that really happened to me when I gave that first talk was I remembered some, I remember two things.

[00:08:15] Annie Duke: The first thing is I remembered that I really do love cognitive science, but the more important thing that I remembered was I love teaching and I hadn’t been doing it, obviously I’ve been teaching myself, but I hadn’t been teaching other people and I really want to do more of that. I started sort of getting asked to come do these talks originally a little bit me pushing it, but then just through, word of mouth and whatnot, started building up that business, actually ended up teaching a little poker on the side too, because I found a way to do a lot of teaching.

[00:08:43] Annie Duke: And then sort of around 2012. So I’m doing that in parallel, right? I’m a poker player. And then I’m, I also have developed this whole other thing that I’m doing, which is thinking about the intersection between cognitive science and poker and how those two disciplines might inform each other to make us actually better at both of them.

[00:09:01] Annie Duke: So I’m now doing those things in parallel. I start getting asked by some of the people that I speak to, if I do consulting, that might be interesting. So I started doing some of that and then in 2012 really made the decision to retire from poker completely so that I could focus on this other thing that I was doing and that I did.

[00:09:25] Annie Duke: And so my life now is I have a very small roster of clients. I keep them small because I embed. I don’t do short term projects with people. The client that I’ve been with the shortest amount of time, except I just took on a brand new one. So I’ll exclude them, but in terms of any of my older clients, the shortest amount of time that I’ve been with any of them is now three years.

[00:09:49] Annie Duke: So I really go deep and long with the people that I work with. And then I also just was really kind of burning to start writing down the things that I was talking about. The, what I was doing in terms of the work in my consulting work. And that became first thinking in embeds, then how to decide. and then quit thinking about my next book right now.

[00:10:11] Annie Duke: I’m just starting research on it actually. And then the other fun thing that I did was last year, I had been doing research with Phil Tetlock and Barb Mellers, and he wrote super forecasting, which I’m sure you’re familiar with. Both of them completely brilliant. And during COVID, they had asked me to collaborate with them on some research on forecasting.

[00:10:31] Annie Duke: And so I did that and I sort of became lead investigator on a series of four pretty large scale studies. We found really fun results. And at the end of it, Phil said to me, why don’t you just write this up? Like it’s more than most people do for a dissertation by a lot. And so last year on June 15th, I successfully defended my dissertation and was officially PhD on August 4th.

[00:10:57] Kyle Grieve: So you mentioned Eric Seidel in Thinking in Bets and some of the key lessons he imparted to you. And you said that you had a crush on his intellect at the time. So I really enjoy learning about, mentors of people I highly respect like yourself. So I’m just interested. I mean, I probably sure you could go hours speaking about the mentors, but maybe could you give me a little bit of information about some of the mentors who had the biggest impacts on you?

[00:11:19] Kyle Grieve: That’s whether that’s inside the realm of poker or outside and why they were so influential for you. 

[00:11:25] Annie Duke: Academically, it has to start with Barbara Landau, who is now at Johns Hopkins. She was at Columbia at the time. It was her first job out of graduate school, I believe, actually. And the first week of college, I was looking for a work study job and she was looking for a research assistant and the rest is history.

[00:11:47] Annie Duke: and so I started working with her and I actually stayed with her for the whole four years that I was in college and, she had actually just come from Penn where her advisors were Lila and Henry Gleitman and she was amazing because she did something that a really good mentor will do, which is I really wanted to stay at Columbia.

[00:12:05] Annie Duke: I did not want to leave New York, so I wanted to do graduate school and do my dissertation under her. And she sort of put her foot down and said, no, you really have to go to Penn. I don’t think it’s good for you to stay with the same person for eight years. which I thought was, or nine years, actually, which I thought was pretty amazing, I did end up going to Penn and I studied with Lila and Henry.

[00:12:27] Annie Duke: So they’re, my next really big mentors. In my life and just, Lila, I was close with until she was 91, just really showed me a model of not just incredible intellect, but also compassion. And again, as a good mentor does. While I was, I sort of felt like I had really let her and Henry down by leaving graduate school and going off to become a poker player.

[00:12:55] Annie Duke: She was just so beaming with pride about it, So for her, successful mentorship was me going off and excelling at whatever it was that I did. And it was funny because I had such a, I was so focused on the letting down because I didn’t follow in her footsteps that I didn’t think of course she would be proud of a student who went off and became a world champion at the thing that they were doing.

[00:13:14] Annie Duke: So I have to, obviously there are huge influences in my life. In poker, my brother, clearly he’s the one who taught me to play. He was the person who I most learned from and bounced hands off of and then Eric Seidel would be the second one. who taught me a lot about how to play poker, but, more of his mentorship was so much in that kind of like emotional control.

[00:13:36] Annie Duke: And how are you processing like good and bad outcomes? And how are you treating like your fellow human beings in terms of the way that you’re communicating to them and generosity of intellect and, that kind of stuff. Like he was more My brother was more teaching me poker and Eric Seidel was teaching me much more about how to behave as a poker player, which I, think it was so important because you do so much, there is so much emotion in poker that if you can’t get that in check, you’re going to be in really big trouble.

[00:14:06] Annie Duke: And he’s so good at it, so he’s such a huge influence for me. And then we start to get to, as I move into this next, The phase that I’m in right now, which, in terms of writing and, you know, and the consulting work and so on and so forth. I mean, obviously, Phil and Barb, Phil Tetlock and Barb Mellors in terms of the mentorship around like my dissertation and actually completing that work.

[00:14:28] Annie Duke: But then the people that you mentioned, Danny Kahneman has been an amazing mentor to me. Michael Mauboussin and I are just, catch up all the time. He’s one of my idols. I, he, I think that just the way that he thinks and his intellectual curiosity is something to completely aspire to.

[00:14:48] Annie Duke: And so he’s definitely been a mentor. To me in that way as well, and and of course at the age where mentorship and friendship a little bit gets melded together. Do you know what I mean? Because they’re also like, Michael’s also a very good friend and Danny’s a friend. and then Katie Milkman sort of in that category where, I look to her for mentorship, but she’s also, my friend.

[00:15:10] Annie Duke: So I’m kind of at that age where it sort of mixes. together a little bit. And then I would say the last person in terms of mentorship would be my husband, who really helps to guide me. And a lot of what he does is, help me to find permission to say no to things, which I really do need a lot of mentorship on.

[00:15:28] Annie Duke: So I’ve been very lucky actually, and I know I’m leaving people out, but I’ve been very lucky to have some Just really awesome, like really incredible mentors in my life, and it’s so important, obviously, to where I am now. 

[00:15:42] Kyle Grieve: That’s an incredible list. In your chat with my colleague, William Green, you mentioned that we all think probabilistically, even if it’s not explicit.

[00:15:50] Kyle Grieve: So you used a great example of when we drive to work and decide which route to take. We’re using probabilistic reasoning to determine which route is going to get us there the quickest. So we are using a form of, an implicit form, sorry, of expected value to help us determine which route makes the most sense, given the information that we know at the time.

[00:16:10] Kyle Grieve: But as you pointed out, the real magic happens when we explicitly use probabilistic thinking. So I’m interested, from an investing lens, What can investors take from this lesson about probabilistic thinking to help them minimize risk in investing? 

[00:16:25] Annie Duke: First of all, let me just say this, that if you have the expected value right, risk becomes a much less important issue.

[00:16:33] Annie Duke: This is a concept that really was hit home for me from Jeff Yoss. Who’s the founder of Susquehanna International Group. And he actually said, Rich Smith’s what I care about is I’m worried that I think I’m winning when I’m actually losing, there’s actually a Don Moore and Max Bazerman has have written about this, where they think everybody should be much more focused on expected value, right?

[00:16:56] Annie Duke: Because if the expectancy is positive, you’re probably not going to make too big a mistake. Now, obviously you have to think about risk in the sense of, am I going to have more money to churn through? This positive expectancy thing. We definitely want to think about it when we’re in that sort of risk of ruin category.

[00:17:17] Annie Duke: So that’s going to be particularly important when we’re in a higher volatility situation. So in poker, we do actually think about this quite a bit, but you have to know what your edges, right? I mean, that’s, the thing, right? So you have to have a good sense of what your edges, which is really an expected value.

[00:17:34] Annie Duke: Problem, right? because you can’t expect, calculate expected value without knowing what, your edge is, right? So you have to know what your edge is in order to start to manage risk well. Obviously, assuming that you’re in a low vol, if you’re in a low vol situation, you’re probably never going to make too much of a mistake as long as you have a positive expectancy.

[00:17:53] Annie Duke: But if you’re in a high vol situation, you do actually have to start to think about that more deeply. In poker, we were just applying something that was very similar to Kelly, basically, which is like essentially bet your edge. And a little bit, the way that I think about Kelly is First of all, just how confident are you of what your edge is?

[00:18:11] Annie Duke: Because I think that the less confident that you are about your edge, the more that you should be moving into like half Kelly, quarter Kelly, that kind of thing. Because you really just have to give yourself a cushion on that, and then there’s just also your tolerance for going broke. How easy is the money to replace and things like that.

[00:18:28] Annie Duke: So I tended in poker, I tended to bet somewhere around, Never more than 5 percent of my total bankroll would be in play at any time. but usually it was more in the two and a half percent range. That would have been a foolish to half Kelly kind of situation. But again, the thing about all of this is that.

[00:18:49] Annie Duke: With risk, it’s like you can’t manage risk unless you know what your edge is. You have to know what your expected value is in order to be able to manage risk. And I think the big mistake that people make in investing is that you have risk managers and there are formulas that you can apply to risk. And I think they make an assumption about their expected value and they get really focused on the risk management side of things because it’s a problem that I think is easier to solve.

[00:19:15] Annie Duke: In other words, if you have an assumption about what your EV is, And what the vol is, right? Now you can actually just apply a formula and you can start to get in to some sort of dive headlong into the risk management side of things. And that’s Jeff Yoss’s point, is that he feels like people are making too many assumptions about the EV part, just like what is your edge in the first place, right?

[00:19:38] Annie Duke: And I think everybody comes in assuming they have an edge. And then they go to all of these risk management formulas. And he’s I don’t even care about that. I’m just so afraid that I think I’m winning when I’m actually losing. And I think that’s the thing that people really need to be thinking about when it comes to expected value.

[00:19:55] Annie Duke: And it’s one of the reasons why you really want to make it explicit. 

[00:19:59] Kyle Grieve: So how do you, how would someone really go over finding their edge? I mean, I guess you’d have to just, because it’s all from experience, right? And you can’t just kind of snap your fingers and have this number in your head. You kind of have to base it off of your prior experiences and kind of go from there.

[00:20:15] Kyle Grieve: Is that how you would coach someone to do it? 

[00:20:17] Annie Duke: So prior experience matters, but not as much as the base rate. This is something Michael Mauboussin just like hits home so well, the thing that he is always saying, and I’ve seen a lot of talks that he’s given, is the way we normally go about making decisions is we think about the problem that we’re approaching and then our own experiences.

[00:20:37] Annie Duke: And then we use that to decide what we think our EV is going to be. if I’m in, if I’m trading and I’m thinking about a particular thesis that I might trade or whatever, I’m going to be thinking about how smart I am and how unique the thesis is and, so and so forth. And I, my past experiences with winning and all of that.

[00:20:56] Annie Duke: But what I should really do to start with is his point is before I get to my own experience, I have to start with base rates. Let me give you an example. Let’s imagine that I’m thinking about opening up a restaurant, and I’ve worked in restaurants before, and the chefs that I’ve worked for have said that I’m amazing, and I’m a great cook, and the customers are always complimenting my food, and now I’m opening up my own joint.

[00:21:24] Annie Duke: And I’m thinking about how much people love my food and how busy the other restaurants I’ve been in have been. And I estimate that the probability that my restaurant is going to thrive is 80%. Because notice I’m thinking about all these things that are personal to my experience and how much people love the food that I make and how good a cook I am and how well the restaurants that I’ve been in have done and so on and so forth that I’ve cooked in I’ve done.

[00:21:52] Annie Duke: But that’s not the place that I want to start. That, that stuff matters, but what I actually want to start is with the base rate. So what I want to think about is, what usually happens in a situation that’s similar to the one that I’m considering. So this is going to give you, in investor speak, beta.

[00:22:10] Annie Duke: You need to know what beta is. In restaurants, if I think about, if I actually look up what percentage of first time restaurants are still open at the end of the first year, survive a year, it’s 40%. Okay, so now I’m going to start with 40%. And now I can say all those same things. I’m a great cook. People love my food.

[00:22:30] Annie Duke: Customers are always complimenting me at the restaurants that I’ve cooked for. The chefs that I’ve worked with have said that I’m amazing and I should open my own joint. And so I think that my chances are better than 40%, but not 80%, right? Like I’m not doubling my chances here. So this allows me to get grounded in a reality that I can now toggle up or down from, so we can look at that, we can find the base rate and then we can say, do my particular circumstances, do I think it makes it more likely or less likely that I’m going to succeed independent of the base rate?

[00:23:10] Annie Duke: Now, we can use this concept, in terms of, historical averages also. if I want to make a guess what my Q2 target should be, I shouldn’t go by what my board wants it to be. I should look at what has, on average, growth been year over year, and then I can actually historically look, because we, again, Base rates are interesting, right?

[00:23:36] Annie Duke: cause it’s not, there’s some art to base rates cause you have to find the right, what’s called the reference class. You have to find the right sort of situation similar to the one that you’re considering. So I can look at growth year over year, but then I can also look at what happened, what has historically happened between Q1 and Q2.

[00:23:51] Annie Duke: If my company has been around a long enough, I can look what’s historically happened within my own company because there may be seasonality, for example, or I can look in the industry in general. So this is now you can see it’s going to help me get a starting point. So let’s imagine that quarter over quarter growth in, net new ARR has been recently 10%.

[00:24:15] Annie Duke: But then I also see that there’s a seasonality component where my sales actually go up in June. Q2 more than I would expect them to go up between Q3 and Q4 and Q1. So maybe I bring it up then and I say actually the seasonality is working in my favor because that usually gives you an extra 50%. So I say, okay, I think it’s actually going to go up 15%, but my sales leader is about to exit.

[00:24:39] Annie Duke: So that’s something else that’s personal to me. My sales leader is about to exit, so maybe I should toggle it back down. So you can see how you can now use this to ground you in reality to get to a more reasonable place in terms of what your forecast might be. 

[00:24:55] Kyle Grieve: So I know that feedback loops are your obsession.

[00:24:58] Kyle Grieve: So when I think about feedback loops myself, I think a lot about jiu jitsu, an activity that I’m very fond of. The beautiful part about jujitsu is that feedback loops are pretty much instantaneous. I can try a new move or a new submission and instantly I know it works that my opponent taps or it doesn’t work, they escape.

[00:25:16] Kyle Grieve: But obviously in investing, those feedback loops aren’t, instantaneous. They, you might make a decision today and you might not actually know if that decision was good or bad a couple of years out. So I’m really interested in knowing how do you best close feedback loops on decisions where the outcomes won’t be clear for a few years?

[00:25:37] Annie Duke: Oh, I’m so excited that you asked me that because it’s one of my favorite things to talk about. Two of my very long term clients are venture firms and they’re both early stage. One is focused on seed. That’s first round capital partners. I’m a special partner there. And then, the other is focused more kind of in the series B area, and that would be Renegade.

[00:26:00] Annie Duke: Love them both. Prior to my working with them, I had lots of, I was invited to talk to partners at a variety of different venture firms. And they all kind of said two things to me, which I thought was interesting because I heard an echo through the whole industry. One is, how you can’t really close feedback loops appropriately.

[00:26:21] Annie Duke: in the way that you talk about in Thinking in Bets. So this was after Thinking in Bets came out where I talked, there’s this obsession in Thinking in Bets about closing feedback loops. So you can’t do that when there’s power law, like when power law applies. And just for those people who, might not know what power law is, it’s when you have a very small number of winners that win a ton, but most things die.

[00:26:43] Annie Duke: So this should sound very much like power law. Venture. It’s actually a little bit like social media where 2 percent of the users are producing all of the content and everybody else is kind of quiet. the power law applies in a variety of different places, but it definitely applies to venture.

[00:26:58] Annie Duke: And so their point was, if everything’s dying, then, and you only have a couple winners, like you could never tell anything about the quality of your decisions. And the second thing that they said was the feedback loops are too long. If you’re investing at seed, it’s going to be 5 or 10 years really, before you get whatever the outcome is.

[00:27:17] Annie Duke: I said the same thing to all of them and it was only when I got to first round and to renegade that they went, Oh, okay. I hear what you’re saying. This is why I work with them. And in particular, Renegade was new. Josh Koppelman was the one that I originally talked to who’s the founder of First Round and he’s so tremendously successful.

[00:27:39] Annie Duke: So I just want to give like a big shout out to him because somebody that successful doesn’t need to be open to changing the way that they think about things, right? And very often aren’t open. And he, was completely open to changing the way that he was thinking about this. So let me tell you what I said to them, because this is the answer to your questions.

[00:27:56] Annie Duke: I said, what do you mean the feedback loops are long? And they said, what do you mean? We don’t know if it excerpts for, and I said, I’m sorry, do you invest in the company? And then you go to sleep like Rip Van Winkle and 10 years later you wake up and you find out what happened when you invest in a company, a couple of things are true.

[00:28:16] Annie Duke: Two different category of things are true, all of which, both of which allow you to close the feedback loop more quickly. Thing number one is that you actually know objective things about the company. You know whether ARR is growing, you know whether they’re hiring top talent and retaining the top talent, you know whether they fund a series A, you know whether it’s an up round, a flat round, a down round, you know what the quality of the syndicate is, same thing for B, same thing for C, if you’re depending on the speed of the market, if you invest at Seed, for example, You’re going to know something very significant about that company between 6 and 16 months later.

[00:28:57] Annie Duke: That sounds like a lot faster than 10 years. the first thing that you know. The second thing, and this is true across all investing, is that you’re investing in the company because you’re making a particular bet. And the bet is your thesis. If it’s in the market, you’re saying, I think that I know something that the market doesn’t know.

[00:29:17] Annie Duke: Why do I know that’s what your thesis is? Because otherwise you would be indexing the market. You’re not indexing the market. So you’re saying, I believe that the market has this mispriced temporarily. I believe the market is efficient, but you know, not every single moment, right? Like it’s overall efficient, right?

[00:29:35] Annie Duke: So I believe that at this moment, when it comes to this stock or whatever, this stock or this option, the market is not, does not have this price efficiently. So you have a thesis about why that is true. It’s true when you’re investing in a company. I believe that this market is going to be a great market to be in.

[00:29:55] Annie Duke: This product is going to have a competitive advantage. They’re going to execute in this particular way, so on, so forth. And those kinds of things you can find out very quickly, even when you’re investing in a seed stage company. You can see, are they executing in the way that I thought they were? Is their product gaining traction?

[00:30:15] Annie Duke: so on and so forth, right? So all of these things, look, is it like poker or jujitsu where you’re going to find out two seconds later? No, but you’re going to find out way more than ten years. And that’s what we’re obsessed with, right? Is how are we thinking about the way that we can grade these companies as they develop, where we know things about the quality of the decision?

[00:30:45] Annie Duke: What long before 10 years is that? And the other thing that’s really important to know is that again, because of the power law issue, there’s lots and lots of companies that die that were great investments because there’s a lot of luck that’s happening, right? do you have a company that COVID, it just destroys, right?

[00:31:06] Annie Duke: Or whatever, right? So you want to be able to see the companies that you wanted to have in your portfolio, regardless of whether they ended up being fund returners or exited for over a billion dollars. And you can only do that if you’re actually tracking these things that you know are necessary, but not sufficient.

[00:31:24] Annie Duke: For them to get a billion dollars to a billion dollars and that all translates perfectly on to, for example, people who have long short funds, right? Where most of the long that if you’re in a value investor is long hold. And so you need to start saying, okay, but I don’t want to not know for five years, what are the things that’s happening with this company that I’ve invested in, given what my thesis was for why the market was inefficient here, and are those things unfolding?

[00:31:55] Annie Duke: So I just never accept, ever, that the feedback loop is too long to be able to do anything with. I’m sorry, I got really passionate about that. But it’s that’s my thing. that’s the thing that frustrates me the most. So I apologize for getting so excited. 

[00:32:14] Kyle Grieve: No, please don’t. That was an excellent explanation.

[00:32:17] Kyle Grieve: In your conversation with Howard Marks, you discussed a lot about the role of luck that you just brought up and how that plays in outcomes. So this has been a fascinating area for me that I spend a lot of time thinking about. So you said, quote, in the short run, there’s just way too much luck. So as an example, if Howard and I were betting and he said, I’m going to lay you two to one on a coin flip, a coin being 50 50, I’m going to make 50 percent on every dollar that I bet there.

[00:32:40] Kyle Grieve: If I call heads and it lands tails, it means nothing. Now, if we were to do it a thousand times and I kept losing, then I could start to draw some conclusions from the outcome, like maybe Howard isn’t using a fair coin, unquote. So from an investing view, this shows how hard it can be to evaluate ourselves on a single decision.

[00:32:56] Kyle Grieve: was the outcome a product of luck or skill? I think you would agree that a lot of outcomes, like you’ve already said, are probably a combination of both of those things. But what steps do you think we need to take to better understand if our process is generating the outcome we want rather than just plain luck?

[00:33:13] Annie Duke: Obviously, like in poker, at the end of a year, assuming you’ve played 1000 to 1500 hours of poker, your results are going to be, have very little influence of luck on them. Very different than one handed poker, right? What we’re trying to do is generate enough data points to start understanding the luck skill differential.

[00:33:33] Annie Duke: This goes back to what I just said. let’s imagine that I’m a venture fund and I’m only going to make 20 bets in a fund. Or I’m a long short investor and maybe I’m, at any given 7 positions. So now, you’re sitting here going, how am I ever supposed to know if my process is good? Because it’s seven coin flips also doesn’t tell you very much. if I’m sort of just waiting for the outcome of those things, it’s not going to help me very much. so a great investor, if their fund has 20 companies and it could have a fund that returns 0. 8, could have a fund that returns 40x and it’s the exact same investor.

[00:34:14] Annie Duke: So it’s just a kind of, it’s did you get Uber in there? So the way that we deal with that is to create more outcomes. In those types of environments. So notice, I don’t have to do that in poker because in poker, I’m turning money through that system so quickly that I’m getting enough outcomes to start to draw conclusions, particularly if I combine that with really deep dives on a process or my thinking process during a hand where I’m hiding the outcome from the person that I’m doing that deep dive with.

[00:34:45] Annie Duke: So I can kind of do that in poker, right? But how do we deal with that in a situation where your portfolio might have seven positions at a time, for example, or in venture, where maybe you have 20 companies or 25 companies and you’re in a fund, right? And it’s, you’re generating more outcomes than the final outcome.

[00:35:04] Annie Duke: So it goes back to what I’m saying. So let’s imagine this. Let’s imagine that you’re an investor investing in Series A. That’s your specialty. And for every single company. Every company that comes into partner meeting, you have to make a forecast of the probability that company will fund at series B.

[00:35:26] Annie Duke: Now, you can put restrictions around it, right? you can say the probability that it will fund at series B with an up round. So you can do whatever you want. So we can think about what are those things that are really necessary for this to survive? Because the thing is, I can guarantee you, you won’t ever have a fund returner if you invest at series A that doesn’t invest at series B.

[00:35:45] Annie Duke: likely not a down round, which is a bad sign. Let’s take that one data point, right? But then you can now generate, you can actually think about what are all the data points that I now want to be making predictions about that, where I’m going to start to know those things more quickly. Now, what’s nice about that is it does double duty.

[00:36:05] Annie Duke: What is it massively increases the outcomes that you have so that you can get more into the thousand coin flip situation. Because if you have, let’s imagine that you have a hundred companies come into partner meeting in a year and you’ve made that, that we’re just talking about the one forecast right now, you’ve made that one forecast for all hundred companies in the space of 18 months, you’re going to know for every single one of those companies, because you can, it’s public, whether they, do a B.

[00:36:31] Annie Duke: How good about a predictor am I? of this particular thing that really, matters. Okay. So, you, so now you start to be able to close these feedback loops faster because you’re generating so much more data that we start to remove the luck and it becomes about sort of your predictive power, right?

[00:36:47] Annie Duke: Do you actually have the situation modeled correctly? Now, the other thing that’s really good about that, if we go back to this idea of implicit versus explicit, is that if you’re going to do that, you’re actually making the decision process explicit instead of implicit. In other words, When you make an investment, when you put something in your portfolio, there are, it is implicit that what you’re doing is making a prediction about the future.

[00:37:16] Annie Duke: You’re judging certain things. So in venture, it might be the quality of the market, the quality of the team, the quality of the products, whatever, so and so when we can think about components of those things, what’s the competitive landscape look like? Is it favorable to the company?

[00:37:32] Annie Duke: Obviously, in, if you’re putting a company in your portfolio, in the form of a stock, you’re going to have some sort of thesis about, it might be that this company is going to perform really well in a high interest rate environment, and you have a, so you have a prediction that interest rates are going up, as an example.

[00:37:49] Annie Duke: You believe that there’s been a bottleneck in terms of supply chain and you believe that it’s closer to solve, the company’s closer to solving the bottleneck than the market believes. And so your, prediction of the number of widgets that company is going to produce in the next two quarters is just way higher than, for example, what the guidance might be.

[00:38:12] Annie Duke: So, we can think of a variety of things. I just made those up, but you can imagine what those things are, right? If we know that those things are included in the decision, then making explicit predictions and judgments about those things. just for example, with a company, I can make a judgment of on a scale of 1 to 7, how strong do I think this market is?

[00:38:32] Annie Duke: That the company is going to be entering into, right? So I can make that judgment about every single company that I’m seeing. Now, not only do I have tons and tons of predictions, where those outcomes are going to come more quickly so that I can get more to a thousand coin flips more quickly and figure out if my thinking is pretty good around this.

[00:38:53] Annie Duke: But I’ve also made the components of the decision explicit, which is really important for reducing bias and noise. So I’ve said explicitly, this is the, here are the, here’s what would have to be true for me to invest in the company. Let me actually make judgments about those things that would have to be true that I would care about.

[00:39:13] Annie Duke: And it makes it less likely that you can tell a narrative that just gets you to a conclusion that you want to get to. And the fact that you know you’re going to be accountable to it also disciplines you to reality. 

[00:39:25] Kyle Grieve: So you were asked how you thought lessons from poker transferred to other areas of life, and you went into some excellent details, but I want to highlight a few things you said, quote, The transfer of training from one domain to another is pretty dismal, unquote.

[00:39:38] Kyle Grieve: And then you talked about some concepts of near transfer and far transfer. So I’m just interested in knowing if you can, explain these concepts of near and far transfer in a little more detail. 

[00:39:48] Annie Duke: I can, because it was part of my dissertation. I’m going to do something weird. I’m going to go back to Plato.

[00:39:53] Annie Duke: Plato wrote about this idea that I think would, it’s probably intuitive to you. It’s intuitive to me, which is if you teach people to solve hard problems, like hard math problems, That, they’ll be better at other problems as well. I was actually just talking to a researcher yesterday who said there was an intuition that if you teach people to get really good at chess, that they’ll be really good thinkers, and they’ll be great at other things.

[00:40:19] Annie Duke: And it turns out that Plato was wrong, and that’s not true, and we’ve known that way back since the early 1900s with a guy named Thorndike who, just showed that, right? if I make you solve trigonometry problems, and I get you to be pretty good at that, it doesn’t make you better at anything else.

[00:40:35] Annie Duke: Now, I assume, Kyle, you find that intuitively not, really? That doesn’t seem like it’s true. But it is. So if you become good at chess, you’re becoming good at chess, is basically the answer. It’s not helping you with other stuff. And if you become good at trigonometry, you’re good at trigonometry. That’s sort of in domain learning, not transferring.

[00:40:56] Annie Duke: Okay, so you can get some near transfer. So near transfer would be problems that are very similar. If you learn to be really good at checkers, that probably helps you with chess. But those things are close to each other. But the question is, can you get far transfer? So far transfer is domains that are very unrelated to each other.

[00:41:15] Annie Duke: So if you learn to be very good at chess, does that mean that when I go and ask you to go be a stock trader, so those things are pretty far away from each other, that’s going to help you? And the answer is no, it will not. That’s that idea of near transfer versus far transfer. Now here’s where I think poker is really helpful.

[00:41:33] Annie Duke: And where I think that our education system could do a better job. In fact, I co founded a nonprofit called the Alliance for Decision Education, which is really trying to teach kids to be better decision makers broadly. There is a way to get transfer of training, but the way to do it is that you have to dig down into the conceptual level.

[00:41:54] Annie Duke: The issue with transfer is that, if you tell most people about Archimedes getting in a bathtub in the water displacing, They’re not going to relate it to anything except for like bathtub water displacing. So they’re going to understand that if you put something in a sink, it’ll displace the water, right?

[00:42:12] Annie Duke: But, obviously that’s, he wasn’t talking about water to place. He was trying to figure out how to figure out if a crown was made of gold, right? So it was a measure, broadly about measurement and how can we measure things and density and so on and so forth. So you have to, get down to this very conceptual level.

[00:42:30] Annie Duke: The kinds of concepts actually that transfer really well are things where you’re sort of taking advantage of the fact that we’re intuitive statisticians. Now, we’re not great intuitive statisticians. I want to just say that, our sense statistically of what should be true or not isn’t fantastic, which is why people can make money in the markets.

[00:42:49] Annie Duke: Because if people were good intuitive statisticians, they would obviously be better at that. But we do all sort of apply a little bit like these concepts. take, for example, the law of large numbers, right? We all sort of intuitively know that one data point probably isn’t enough to tell you anything, even though we reason by anecdote a lot.

[00:43:09] Annie Duke: But if you sort of step back from that intellectually and say, Yeah, but that’s one person, people are like, yeah, they kind of get that. So what the research has shown is that if you want to get transfer, bar transfer to very disparate domains, teaching statistical concepts. is actually the best way to do it, right?

[00:43:28] Annie Duke: So you have to get way down deep to that underpinning. If I teach you about the law of large numbers as applied to pulling ball, cut different colored balls from an urn, then when I ask you what you can surmise about a baseball player’s batting average for the year from their first seven at bat, you will recognize pretty quickly that probably not too much.

[00:43:51] Annie Duke: So that’s true, for example, for concepts like. Equilibrium or regression to the mean. I can teach you about regression to the mean and you can apply that across all sorts of different domains. I can teach you about base rates. And you can apply that across all sorts of different domains. So it turns out if we can actually teach people sort of the underlying idea of the statistical concepts, proportionality would be another one that you don’t want to just think about the numerator, probably want to think about the denominator as well, but that’s really helpful.

[00:44:20] Annie Duke: Then you can get this far transfer. And I think that’s where poker is really helpful. Because it’s very hard to be really good at poker if you’re not actually thinking about those types of statistical, like the statistical concept. That’s not to say that there aren’t people who aren’t good at poker who don’t think that way, who really are kind of just good at poker.

[00:44:40] Annie Duke: But somebody like Eric Seidel, for example, who’s good in sort of all different types of poker and so on and so forth, he understands the underpinnings and that’s going to transfer well to other domains. For example, he used to trade options and did a great job of it. 

[00:44:53] Kyle Grieve: There is a very successful investing partnership, I’m not sure if you’ve ever heard of it, called the Nomad Investment Partnership, that published its annual letters, and in it, they discuss the concept of destination analysis, which is an analytical tool that they use to help them determine if the destination of a business was good or bad, and if it was obvious to them that they could actually see what that would look like years into the future.

[00:45:13] Kyle Grieve: So when I do my own destination analysis, I like to add a segment where I would list off the events that have to happen for them to never reach that destination. that I thought that they could hopefully achieve. So I call it my losing playbook, which helped me be more objective when an investment started going bad.

[00:45:28] Kyle Grieve: So when I read about your concept of the kill criteria, it immediately resonated with me. So one thing I noted was that you emphasize that the kill criteria needs to have a state and a date. So do you mind going over the concept of kill criteria and discussing it in a little more detail in terms of improving decision making for long term oriented investors?

[00:45:47] Annie Duke: Yeah, so there’s two separate concepts in here in terms of what you just talked about, like what, what would need to be true in order for them to not succeed, what would need to be true in order for them to succeed. So we can think about that generally as a bottleneck problems. Like they have to solve, there are certain bottlenecks that are going to, either they, solve them or they don’t.

[00:46:07] Annie Duke: And if they can’t, then they’re not going to succeed. And if they can, then necessary, but not sufficient for success. All right. So the first place that we want to start before we get to kill criteria is a concept called monkeys and pedestals and monkeys and pedestals is basically, it goes like this, let’s imagine that you’re, you’ve decided that you’re going to leave your podcast.

[00:46:29] Annie Duke: And stop investing because what you really want to do is train a monkey to juggle flaming torches while standing on a pedestal in the town square. And if you did that, I, people would throw a lot of money in your hat. That’s pretty spectacular thing to train a monkey to do. So we can say there’s two parts to this act that you’re going to build.

[00:46:44] Annie Duke: One is training the monkey to juggle the flaming torches and the other is building the pedestal. And the pedestal. Is not the place that you should start. And the reason that you shouldn’t start there is that you already know you can build the pedestal. The thing that you don’t know the bottleneck to success is whether you can train that monkey or not.

[00:47:02] Annie Duke: first of all, you should start there. And second of all, when you’re thinking about, is this an investable business? Should I actually spend my time doing this? That’s where you want to say, what’s the probability that I can actually get the monkey to actually juggle these flaming torches? Because that’s the thing that matters the most.

[00:47:16] Annie Duke: In any project that you approach, there are, monkeys. Or any investment that you make as you’re thinking about what needs to be true for this to be a successful investment or for it to fail. One is a premortem, one would be a back cast. There are monkeys and then there are pedestals. for example, coding is a pedestal because you can kind of code everything.

[00:47:36] Annie Duke: Achieving product market fit is a monkey. That’s sort of the idea, right? So it’s not, I’m not saying coding is not hard and it’s not work. It’s that you already know you can do it. So once we’ve identified the monkeys, now we can start to develop kill criteria around those things. And basically it’s, I didn’t solve this monkey.

[00:47:55] Annie Duke: Looking back, I saw that there were early signals that there was no way that I was going to solve that monkey. What were those early signals? And now you can sort of write down a list of what those things are. Like, what are the things that would tell me that I ought to kill? An example that, that I give sometimes it’s let’s imagine that you have a thesis.

[00:48:15] Annie Duke: around Bitcoin. So you invest in Bitcoin and your thesis is that it will be a hedge against inflation. Okay. So we, automatically that’s a very, we know what the bottleneck is, right? what if it isn’t? So now we can just say, okay, let’s imagine that inflation soars. And Bitcoin goes down while inflation is soaring and looking back, I could kind of see that was going to happen.

[00:48:41] Annie Duke: And I held on to it too long. What are the signals? And then you can, you set those things out. So you can say the correlation has to be this high and persistent for this period of time. And then I’m going to kill. So that brings up the states and dates problem is it’s not enough just to say. if it turns out that it’s, not a good hedge against inflation, I’m going to sell.

[00:49:03] Annie Duke: And the reason is that once you’re kind of in it, particularly as you’re taking on losses, you’re just very like unlikely to actually sell unless you set a state and a date. A date is the state and date here. For example, if we say the correlation has to be, this has to be above this correlation. So that’s the state for this period of time.

[00:49:26] Annie Duke: That’s the date. So the simplest example of this would just be a stop loss. You can set a date to it, but a stop loss is really just a state, right? And in this particular case, it’s fine because it’s a trigger. I bought it at 50. If it’s trading at 40, I have to sell it. So that’s a very, simple version of a kill criteria.

[00:49:46] Annie Duke: Another one is if you’re summoning Everest, if I’m not at the summit by 1 PM, I have to turn around, right? And so that, again, combines a state and a date, and it creates a pre commitment device. That makes it much more likely that you’re going to walk away when you ought to instead of hanging on to things too long.

[00:50:02] Annie Duke: And we know that people tend to hang on to things way too long. And so you’re trying to basically counteract that particular bias, hanging on to your losers, which I’m sure everybody who listens to this has done before and has felt the one thing that I just want to just sort of say explicitly is that every thesis, every investment thesis, every bet that you make, kill criteria are implied.

[00:50:25] Annie Duke: If I invest in Bitcoin because I think it’s going to be a hedge against inflation, you sort of, you think intuitively that, then if it turns out that it’s correlated with inflation, then obviously I’m going to, in this case, negatively correlated, then obviously I’m going to sell. And it just turns out that intuition is just really bonkers and wrong, and you just need to let go of the idea that you’re going to behave rationally under that.

[00:50:47] Annie Duke: So by setting kill criteria in advance, you just make it much more likely that you’ll actually act rationally. And here’s the important thing to what your own thinking is, because it’s your thesis, not somebody else’s. So you’re more likely to behave rationally in relation to your own thesis, if when you generate the thesis, when you generate the bat, that you actually write down what these criteria are for why, when you would exit.

[00:51:11] Kyle Grieve: I really enjoyed your detailed breakdown of the endowment effect, especially how it’s affected by consensus versus non consensus views. So value investors tend to explicitly try and hold non consensus views, so it would appear that they would put themselves at a higher risk of suffering from the endowment effect as they are more likely to double down on their thesis, even in the face of disconfirming evidence.

[00:51:32] Kyle Grieve: So I’m interested in knowing some of the best strategies to use so we can try to detach our identity from our investments to help reduce the negative impacts of the endowment effect. 

[00:51:42] Annie Duke: Yes. So first of all, let me just explain what the endowment effect is. So the endowment effect is, it was originally about ownership of objects.

[00:51:51] Annie Duke: If I own a particular model of car, I’m going to think it’s more valuable than identical car that I do not own. and I think we’ve all had that feeling of, we’re going to sell a used car and they, we get offered a, we look at the Kelley Blue Book and we’re like, highway robbery, Kelley Blue Book is wrong.

[00:52:08] Annie Duke: That’s completely ridiculous. My car is worth more than that. But then when we go to buy an identical car, we’re always like, highway robbery, you have this price too high. But it turns out that the endowment effect is not just about items that we might own. It’s also about ideas. So we value our own ideas more than the identical ideas that other people might generate.

[00:52:32] Annie Duke: We’ve all had that feeling in meetings where we’re like, wait a minute, I just said that. That’s, and obviously what ends up happening is that we can kind of think about it as, we have ownership over our ideas. And our own theses and the investments that we have, we, both own physically, but also it’s our idea.

[00:52:52] Annie Duke: And then when we start to get into this consensus versus non consensus problem, what ends up happening is that identity starts to get mucked up in there. When I have a consensus point of view. My identity isn’t nearly as tied into that point of view as when I have a non consensus point of view, because if I have a consensus point of view, my, my view is not unique.

[00:53:17] Annie Duke: It turns out that’s not true. My identity is not going to be nearly as threatened. I try to think about it. I used to believe Pluto was a planet, but so did everybody else. So when scientists told me that Pluto wasn’t a planet, I was like, okay, I don’t care. Pluto’s not a planet, whatever.

[00:53:32] Annie Duke: But for flat earthers. it kind of doesn’t matter how much evidence you give them. They just believe the earth is flat. It’s a real stake in the ground, right? Like it’s part of who you are. And so this happens when we’re, sort of contrarians, right? When we’re taking a contrarian point of view, it’s much harder for us to update rationally in the face of new evidence than when we have a non contrarian point of view.

[00:53:54] Annie Duke: So this is really, as you pointed out, like an investor’s dilemma, at least unless you’re indexing. Otherwise it’s an investor’s dilemma. The way that we solve for these kinds of things is twofold. One, we already talked about, which is you really have to set really good kill criteria, because when you’re going to be at your most rational is when you’re thinking about something that’s happening in the future.

[00:54:15] Annie Duke: Because when you think about something that’s happening in the future, it doesn’t feel like you, I’m sure you’ve had that feeling of committing to something that’s six months away. Because it’s you don’t feel like you’re the one who’s actually going to have to do that. And then it gets to be the day before.

[00:54:29] Annie Duke: And you’re like, Why did I ever agree to this? Because all of a sudden now it’s about you. So by thinking about kill criteria in advance, you’re sort of becoming a good advisor to a future version of you who’s going to be subject to the endowment effect in these issues of internal and external validity, which is this fancy way to say your identity.

[00:54:48] Annie Duke: So that’s the first thing you can do. The second thing, and this, I think is so incredibly important is to get yourself an outside advisor, some sort of coach, acquaintance coach, that’s going to help you to see the situation more clearly than you might see it yourself. Like Eric Seidel was very helpful for me.

[00:55:09] Annie Duke: In that role, sort of as an outside person that I could go and say, Hey, what do you think about how I played a hand? Because when I play a hand that’s my idea, that’s my identity. I don’t want to be told that I’m wrong about it. So I go and talk to him so he can help me figure out if that’s a particular idea that I ought to quit.

[00:55:26] Annie Duke: One of the things that I think is so fun is Daniel Kahneman has told me that he has a quitting coach because researchers start lines of research. And then tend to hang on to them too long. And he doesn’t want to do that because he wants to use his time efficiently. And his, his, quitting coach is Richard Thaler.

[00:55:43] Annie Duke: Who’s also a Nobel laureate. So we all be so lucky. But he actually uses that to figure out, should I still be pursuing this line of research? Is this too much of a dead end? Are there enough signals that tell me that I should drop it? So we all kind of need those types of people. Now, the best thing that you can do is combine the two.

[00:56:01] Annie Duke: When you’re going into an investment, You’re putting a position on. Set out kill criteria. It’s, actually very helpful to have an outside view on the kill criteria itself. And then now you’re committing with somebody else that you’re going to follow the kill criteria. And that’s really helpful because then when you see them later and they’re like, Oh, you must have sold Bitcoin because I saw what happened in terms of that correlation.

[00:56:26] Annie Duke: Boy, are you embarrassed if you didn’t actually do that? Because you committed to do that with them. So these are all things, the way that you can think about it is like, you want to help your future self by getting an actual outside view, somebody who’s not, isn’t endowed to the position in the way that you are, but then also by advising that future version of you, which from your cognitive standpoint is not even you in the first place.

[00:56:51] Annie Duke: So it’s, you sort of become an outside advisor to yourself. And both of those things will help you to act much more rationally. 

[00:56:58] Kyle Grieve: You mentioned the outside view and I actually I wanted to ask you specifically about that. So the first person who I learned that from was Michael J. Mobison from his excellent book, Think Twice.

[00:57:08] Kyle Grieve: For me, the kind of the one of the, one of the ways I like to look, use the outside view is I’ll look at my portfolio and then I’ll imagine myself inheriting the portfolio from somebody else. And I’ll ask myself, if I got this today, what would I want to sell? What would I want to hold on to?

[00:57:23] Kyle Grieve: What would I want to add more of? So I’m just interested in, in, in learning more about what you think of the outside view and how you use it to help combat some of the biases. Obviously, the endowment effect works great with that, but I’d love to know more about your thoughts on that. 

[00:57:35] Annie Duke: I’m now going to make this podcast like super worth your time.

[00:57:39] Annie Duke: The thing that you just said doesn’t help. So let me explain. But it’s very common, right? So I hear people all the time say, imagine that you didn’t own any of this. What would you buy? And the reason that you intuitively think that will help is because you do know about the endowment effect.

[00:57:58] Annie Duke: You do know about the sunk cost effect. You know about all those things so that you think that doing this thought experiment, what if I were fresh to the decision? You’re sort of imagining you’re fresh to the decision that it would be helpful. The science shows very clearly that is not helpful.

[00:58:13] Annie Duke: It’s actually in some ways counterproductive because you fooled yourself into thinking that you’re being rational about it. So it’s a question of like, when you say, would I start this today? You think that you’ve solved the problem. Let me just take it back a step. What we’re trying to do to get to be rational is only continue to do things that we would start today.

[00:58:33] Annie Duke: Assuming there’s no transaction costs, obviously, otherwise you have to take those into account, but let’s be simple and say there’s no transaction costs. So you only want to hold positions that you would buy today. If you wouldn’t buy them today, you want to sell them. That’s clear. Okay, great. So now, obviously, the best thing that you could do is clear the decks, okay, so we don’t like to close accounts in the losses, so you could actually just sell everything.

[00:59:00] Annie Duke: Let’s imagine there were no transaction costs, so it’s free to sell everything. Then I would tell you to sell everything every Monday morning, so that would solve it. And then you have to now decide whether you’re going to buy these things back. So clearing the account allows you to reset and start back fresh.

[00:59:19] Annie Duke: Now, obviously, that’s impractical because there are transaction costs. So I don’t actually want you to sell everything every morning. Please don’t do that. But if you can, I mean, this is something that you see with poker players. Actually, there was a really wonderful study that was done on people who were playing slots.

[00:59:36] Annie Duke: Where they had their like player card data and during the day, if they got into the losses, they started, they would keep playing, they play longer, they’d increase their risk and whatnot, and they start betting more and being more and more rational. But then if they quit for the day and came back the next day, they would go back to what their normal risk attitude was.

[00:59:57] Annie Duke: So it’s the clearing the decks, right? Okay, I’ve had to cash out. Now my, I’ve cleared the losses out of the mental account. Now I’ve got to start again. So this is, something if people want to sort of look up a lot of Richard Thaler’s work, it’s this mental accounting problem. And we can’t kind of trick ourselves.

[01:00:13] Annie Duke: We can’t do this Jedi mind trick to make it better. If you can sell it with no transaction costs, you should probably do that. And then, start fresh. Outside of that, you’re just going to have to go back to Kill Criteria and Acquitting Coach. What the Kill Criteria allow you to do is to say in advance, Is this something that I would hold?

[01:00:32] Annie Duke: If you’re doing, say, a Monday portfolio review, part of the, that portfolio review, depending on your, the duration of the hold should be to review the kill criteria. is there anything new? Is there a new monkey that’s come up? Is there an adverse signal that I’m now thinking about that I’m going to see in the future?

[01:00:52] Annie Duke: And so on and so forth. So you’re refreshing on some sort of regular cadence, what those kill criteria are. And it’s really good if you partner up with somebody else who can see your portfolio more clearly than you can. And those things are all going to help you, but it’s all this pre commitment devices that are actually going to help you.

[01:01:09] Annie Duke: Everybody does what you do. Everybody does what you do. So I’m going to ask you to stop doing that. 

[01:01:15] Kyle Grieve: Will do. So you mentioned Daniel Kahneman, how you spoke to him and how he has Richard Thaler as a quitting coach, which obviously, like you said, we should be so lucky. 

[01:01:26] Annie Duke: We should be so lucky. And you know why?

[01:01:27] Annie Duke: Because he knows he’s not rational when he’s in the middle of a research program at knowing would I start this today? He’s not going to give a good answer to that. 

[01:01:35] Kyle Grieve: There you go. My question more is just on, I guess, on, the nitty gritty of finding a quitting coach. So for instance, investing.

[01:01:44] Kyle Grieve: It’s not exactly the type of thing that everyone, who you associate with, your friends and family care about at all or have any, 

[01:01:52] Annie Duke: Don’t have them be your quitting coaches. 

[01:01:54] Kyle Grieve: Yes, exactly. So I’m just interested, you’d have to get a quitting coach who hopefully has some something invested in me, hopefully a friend of some sort, but once you get them, what are kind of the steps?

[01:02:07] Kyle Grieve: What are the, what are you asking them specifically to do just to help you make better decisions? I was interested in getting into the nitty gritty of that a little bit more. 

[01:02:16] Annie Duke: Okay, first of all, so when you find it, when you find someone who you think is going to be good. I, for example, I think that solo PM should pair up.

[01:02:23] Annie Duke: One acting as the quitting coach for the other, and vice versa, that’s why they should pair up. One of the really important things about a quitting coach, and this is true whether you’re an investor, or you actually have someone that you’re going to, because you’re wondering if you should leave a relationship, or a job, or whatever, is that there has to be an act of permission giving.

[01:02:42] Annie Duke: And the permission has to be for you, Kyle, to tell Annie, What’s in my long term best interest. And the reason for that is that you’re going to have a tendency to think that you’re being nice to me by telling me what you think I want to hear. If I’m in a relationship. And I’m asking your advice, you’re probably not going to tell me you should break up with that schmo, because it’s you think that you’re hurting my feelings, right?

[01:03:08] Annie Duke: Or no, this, your startup is going nowhere. Why don’t you shut it down? No, you’re going to be like, rah, You should keep going at it. I know you can do it. Keep trying. You’re so smart. You’re so great. The first thing is that I have to tell you, don’t worry about my short term bested, like what you think is my short term feelings are going to be.

[01:03:27] Annie Duke: I don’t care if you hurt my feelings in the short run, because I want to know that I’m doing well in the long run. So I don’t want Eric Seidel to tell me I played a hand great because he doesn’t want to hurt my feelings. That’s really bad for me in the long run. So I have to tell him, I want to know what you really think.

[01:03:42] Annie Duke: And if you think I butchered the hand, tell me that. Because otherwise you’re really hurting me in the long run. So that’s the first thing is that there has to be this permission given. And then the second thing, it’s really, as I said, it’s about figuring out what’s realistic in terms of the setting of kill criteria, kind of running that by them in a neutral way and saying, so what do you think are the signals or do you think these things are reasonable?

[01:04:07] Annie Duke: And then committing to actually following those and maybe you can actually do a little, like a kill criteria review with them and sort of set that up on a regular basis and do the same thing for them. and, the reason why you really want, part of the reason why you want an outside view, and I’ll give you like a simple example of this.

[01:04:24] Annie Duke: On the setting of the criteria in the first place is let’s imagine that you’re 21 years old and you just got your first job out of college and you come to me as your mentor and you say, I’m going to quit this job if I don’t get a promotion within six months. I have a lot of experience and I can tell you, hey, seems to me that’s unreasonable.

[01:04:43] Annie Duke: I don’t think that you, 21 year old Kyle should have the expectation that you should be getting a promotion within six months. And why don’t we delete that as a kill criteria? And then I can say it’s more reasonable for you to say like within 18 months that you get some sort of something like some bump or a title change or whatever, but we can figure out whatever that is, but so I can sort of tether you to reality like when we go back to way back when we were talking about this idea of applying your own experience to what are the chances my restaurant is going to succeed.

[01:05:15] Annie Duke: If you go talk to another person, you can look up the base rates and see that only 40 percent succeed. But you can also talk to someone who’s opened a lot of restaurants and they’ve been in the industry for a long time, and say, I think there’s an 80 percent chance my restaurant’s going to succeed. Tell me if I’m being unreasonable, and don’t tell me because what you think I want to hear.

[01:05:33] Annie Duke: I really want to know. I need you to tell me the truth here. They’re probably going to say, that’s nuts. Yeah, like 80 percent is a ridiculous number. No, you have to downgrade that. That’s where that they can be really helpful. If you give them permission to tell you the truth, they can help you both on the setting up of the Kill Criterion.

[01:05:51] Annie Duke: Then they can also help you to adhere to it. So it’s like getting you into the right area of like expectations and criteria and that kind of. 

[01:05:59] Kyle Grieve: So Annie, I just want to thank you so much for coming on the show today. This was an incredible conversation. Where can the audience learn more about you and your books?

[01:06:07] Annie Duke: Okay, I’d love for the audience to go to the Alliance for Decision Education. That’s the nonprofit that I mentioned where we’re trying to bring these types of concepts to K through 12, right? Like, how do you become a better decision maker? I think that much more important than teaching trigonometry, going back to our transfer training thing, which doesn’t really do anything for you unless you’re going to be a structural engineer.

[01:06:28] Annie Duke: So I really love for people to visit that. but other than that, I have a sub stack called thinking and bets. People can catch me there. I teach on a platform called Maven. That’s where the public can get my teaching. I also teach at Wharton, but you know, you could enroll in one of my classes at Wharton, but you got to travel there and whatnot, but Maven is online.

[01:06:46] Annie Duke: So I’d love people to go check out maven. com. And then AnnieDuke. com. you can go find out my goings on and you want to hire me for something. You could contact me there and I’m active on social media, but, most of the things that I’m really doing is are on Substack. 

[01:07:05] Kyle Grieve: Okay, folks, that’s it for today’s episode.

[01:07:07] Kyle Grieve: I hope you enjoyed the show, and I’ll see you back here very soon. 

[01:07:11] Outro: Thank you for listening to TIP. Make sure to follow. We Study Billionaires on your favorite podcast app and never miss out on episodes. To access our show notes, transcripts or courses, go to theinvestorspodcast.com. This show is for entertainment purposes only.

[01:07:28] Outro: Before making any decision, consult a professional. This show is copyrighted by The Investor’s Podcast Network. Written permission must be granted before syndication or rebroadcasting.