The Bitter Rivalries Behind the Race to Build Superintelligence

In the following excerpt, Derek and Kevin Roose discuss building superintelligence, Sam Altman, and AI safety.

Derek Thompson: Kevin, what is superintelligence and why do you think we’re on the precipice of building it?

Kevin Roose: Superintelligence you could think of as an AI system that is smarter than everyone at everything. There are sort of these gradations of AI capabilities. People talk about AGI, artificial general intelligence, as a computer program that is human level or better at most tasks, or most intellectual tasks. Superintelligence would be the level beyond that where we have something that is vastly smarter than us at everything that we could conceivably do.

Thompson: Your new book is called The AGI Chronicles: The Inside Story of the Race to Create an Artificial Superintelligence. It seems from your reporting that you think this is a race that is on the precipice of achieving something quite dramatic in the near future. What have you seen, let’s just say in the last 12 months, that have made you think we’re really making progress toward this goal, as you’ve defined it?

Roose: Well, let’s back up more than 12 months to 10 years ago when the book starts, which is we had these AI systems that could do things like identifying objects in images or translating between languages or helping you with autocorrect on your email. But those were narrow systems. They could only do one thing. They couldn’t be repurposed to do 20 different tasks.

Now, we have these general models, Claude, ChatGPT, Gemini, that can do a wide range of things near or above human-level. We have systems that are capable of solving Millennium Prize Problems in math, of discovering new protein structures, of helping people, billions of people around the world every day in various ways. So I think that’s what I’ve seen, is just the continued march of this technology.

And then the non-public thing that I’ve seen is that the people at the companies who are building this stuff, I guess this is not a secret anymore, are terrified of it. They have been for years, and they’re not getting any calmer as we approach what they think could be some kind of intelligence explosion, which we could talk about. That’s sort of another concept that they think is not far off, and I’m inclined to believe them.

Thompson: One thing I think is so strange about this entire project is how small this world is. And when we talk about the Frontier Labs, we were talking about three companies leading the race to build superintelligence. They are OpenAI, Anthropic, and Google DeepMind. Those companies are run by, or have until very recently been run by three particular and peculiar dudes, respectively Sam Altman, Dario Amodei, and Demis Hassabis. I want to spend the bulk of our time trying to understand these men, who they are, what motivates them, their strengths, their weaknesses. And let’s just jump right into it. Sam Altman. Paul Graham, the former head of Y Combinator, once famously wrote this about Sam Altman: “You could parachute him into an island full of cannibals and come back in five years and he’d be the king.” What makes Sam Altman so good at becoming powerful?

Roose: He has this superpower, which everyone he’s ever worked with, all of his mentors, they have recognized this in him, and he had it from quite a young age, of figuring out exactly what someone wants and giving it to them, or at least appearing to give it to them. So back in his days running Y Combinator, people would sort of talk about this power of his, and they didn’t know where it came from, but people like Paul Graham recognized it very early. He was very good at reading the room, at figuring out, “This person expects me to do this or wants me to do this, so I will do that.” And it led to him becoming one of the most successful young entrepreneurs and investors in the Valley.

The flip side of it is that he would often get into trouble because he would tell one person one thing that they wanted to hear and then tell a different person something completely different that they wanted to hear. And he sort of got this reputation as a sort of slippery people-pleaser, which is, I think, a criticism that many people who have worked with him over the years have identified and have made in pretty consistent ways. He just seems to be in the business of both being quite ambitious, but also of being quite eager to please.

Thompson: I mean, one way to say this is that he is a brilliant dealmaker in the mold of a J.P. Morgan. Another way to say it is that he’s a bullshitter. And I wonder, after all of your reporting, whether you feel like underneath all of that dealmaking, there is a core philosophy.

Just to put my thumb on one distinction that might help you answer that question, there’s a 2015 blog post that Sam Altman wrote where he called machine intelligence, “Something we should be afraid of.” And when he founds OpenAI with Elon Musk, he founds it as a nonprofit because he thinks that this is a project that should not be a typical commercial business, but now he’s running a business that is by some accounts valued at 800, 900 billion, a trillion dollars, and later colleagues who work with him are very skeptical of the degree to which he’s serious about AI safety at all.

So, one way to ask this question is to say, does he believe in AI safety? But what I’m really asking here is does he believe in anything outside of just making the deal that increases his power?

Roose: I believe he has a core. I don’t know what it is. I’m not in his inner circle of people that he confides in. I’ve spent time with him, I’ve talked to him, I’ve had encounters with him as a journalist, but I’ve always sort of understood that he was playing the part in those interactions and wasn’t necessarily showing me his true self. And for what it’s worth, I think most executives do this. They have their persona that they put on.

I think the thing that kept coming up in reporting on Sam and the people around him is that he is a person who believes first and foremost in agency. He has this famous phrase, “You can just do things.” And this is something that he believed at Y Combinator. He would tell these young startup founders that he was investing in, “You don’t have to wait to change the world. Nobody has to give you permission. You can just go do things.” And that is a core part of his belief about himself and about those around him. Another way of saying it is there’s no grown-ups. You are the grown-ups, and you can just do things that change the world without permission.

He’s also a fervent believer in scale, the power of scale. People would tell me these stories about he would walk into a meeting and he would just tell the people in the meeting to add a zero or two zeros to whatever budget they had for their project going forward, or increase the scale of a training run just because he thought that bigger was better.

So I think that love of scale is probably part of his experience running Y Combinator and being an investor before OpenAI, but certainly OpenAI’s success has confirmed that just having this kind of delusional self-belief and belief in the power of scaling is actually what you need to win in AI because, this is something that didn’t really occur to me until midway through the book, belief is an active ingredient in AI’s success. You have to believe that the scaling laws will continue, that putting billions or trillions of dollars into building data centers and acquiring compute will pay off. The belief is the enabling force that allows you to win, and if you don’t have it, no amount of technical expertise can save you.

This excerpt has been edited and condensed.

Host: Derek Thompson
Guest: Kevin Roose
Producer: Devon Baroldi
Additional Production Support: Ben Glicksman