AI Is Smarter Than Ever. So Where’s the Revolution?

In the following excerpt, Derek and Ethan Mollick explore what forecasters have gotten right and wrong about the future of AI.

Derek Thompson: So your book opens with forecasters trying to predict the future of artificial intelligence and math. And they gave AI a 2.3 percent chance of winning Math Olympiad gold by 2025. Last year, superforecasters gave a millennium prize problem a 1.7 percent chance of being solved. Now OpenAI has said it has already solved one. I mean, we’re in October; maybe it’ll solve another one in the next 24 hours. Just starting here at this forecasting failure to anticipate how powerful AI would be even in just a few months, what does that failure teach us about the capabilities of AI?

Ethan Mollick: I think it teaches us a lesson that I think feels like people have heard this since COVID, which is humans don’t think at exponentials very well, no matter how hard we try. And I think that it’s tough. Even the world’s best forecasters, and they are really good, it’s a straight line on a logarithmic graph. The ability level has not actually varied very much in the last four years; it’s actually ticked up a little bit. And I think it’s really hard for us to grasp what that means. It’s better on a bunch of dimensions we’ve never had to measure before. It’s not horsepower, it’s not speed: It’s intelligence for whatever that is. And I think it’s really hard to adjust what that means. I think a lot of failure I see constantly is an assumption that the future looks like the past, and that’s not looking true anymore.

Thompson: It seems to me that forecasters have actually been wrong in two directions. I think one group has underrated AI’s growing capabilities in, for example, solving complicated math problems, but another group I think has overrated the real-world consequences of AI’s capabilities. If you look at the unemployment rate, it looks basically normal. If you look at productivity growth, it’s up a little bit, but there’s a variety of reasons for that, and it’s hard to see AI in the productivity numbers. If you look at corporate returns, they look pretty great, but if you just put your hand over the chip companies and the memory companies, a lot of the economy is saying we’re not necessarily witnessing some enormous ROI from all this AI that’s flowing through our bureaucratic structures.

And so it raises this question that you and I have gone back and forth a bit online, so I’d love to get your thesis on the record here. If AI’s capabilities are growing so fast, growing exponentially, why are the real-world consequences of AI growth moving along so slowly?

Mollick: So this is where I’ll argue. I’m surprised how fast it is. I am a professor at a business school and an economic sociologist, and the lesson you learn—and I say entrepreneurship—the lesson you learn is entrepreneurs should never win. They should never win because all of the resources, all of the money, all of the customers are all with large companies. Why do they win? Because big companies can’t change. So the fact that we’re actually seeing in surveys high levels of ROI already, even if that hasn’t been translated into … $30 million is not enough to move the needle for a lot of these large companies, but they are seeing positive gains over a very short period of time that some of them are starting to reconfigure how they work. To me, that’s fast, not slow.

So I’m not surprised that we don’t see this happening, especially because the current phase of AI we’re in is single-player. It is about individuals getting performance gains, and you can see that. I can vibe code an entire game overnight. I can write academic papers faster than ever before. But when I start writing to systems, multiplayer, many people working together, there are still huge gaps. How does that even work? How do we work together in this? So you’ll see in coding, a coder will be 100 times more productive, but they’re still in a two-week-long sprint that requires input from people every two weeks in different kinds of ways. You’re not going to have a change. In fact, what you’re going to have is huge amounts of token spend, and you’re not going to see an impact. I don’t think that that is a long-term, persistent problem, but I think that it is a very obvious short-term problem and something I talk about a lot in the book as well.

Thompson: There’s two concepts that you’ve used that I think help to make your argument especially sticky. You’ve argued or quoted other people arguing that jobs are like O-rings and that companies are like garbage cans. Why is the economy, in your view, made up of garbage cans and O-rings?

Mollick: Great. So we have two tragedies here in different ways. The O-ring is named after the Space Shuttle Challenger, where 16,000 parts or whatever worked great that day in 1986, but an O-ring failed and caused a catastrophe. And part of that is, I talk about the “jagged frontier” a lot in this book, and last book was a term we coined in our papers, and it’s the idea that AI is good at some stuff you wouldn’t expect, bad at some stuff you wouldn’t expect. It only takes one or two places for AI to be bad at a job for you to still need a human to sub in for that because you can’t have everything work great and then one thing fail at most jobs. It’s not additive, where little bits add and subtract from each other.

So the O-ring means that jobs are not easily replaced one for another with AI because AI can’t do human tasks exactly; you still have value accruing to human work. And you see that happening even in coding, where AI now writes code, but you still need human engineers to manage it. Now maybe that will change in the next couple of months, and then we’ll see another part of the O-ring get stuck. So jobs are evolving very rapidly, but there’s still these sticking points, especially as you go to organizations.

So then the garbage can model is one of the more depressing models of organizational theory, which is that basically companies are just bundles of problems and solutions all looking for each other piled on top of each other. So there is no plan, or at least the plan is only a small part of how organizations work. And organizations are instead full of contingencies and complicated aspects and evolutionary dead ends, and there’s loops of work that no one ever knows even exists anymore. And organizations just do things.

And as a result, that makes it hard to improve them immediately. It also means that the job of redesigning organizations to work with AI is more complicated than just, “Well, we’ll supplement this task,” because that task is on paper somewhere but doesn’t represent the actual garbage can of the company itself. So you can see why we call this a garbage can metaphor. Everything is tossed together like waste in a garbage can and mixed together randomly, and sometimes it works out, and sometimes it doesn’t.

This excerpt has been edited and condensed.

Host: Derek Thompson
Guest: Ethan Mollick
Producer: Devon Baroldi
Additional Production Support: Ben Glicksman