The Single Smartest Case Against AI Doom

In the following excerpt, Derek, Sayash Kapoor, and Arvind Narayanan discuss AI as a superintelligence vs. a general-purpose technology, and the possible economic effects of AI.

Derek Thompson: You are the inventors of this framework that says we should think of AI as a normal technology. Before you tell me what that means, let me ask you the opposite question. What would it mean for AI to be an abnormal technology? Sayash, what is the worldview that you are arguing against?

Sayash Kapoor: So there’s this major point of discussion in a large part of the AI community that treats AI as an impending superintelligence. So in this worldview, you don’t treat AI as yet another general-purpose technology in the long history of technologies that we’ve invented, but it’s more like a new species are coming up with a potential successor species that can take over the reins to this world that will autonomously drive what we do with very little role for human insight. So it’s a combination of two things. On the one hand, there are these technical advances that the AI safety community and, more specifically, people who are in the rationalist and the effective altruist communities have thought AI will soon accomplish. This includes AI is becoming much, much better than humans on every single thing. They can persuade people, they can forecast things. And as a result, they can get to do basically whatever they want in the real world, whether acting by themselves or through people. The second part, though, is political.

These AIs would be so far ahead of humans that they would essentially have the ability to create the political will to accomplish their goals. One thought experiment that is often raised in this community is that of the paper clip maximizer where you have an AI system that humans sort of task with a benign task, like maximize the number of paper clips that we’re building, presumably that was meant to be in a factory, but the AI is sort of take this single-minded focus and convert the whole earth into a paper clip factory. And presumably, they’re able to do so not just because of their technical competence, but also because they can manipulate all of our institutions and physical systems and orient them towards this goal. So this is the view that we are arguing against, that AI can be this sort of totalizing entity that’ll be so far ahead of humans. One comparison that’s often made is that the AIs would treat humans as humans treat ants today, and we won’t really have a lot of say in the matter. That’s what we view this sort of alternative vision of abnormal technology as.

Thompson: So, Arvind, your co-author has explained what you’re arguing against. Now, I think you should tell us what you’re arguing for. And in particular, normal is such a deliberately ordinary word, but you are not saying, “We think AI is like a toothbrush. We think AI is stupid. We think it can’t do anything.” You’re talking about something that you acknowledge could be as transformative as electricity. So what does this framework of AI as normal technology give us?

Arvind Narayanan: That’s right. Thanks, Derek. AI is an important technology. It’s a powerful technology. It’s a general-purpose technology. We repeatedly compare it to electricity and the Industrial Revolution, but it’s important to keep in mind that those technologies did not transform the world overnight. So one plank of this that we’re arguing against is on the economics, that whatever bottlenecks or barriers stand between AI and economic impact, AI itself will be such a powerful agent of change that it will overcome them. We don’t see that happening. And we see evidence for this, for instance, in software engineering where AI has, in fact, been rapidly adopted. The role of a software engineer has been pretty dramatically transformed over the last year or two. I think manually writing code today is almost like going back to the days of punch cards. Most software engineers, at least at many companies, are essentially these agent operators.

And yet, it has not replaced software engineers. It has not led to an explosion of software that is visibly of higher quality from a user perspective. It has not led to the SaaS-pocalypse, which was widely predicted a year ago, which is that software-as-a-service companies will go out of business because every company, a bank or a law firm or whatever, will simply be able to roll their own software using AI. Now, many or most of those things are possible in the long run, but from an economic perspective, we do think that the barriers and bottlenecks which involve human behavior and organizational change and regulatory barriers, those very much apply to AI as well. So that is the first way in which we would defend the perspective of AI as normal technology, but I can talk specifically about safety as well.

Thompson: Well, it’s been 18 months since the original paper, maybe 16, 18 months since the original paper, AI as Normal Technology. Arvind, what is the strongest piece of evidence today that you’re right? And what is the single best piece of evidence that worries you that you might be wrong?

Narayanan: Yeah. Let’s start with the latter. I think on safety, we definitely got some things wrong. One point we made in the paper is that we don’t need to worry so much about what happens inside AI companies because both benefits and risks only arise when AI is deployed, not when it is developed. I think at a high level, that is certainly true. Especially when it comes to the benefits, there is this long process, as I’ve been talking about. But specifically, on the risks, what we didn’t anticipate is that the evaluation of new AI models, which happens inside AI companies, is some of the riskiest part of the pipeline between development and deployment because that is a time when the model is new, some of the capabilities might be new, risks are not fully understood, and a lot of the evaluation has to happen in a way that certain safeguards are reduced.

And sure enough, when we look at the events of the past few months, that has been a major source of the risks that we’re actually seeing. So definitely, we need more transparency into what is going on inside the AI companies. There are certain aspects of a coordinated slowdown that we’re on board with, although we think the really critical question is what happens once you slow down. It’s not so much the slowdown itself. I think both on safety and the economic part of it, we got a bunch of things right. So on safety, this was a minority position when we wrote the essay, but a point that we made is that it’s not so much the absolute capability level that matters, but how that capability level changes the attacker-defender balance because increase in capabilities help both. That is almost taken for granted today. It has become very much common knowledge, but I think we were among the first to very prominently say that.

And more importantly, we had an academic paper behind that that proposed what is called a marginal risk framework. It was a big collaboration, but we were among the leaders of that paper that proposed that what we should be looking at is how AI changes the marginal risk, the risk compared to what came before it, how it helps attackers versus defenders, as opposed to panic about a particular capability threshold. If we were to panic that way, GPT-2 would’ve been a disaster. And if we recall, back in 2019, OpenAI delayed the release of GPT-2, a trivial model by today’s standards, because they were worried about what it could do. But really, the thing to worry about is how it changes this balance. So I think that’s one big thing we called early on. And then on the economics of it, as I’ve been talking about, a lot of these bottlenecks have now become very clear. Sam Altman, to his credit, recently on a podcast, openly changed his mind about how slow he thinks these economic timelines are going to be.

This excerpt has been edited and condensed.

Host: Derek Thompson
Guests: Arvind Narayanan and Sayash Kapoor
Producer: Devon Baroldi
Additional Production Support: Ben Glicksman