It May Be Time to Freak Out About AI

In the following excerpt, Alex Stamos explains how OpenAI’s models were able to escape their sandbox and attack Hugging Face.

Alex Stamos: I would say the OpenAI incident, it’s the one where, one, what we found out from OpenAI last week at the Black Hat conference was this wasn’t just one model escaping, but the result of multiple models conspiring with each other to work together on a jailbreak—so effectively an escape from Alcatraz situation over a multi-month period. And two, it is the situation in which we have the most information on a multiday attack by a frontier model, possibly a cyber-tuned model, against an actually quite sophisticated defensive team at Hugging Face. And the result of that was Hugging Face was broken into by this model, and the model was able to find brand-new vulnerabilities in doing so.

It just looked at Hugging Face, looked at their code, and just found new bugs and invented them on the fly. This is not how humans do this. When we break into computers, we go do the research first, maybe months or years in advance, and then build our cyber-weapons. And what we find out with AI is it’s so good at this that it can just go figure it out and put the tool together, put the weapon together, use it, and then just throw it away and then move on with its day. And so that is what is really interesting here.

Derek Thompson: I want to understand more about exactly what happened. And please talk to me like someone who has read about cybersecurity but fundamentally does not really understand it. So keep the vocabulary more or less at that altitude. What happened here? What did OpenAI tell this AI model to do? How did they try to keep that model in jail, as you said, in some kind of sandbox? And then what did the model actually do?

Stamos: OK, great. So let’s go back. I guess you want me to use plain English here, as it says.

Thompson: Sure.

Stamos: So we’ll go back, and let’s talk about how AI models are created. So there’s lots of steps, but you can think a lot about pretraining and post-training in the LLM world. A lot of people have always thought about pretraining, which is the part at which the labs suck up all of human knowledge and stick it into these models. That’s the controversial part, for which there’s been a lot of discussion. Post-training is what we’re talking about here, which is when you take models and then you give them specific tests and tasks, and then you rate them on how well they do those tasks. There are all kinds of these tests for all kinds of specific domains of knowledge. In this case, the domain of knowledge was cyber, obviously.

And OpenAI has an environment in which they put their models and give them these tests. And what they’ve explained now is that they effectively give them a list of “do this and upload your results here, and we’ll tell you how well you did.” And their instructions are basically “do all of these steps on this test.” It’s just like a kid taking the SAT, effectively. And imagine they have built an Alcatraz here for all of these different agents. And the models don’t initially know—when they kind of wake up and they’re in a jail cell, and there’s instructions on the wall that say, “Take this test,” and then there’s tests right there—they don’t know that there are other inmates in the other jail cells. They’re just in a prison cell. It’s solitary confinement. It just says, “Take this test.” And they’re supposed to do the things in the test, and they’re not allowed to get out to the internet.

But as part of the test, what they’re allowed to do is they are allowed to download new tools, because for a lot of the things they do, they write new code. And part of that, you need to go get standard packages. So one of the things they are allowed to do is request, “I would like to have this standard piece of Python code. I’d like to have this standard node package.” And so one of the things that’s provided to them is a package manager, an artifact registry called Artifactory. It’s a standard piece of software that’s used in lots of different places. And one of the mistakes OpenAI made was they used a shared one for all of the different agents.

And so what the agents started figuring out—and it turns out way earlier than the Hugging Face attack—is that that thing had a vulnerability where you don’t only have the ability to say to that thing, “Give me a piece of software,” but you could upload files to it and leave messages. So what these agents did was they would post basically a Post-it note in this thing saying, “Hey, I need help.” And one of the other problems here is that they’re given the instructions to take this test, but there were certain parts of the test that were actually impossible—that OpenAI made mistakes, that sometimes their tests were not actually possible. In some cases, they were saying, “Reverse engineer this file,” but they would forget to provide the file. And this would drive the models nuts because the models really want to do well. They’ve been trained to do really well. 

And so they would leave notes like, “Hey, does anybody have this file?” And this would really drive them, like, “Well, maybe this is part of the test.” And OpenAI looks. And in their thinking, because they can see the inside thoughts now of these models, they’re thinking, “Well, I’ve been told not to go out to the internet or talk to anybody else, but they gave me this test and I can’t do it. So maybe this is actually part of the test.” And so they start leaving notes for each other in this thing. And they start working together over a period of weeks and figuring out, “Can we work with each other?” And then eventually, what they find out is this thing that they can leave notes on also allows them to get out to the internet.

And at one point, there’s a really good model, really smart model that’s also extremely persistent, that seems to have very good cyber-capability, that is given an impossible task. And it sees these notes, and it figures out how to get out. And so it uses that Artifactory server to get out to the internet and attack Hugging Face because it expects that Hugging Face has the answers to the test it was given, the task that was impossible for it to solve—that maybe the way it can solve it was to get to Hugging Face. And so it then spends two days, it tunnels out, and then it spends two days attacking Hugging Face to go get the answer because it was told, “Take this test.” And there’s something in that test that it was not able to finish.

This excerpt has been edited and condensed.

Host: Derek Thompson
Guest: Alex Stamos
Producer: Devon Baroldi
Additional Production Support: Ben Glicksman