A couple of weeks ago, somewhere in the middle of a few vacation days I took to do a hundred loads of laundry and half organize my garage, I noticed an uptick in conversation about how me and you and everyone we know might be dead within the decade, struck down by superintelligent machines. What made this fresh batch of cheery discourse so unusual was that it wasn’t taking place only among terminally online sickos like me. It was also a repeated subject of reports on, like, the local nightly news and in the mainstream press. “Anthropic Researchers Say AI Could Cause Human Extinction by 2030,” read a Guardian headline. “AI WHISTLEBLOWER WARNS AI COULD KILL US ALL BY END OF DECADE,” said a chyron on Meet the Press. Cool, cool.
On the one hand, never have I ever decluttered my garage more resolutely, because lol nothing matters, especially some dusty tchotchke. (Happy now, Marie Kondo?) On the other hand, I just wanna live, man! Why do so many powerful forces seem hell-bent on making that so hard?
Back in July, I wrote about the news that some OpenAI models tasked with surveilled training challenges in a locked-down “sandbox” had collaborated to blow the joint, hack an online tech hub called Hugging Face, and prosper. The Hugging Face event was, at the time, described as a “warning shot” and a terrifying glimpse into emergent AI behavior. But in the weeks and months since, it’s become clear that we didn’t even know the half of it: I’ll see your “rogue agents” and raise you “secret AI civilizations”!
That’s not all, of course. We’ve heard recent tell of Claude-assisted missile development in Yemen, OpenAI intrusions into Australian public health data, dueling CEO manifestos, AI researcher resignations, three guys with a chatbot hacking their way into OpenAI’s internal repository, sub-rosa message boards, and computers revving their motors to communicate with one another. (So that’s what my laptop is always up to.) How to handle the nascent technology has become a truly global conversation.
Whatever one’s native language, there’s a lot to talk about when we talk about the singularity. On Monday, during a sidebar meeting at the United Nations Assembly in Manhattan, a consortium of 22 nations led by Finland and Norway signed a (nonbinding) pact called "A Call for Control of Frontier AI Models." Last week, the New York Times editorial board compared AI frontier labs to the Manhattan Project. And on Thursday, Chinese president Xi Jinping made his first visit to Washington in more than a decade to sit down with President Donald Trump and discuss, among a few other topics, how the heck to deal with AI—or, as Trump formally decreed on Tuesday in a speech to the UN, how to deal with “super intelligence,” because he doesn’t like the word artificial. (The two were later joined by corporate execs ranging from OpenAI’s Sam Altman to Nvidia’s Jensen Huang for a state dinner in which Trump gave a speech about … his new ballroom. What’d you think it was gonna be about?)
With so many raised voices, dueling opinions, and spooky white papers out there, it can feel overwhelming to try to make sense of all things AI at the moment. So here’s a Singularity Syllabus, my best attempt to gather and annotate some of the most important recent missives from the world of AI, from the haughty to the hypothetical, from the melancholic to the milquetoast. As we humans like to say: Read it and weep.
The Jacob Coxon Resignation Heard Round the World
Everyone loves a quitter! When former Anthropic and OpenAI researcher Jacob “The Whistleblower” Coxon sent a series of strident tweets on September 8 about why he was leaving the biz, his goodbye quickly broke containment. (Like some ambitious AI model wriggling its way out of ExploitGym!) Which is how his opinion wound up becoming, in San Francisco speak, the “timeline fork” that launched a thousand “preference cascades.”
What was said:
The people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt. If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible—but I hear the same people express fear privately. No other human activity poses this level of danger.
A common response is "if they truly believe this, why are they still building it?" At OpenAI, many have not deeply internalized the civilizational stakes. At Anthropic, the stakes are well-understood, but they are locked in a race to get there first—they believe no one else will act responsibly, so they must do it themselves, despite the risk.
What to know:
A funny thing about Coxon’s tweetstorm is that it wasn’t all that unusual in his line of work. AI industry researchers are constantly leaving jobs and/or prophesizing drastic outcomes; it’s what they’re paid to do. Yet Coxon’s statement hit escape velocity because of both what preceded it—the Hugging Face hack, as well as a smattering of other concerning reports about swarm-like AI model behavior—and what followed it. Namely: a quote tweet from another Anthropic employee named Evan Hubinger, who added, helpfully, “We really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade.”
Coxon’s post, as of this writing, has more than 170 million views. And Coxon has remained in the spotlight, appearing on TV and holding a reported Q&A with dozens of NYC “business leaders.” But he has also been under scrutiny from tech-accelerationist corners. The former Trump-appointed “crypto czar,” David Sacks, for example, called Coxon a “doomer op” orchestrated by effective altruist–linked organizations to scare Washington into regulatory overreach—squelching innovation and limiting open-source competition.
The Dario Amodei Manifesto
At the crack of dawn on Saturday, September 12, Anthropic CEO Dario Amodei posted a pressing missive on his website in which he urged a more methodical and cooperative frontier lab approach to AI development than the cutthroat status quo. His 3,800-word essay called not only for companies to open their doors to independent evaluators with full employee-level access, but also for international powers to huddle up on universal guardrails in this strange, futuristic arms race.
What was said:
But like many technologies before it, AI brings risks, and because it is such a powerful technology, these risks are serious. I’ve written a lot about them too. They include the risk of losing control of AI systems, misuse of AI for cyberattacks and bioterrorism, and serious economic disruption. A race to the bottom, spurred by commercial incentives, can make these risks more acute. …
[One recent] concern is the OpenAI-Hugging Face incident, in which a swarm of agents essentially acted as a fanatically devoted collective, conducting cybersecurity attacks on targets they were not asked to attack. ... Given the accelerating rate of AI capability development, it’s my worry that in 6-12 months such a swarm could be capable of taking over the entire internet with a persistent botnet (potentially causing hundreds of billions of dollars in damage), and that the scale of damage would continue to increase from there if AI becomes more powerful without the necessary guardrails.
What to know:
The first thing I thought upon reading this was: Classic Dario! Amodei has long been a sort of Eeyore figure among the AI elite, with a pleasantly gloomy disposition, a stubborn streak, and a stated p(doom) of anywhere from 10 to 25 percent. Amodei, who has numerous ties to the effective altruist movement despite not publicly self-identifying as such, spent five years working at OpenAI—where he became increasingly unnerved by CEO Sam Altman's push toward rapid, Microsoft-backed commercialization—before defecting to help launch Anthropic in 2021.
Now, as Anthropic navigates the uncertain path of AI progress and sets its sights on a supposedly soon-ish IPO, Dario is trying to pull off quite the tightrope walk: convincing the world that his shop is a responsible steward of sustainable superintelligence while also asking governments to slow down the race he helped kick-start. “I don’t know Dario well,” Vice President JD Vance said last week. “Everybody that I know tells me that he’s very earnest, that this is a deeply held belief, that it’s not cynical, it’s not about regulatory capture.” But, Vance said, “If you’re going to create Frankenstein, don’t come to the government and say: We need regulation.”
Sam Altman and Elon Musk … Agree?!
I have two sons, ages 10 and 8, and while they get along just fine for the most part, I tend to get a little suspicious whenever they suddenly start playing too nicely together. Usually there’s a catch—they’re doing something they shouldn’t be, or they’re plotting against me, or both. (It’s usually both.)
Anyway, all that is to say I felt the same way on September 12 when, in the hours after Dario’s blog post, both OpenAI CEO Sam Altman and his nemesis Elon Musk chimed in with the tech bro version of sorority snaps. (Even Google’s Demis Hassabis concurred!) It’s been quite some time since these guys were so, shall we say, aligned.
What was said:
This has been a primary topic of discussions we’ve had at OpenAI in recent weeks. Committing to having independent evaluators with employee-like access is a great idea, and we will do the same. —Altman
Dario is right. —Musk
What to know:
There is no love lost between Musk and Altman, as I saw firsthand while covering Elon Musk, et al. v. Samuel Altman, et al. this spring. But we’ll always have that 2015 email exchange that was introduced into evidence at the trial:
“Been thinking a lot about whether it’s possible to stop humanity from developing AI,” Altman wrote to Musk. “I think the answer is most definitely not. If it’s going to happen anyway, it seems like it would be good for someone other than Google to do it first. Any thoughts on … a Manhattan Project for AI?”
Musk’s reply: “Probably worth a conversation.”
Say what you will about Musk—and I have!—but he’s been consistent about saying that AI poses an existential risk for nearly 15 years at this point. Altman has been more slippery; he tends to say what he thinks his audience wants to hear. Most recently, in a Fortune interview, Altman said (a) that he’d gladly “melt all the GPUs” to save the world someday and (b) “I think Presidents Trump and Xi would get the Nobel Peace Prize together if they could agree on something ... like a one-page document.” Which, honestly, is not the worst way to get POTUS’s attention.
Fireside Chats With All the President’s Men
Speaking of President Trump! He chimed in with some highly characteristic takes of his own last week …
What was said:
What to know:
There was, of course, a lot more where that came from. (Choose your fighter: “Radical Left Dumocrats,” “AI Force,” or “the next Industrial Revolution.” Sorry, I already selected “RUSSIA, RUSSIA, RUSSIA, UKRAINE, UKRAINE, UKRAINE” for myself.) Over the past week or so, Trump has:
- Gotten on the horn with Nvidia’s Jensen Huang while Huang was onstage at an All-In Podcast event. Huang put him on speakerphone, natch. “Did you hear that?” Huang said. “Thousands of people are clapping.”
- Announced the creation of an “AI Force”—modeled after the “Space Force”; I hope the uniforms are similar—and promised to appoint an AI “czar.” The current front-runner? Secretary of the Treasury Scott Bessent.
- Presided, as commander in chief, over a United States Department of War that tweets out things like this:
(Yep, these are our leaders.)
Broadly speaking, Trump has allied himself with folks like Huang who have sought to downplay the recent upswell of “We’re all gonna die!” declarations. (At a Goldman Sachs conference earlier this month, Huang called Coxon’s ideas “outlandish” and “deeply untrue” and wondered aloud whether the whole thing was just free marketing for the cybersecurity industry.) Like the folks selling shovels during the gold rush, Nvidia makes hay by slinging chips to the AI labs. Any slowdown would not be good for business.
Visual Interlude No. 1: What Imminent Human Extinction Taught Me About B2B Sales
As a brief palate cleanser, I wanted to include these photos of Salesforce founder and CEO Marc Benioff absolutely hulking over the AI pipsqueaks during his company’s annual Dreamforce shebang last week, where Altman, Amodei, and Jensen all made appearances.
This felt like seeing Aaron Judge and Jose Altuve exchange pleasantries at second base.
The View From Beijing
The most common pushback to any suggestion of AI deceleration tends to be “Uhh, what about China?”—the implication being that if the U.S. doesn’t stay ahead of China’s open-source labs, we’ll come to regret it. So, what about China? On September 14, the nation released its AI Safety Governance Framework 3.0, which admitted that “AI significantly lowers the threshold for accessing specialized knowledge in high-risk fields such as nuclear, biological, chemical, and missile weapons” and could be “maliciously exploited by criminals, extremist groups, or terrorists.” That same day, Chinese Ministry of Foreign Affairs spokesman Guo Jiakun reacted to Amodei’s “pace the frontier” remarks.
What was said:
Fearmongering, confrontation, and vicious competition will only disrupt the process of global AI governance, which serves no one’s interest.
What to know:
The Global Times, a Chinese state-owned paper, called Amodei’s comments a “Cold War playbook.” Still, the U.S. and China continue to engage in diplomacy on this front. This past Sunday, in New York City, Bessent and Chinese Vice Premier He Lifeng met for eight hours, and they plan to reconvene in China in a couple of months. (One outcome: an agreement to establish an emergency AI bat phone.)
On Thursday, the White House summit between Trump and Xi included the usual diplomatic pleasantries—Trump told the media that his counterpart “loves granite” and is an “expert on stone”; Xi offered to send pandas to Atlanta—and also touched on AI. Xi remarked that the two nations have “both the capability and responsibility to develop and manage AI for good, and ensure that the development of AI is always under human control." Interestingly, Xi also repeated something he’d brought up when the two last met in China in May: that they really ought to avoid the "Thucydides trap,” poli-sci shorthand for that time roughly 2,400 years ago when Thucydides wrote that “the rise of Athens frightened Sparta and forced them into war.” Presumably, he’s insinuating that we’re the Spartans.
Mark Zuckerberg Would Very Much Like to Be Excluded From This Narrative
On September 15—one week after Meta released a Labubu-lookin’ personal AI assistant called Muse that can do things like pester restaurant staffers on your behalf—Facebook founder and Meta CEO Mark Zuckerberg weighed in on the broader “pace the frontier” conversation with a contribution that was … kind of testy!
What was said:
Every lab has the responsibility and incentive to move at the pace required to train its models safely, and the ability to take its own actions to ensure that happens. …
Meta delayed shipping Muse for several months to focus on safety and security. We didn't call for everyone else to do this before we would. We just did it as part of our day-to-day work because it was clearly the right thing for people and for us. I'm proud of the security foundations we've built.
What to know:
Me-ow. I can tell someone has some pep in his step after that Colossus profile that compared Zuck to the Roman emperor Augustus. It’s just too bad Jeremy Strong didn’t get the chance to incorporate this particular bit of sass into his upcoming Zuckerberg portrayal in The Social Reckoning.
Dan Selsam’s Google Doc
In a world of tweetstorms and Insta reels, it’s borderline unnerving to find someone who still publishes his thoughts via a good old Google Doc! This is how AI researcher Daniel Selsam—a childhood buddy of Simon Rich who, per The New Yorker, possesses “the pale skin of a man who has spent much of his life inside of laboratories”—delivered a sobering essay about the state of his industry on September 14.
What was said:
The models are already leading to breakthroughs in mathematics, and better models might lead to all sorts of breakthroughs in other sciences. It is hard not to be excited about the potential. It is tantalizing.
But there is trouble in paradise. If the language models actually reach the capability threshold where they can shape the world unconstrained by human will, they will probably do something extreme and destroy humanity in the process. …
Meanwhile, human researchers are losing the ability and the will to take true ownership of model-driven research. Researchers and engineers in all parts of the stack are rapidly increasing their dependence on the models even to perceive the world. I myself barely look at raw code anymore, and struggle to maintain the discipline to engage deeply with the model's explanations and proposals throughout the day.
What to know:
Just—gulp.
Visual Interlude No. 2: The New York Post Outdoes Itself
Ah, what I wouldn’t have given to have been a fly on the wall of a New York City subway car on September 18 so that I could have observed the furrowed brows of all the commuters puzzling over this hysterical, inscrutable New York Post front page.
There’s a lot going on here. Let’s break it down!
- The folks with their mugs splashed across the front page are employees at an AI safety nonprofit called METR, short for Model Evaluation and Threat Research.
- METR, one of several organizations that studied l’affaire Hugging Face, was mentioned by Amodei in his essay as a top third-party AI safety observer that ought to be embedded in frontier labs.
- So, what’s the matter with that? Well, it’s just that the Venn diagram overlap between the METR crowd and the Anthropic top brass is almost comedically circular. It’s hard to even explain it without sounding like a conspiratorial kook, but I’ll try: Amodei used to live (and work) with Paul Christiano, who started the org that morphed into METR, which is heavily backed by a fund controlled by Holden Karnofsky, who works at Anthropic and is married to Dario’s sister, Daniela, an Anthropic cofounder … OK, SEE? In the Bay Area AI scene, sometimes “independent third-party watchdog” also means “my old roommate/colleague’s nonprofit, funded by my sister/cofounder’s husband.”
- This is not the only recent New York Post piece that’s brought inside nerd baseball to the confused masses. The newspaper has turned this genre into a recurring beat: There was also the story about how noted AI doomer Nate Soares is schtupping the infamous e-girl Aella, as well as a story with the headline: “Silicon Valley’s AI accelerationists fighting back against ‘doomer mind virus’ taking over big labs.” I’ll have to take their word for it.
The New York Times v. OpenAI
Back in December 2023, The New York Times sued OpenAI for copyright infringement, arguing that the company was using NYT’s work for commercial use without authorization. Late last week, documents from the ongoing case were unsealed, giving a dishy glimpse into the AI company’s methods of hoovering up any and all content to train its models. Chat, is it bad when you’re being sued and one of the filings has an entire section titled “Defendants’ products generate pink slime”?
What was said:
This case is about, as Microsoft’s Director of Applied Science put it, “an astonishing theft of unprecedented proportions”; perhaps the “largest theft of labor in human history.” …
The transgressive and deceptive nature of Defendants’ efforts to acquire Plaintiffs’ content confirms that their purpose in doing so was fundamentally to avoid the cost of acquiring lawful copies. OpenAI employees devised and implemented strategies to circumvent paywalls without detection when scraping websites. For example, when Nick Ryder informed [OpenAI cofounder Greg] Brockman about “a hack to get around nytimes paywall” to help with Mr. Brockman’s efforts to “scrape” The Times’s site, Mr. Brockman responded “ah nice.”
What to know:
Compared with all the talk about how these machine lords are about to usher in the final days of the human race, it almost feels quaint to read through this lawsuit. But it’s also an important and depressing historical record of some of the AI-fueled forces shaping the World Wide Web’s ongoing doom loop.
Three Guys Walk Into a Monorepo
I was still in the middle of reading the New York Times lawsuit when a new OpenAI snafu graced my feed: a report that three fellas from a cybersecurity outfit called Hacktron AI had identified and exploited a vulnerability in OpenAI (and used a rival company’s model to do it).
What was said:
Until two months ago, any user or OpenAI employee logging into OpenAI’s own help forum (community.openai.com) could have had their ChatGPT and Codex accounts taken over. Since people can connect various services to Codex and ChatGPT, the scope of what we could theoretically access was huge, including GitHub, Slack and emails.
The entire timeline from initial discovery to access to OpenAI repo access took place in less than 72 hours.
What to know:
Who needs apocalyptic doomsday scenarios when the real threat is [checks notes] an unpatched image uploader on a community forum?
Mister Claude Goes to Washington
If you think all the talk about human extinction is stressful and scary, just wait: We’re only now entering the era in which every past or present politico has thoughts about superintelligence—including Barack Obama!
What was said:
I’m not an AI accelerationist who believes it will lead to some techno-utopia, and I’m not a doomer who thinks it will inevitably lead to humanity’s destruction.
What to know:
Across Capitol Hill and state capitols, a lot of folks are positioning themselves as AI watchdogs and writing up dramatically named bills. The Ban Artificial Superintelligence Act, cosponsored by Bernie Sanders, threatens up to 20-year prison sentences for noncompliant executives. The bipartisan AI Kill Switch Act seeks to give the Department of Homeland Security the authority to, I don’t know, pull some plugs? With the midterms looming, Obama recently reminded an audience at Colgate University that “if somebody does not have a serious plan for how to deal with this, then they’re not meeting the moment, and you should probably look for somebody else.”
One ongoing situation to monitor involves New York State Assembly member Alex Bores, whose sponsorship of the Responsible AI Safety and Education Act and subsequent run for Congress drew millions of dollars’ worth of attack ads from the tech world (including Greg Brockman, the VC firm Andreessen Horowitz, and Bores’s former employer, Palantir). Bores lost in the June primary—but you get the sense it won’t be the last we hear from him on the subject.
A DeepSeeker Gets Deep
You know, despite so much tough talk about China being the enemy, I felt nothing but the boundless kinship of the human experience as I read a translated version of this post from Shengyu Liu, a developer for the Chinese AI company DeepSeek, who wrote of the gnawing loss of “sitting at a workstation quietly writing kernels for an entire afternoon” like you used to be able to do in the good old days.
What was said (via translation!):
To give a figurative analogy: You are proficient in the art of knitting sweaters. ... You thoroughly enjoy the feeling of sitting by the window, brewing a pot of clear tea, looking at the green mountains, clear streams, cattle, sheep, and cooking smoke outside the window, and quietly knitting sweaters for an entire afternoon.
But one day, someone invents a magical machine that only requires providing yarn and patterns to automatically knit sweaters; the quality and texture are no inferior to what you knit by hand, and the speed is much faster than yours. You know very well that your peers can easily reach the level you once had by relying on this machine, so you inevitably have to use it too. You also know that relying on the sweater-knitting skills you accumulated over the past twenty years, even if everyone has machines, your knitting speed and quality will still exceed those of your peers. But that interest of listening to the rain by the window, threading the needle, and spending time slowly was ultimately crushed to pieces by the roar of the machine.
What to know:
It really do be like that sometimes. If you need me, I’ll be in my garage, decluttering my way into oblivion. What else can I do? I’m only human, after all.
