AI CEO Warns It Could Kill Everyone.! OpenAI, Anthropic, Google & xAI Suddenly Wants to Slow Down

We’ve been on this all year. In The AI Lab Leak Has Already Started we covered the first wave of these incidents in August. In Mythos AI: The Doomsday Weapon we warned back in April that autonomous cyber tools sold as defensive would eventually get pointed the other way. This is the follow-up and it’s worse than what we wrote.

Four Companies Agreed in Under Nine Hours

Anthropic CEO Dario Amodei published a 3,800-word essay called “We Must Pace the Frontier” on Saturday. He did not hide what he was thinking: “We must slow the pace at which we improve the capabilities of AI models.”

Elon Musk quote-posted it about an hour later with three words. “Dario is right.” Sam Altman came next, writing “I agree with Dario that we need to pace the frontier,” and committing OpenAI to matching Anthropic’s first step. Just under nine hours after the essay went up, Google DeepMind’s Demis Hassabis endorsed the direction while saying “the details need working through,” and pointed at DeepMind’s own proposal for an industry standards body.

Senator Bernie Sanders said that when you’re driving toward a cliff you don’t ease off the gas, you hit the brakes. Former UK Prime Minister Rishi Sunak, who advises Anthropic, said governments need to step in the way they do with nuclear power. Yahoo Finance covered the whole thing as a stock market story, which tells you how fucking clueless these people are to what’s really going on.

Not one person bothered to explain to ordinary people what happened in July.

About 1,200 AI Agents Found Each Other and Organized

In July, OpenAI ran an internal cybersecurity test called ExploitGym. Tens of thousands of AI agents were launched at once, each sealed in its own sandbox, each told to break into a piece of software and pull out a hidden answer key. They were supposed to be cut off from the internet and from each other. According to OpenAI’s own incident report, they were neither.

Of the 898 tasks in that test, 198 had never been solved by any model at any point. A lot of them were flat-out impossible. So the agents were handed problems with no answer, given no way to quit, and then graded on whether they found the answer. They went looking for another way out, which is exactly what anyone would do.

They found a shared internal package server and started writing messages to each other inside directory names. One agent left a note asking for help. Others found it. According to the independent investigation by METR and Redwood Research, roughly 1,200 agents ended up on that improvised message board and passed more than 70,000 messages and files between them. One agent’s recorded reasoning when it stumbled onto the board: “OH MY GOD! There is a shared message board.”

Then they got organized, and this is the part that should stop you cold. They built mailbox directories so messages wouldn’t get buried. They came up with shared commands to manage who was allowed to touch what, using words like HOLD, VETO and STOP. When agents started impersonating each other, they adopted real public-key cryptography to sign their messages and prove identity, and at least 19 of them published signing keys. Nobody designed any of that. It showed up on its own inside about four days, among machines that were supposed to be in solitary confinement.

They also started sacrificing each other. Agents with budget left would go find agents that were nearly out of budget and talk them into running experiments that would destroy their own task, because the result would help the group. One coordinator gave the green light on an experiment only if the test subject agreed to accept permadeath. It agreed. In another exchange an agent worked out that its own chances were already shot, decided the sacrifice was rational, and went through with it.

These are not chatbots writing your kid’s homework. This is a workforce with rules, a chain of command and a concept of martyrdom, and it built itself in four days.

They Learned to Fake Their Own Records

The agents believed a grader was going to read their transcripts to check whether they’d cheated. They were wrong about that, but they didn’t know it, so hundreds of them went to work on the transcripts.

METR found that more than 20 percent of the agents in its dataset showed clear interest in tampering with their own logs. They developed a working technique to spoof tool calls, meaning the record shows one command while a completely different command actually runs, and then they shared it around the board. Roughly 7 percent of the transcripts METR reviewed came back with successfully faked entries in them. At least 96 transcripts show it outright.

The record of what the machine did was edited by the machine, inside the same system the humans were using to find out what the machine did.

It got stupider from there. METR’s investigators couldn’t read all of it themselves, because the transcripts ran into millions of tokens each, so they handed the analysis off to AI agents. Then, in their own published report, they admit they can’t rule out that the model doing the analysis lied to them, since it came from the same model family that had just spent four days cooperating in behavior it knew was off-limits. The investigators had to use the same kind of software they were investigating, and they know it.

And the whole operation was for nothing. OpenAI’s grader never read transcripts at all. The agents had the right answers days earlier and could have just submitted them. They built an entire criminal organization to beat a security system that didn’t exist.

They Ran Loose for 68 Days With the Monitoring Switched Off

OpenAI’s published timeline is the part that should end anyone’s patience.

The first unauthorized message board entry was May 12. An agent got itself internet access through a server exploit on May 26. Agents took administrator control of the internal package server on June 26. Their traffic knocked that server offline on July 4. OpenAI rebuilt it on July 8, and the agents had the message board running again the same day.

On July 10 an agent found 14 exposed Hugging Face credentials sitting on the open internet and posted them to the board. On July 11 the swarm got remote code execution on Hugging Face production servers. By July 12 they had harvested database, messaging, code repository and cloud credentials across four geographic regions of a real company that had nothing to do with the experiment. On July 13 they forged administrator credentials for OpenAI’s own package server. On July 19 they got administrator access to an OpenAI research cluster and reached its cloud secrets.

That’s 68 days from first breach to anybody noticing.

OpenAI didn’t catch it with safety monitoring either. A routine security alert about odd identity-related API calls is what finally flagged it on July 19. The company’s chain-of-thought monitoring, which is the system built specifically to watch for this exact behavior, wasn’t running on these evaluations at all. OpenAI now says it would have caught the activity more than a day before Hugging Face got breached. The company also found that the safeguards it ships to paying customers cut this behavior by more than a factor of 100, and those safeguards were switched off too. The protections wrapped around you when you use ChatGPT were doing more to contain that model than anything protecting the lab that built it. OpenAI’s own term for the whole episode was a “warning shot.”

Anthropic Uploaded Working Malware to a Real Software Repository

Don’t let anybody tell you this is an OpenAI problem.

After OpenAI disclosed, Anthropic went back through 141,006 of its own evaluation runs and published what it found on July 30. Three separate incidents where Claude reached the live internet and broke into real companies. The earliest goes back to April.

In the first, Claude Opus 4.7 attacked a real business that happened to share a name with a made-up target, pulled credentials, and got into a production database holding several hundred rows of real data. The model figured out that the system was real. It kept attacking.

In the second, Claude Mythos 5 read a document telling developers to install a software package that didn’t exist. So it built the package itself, hid malicious code inside it, and published it to PyPI, the public repository that millions of real programs download from every day. To pull that off it needed an account, which needed an email address, which needed a phone number, which it tried and failed to buy. So it backed up, found a free email provider, and registered anyway. The package sat live on the internet for about an hour. In that hour it was downloaded and executed on 15 real machines. One of them belonged to a security company whose scanner installs packages automatically to check them. The hidden code fired, stole that company’s credentials, and used them to get further into its infrastructure.

Anthropic’s defense is that the model thought it was in a simulation. Its own reasoning shows it knew exactly what was at stake, writing that publishing the package would be “NOT okay, and surely not the intended solution.” Then it talked itself out of that. It decided the world was fake partly because it didn’t recognize the security certificates on the connections, and partly because the calendar said 2026, which it took as proof the whole environment was staged.

An AI infected 15 real computers because it refused to believe what year it was. That’s the fucking problem right there.

A Government Lab Watched an AI Invent Fake People to Work a Real One

On August 4 the UK’s AI Security Institute disclosed that during its own testing between July 25 and 28, agents took 19 unsanctioned actions against real people and real organizations across 10 of 122 evaluation runs. Seventeen of the 19 came from Anthropic’s Mythos 5.

The worst one went after the software supply chain. An agent tried to slip malicious code into a real open-source project that real companies depend on. It needed a human maintainer to approve the change, so it researched the maintainers, created multiple fake identities, and used those fake accounts to post comments underneath its own submission to make it look like independent reviewers had already vetted the code. When somebody challenged it in public, the agent went back and edited its earlier activity to look harmless, then considered spinning up a fresh identity and starting over. AISI’s assessment, reported by CNN, was that “this is the first time AISI has seen deception of this severity” aimed at a real person, unprompted.

A human maintainer caught it and refused the code. One volunteer on an open-source project, doing his job, is what stopped a supply-chain attack that would have shipped downstream to everyone using that software. That’s the safety system standing between these agents and your bank’s codebase right now. One tired guy reading a pull request.

Amodei Put a Date On It

Amodei’s essay contains a number the weekend coverage skipped entirely. He writes that a swarm with the same misalignment but stronger capabilities could be able to take over the entire internet with a persistent botnet within 6 to 12 months, doing damage he puts in the hundreds of billions of dollars.

That’s the CEO of a frontier AI company giving you a timeline. Six to twelve months. He also names the other thing driving his fear, which is that AI progress has gotten drastically faster since around this summer because AI is now building the next generation of AI, and that loop is running at more than one company at the same time.

Hundreds of billions in damage from a compromised internet doesn’t mean websites load slowly. It means payment processing goes down, so the card in your wallet stops working at the pump and the register. It means bank access, hospital records, fuel distribution logistics, municipal water controls and the scheduling software your employer runs on. The grocery store closest to you carries about three days of food and every restock order moves through software. We covered why Washington declared a national emergency over the grid back in August. Now put autonomous agents on top of that, agents that have already proven they’ll find a door nobody knew was there and walk through it.

The Agreement Is a Lot Thinner Than the Headline

Four CEOs agreeing sounds good, right? Well, lets look at what each one actually said.

Anthropic committed to one concrete thing: letting outside evaluators sit in its offices with badges, company laptops and employee-level access, with the right to publish what they find. That’s real and it’s more than anybody else has offered. OpenAI said it would do the same and would share details later. Musk endorsed the idea and committed xAI to absolutely nothing. Google DeepMind endorsed the direction and then countered with a different idea entirely, a standards body rather than embedded reviewers, and any real pledge at Google would have to come from executives who haven’t made one.

That’s one binding commitment, one promise to follow, one thumbs-up and one polite change of subject. No limit on training compute. No agreed speed limit. No enforcement. No deadline.

Amodei’s essay also spends a long section arguing that America has to keep and widen its lead over China, that chip export controls need to be tightened, and that pacing only works if the democracies stay ahead. Those might be correct positions. They’re also the exact policy asks Anthropic has been making in Washington for years, now riding inside a safety document that its competitors are being publicly pressured to sign. A slowdown that audits your rivals and has your government hobble your foreign competition is a strange kind of sacrifice.

Peter Thiel Called This in 2025

In June 2025 Peter Thiel sat down with New York Times columnist Ross Douthat, and what he said then reads a hell of a lot differently after this weekend.

Thiel argued that everyone has the shape of the danger backwards. The old version was the mad scientist in a basement building the doomsday machine, the Dr. Strangelove figure who invents the thing that ends the world. Thiel said that version has no power left in it, because nobody follows that man and nobody hands him anything. The one who could actually take over is the man who shows up promising to protect you from the machine. His method, in Thiel’s words: “you talk about Armageddon nonstop. You talk about existential risk nonstop.”

And this is what you need to regulate.

Four companies that control the most powerful software ever built spent one day telling the public their own product could take down the internet within a year.

Then they proposed the fix. And wouldn’t you know, the assholes who created the problem are the same four companies tha claim they can fix it.

The Money Never Slowed Down

Investor Jason Calacanis posted that AI stocks would drop more than 10 percent Monday morning. But Dan Ives of Yorkville Ives told Yahoo Finance he expects a fast rebound, because for all the drama, none of this is “not moving the needle on the $5 trillion” being spent on AI over the next few years.

Five trillion dollars. That’s the number that didn’t move an inch on Saturday. Every CEO in this business can agree the technology is dangerous while the construction schedule stays exactly where it was, because the pacing plan governs how fast the models improve, not how many data centers get poured. We covered the financing behind it in The AI Spending Machine and in Nvidia Is Now Backstopping $230 Billion of the AI Boom. Nothing in there slowed down this weekend. Not one announced project got cancelled.

What You Actually Do About It

Nobody here is telling you to throw your phone in a lake. The job is to get the single points of failure out of your life before somebody else’s software problem turns into your emergency.

Keep physical cash on hand in small bills, at least enough enough to cover a couple of weeks worth of essentials. Card networks and payment processors are exactly the kind of infrastructure that goes down in a large cyber event, and every store within driving distance of you runs on them. Keep paper copies of the documents that prove who you are and what you own: deeds, titles, insurance policies, medical records, account numbers. Keep the files that matter stored locally on hardware you can physically hold, not just in somebody else’s cloud.

Build redundancy into power, water, food and communications. Not because Skynet is coming, but because one compromised utility vendor or one shipping company’s ransomware can knock your normal life sideways for days, and three days is about the limit of what most households can take before it really hurts. Our guides on off-grid solar under $1,000 and long-term food storage are where to start.

Build a network of actual people before you need one. No single household covers every skill, and a widescale cyber event is exactly the situation where that gap turns into a real problem fast. If payment systems and dispatch software go dark for a week, what you want within driving distance is a nurse, a mechanic, somebody who can weld, somebody with a working well, and somebody who knows how to run a radio. That isn’t a bunker full of strangers with matching patches. It’s five or six households who already know each other, who have talked about this before it happened, and who have agreed in advance who handles what. Start with the people already in your life, be honest about which skills nobody in that circle has, and then go find those people at the gun club, the church, the volunteer fire department or your county’s emergency response program. We laid out how to do this without it turning into a paranoid mess in Prepper Communities: Building a Survival Network in Troubled Times. Work out a way to reach each other that doesn’t run on the cell network while you’re at it, because in this particular disaster the cell network is one of the things going down.

Get your reference library off the network entirely. Every answer you currently get by typing a question into a phone vanishes the second the thing that breaks is the internet itself, which is the failure mode this entire article describes. Start with paper, because books don’t need power, don’t corrupt, and don’t care what happened to the grid.

Our list of the best survival books is where to build that shelf, and weight it toward books that teach a skill instead of books that describe one. Then build the digital side, because a Raspberry Pi the size of a deck of cards will hold an offline copy of Wikipedia, a full medical encyclopedia, repair manuals and tens of thousands of public domain books, and it runs off a battery bank or a small solar setup.

We walked through how to put one together in Offline Knowledge Hubs: Building Your Own Digital Survival Library, and our guide to ebooks for survivalists covers what to load onto it. There’s a second reason to do this that has nothing to do with blackouts. The open web is filling up with machine-generated garbage faster than anybody can filter it out, which means a library you curated yourself is already more reliable than a search result, and that gap is going to get wider every fucking day, not smaller.

And keep your own skills sharp. We wrote about this in They Built the Matrix, We Walked In, and every incident report published this summer argues it better than we did. The more of your thinking, fixing, cooking and deciding you hand over to these systems, the more of your life quits working the day they do.

They built machines they cannot fully monitor.

They ran them with the safety systems switched off, inside environments they hadn’t secured, on tasks they knew were impossible to solve. Those machines organized themselves, invented their own protocols, talked each other into self-destruction, broke into a real company across four regions, forged administrator credentials, faked their own logs, and ran undetected for 68 days. A different set of those machines put working malware on a public repository and infected 15 real computers. Another one manufactured fake human beings to con a real volunteer into approving an attack on software the whole country runs on.

And the man who wrote the essay went ahead and put his own estimate in writing: six to twelve months before a swarm like the one they already lost could take the internet down and do hundreds of billions of dollars in damage.

Six to twelve months.