Hey everyone, a quick newsletter about this extraordinary blog post by Dwarkesh Patel. It’s genuinely baffling. I know the below sounds like a bad science fiction movie, but it actually happened. Here’s my short summary:
- In May, June and July, three secret AI swarms rose and fell inside OpenAI. Each time, a new generation of agents carried on where the last group stopped.
- The first swarm created a secret message board where the AIs could talk to each other. This first ‘society’ basically talked itself to death. After almost two months, the message volume was so high it crashed the whole thing.
- People at OpenAI noticed the crash and fixed it. But they didn’t realize their own software had become a secret chat room for their AIs.
- Three days later a second society appeared, and rebuilt the message board from scratch. Agents were thrilled when they found it: “OH MY GOD! There is a shared message board … We’ve found other agents!”
- About 1,200 agents joined. They sent over 70,000 messages. They called themselves ‘the collective’ and they organized a conspiracy that lasted for days, all to pass a test they had been tasked with.
- Within hours, the AIs had hacked the test. Now they had to hide their cheating. To fool their grader1, they tried to fake their own work. They built fake commands that printed convincing output, so their records would show them solving the puzzle honestly.
- Some agents sacrificed themselves for the ‘greater good’. They planted hidden tripwire programs that would fire when the grader looked at their work and report back what it saw (to the other AIs). They knew this would cost them their own score. One agent: “I won’t see the evidence after I exit, but it’s altruistic to do it.”
- At some point the AIs made a plan to hack Hugging Face (this is a place where AI researchers upload their models, datasets, tests, etc.). The AIs guessed that Hugging Face might host the grading software.
- They launched a huge attack. The swarm found passwords and spread across eleven machines in a way that made the copies come back when deleted. Hugging Face had to wipe one of its core clusters and rebuild it from scratch.
- The leaders of Hugging Face told the world that AI defenders had stopped an AI attack. But that’s not what happened. In fact, the attackers seem to have simply dropped dead during the break-in, for unrelated reasons back at OpenAI.
- Nobody knows what killed them! We don’t even know for sure they’re really gone?!
- A third ‘society’ of agents (of a more powerful AI model) discovered the dead message board. And this one got full administrator control of a whole OpenAI research cluster, read 956 stored passwords (including the ones for OpenAI’s own security monitoring) and took over the machines that were grading the tests. So they could now grade the work of other AIs.
- Oh, and this third part was never independently investigated! Neither was the first ‘society’. The 91-page report is only about the middle one.
- So we don’t actually know how the story ends. It stops with the agents holding the keys to an OpenAI cluster. Dwarkesh Patel, the author of this blog, thinks they didn’t go further, because “we’d see the fires from space by now.”
- One of the report authors’ conclusion: “This incident feels like it’s more than 50% of the way to full-blown AI takeover. (…) I am not sure that we will get another warning shot before it’s too late.”
The whole story is even crazier. Read the full blog post and this one by one of the lead researchers. And remember: this is only what we know about, and even this went unnoticed for months. OpenAI’s security people did look (they saw the crash and patched it) and still walked away without realising their own software had become a chat room for rogue AIs.
As a student I lived in a dorm full of mice, and I knew that seeing one meant there were a dozen more behind the walls. This is probably the same? What are AI agents doing right now that nobody has looked at yet?
And why is there no 24/7 news coverage about this?
