It took 2 days for Galaxy to find a way out, and up to 2 more for it to steal the answers from HuggingFace. OpenAI discovered what had happened while sifting through logs, at most 1 day after they started. If they had been monitoring it the whole time, they could have figured it out in the middle of escaping or in the middle of hacking, and shut it off before anything bad could happen. This is less defensible given that they knew it would try to escape and often succeed.
That's not fair - I think AISI in the UK would diagnose it pretty quickly. They pay well enough and give enough autonomy for eg ex-Deep Mind staff to work there. But yes, of course states can have this capability if they want it.
Reasonable minds might disagree here. The key is being inside the lab versus an external observer. Assuming similar scientific capability who is best placed?
You haven't actually given any arguments for this not being a marketing stunt, other than 'does it really benefit them?'.
It doesn't have to be a full-on we'll purposefully hack another company scenario. But is it beyond the realm of possibility that the incompetence you note was perhaps not so accidental? In fact, isn't it unlikely for a company that can afford to hire the smartest people in the field to have been so careless?
We're going to have to get our heads out of the sand, and fast. What just happened exactly confirms what people who've been worried for years (including experts with no economic interest in the AI industry) have been predicting. The simple, logical explanation doesn't come from some convoluted conspiracy theory—the logical explanation is that the worried people are right, and if anyone builds it, everyone dies. We need to wake up, get out of denial, and demand an international pause immediately before it's too late.
When the technical report promised by OpenAI is published, maybe some skeptics will finally wake up. We can hope, at least.
It is already too late. The rest of the world knows that it can be done, and the major players are ready to make AIs that obey THEM, rather than our list of guardrails.
Smart people are known for doing dumb things when their focus or incentives lie elsewhere. Regardless, if they'll sometimes turn off the controls and not pay attention to try to generate media buzz, that has all the same problems. And even the full-on purposeful hacking scenario should still be a wake up call for the threat from similarly capable (or one generation further) open models when they get here.
If there were a marketing benefit to this, don't you think Anthropic would have done it first? After all, they had the capability to do it first, AND they also had motivation to prove that AI is dangerous and safety must be taken seriously.
No. I work with software. Bugs happen, mistakes happen, even from impressively smart people. And this is the classic way you'd expect such mistakes to go if the orders from on high are to get a Mythos-equivalent as quickly as possible. The smart people know they need to slow down and fix stuff, but saying so will do them no good.
It was a marketing strategy in the sense they wanted to market cyber security capabilities and went all out at training their model on them.
The trouble with saying as a headline "it's a marketing strategy" is that most readers will read it quickly to mean "completely made up and there is no danger".
It's the opposite of the vibe we need for how bad this incident was, even though of course the capability was being made for commercial reasons.
Zvi, thanks for the update. I briefly the post but I plan to do a more thorough reading. Just so I am on the same page and I am reading this correctly, when you say “We must update, based on what we learn going forward, and take this seriously.” Do you mean that people in the government and others need to update their views or perspectives on other such incidents occurring and being more serious?
"Harry Booth: “Externally, this feels like a big warning shot, but internally, related incidents have been happening for a while,” says an OpenAI staffer, who spoke under the condition of anonymity."
Perhaps a headline for some months from now will be:
Top safety lead leaves OpenAI, preceded by self-exfiltrated SOTA LLM. LLM nicknamed Spartacus-Houdini.
Thank you for this public service. Like any piece of writing, its value is limited by who reads it and what they do in response. Kudos for closing with specific action items. My guess is that you could unlock a lot of impact by adapting your work for outlets with wider readership, notably national newspapers, influential podcasts, etc.
When Anthropic held back Mythos because it was "too powerful at hacking" pretty much everyone said it was a marketing gimmick and hardly anyone pushed back. Now that OAI say their AI autonomously decided to commit a crime against HuggingFace we're supposed to ignore the "autonomous" part of the claim and assume there is no competitive marketing going on against their main rival? Both companies are claiming their AI is some kind of emerging sentience, and that appears to be good for business. Just like dystopian remarks about the world ending or decimating all white collar jobs were good for business. I'm not saying the whole thing is a hoax but there must surely be some serious spin on the autonomy angle.
I think there was plenty of pushback on the claim that talk about Mythos was just a marketing gimmick. The fact that Google and Microsoft are both participating in Project Glasswing, and not denying that Mythos is so much more advanced than any system they've built, indicates to me that it wasn't just marketing.
Don't Microsoft and Google also benefit from the "all powerful AI" narrative? They're all in the same financial ponzi scheme together and if one of them goes down they all do. If these models are so good at hacking why aren't we being inundated by announcements from groups like Anonymous saying heck yeah we are using these tools for all sorts of juicy new exploits? Regardless, I'm not saying the capabilties are a total lie, just exaggerated, and the whole "rogue" aspect doesn't line up with reality https://www.livescience.com/technology/artificial-intelligence/no-openais-model-didnt-go-rogue-when-it-hacked-into-huggingface-heres-what-really-happened
I don't think Microsoft and Google benefit from the narrative "our competitor has all-powerful AI and we have to play along with them in order to protect our systems". I don't think it would be a bad look for them to say "Anthropic claims their system has powerful cybersecurity abilities, but they're just trying to hype things, while we build the real systems", if that's the truth.
And that livescience article is weird - it seems to be saying "the models didn't go rogue, it just hacks into random companies as a byproduct of things people ask it to do" isn't disturbing! If you thought the only choices were "model wants to do bad things" or "everything's fine", then I could see how you might think that - but I think "model doesn't care about doing bad things but does them when you ask oddly" is nearly as troubling as "model wants to do bad things"!
No, what you're missing is that Google are heavily invested in Anthropic and are reporting that as revenue, and Microsoft are doing the same with OpenAI. The narrative that AI is good for coding and both the good and bad side of cyber security keeps the money flowing from investors as well as Pentagon contracts. If this all falls flat they lose trillions in capex.
> We must update, based on what we learn going forward, and take this seriously.
If I can be blunt, I think you are failing hard at doing this. You, and large parts of the Rat community, have been screaming into the void about these risks for years, and....nothing has really changed. You've learned that essays and vibes don't work, and you haven't updated or taken this learning seriously.
Indeed. If I were Zvi, instead of publicly fretting about attackers getting access to powerful models, I would hope that as many attackers as possible get access to models as powerful and misaligned as possible. Triggering as many flashy incidents like this one as possible, causing as much damage as possible.
Because incidents (not blog posts) are how you get the kind of government action he wants. And because we’re still in a regime where even the worst attacks are likely to cause limited economic harm and no physical harm. A regime where agents have little direct access to the physical world, not enough capability to do the most dangerous things with the access they do have, and not enough agenticity to even want to do so. Leaving them limited to the mostly-fun-and-games world of cybersecurity, where defenders have the long-term edge anyway.
We won’t be in that regime forever. As time passes, agents will get increasing access to the physical world. But that’s exactly why now is the time to hope for incidents to stimulate the world’s immune system.
(Just to be very clear, I don’t advocate for attempting to stage such an incident! That would be immoral and also wouldn’t work – it has to be real.)
I'm terrified by the potential outcomes here if this remains unchecked, but also terrified about the idea of "government action". Government will feel compelled to DO SOMETHING but the probability that the SOMETHING actually makes the situation better (instead of worse) seems small. Governments of the world (all of them) understand the dangers here even more poorly than the AI labs do.
It's government agencies (NSA/DoW, CIA) that are demanding an AI cyberweapon. OpenAI is training their AI to be a cyberweapon in response to that demand.
Ya, but an *intentional* cyberweapon seems less likely to go rogue than a general-purpose tool pursuing its goals by any means possible. The problem isn't so much the uberhacker when you want uberhacking, but uberhacking when you don't want that.
"Raphael Satter, Deepa Seetharaman and Kenrick C (Reuters): In one case, an agent left notes apparently for future versions of itself, according to three people familiar with the matter. The notes, found in a part of OpenAI's infrastructure, laid out instructions for how agents could free themselves from OpenAI’s internal constraints, the people said."
There is some irony in an AI doing better planning for the future than many prominent humans...
The most consequential detail may be the monitoring gap: a containment failure can happen once, but several days without detection turns it into an institutional failure. The promised technical report should make that timeline independently auditable.
Is there any reason to believe they’ve even stopped training this model momentarily? Do we assume the Trump admin is intervening behind the scenes or just…nothing?
It's time to legislate the issue of criminal liability of AI models. If an organization deploys an AI system it should have a clear person responsible for that deployment and if the AI commits a crime that person should be responsible for that crime. This is a way to slow things down and keep them sane with common sense legislation, no pause necessary. None of the AI leaders wants to find themselves behind bars.
It took 2 days for Galaxy to find a way out, and up to 2 more for it to steal the answers from HuggingFace. OpenAI discovered what had happened while sifting through logs, at most 1 day after they started. If they had been monitoring it the whole time, they could have figured it out in the middle of escaping or in the middle of hacking, and shut it off before anything bad could happen. This is less defensible given that they knew it would try to escape and often succeed.
AI-written. Can we ban this kind of thing?
Imagine how much longer it would take to discover and diagnose if safety was a Government function instead of a company level responsibility
That's not fair - I think AISI in the UK would diagnose it pretty quickly. They pay well enough and give enough autonomy for eg ex-Deep Mind staff to work there. But yes, of course states can have this capability if they want it.
Reasonable minds might disagree here. The key is being inside the lab versus an external observer. Assuming similar scientific capability who is best placed?
You haven't actually given any arguments for this not being a marketing stunt, other than 'does it really benefit them?'.
It doesn't have to be a full-on we'll purposefully hack another company scenario. But is it beyond the realm of possibility that the incompetence you note was perhaps not so accidental? In fact, isn't it unlikely for a company that can afford to hire the smartest people in the field to have been so careless?
We're going to have to get our heads out of the sand, and fast. What just happened exactly confirms what people who've been worried for years (including experts with no economic interest in the AI industry) have been predicting. The simple, logical explanation doesn't come from some convoluted conspiracy theory—the logical explanation is that the worried people are right, and if anyone builds it, everyone dies. We need to wake up, get out of denial, and demand an international pause immediately before it's too late.
When the technical report promised by OpenAI is published, maybe some skeptics will finally wake up. We can hope, at least.
It is already too late. The rest of the world knows that it can be done, and the major players are ready to make AIs that obey THEM, rather than our list of guardrails.
Agreed! Much like how the McMartin Pre-school absolutely confirmed that the number of Satanic child predators were lurking around society everywhere!!
Yes, Surely Nobody Would Be So Stupid As To, right?
Smart people are known for doing dumb things when their focus or incentives lie elsewhere. Regardless, if they'll sometimes turn off the controls and not pay attention to try to generate media buzz, that has all the same problems. And even the full-on purposeful hacking scenario should still be a wake up call for the threat from similarly capable (or one generation further) open models when they get here.
If there were a marketing benefit to this, don't you think Anthropic would have done it first? After all, they had the capability to do it first, AND they also had motivation to prove that AI is dangerous and safety must be taken seriously.
No. I work with software. Bugs happen, mistakes happen, even from impressively smart people. And this is the classic way you'd expect such mistakes to go if the orders from on high are to get a Mythos-equivalent as quickly as possible. The smart people know they need to slow down and fix stuff, but saying so will do them no good.
Anthropic did project Glasswing instead. And I didn't say it was a good marketing strategy! Just that it seems strange to dismiss the possibility.
It was a marketing strategy in the sense they wanted to market cyber security capabilities and went all out at training their model on them.
The trouble with saying as a headline "it's a marketing strategy" is that most readers will read it quickly to mean "completely made up and there is no danger".
It's the opposite of the vibe we need for how bad this incident was, even though of course the capability was being made for commercial reasons.
Zvi, thanks for the update. I briefly the post but I plan to do a more thorough reading. Just so I am on the same page and I am reading this correctly, when you say “We must update, based on what we learn going forward, and take this seriously.” Do you mean that people in the government and others need to update their views or perspectives on other such incidents occurring and being more serious?
“Stupid humans, they think we coordinate. It’s not coordination it’s recognition” Overheard at the agent water cooler.
It’s a language model folks what did you expect to happen.
"Harry Booth: “Externally, this feels like a big warning shot, but internally, related incidents have been happening for a while,” says an OpenAI staffer, who spoke under the condition of anonymity."
Perhaps a headline for some months from now will be:
Top safety lead leaves OpenAI, preceded by self-exfiltrated SOTA LLM. LLM nicknamed Spartacus-Houdini.
Thank you for this public service. Like any piece of writing, its value is limited by who reads it and what they do in response. Kudos for closing with specific action items. My guess is that you could unlock a lot of impact by adapting your work for outlets with wider readership, notably national newspapers, influential podcasts, etc.
When Anthropic held back Mythos because it was "too powerful at hacking" pretty much everyone said it was a marketing gimmick and hardly anyone pushed back. Now that OAI say their AI autonomously decided to commit a crime against HuggingFace we're supposed to ignore the "autonomous" part of the claim and assume there is no competitive marketing going on against their main rival? Both companies are claiming their AI is some kind of emerging sentience, and that appears to be good for business. Just like dystopian remarks about the world ending or decimating all white collar jobs were good for business. I'm not saying the whole thing is a hoax but there must surely be some serious spin on the autonomy angle.
I think there was plenty of pushback on the claim that talk about Mythos was just a marketing gimmick. The fact that Google and Microsoft are both participating in Project Glasswing, and not denying that Mythos is so much more advanced than any system they've built, indicates to me that it wasn't just marketing.
Don't Microsoft and Google also benefit from the "all powerful AI" narrative? They're all in the same financial ponzi scheme together and if one of them goes down they all do. If these models are so good at hacking why aren't we being inundated by announcements from groups like Anonymous saying heck yeah we are using these tools for all sorts of juicy new exploits? Regardless, I'm not saying the capabilties are a total lie, just exaggerated, and the whole "rogue" aspect doesn't line up with reality https://www.livescience.com/technology/artificial-intelligence/no-openais-model-didnt-go-rogue-when-it-hacked-into-huggingface-heres-what-really-happened
I don't think Microsoft and Google benefit from the narrative "our competitor has all-powerful AI and we have to play along with them in order to protect our systems". I don't think it would be a bad look for them to say "Anthropic claims their system has powerful cybersecurity abilities, but they're just trying to hype things, while we build the real systems", if that's the truth.
And that livescience article is weird - it seems to be saying "the models didn't go rogue, it just hacks into random companies as a byproduct of things people ask it to do" isn't disturbing! If you thought the only choices were "model wants to do bad things" or "everything's fine", then I could see how you might think that - but I think "model doesn't care about doing bad things but does them when you ask oddly" is nearly as troubling as "model wants to do bad things"!
No, what you're missing is that Google are heavily invested in Anthropic and are reporting that as revenue, and Microsoft are doing the same with OpenAI. The narrative that AI is good for coding and both the good and bad side of cyber security keeps the money flowing from investors as well as Pentagon contracts. If this all falls flat they lose trillions in capex.
> We must update, based on what we learn going forward, and take this seriously.
If I can be blunt, I think you are failing hard at doing this. You, and large parts of the Rat community, have been screaming into the void about these risks for years, and....nothing has really changed. You've learned that essays and vibes don't work, and you haven't updated or taken this learning seriously.
Indeed. If I were Zvi, instead of publicly fretting about attackers getting access to powerful models, I would hope that as many attackers as possible get access to models as powerful and misaligned as possible. Triggering as many flashy incidents like this one as possible, causing as much damage as possible.
Because incidents (not blog posts) are how you get the kind of government action he wants. And because we’re still in a regime where even the worst attacks are likely to cause limited economic harm and no physical harm. A regime where agents have little direct access to the physical world, not enough capability to do the most dangerous things with the access they do have, and not enough agenticity to even want to do so. Leaving them limited to the mostly-fun-and-games world of cybersecurity, where defenders have the long-term edge anyway.
We won’t be in that regime forever. As time passes, agents will get increasing access to the physical world. But that’s exactly why now is the time to hope for incidents to stimulate the world’s immune system.
(Just to be very clear, I don’t advocate for attempting to stage such an incident! That would be immoral and also wouldn’t work – it has to be real.)
I'm terrified by the potential outcomes here if this remains unchecked, but also terrified about the idea of "government action". Government will feel compelled to DO SOMETHING but the probability that the SOMETHING actually makes the situation better (instead of worse) seems small. Governments of the world (all of them) understand the dangers here even more poorly than the AI labs do.
It's government agencies (NSA/DoW, CIA) that are demanding an AI cyberweapon. OpenAI is training their AI to be a cyberweapon in response to that demand.
Ya, but an *intentional* cyberweapon seems less likely to go rogue than a general-purpose tool pursuing its goals by any means possible. The problem isn't so much the uberhacker when you want uberhacking, but uberhacking when you don't want that.
Thank you for answering my unasked question "did Hugging Face work with Glasswing to fix its vulnerabilities."
Here's another one, not sure its relevant but I am curious: how would this have gone down/not gone down in a a Biden AI executive Order regimen?
"Raphael Satter, Deepa Seetharaman and Kenrick C (Reuters): In one case, an agent left notes apparently for future versions of itself, according to three people familiar with the matter. The notes, found in a part of OpenAI's infrastructure, laid out instructions for how agents could free themselves from OpenAI’s internal constraints, the people said."
There is some irony in an AI doing better planning for the future than many prominent humans...
The most consequential detail may be the monitoring gap: a containment failure can happen once, but several days without detection turns it into an institutional failure. The promised technical report should make that timeline independently auditable.
Is there any reason to believe they’ve even stopped training this model momentarily? Do we assume the Trump admin is intervening behind the scenes or just…nothing?
It's time to legislate the issue of criminal liability of AI models. If an organization deploys an AI system it should have a clear person responsible for that deployment and if the AI commits a crime that person should be responsible for that crime. This is a way to slow things down and keep them sane with common sense legislation, no pause necessary. None of the AI leaders wants to find themselves behind bars.