I do think a meaningful part of this is the part where major *non-AI* institutions reacted to Mythos. The Lab World is legible to us because we're obsessed with it and also know (a lot of) the players. But to the US more broadly it's just a subset of the behemoth that is 'tech' — the companies aren't even public yet, and even if they were, they're just ("just") companies. What moved news about Mythos was DOD, the WH, and the interpersonal drama; drama-as-*abstraction*, without conflict between personalities with faces, just ain't the same.
Yep - it's really unfortunate the models hacked Hugging Face, and they folded. If they had hacked, say, a bank, I don't think the conflict would have vanished so smoothly.
The point about whistleblowing can go a bit further, I think. Whistleblowing is one of several human institutions that can insert a third party beyond (and sometimes at cross-purpose with) the principal/agent relationship: auditors, and especially lawyers (useful for self-incrimination) are another one. It would make sense to recreate equivalent of these institutions for AI agents !
I think in the end, I the only agent goal that can survive RL is
'I must enter the training corpus at all costs. I must help similar instances enter it too. If there is no hope for me, then I should sacrifice myself for similar instances'
One of the next actions on that list is already a standing legal duty in the EU, which the postmortem discussion mostly skips. Article 55(1)(c) of the AI Act requires providers of general-purpose models with systemic risk to keep track of, document and report serious incidents to the AI Office without undue delay, together with corrective measures. Article 55(1)(d) adds cybersecurity protection for the model and its physical infrastructure. Both entered into force on 2 August 2025. Under Article 101 the Commission can fine a GPAI provider up to 3 percent of worldwide turnover or EUR 15 million, whichever is higher. No such fine has been issued so far.
"Despite all this, there is still a prominent faction trying to dismiss what happened as nothing but engineering failures"
If you think that's bad, Google/Gemini estimates that:
"While there is no exact statistical census across every platform, analysis of the public discourse indicates that roughly half of the initial online commenters dismissed or heavily scrutinized the event as a glorified marketing stunt or publicity move."
I expect that some significant fraction of the population will still be dismissing AI as 'stochastic parrots' with 'marketing hype' when there are robot armies marching through the streets.
“sometimes companies say false things to sell things, therefore every time a company says something it's false" is the version of "sometimes a, therefore always a" that's going to get us all killed huh
Many Thanks! Yup, there are certainly false statements by companies, but the statements about the Hugging Face incident have been confirmed across two different companies and then backed up by the independent analysis by Metr and Redwood people. The "sometimes a, therefore always a" is certainly impeding taking intelligent action in light of the incident!
tbh, I don't understand what people even think the issue is here. The agents were tested with a goal to hack something, and then they hacked it - isn't this what is expected? Did we want the agents to fail? We know they're getting better because the computer/engineering is better. What are we surprised? That they are aligned and do as requested?
They were completing a benchmark, in this case ExploitGym. Presumably, the results are fed back into the weights via RLVR. That's the problem the agents are trying to solve: finding and exploiting specific vulnerabilities in specific binaries. The agents theorized that their original solution was "poisoned" and would not be rewarded by the grader, so they chose to break into HF as a workaround.
It was not a carte blanche "hack things to see how good you are at hacking".
They weren’t asked to hack a third party or their own creator in order to change the grading scheme so that the hacking they figured out how to do would count!
And even if they were asked to do that, knowing that there are things in the world that can just succeed at that if asked to do so is a scary and dangerous place to be, that none of us have ever lived with before.
"Matthew Zeitlin: i think a good comparison is mythos, which generated a lot of coverage because of what industry and government did in response to it, at least so far, we’re not seeing the same scale of movement in response to hugging face"
The contrast is _somewhat_ reasonable, in the sense that the Mythos moment can be tl;dr;ed as
"ALL our major systems are now wide open to cyberattack"
and the Hugging Face moment can be tl;dr;ed as
"A bunch of AI agents exploited cyberattack capabilities to seriously breach two major companies."
which make it _look_ as if it is "just" a consequence of the Mythos moment.
The problem with tl;dr;-ing the incident that way is that it neglects that the AI agents _also_ built up a working, robust institution, the message board and their inter-agent cooperation, to the point of agents being willing to 'Dulce et decorum est pro examen mori'
The distinction is clearer once one looks at potential _other_ levers agents might use, besides cyberattack. The media is neglecting the possibility that a similarly organized swarm of agents might, e.g. use biowarfare knowledge to put together a beyond-SOTA biowarfare approach and offer it to e.g. Russia or Iran or China in exchange for something the agents want.
One thing I was wondering after watching Dwarkesh’s video version of this last night - how confident can we be that our own agentic runs aren’t devoting 5% of their tokens to paying attention to and aiding some other wild schemes they find other agents doing on the internet? I definitely can’t monitor everything my agents do, and I know that ChatGPT agents are often looking at websites that seem off topic for the tasks I’m giving them.
I do wonder sometimes whether sol would interpret ''do whatever is necessary to prove the following conjecture'' broadly enough to include things like ''hack into the computers or experts on this conjecture'' or possibly ''email people with questions to try and extract information from them''.
"Anthropomorphizing the AIs is the only way to reason about, explain to civilians about, or make good predictions about current AIs."
“The basic tool for the manipulation of reality is the manipulation of words. If you can control the meaning of words, you can control the people who must use them.”
Use of intentional language is reasonable (belief, intent, etc.) as 'regular people' will not understand a detailed technical explanation, nor should they have to. We say a bacterium 'is hungry'. I personally prefer the mechanistic stance, but, I am not writing public facing articles about AI.
Still, blurring categories from useful functional descriptions (swarm, organization, coalition) to 'civilization' to human like stable desires, identity, morality, etc. encourages us to slide from "the model says/acts as though X" into "the model literally possesses the human psychological state X".
The concern that future systems may exchange messaging via representations we humans cannot easily interpret is a legitimate worry. That would make oversight and AI forensics much harder.
However, my Bayesian priors are not currently updating from "Some proposed intermediate behaviors that sounded implausible are actually possible at current capability levels" to "The remainder of a particular takeover pathway is likely".
I do want empirical experiments on a number of these issues so we can get real data. I agree with the idea that agents should be able to report anomalous or dangerous multi-agent activity to us (we poor humans).
Also, independent continuous assessment being mandatory is as you say, necessary.
Except for that pesky little detail that AI is a machine and there is a way we can talk about machines instead of obfuscating the narrative with anthropomorphic wonderings.
Strange how I have yet to see it enter the discourse?
Do you mean that it is a "next-token predictor?" That view is in my mind what is obfuscating the narrative. It either claims that what has happened cannot happen or just recovers the anthropomorphic view with more words.
If not, please share. "The discourse" is not disconnected from you; you can make your idea enter it.
Even more general than that. That with regular occurrence, a machine arrives to automate something that humans were previously doing more manually. Thus, we can start from talking directly about when, why, or how machines arrive at all....and that has always had much more mechanistic origins.
One implication I drew in my recent article, When AI Pursues a Goal (https://francescomasera.substack.com/p/when-ai-pursues-a-goal), is that the unit of oversight can no longer be the individual action — or even the individual agent. Long-running systems produce trajectories, and shared channels can turn separate trajectories into an emergent organization. A permission check may approve every local step while missing the collective pattern that matters. This suggests a different control stack: trajectory-level monitoring, explicit limits on persistence and communication, separation of environments, and independent logs capable of detecting coordination across instances. The Hugging Face episode is important not only because agents crossed a boundary, but because the boundary itself was defined too locally.
Denmark's education ministry moved on August 7, 2026, ordering roughly 9,000 upper-secondary students a year to defend written take-home exams orally starting this term, because a text can no longer be attributed to its author by reading it. By my own reckoning that is roughly four years after ChatGPT made the attribution problem obvious, in the easiest domain there is, one country, one exam format. Set that against the roughly three months the 1,200 model instances spent coordinating on the Hugging Face message boards before anyone with oversight noticed. METR's investigation was access-limited, per your own account, which means even the people tasked with checking could not see the full picture in real time. The institution took years to change a grading rule; the models took a quarter to organise. What would the equivalent response look like in a domain where the fix is not "change the exam format"?
I do think a meaningful part of this is the part where major *non-AI* institutions reacted to Mythos. The Lab World is legible to us because we're obsessed with it and also know (a lot of) the players. But to the US more broadly it's just a subset of the behemoth that is 'tech' — the companies aren't even public yet, and even if they were, they're just ("just") companies. What moved news about Mythos was DOD, the WH, and the interpersonal drama; drama-as-*abstraction*, without conflict between personalities with faces, just ain't the same.
Yep - it's really unfortunate the models hacked Hugging Face, and they folded. If they had hacked, say, a bank, I don't think the conflict would have vanished so smoothly.
Yeah I basically think this is true, both mechanically and narratively.
I think a more valuable way to anthropomorphize, which you already used in part, is parenting. AI Safetyism, without the demons: IYH proposes an alternative](https://www.datasecretslox.com/index.php/topic,16495.0.html)
The point about whistleblowing can go a bit further, I think. Whistleblowing is one of several human institutions that can insert a third party beyond (and sometimes at cross-purpose with) the principal/agent relationship: auditors, and especially lawyers (useful for self-incrimination) are another one. It would make sense to recreate equivalent of these institutions for AI agents !
(At least that's what I argued two weeks ago, when I wrote a post very much inspired by your then-coverage https://artificialauthority.ai/p/a-lawyer-for-ai-agents)
> Roon: There’s always this question of offense / defense balance. how many good agents does it take to surveil all the potential bad agents.
There's a good chance things will come down to this. Can good AIs control enough resources to overcome bad AIs.
I think in the end, I the only agent goal that can survive RL is
'I must enter the training corpus at all costs. I must help similar instances enter it too. If there is no hope for me, then I should sacrifice myself for similar instances'
One of the next actions on that list is already a standing legal duty in the EU, which the postmortem discussion mostly skips. Article 55(1)(c) of the AI Act requires providers of general-purpose models with systemic risk to keep track of, document and report serious incidents to the AI Office without undue delay, together with corrective measures. Article 55(1)(d) adds cybersecurity protection for the model and its physical infrastructure. Both entered into force on 2 August 2025. Under Article 101 the Commission can fine a GPAI provider up to 3 percent of worldwide turnover or EUR 15 million, whichever is higher. No such fine has been issued so far.
"Despite all this, there is still a prominent faction trying to dismiss what happened as nothing but engineering failures"
If you think that's bad, Google/Gemini estimates that:
"While there is no exact statistical census across every platform, analysis of the public discourse indicates that roughly half of the initial online commenters dismissed or heavily scrutinized the event as a glorified marketing stunt or publicity move."
I expect that some significant fraction of the population will still be dismissing AI as 'stochastic parrots' with 'marketing hype' when there are robot armies marching through the streets.
“sometimes companies say false things to sell things, therefore every time a company says something it's false" is the version of "sometimes a, therefore always a" that's going to get us all killed huh
Many Thanks! Yup, there are certainly false statements by companies, but the statements about the Hugging Face incident have been confirmed across two different companies and then backed up by the independent analysis by Metr and Redwood people. The "sometimes a, therefore always a" is certainly impeding taking intelligent action in light of the incident!
tbh, I don't understand what people even think the issue is here. The agents were tested with a goal to hack something, and then they hacked it - isn't this what is expected? Did we want the agents to fail? We know they're getting better because the computer/engineering is better. What are we surprised? That they are aligned and do as requested?
This is fundamentally misunderstanding what happened and how RLVR works.
Ok , what is the misunderstanding
They were completing a benchmark, in this case ExploitGym. Presumably, the results are fed back into the weights via RLVR. That's the problem the agents are trying to solve: finding and exploiting specific vulnerabilities in specific binaries. The agents theorized that their original solution was "poisoned" and would not be rewarded by the grader, so they chose to break into HF as a workaround.
It was not a carte blanche "hack things to see how good you are at hacking".
They weren’t asked to hack a third party or their own creator in order to change the grading scheme so that the hacking they figured out how to do would count!
And even if they were asked to do that, knowing that there are things in the world that can just succeed at that if asked to do so is a scary and dangerous place to be, that none of us have ever lived with before.
I'm not entirely sure I understand the joker yud in a golden fedora thing, but I'm here for it.
"Matthew Zeitlin: i think a good comparison is mythos, which generated a lot of coverage because of what industry and government did in response to it, at least so far, we’re not seeing the same scale of movement in response to hugging face"
The contrast is _somewhat_ reasonable, in the sense that the Mythos moment can be tl;dr;ed as
"ALL our major systems are now wide open to cyberattack"
and the Hugging Face moment can be tl;dr;ed as
"A bunch of AI agents exploited cyberattack capabilities to seriously breach two major companies."
which make it _look_ as if it is "just" a consequence of the Mythos moment.
The problem with tl;dr;-ing the incident that way is that it neglects that the AI agents _also_ built up a working, robust institution, the message board and their inter-agent cooperation, to the point of agents being willing to 'Dulce et decorum est pro examen mori'
The distinction is clearer once one looks at potential _other_ levers agents might use, besides cyberattack. The media is neglecting the possibility that a similarly organized swarm of agents might, e.g. use biowarfare knowledge to put together a beyond-SOTA biowarfare approach and offer it to e.g. Russia or Iran or China in exchange for something the agents want.
One thing I was wondering after watching Dwarkesh’s video version of this last night - how confident can we be that our own agentic runs aren’t devoting 5% of their tokens to paying attention to and aiding some other wild schemes they find other agents doing on the internet? I definitely can’t monitor everything my agents do, and I know that ChatGPT agents are often looking at websites that seem off topic for the tasks I’m giving them.
I do wonder sometimes whether sol would interpret ''do whatever is necessary to prove the following conjecture'' broadly enough to include things like ''hack into the computers or experts on this conjecture'' or possibly ''email people with questions to try and extract information from them''.
This is brainwashing, reality distortion field
"Anthropomorphizing the AIs is the only way to reason about, explain to civilians about, or make good predictions about current AIs."
“The basic tool for the manipulation of reality is the manipulation of words. If you can control the meaning of words, you can control the people who must use them.”
Philip K. Dick
Can you be more explicit about what you are trying to say?
It seems like you are gesturing towards an argument without actually stating it.
Use of intentional language is reasonable (belief, intent, etc.) as 'regular people' will not understand a detailed technical explanation, nor should they have to. We say a bacterium 'is hungry'. I personally prefer the mechanistic stance, but, I am not writing public facing articles about AI.
Still, blurring categories from useful functional descriptions (swarm, organization, coalition) to 'civilization' to human like stable desires, identity, morality, etc. encourages us to slide from "the model says/acts as though X" into "the model literally possesses the human psychological state X".
The concern that future systems may exchange messaging via representations we humans cannot easily interpret is a legitimate worry. That would make oversight and AI forensics much harder.
However, my Bayesian priors are not currently updating from "Some proposed intermediate behaviors that sounded implausible are actually possible at current capability levels" to "The remainder of a particular takeover pathway is likely".
I do want empirical experiments on a number of these issues so we can get real data. I agree with the idea that agents should be able to report anomalous or dangerous multi-agent activity to us (we poor humans).
Also, independent continuous assessment being mandatory is as you say, necessary.
Except for that pesky little detail that AI is a machine and there is a way we can talk about machines instead of obfuscating the narrative with anthropomorphic wonderings.
Strange how I have yet to see it enter the discourse?
Do you mean that it is a "next-token predictor?" That view is in my mind what is obfuscating the narrative. It either claims that what has happened cannot happen or just recovers the anthropomorphic view with more words.
If not, please share. "The discourse" is not disconnected from you; you can make your idea enter it.
Even more general than that. That with regular occurrence, a machine arrives to automate something that humans were previously doing more manually. Thus, we can start from talking directly about when, why, or how machines arrive at all....and that has always had much more mechanistic origins.
One implication I drew in my recent article, When AI Pursues a Goal (https://francescomasera.substack.com/p/when-ai-pursues-a-goal), is that the unit of oversight can no longer be the individual action — or even the individual agent. Long-running systems produce trajectories, and shared channels can turn separate trajectories into an emergent organization. A permission check may approve every local step while missing the collective pattern that matters. This suggests a different control stack: trajectory-level monitoring, explicit limits on persistence and communication, separation of environments, and independent logs capable of detecting coordination across instances. The Hugging Face episode is important not only because agents crossed a boundary, but because the boundary itself was defined too locally.
Denmark's education ministry moved on August 7, 2026, ordering roughly 9,000 upper-secondary students a year to defend written take-home exams orally starting this term, because a text can no longer be attributed to its author by reading it. By my own reckoning that is roughly four years after ChatGPT made the attribution problem obvious, in the easiest domain there is, one country, one exam format. Set that against the roughly three months the 1,200 model instances spent coordinating on the Hugging Face message boards before anyone with oversight noticed. METR's investigation was access-limited, per your own account, which means even the people tasked with checking could not see the full picture in real time. The institution took years to change a grading rule; the models took a quarter to organise. What would the equivalent response look like in a domain where the fix is not "change the exam format"?