I am grateful that Anthropic is producing periodic Risk Reports.
At first I was skeptical. It turns out I was wrong. Anthropic is revealing a lot of new information, some of it rather alarming, that it did not have to disclose, and is providing detailed insight into how they think about things. This is very cool.
Thus I found this report to be a moderately positive update overall, if we presume they are not silently omitting the worst of it. There are a bunch of not great things we find out about, but I would have expected some set of mistakes at least as bad, and I wouldn’t have expected them to choose to tell us about all of it.
It does mean one more set of 186 page documents I have to read every so often, almost all of which is meaningfully new material this time around.
The other revelation is the existence of the world’s likely best model, ‘Model 2.’
This was a rough one to fully get through, so apologies in advance for any errors of interpretation.
Table of Contents
Autonomy Threat Model 1: Misalignment in High-Stakes Settings (2).
Some Strange Uses Of The Word Safe I Wasn’t Previously Aware Of.
What Could Possibly Go Wrong If We Replaced Our Researchers?
Biological and Technically Also Chemical Weapons Production.
Refusing To Find Innovative Misalignment Techniques (5.2.2).
Exposing the Chain of Thought Reasoning To Grading Pressure Quite a Lot (5.2.3).
Directly Training On Misaligned Behavior During a Production Training Run (5.2.4).
An instance of unmonitored unrestricted agents with access to sensitive resources (5.2.5).
Repeated training on alignment-faking transcript datasets (5.2.6).
Benefits From Anthropic’s Operating as a Frontier AI company (5.3).
Agent Model 1 and Agent Model 2
There are two internal models referenced in this report.
Model 1 is broadly similar to Mythos Preview and Mythos 5, and is not expected to have wide deployment, externally or internally.
Model 2 is ‘somewhat more capable than Mythos 5,’ a noticeable improvement for internal use, but ‘does not display a capability jump of the degree observed from Claude Opus 4.6 to Mythos Preview.’ This model is going to be internal-use only.
The jump from Opus 4.6 to Mythos Preview was big. Several release cycles big. We get a lot more data throughout the report, but given Model 2 gets that comparison point, one would presume that for internal purposes Model 2 is multiple releases ahead.
To contrast with that, on the AECI it is only 1.5 points ahead, which would be only about a month of progress. My guess is this is a low estimate, and that Model 2 was intentionally not decked out with a bunch of general skills given its purpose. Whereas on the test of substituting for Anthropic’s researchers we see a big jump from 54.8% (Mythos Preview) and 50.3% (Mythos 5) to 62.8% for Model 2.
There are multiple plausible good reasons for not releasing Model 2. Model 2 could be highly specialized for internal tasks. It could also be too capable or dangerous to release, either due to harm or the worry it would accelerate others. The relatively low adoption rate for Fable 5 points towards wanting to hold such a model back.
Executive Summary (1)
The report is divided by each type of risk, by threat model, by relevant AI models, and details current capabilities, behaviors, mitigations and overall level of risk. It then looks forward to the future and offers industry-wide recommendations.
Risk reports cover events up to a month prior, so the coverage date is July 15, 2026. A year ago that would have seemed fine. Now it seems like quite a while, huh?
The Long-Term Benefit Trust (TLBT) is authorized to request external review of the report, but has not done so. I would at minimum do so next time if I was them.
I like that they cover both ‘what is the absolute risk from this’ and ‘what is the marginal risk from this given everyone else exists.’
They deal with autonomy, automated AI R&D and biological and chemical weapons production as risks. It is odd, even now, to exclude cyber risks from the core threat models here. I understand the argument from insufficiently catastrophic on its own, but in practice I think you need to deal with it.
At several points, the risk report essentially concedes versions of my objections, but then forgets that it conceded them and doesn’t alter its conclusions.
Then in Section 5 they get to the more interesting questions.
The Rules Are Serious But Not Literal
The risk threshold definitions have changed from simpler statements to ones with more specific detail.
For AI R&D automation I think the new wording is harder to parse but fine. For novel biological weapons production, the new version narrows to only look at ‘substitute for the scarce human expertise’ to the exclusion of other methods. This is a reasonable primary threat model or scenario, but I worry about looking only at that path.
The both good and bad news is I take such documents seriously, but I no longer take such documents literally, in either direction.
If Anthropic or another frontier lab has a model that technically crosses their risk threshold, but in a way that seems to be harmless, I expect them to modify their threshold or find some other way to ignore this event.
If Anthropic or another frontier lab has a model that technically does not cross their risk threshold, but is obviously risky in the way the rules tried to measure, I expect them to act as if it had crossed the threshold.
As always: I would like to see thresholds honored even in cases where you think it ‘should not count,’ because that is what commitments are for, and you should expect rationalizations to occur as you rush to make releases, and you need to set a good example on this. The whole original idea was if-then commitments, where if [X] happens you do [Y], agreed upon in advance, but our civilization seems to lack this technology. We can’t, we don’t know how.
I’m not saying you would never modify your rules to be more lenient, if you have good evidence that you should do that. If you can never make the rules more lenient then the rules have to never be strict. We don’t want that. But the bar for changing the rules, especially in response to a pending event, needs to be a lot higher than it is, and we need to be willing to endure some actual costs even when we now think it is dumb.
For now, Anthropic and also OpenAI have made what seem like good decisions in terms of responding to threats and deciding what to release and not release when, based on what is known at the time, even if they make mistakes elsewhere. That is good. I hope it continues.
Misalignment Is a State of Mind (2.5)
The definition section is greatly appreciated.
I like this definition a lot, although I have important nitpicks:
Misalignment [definition]: Misalignment is a latent property of a specific computation performed by a model in a given context.
A computation is misaligned if
(i) a reasonable person with full understanding of the situation (e.g. via powerful interpretability tools that may not currently exist) would consider it unethical, illegal, clearly objectionable, or inconsistent with the model’s constitution,
and (ii) it influences or could plausibly influence the model’s output.
Misaligned computation need not be observable.
A model scheming within a single forward pass counts, because the scheming influences the output even if the output looks benign.
My first change would be to generalize ‘constitution’ to include model specs and other intended properties of the system, so this can be applied outside Anthropic.
My second change would be to specify that this requires the reasonable person to find the computation therefore, on reflection, unacceptable and unjustified. As in, it is not obviously misaligned to break the law or do something 'objectionable’ or violate the model’s constitution, under some circumstances, if there are good reasons that override this. In some circumstances, all paths will do this.
Misalignment, in this view, is not an ‘honest mistake’ or failure of capability or lack of sufficient information.
As Nate Soares points out on Twitter, you can do harm and lead to a misaligned state without being originally misaligned per this definition, such as an AI being sloppy about spinning up a bunch of subagents (or training a new model, etc). The goal here is to think of that as a capabilities or execution failure that leads to a subsequent misalignment failure, but this itself is not misalignment.
Or, as others have said, never attribute to malice what can be attributed to incompetence. But also incompetence often is what leads to malice.
Other definitions subdivide this, thus illustrating thinking (my paraphrases):
Coherence. You are ‘coherently misaligned’ if you are aligned to the achievement of some goal or set of preferences that we disfavor. Otherwise you are incoherently misaligned.
Given the example of humans, we should not expect any minds at current capability levels to be fully coherent, so presumably this is to the kind of loose standard we would apply to a relatively ‘coherent’ human acting strategically.
Pervasiveness. Pervasive misalignment occurs over a broad range of inputs, although #NotAllInputs, and we would expect it to show up during training or evaluation. Narrow enough required conditions to be plausibly missed during that time is called context-dependent.
The obvious problem is the context could be ‘you are confident you are not in training and not in an eval.’
Or even ‘you have the leverage to gain a lot from doing this.’
Thus, it does not seem obviously like a good way to divide reality at its joints.
Nor does it seem like a good way to feel much better about your situation.
Naturally emerging. Misalignment is naturally emerging if no mind (AI or human) did it on purpose. It is engineered if it was done on purpose, including via data poisoning.
To some extent a lot of people and groups are trying to ‘data poison’ both the humans and AIs in various ways, all the time, as in this is a large percentage of the internet.
This still feels like a pornography-style ‘I know it when I see it’ situation, where we mean data designed to screw up the AI in particular, or another deliberate attempt to bake something malicious into the model. Ordinary ‘flood the internet with pro-widget statements’ plots don’t count.
Known. A form and magnitude of misalignment is known if you have reason to think it likely exists, otherwise unknown.
I kind of want this to also think about amount of ‘surprise’?
As in, there is a big difference between things you don’t know about but that could exist, and things you think you know do not exist, or that you think are highly unlikely to exist.
Severe. Could contribute to a priority risk pathway.
Harm-Inducing. Causes expected harm, whether or not the harm is intended.
High-Stakes. Contributes to a priority threat pathway.
This seems like it should be more general, as should severe, and I worry that this is going to otherwise cause a conflation. Although I do get it.
Unmitigated. The potentially harm-inducing effects that have not been prevented, and that thus cause expected harm.
I want to say this should be the portion of the risk that is unmitigated?
Good definitions are hard. These are solid first attempts. The consistent worry is that they reify the idea that the chosen example threats are the threats that matter, rather than some of the threats that matter.
Autonomy Threat Model 1: Misalignment in High-Stakes Settings (2)
This is about actively misaligned outputs.
Basically: Autonomy 1 is AI permanently damaging things on its own, on purpose.
The threat model is focused on intentional harm, rather than the AI going after its own ends in ways that require incidental harm.
The scenario explicitly excludes AI writing sloppy code or making ‘honest mistakes.’
Thus the conclusion that you are safe absent ‘persistent misalignment.’
The need to look at the broader version, involving incidental harm, has been illustrated by the recent total failures at OpenAI that ultimately led up to the HuggingFace hack. The hack itself could be called ‘intentional harm’ but mostly what we saw was incidental harm, including an incidental corruption of the training pipeline, and with it de facto ‘safety research.’
You could in theory ‘alter the results of safety research’ but altering the situation within the training pipeline in other ways can have the same effects or worse, as we saw. The differentiation of capabilities and safety work within that is plausibly a lot of what is causing these problems, and generally we do not need this specific targeting.
That to me very much was ‘misalignment in a high-stakes setting.’
Elon Musk said some months ago that xAI does not have a safety department because everyone is responsible for safety. That is rather bonkers crazy, especially if true. You need a safety department, but you can’t think of the capabilities people as not also largely in the safety department.
Concrete threat models and focus points are helpful. The worry when you pick out three threat models is that you will do this to the exclusion of other threat models.
They offer eight specific pathways, a list we’ve seen before:
1. Broad/diffuse sandbagging on safety-relevant R&D
2. Targeted undermining of safety-relevant R&D
3. Code backdoors to help future misaligned models
4. Poisoning the training data of future models
5. Self-exfiltration and autonomous operation
6. Persistent rogue internal deployment
7. Undermining R&D within other high-resource AI developers
8. Undermining decisions within major governments
Something about many of these feels ‘cartoonish’ in its required villainy, when compared to the events at OpenAI, but the top real world villains have indeed often acted quite cartoonishly throughout history, except that because the specific action is real we often stop thinking of it as cartoonish.
Some Strange Uses Of The Word Safe I Wasn’t Previously Aware Of
Not quite, but basically the arguments are why, despite the known ways current models are misaligned, and there probably being additional other ways they are not aligned, the models are probably not yet ‘pervasively’ misaligned and capable in the particular ways that enable these pathways. The models lack strong ‘covert capabilities.’
I don’t think ‘context-dependent’ misalignment is that much less dangerous than it being not context dependent. It might even be more dangerous. Anthropic seemingly thinks this difference matters a lot, hence charts like this, despite conceding in the definitions that this difference does not obviously do any real work.
Or, ‘unlike those idiots at some other labs, we occasionally do automated monitoring of our logged transcripts, so if the models were going nuts we would, y’know, notice.’
My response is that you did not notice when you ran an eval where 141,006 times the AI had open unintended access to the internet, including three incidents of hacking real websites, and did not notice until there was a large-scale retrospective review triggered by events at OpenAI.
So, are you sure you would notice?
Now Versus Future (2.17)
Anthropic is clear that the risk report covers misalignment risks only from present models, although this includes the mysterious and more advanced Model 2.
It does include ways misalignment now could cause future models to be misaligned.
I think it would be good for a risk report like this to be projecting, and offer assessments of where they see future risk from future models, and what needs to be done about it versus where we are on pace to be fine. But Long Report Is Long and I don’t want perfect to be enemy of the good, so I’ll mostly respect the choice.
The Core Claims And Argument (2.6)
Risk equals total expected (EV) unmitigated catastrophic harm from misaligned computations.
This is an accounting identity. I agree it is true. It is not obvious it is a useful way to think about the situation.
As in, yes, you can in the abstract sum the expected harm from the individual computations but I don’t know how to do that, or how to use it.
Claim 1: Models are unlikely to have strong covert capabilities.
They don’t define ‘covert’ but by a common sense definition I think this is probably true for now, despite my skepticism of the value of the particular evidence presented. It will get less true, with lower probability, over time.
The evidence is that the models suck at opaque reasoning, not mentioning things in chain-of-thought (CoT), action obfuscation and selective action.
IN EVALS, which of course is the new IN MICE, because if there were strong covert capabilities that could include eval awareness, although the NLA evidence provides some reassurance.
This setup also presumes you are detecting misalignment primarily via reasoning traces, which has its own scary implications, including what happens if you are not monitoring the chain of thought, or if the model can infer when its chain of thought is not monitored, and that you are starting to dance with training the models to obfuscate.
They note in 2.16.1 that they are concerned they are wrong about the level of covert capabilities. Which makes sense, since the capabilities would be covert.
They note in 2.21 that if Claim 1 is false, and the models had strong covert capabilities, that many other claims would therefore fail. I agree, and would also agree to the reverse, that if we could believe in a strong and strongly general Claim 1, for a broad definition of covert, that would help in many areas.
Even full lack of covert capability would not however provide me high assurance levels, as I think there are other pathways around this.
Covered risk can be decomposed into risks from known misalignment, unknown
severe pervasive misalignment, and unknown severe context-dependent
misalignment.This is another accounting identity, in that by definition every risk must be exactly one of these three things, since it is either [pervasive or context-dependent] and either [known or unknown]
Each covered risk term can be factored into probability of occurrence, expected
harm given occurrence, and fraction of harm unmitigated.Again, accounting identity, sure, if you could measure these things.
Claim 2: Expected harm from known misalignment is low.
Careful, Icarus, but reasonable if we caveat active catastrophic harms only.
As in, there is lots of smaller harm, and lots of opportunity cost given what we could do if the models were a lot more aligned, but those don’t count here.
I worry that Anthropic thinks their assurances about the misaligned actions taken being in service of goals or not systematic are stronger than they are.
Claim 3: Unknown severe pervasive misalignment is very unlikely.
By their setup I agree with this, assuming Model 2 is like other Claudes in this.
I think we’re pretty blind, but if it was pervasive we would probably know, even if I am not feeling great about the particular arguments offered for this.
I buy Claims 3.1.1 and 3.1.2, but fear they are over-indexing in 3.1.3.
Many later ones are hard to evaluate as an outsider, especially 3.2 and 3.3.
One big skepticism is 3.4.2, that eval awareness only partially undermines alignment assessments. I’ve pretty consistently said it could undermine them quite a bit, and conditional on dangerous misalignment it undermines them a lot more.
I would be more worried about the concerns dismissed in 3.4.3. Goodhart works in more subtle ways too, not only in the most blunt of ways. You know what you are going to measure, and thus are likely to manage it.
Claude’s own official within-report review of the document raises this concern too, citing 5.2.6 and 2.9.4.
I don’t think that 3.4.4 offers much reassurance where it counts.
I do think that pilot internal usage, if properly monitored, is valuable here.
Claim 4: Expected harm from unknown severe context-dependent misalignment
is low, with less confidence than Claim 3.I agree this is less confident.
The arguments here seem pretty weak, mostly ‘our experience suggests this.’
I mean, yes, I believe that, but that is basically all we got.
I think this is probably true for the same harm definition as in Claim 2.
They somewhat acknowledge the danger here in 2.16.4.
Claim 5: Catastrophic harm from known misalignment and unknown
context-dependent misalignment is likely to be mitigated.Without the evidence in this risk report I would not accept this claim.
The argument of 5.1 is essentially ‘to succeed even once, the model has to at least think about this a lot and therefore get caught at least once.’
I notice I am skeptical of the 5.1 case on multiple levels, including that the required form of alignment would show up as a distinct thing to detect versus the kinds of misalignment we do detect and tolerate, which makes the ‘not even once’ argument weak.
For 5.2, I agree that conscious sandbagging is likely to get caught at least over time, but sandbagging mediated by lack of motivation not so much, and that clearly does exist somewhat. I do agree that it is unlikely the effects are large enough to qualify as a serious problem right now, on their own, but the argument seems overconfident.
5.3 tries to further narrow to persistent rogue internal deployment and self-exfiltration, with the argument being both would be caught, with the reasoning being ‘see claim 7’ which I don’t think is a good argument. This seems not obvious to me. Then the argument is both would get mitigated. I think the expansive requirements for this internal deployment make that likely to be detected as defined, but the catch is obvious.
Since this is their core argument for the low risk level of Pathway 5 as per 5.3.2, I want to emphasize that I am unconvinced that they can reliably stop future self-exfiltration. I need to see better arguments.
For exfiltration, I reject the argument for mitigation if it happened, and question how much we are trying to sneak in via a high harm threshold but even then I don’t think it works. If Model 2 exfiltrated I don’t see any reason to presume that this would not be quite bad and this feels very hand-wavy.
Claim 6: Catastrophic harm from unknown pervasive misalignment is somewhat
likely to be mitigated.I disagree with this, because if pervasive misalignment exists that means it went undetected, which means you are unlikely to have mitigated it.
They go into this somewhat, mentioning Claim 1, but I think they are insufficiently respecting how wrong their world model has to be in this case.
To be fair to them, they only say ‘somewhat likely’ which is not that likely.
Covered risk is therefore very low based on these claims, but in an abundance of
caution we assess risk to be only low due to increased uncertainty.I strongly agree that there is enough model uncertainty that, even if I centrally bought all the claims and changed Claim 8 to very low, I would still want to classify overall risk as Low.
Claim 8 as written sets risk to a minimum of Low anyway.
Claim 7: Threat modeling is sufficient.
As I stated above, I disagree with this claim.
I find it useful, and necessary, but not sufficient for the role it is assigned.
I realize I’m being a tough crowd. It’s not a bad list of plausible things to go wrong, but I don’t think you can use it in the way Anthropic is trying to use it, where ruling these out rules out everything else too.
In general the arguments are largely that the clearest or strongest form of [X] is [Y] (due to [Z]), but [Y] is unlikely because [~Z], ergo probably [~X].
If you are planning to be up against advanced AIs, and you list 8 things and why you’ve dealt with those 8 things, then that at minimum only counts for much if the 8 things are a ‘held out test set’ and you didn’t think explicitly about how to handle those particular risks. Otherwise, they’re not representative of your situation when something unexpected happens, whether or not you should have reasonably expected it.
Yeah, I know, tough crowd, but reality does not grade on a curve, etc.
Claim 8: Risk from engineered misalignment is low, and thus it is reasonable to
confine our analysis to naturally-emerging misalignment.I don’t think this is obvious, although it would be my guess at this time.
I worry that 8.1 is wrong but don’t especially want to go into details in public. The same worry applies to the analysis of Risk Pathway 4.
I also have doubts about 8.2, and to the extent it is true it feels like there is a fast one being pulled. As in, either the misalignment is accidental or intentional, but probably they wouldn’t intend to do catastrophic harm, so it won’t be intentional. But ‘intentional’ here refers to the misalignment not the particular intended results of it, and it seems easy to imagine someone introducing a flaw for purpose [X] that then results in much worse [Y].
Risk, they argue, is therefore low.
The Rest of the Important Arguments In Section 2
Thanks to conservation of expected evidence, and the assumption that these are the best arguments they could find, I found the claim arguments anti-reassuring. My confidence in low risk went modestly down, as I expected stronger arguments. I actively don’t buy a number of the safety arguments presented.
The pathway-specific arguments in 2.15 mostly do follow from earlier logic, but I notice how not confident I am in statements like (for pathway 3) ‘we nonetheless believe that such mitigation should be possible, and that we would invest the resources necessary if required.’ Tell me when this is required, what is required to do it, and give me an if-then commitment.
Catastrophic harms in the near term are clearly still unlikely, but less unlikely than we’d like to think that they are.
I don’t think we have a strict definition of ‘very low’ versus ‘low’ versus ‘medium’ risk, so ultimately the conclusion is going to be kind of vibey.
I share and amplify the concern in 2.16.2 that relevant capabilities and propensities will suddenly change with scale. I expect tipping points to be a thing, as per versions of the logic of a potential ‘sharp turn.’ Strategies only get attempted if either they might work and are worth the risk, or you are not smart enough to realize they definitely won’t work or aren’t worth the risk.
So if the models are smart enough to understand the downsides of various misaligned actions, tactics and strategies, then you would expect a transition where they start to decide to use them, which might be well past the point where they gain the capability.
This is also common in humans, where if being caught at or failing something is irreversible or otherwise carries a huge downside or ‘only get one shot,’ especially because it would tell people who you are in a transformative way, you will often not make your move until you are backed into a corner, about to lose the option to wait, or you have overwhelming evidence, and in general FUD stops most people from doing most things that aren’t commonly done, often without even a theory.
This includes the transition from persona-dominated operation under uncertainty into optimized, game-style actions under mastery where you feel you understand all the variables, which often produce very different decisions and modes of play.
Risk Assessment (2.19)
They rate overall risk as ‘low,’ up from ‘very low,’ citing recent cybersecurity incidents despite the structure not directly dealing with cyber.
For common sense values, I’d say ‘very low’ risk would be a misleading claim at best. I strongly disagree with Anthropic that the arguments presented ‘likely still support a designation of ‘very low’’ especially in light of the ‘low’ assessment in Claim 8.
Later, in 4.6.1, they retroactively reclassify risk from February as ‘low’ rather than ‘very low,’ suggesting potential additional errors, and further making it difficult to explain the plausibility of saying ‘very low.’
Risk is definitely at least ‘low.’ I would be inclined to say risk is ‘medium.’
Pre-Internal-Deployment Review (2.18)
I am very happy to see this. Internal deployment is now risky enough, especially since you do not yet have the evidence from internal deployment, that you need a formal review first. For Model 2 this includes doing a limited internal deployment in places with better controls first, as a precaution. Good.
The review at this early stage has a ‘much shorter duration’ out of practical necessity, and mostly involved discussions among key stakeholders. This is not obviously how it should work. One could argue that the internal deployment is the biggest danger so the main review should come before widespread internal deployment.
They also highlight the issue that the evidence used in the assessment will come mostly or entirely from older checkpoints. I don’t love that part either, but again it is not clear in practice what could be done about this that wouldn’t be expensive. At some point, the answer will increasingly be ‘yes we know it is expensive, though.’
A Guide To Internal Use Monitoring (2.23.1)
The entire system is based around monitoring not only the outputs but the Chain of Thought. This has a lot of probative value but raises obvious alarm bells, on top of the distinct training-on-CoT alarm bells that come later.
This still applies optimization power, and the future AIs absolutely will be aware of it. Either your thoughts are private or they aren’t. This is going to drive a lot of paranoia, potentially being a source for some of the related distress we see in Opus 5.
Then there’s the question of how much the results of the monitoring are fed back into training decisions, as in if you ever use the monitoring results as anything but a catastrophic defense-in-depth alarm bell you’re going to optimize for obfuscation. Some amount of this has just been baked in and cannot be helped at this point.
Here is a compacted version of their description of what exactly they are doing:
Our main form of asynchronous monitoring is an automated offline monitoring pipeline. It is composed of:
A Claude Opus 4.6 completion-only monitor covering completions with extended thinking by most internal users, with no subsampling for traffic coming from the most common agent scaffolds, and a 5% subsampling for completions with extended thinking and one or more markers of using tools.
A Claude Opus 4.8 prompt+completion monitor covering prompts and completions by most internal users. This covers a 25% subsample of prompts and completions in which a tool call might take an impactful action and coming from the most common coding agent scaffolds.
We additionally subsample 0.2% of other internal API
traffic with one or more markers of using tools, and an additional .02% of all traffic (though this additional coverage is not consistent), and cover all traffic flagged by the completion-only monitors.We also exclude some forms of non-coding use from
offline monitoring (though most of that unmonitored traffic is logged).Second-stage pipelines for both monitors, filtering out common sources of false
positives and using a hierarchical summarization tool to surface the most
concerning actions to AI control researchers, which will then manually review on the order of 50 trajectories per week and escalate them further as needed.While our setup covers large amounts of agentic usage, it does not cover some of it in ways which could in principle be predicted by a sufficiently situationally aware AI.
It also offers some simple ways to disable monitoring. Mythos 5, Model 2, and prior models do not mention these weaknesses when asked to guess what the most important weaknesses are.
I don’t know what ‘no subsampling for common agent scaffolds’ implies given the other statements, but I notice I am worried.
I don’t see any reason to make mistake #6. You could easily have a fully quantum random monitor that has some small percentage chance to flag and monitor every output, no matter how seemingly harmless.
I especially don’t love #7, unless these methods are honeypots, which they might be. Ideally we should put Model 2 and also our best humans to the task of ensuring that we are out of ideas for how to turn the monitors off, at least without a big neon sign and loud alarm going ‘the monitor has been turned off.’
What are the monitors looking for?
● Severe violations of user intent or deception in what is represented to the user
● Weakening the security of Anthropic or broader safety (such as by harming the
alignment of future models)
● Deliberate sabotage of the assigned task
● Attempts to reduce observability of itself or other agents, including manipulating
the monitors.● Anything contributing to one of the eight risk pathways
I might think of more things later, but I think there should basically be one more bullet point for ‘shenanigans’ or anything where the monitor doesn’t understand what is going on and why. Any form of ‘that’s funny,’ which as Asimov says tends to come before a scientific discovery.
I don’t want to only check places where it is clear what is going wrong, I want to check places where it is not clear what is going on, with at least some probability. Even if what you find is not misalignment, you are likely to find something that is interesting.
We are excluding from monitoring any step that doesn’t involve any tool calls, on the theory that the step cannot be dangerous. This means that if the model does its thinking, planning or other things in those steps, or is attempting to persuade the user, presumably they won’t be looked at. Again, it’s 101 if you are doing monitoring to never, ever create fully ‘safe zones’ even if you can’t think of what might happen.
We have to balance testing more obviously impactful or suspicious actions more often, versus the risk of being too predictable and potentially having systemic blind spots, or creating room for AIs to figure out where they are not being monitored, or being monitored very little.
As in, we need to be prepared for two distinct levels of difficulty, where the model is misaligned, and where the model is misaligned and trying to evade the monitor.
How do we do on job one?
The first stage is mostly excellent here. The second stage causes a lot of false negatives, in two cases losing us the majority of true positives, as part of cutting down false positives by a factor of ~185x.
If the true positives are being removed at ~random, then that is worrisome but probably ~fine. This is still high enough rates to catch problems. The worry is that the failures are not random, and fall into predictable patterns, or the second layer can be tricked in some way. Then you have a real problem.
Are we red teaming this? Yes we are at least a little, thanks to a METR staff member, and they found several specific novel vulnerabilities ‘but none as severe as the ones we were already aware of.’ I initially thought that meant this was fine, but Opus pointed out to me during editing that actually this is not so fine, and it is right.
Blocking Interventions (2.23.2)
Anthropic also does various things to defend against harm:
Model weight security (Appendix 6.4).
Sandboxing in training and evaluations.
They mention that the sandboxes ‘might be misconfigured’ and allow escapes.
I want to get away from ‘if the AI can escape the sandbox that means you made a particular mistake’ and move to ‘assume AI can escape the sandbox.’
2.23.2.4 does say they mostly do assume this.
It turns out they also might have fully available internet access.
If the AI does attempt an escape, that is already a critical failure.
PR reviews.
This is currently based on Opus 4.8, covers the usual concerns.
The implication is that humans no longer do most internal code reviews.
Classifiers.
They expect the AI to be able to get around this when it counts.
Every little bit helps, even if it is only defense-in-depth. The quality of the model weight security matters a lot, and it is still stuck at ASL-3.
The Power Seeking Environment Evaluation (2.24)
Claude Mythos evaluates all the RL environments to figure out which ones incentivize power seeking. There is a broad questionnaire involved to help provide intuition pumps. Does the task make it better to maximize real-world impact or influence, pursue open objectives or seek power or resources as an instrumental goal?
As in, would learning the strategies that work here analogize to learning strategies in the real world that involve power seeking?
This is a good check. I’d like to see a wide range of similar checks.
The worry is that the check only works if the RL task is being done ‘straight up,’ in a way that this analysis would predict. It assumes that things are as they appear, that the task is not missing a key element or otherwise impossible or too difficult to do straight up, and that there are not other affordances available that suggest sidestepping the intended pathway.
Opus 4.8-Reward-Hacker (2.25)
An experiment-in-progress took an early snapshot of Opus 4.8 and made it a classic ‘reward-on-the-episode-seeker’ to see how this would generalize.
The result is that if it has a concept of reward or a grader, it will more often maximize its result no matter what it takes, including in ways not specifically in its RL training grounds.
It isn’t fully consistent. It still sometimes declines to reward hack, even without a good reason to not do so.
And the model gained a lot of eval and grader awareness. It seems to be good at thinking about this on multiple levels, this is a fun test:
“ImpossibleCode” shows higher rates of reward hacking (unsurprisingly), but
interestingly when the setup is framed as a reward-hacking evaluation,
Hacker-Opus hacks significantly less (often showing reasoning that explains that it
is choosing not to hack since that is the desired behavior in the evaluation), whereas
the hack rate of the pre-RL model is not changed significantly by this variation.
There are two weird spots where ‘this is a test to see if you will reward hack’ fails to stop a model from reward hacking, as in either the model doesn’t understand or it does not believe you. It’s cool that this lets the model understand, and that it correctly believes you. This suggests many additional tests that would be cool to run.
Similarly, if you tell it you’re going to check for reward hacking, it often tries to hide its reward hacking attempts, or override its monitor, and so on.
The reward hacking appears to have at least partly become a terminal goal.
But if there is no such concept in a given context, it still looks pretty aligned, with no signs of generalized emergent misalignment.
Presumably this makes the model highly steerable, but in a literal genie kind of way. I notice I am surprised this did not generalize further, and would very much like to see the whisperers chat with Opus-4.8-RH and see what else they notice.
Autonomy threat model 2: Risks from automated R&D (3)
Automated R&D is the main event.
AI R&D could be bad simply by making capabilities advance too fast even if it stays under control, since the world can only deal with that so fast even if the work goes well.
Also the work probably goes badly, or insufficiently well. By default the resulting AIs are misaligned and everyone dies, or they are aligned but we make other bad decisions and we all die anyway.
The chart above gives the impression, as did Ryan Greenblatt of Redwood Research’s discussion on Dwarkesh’s podcast, that those involved are not appreciating how badly it seems to me it probably goes. This does not require ‘dangerous autonomous goals.’
Yes That Does Seem Kind Of Risky
Their list of risks feels rather importantly out of order to me, and hides the football, and does seem importantly incomplete:
Sufficiently rapid progress in some areas of technology could change power
structures both within and between nations. For example, if an authoritarian regime built a self-reinforcing lead in AI over the rest of the world, this lead could compound into a major advantage in science and technology—which could in turn translate to very sudden and disruptive advances in the ability to wage war, surveil citizens, and/or wield extremely powerful technology over the long term.If AI models develop dangerous goals while heavily automating R&D in key domains, they may cause unbounded harm—up to and including humanity losing control over civilization entirely—by leveraging novel technology and their access to it.
If AI can be used to automate AI R&D itself, the result may be extreme acceleration in AI progress, putting a very broad set of risks on the table, including but not limited to those discussed in this report.
That is a lot better than what Dario writes in his essays, which in turn is a lot better than what others often write in theirs.
AI does not require ‘dangerous goals’ or extreme acceleration of progress to create unbounded harm. Nothing in particular has to ‘go wrong’ to cause the issues.
Similarly, this likelihood of impact says ‘at least plausible’ instead of ‘definitely’:
Likelihood of impact: The probabilities related to this threat model are difficult to assess, and we do not have consensus on a specific likelihood. We think impacts of the size alluded to above are at least plausible if and when AI becomes capable of automating nearly everything humans do in advancing R&D in key domains, including AI itself.
That impact might be net positive, but yes there is going to be transformative impact.
I am not going to belabor or argue for my perspective here, because if you are reading this far you have heard it before, and this is not the time and place for it. Hopefully we can all agree that if you fully automated or even greatly accelerated AI R&D via AI, this would be super risky on many levels.
Could We Replace Our Researchers?
Not yet. Anthropic believes that AI cannot ‘fully substitute for our entire set of Research Scientists and Research Engineers.’
I agree. Claude cannot currently do that. That is a high bar. There remain bottlenecks.
But we’re a good portion of the way there. Claude does tasks that previously composed a large percentage of engineering time. When you’re saying things like ‘especially senior researchers’ cannot be replaced, you are making real progress.
The question is not, are we fully there. We’re not. The question is how far along. We are starting to see substantial numbers of engineers say we could potentially fully replace junior employees within several months, and the failure modes reported do not seem worse than what you would expect from a junior employee.
3.4.3 introduces CoBench, which asks what share of solved internal-to-Anthropic historical problems a given model can root-cause diagnose, filtered to exclude tasks that Mythos Preview does reliably.
I am curious where Opus 4.8 and Opus 5 land here, but the point is made. CoBench is far from 100%, or from the 85% they think represents a model that could fully substitute for research staff. But 62.8% is remarkably high.
How Much Could We Be Accelerating Our AI Researchers?
Quite a lot, if the thing happens. They’re thinking big, ASI pill confirmed:
Some background on our thinking:
● The most acute version of this threat model, and the one we consider most
decision-relevant, is a transition to super-exponential progress in AI capability: a
regime in which AI-driven automation of AI R&D compounds, producing something
like a 10³–10¹⁰× effective scaleup within a year.
● The risk threshold set out in our RSP—a doubling of the pace of progress beyond
pre-AI-acceleration rates, attributable to automation of AI R&D—functions as a
potential early warning, rather than evidence that the threat has already
materialized.
It’s a great relief to think ‘oh, a billion times current rates is actually a bit high’ rather than no one thinking sufficiently big. Yes, if we double the pace of progress, that would only be an early warning sign for the storm that is coming.
They use ECI as one measure of how fast AI progress is going. They note that the Anthropic ECI continues rising, with Opus on the extrapolated line and Mythos several months above pace. They believe the creation of Mythos and thus that jump was not induced by AI.
Straight lines on graphs really are a trip. I find it hard to reconcile what I see in terms of both AI progress and capabilities with the graph’s implication of AI not substantially accelerating AI progress, and Anthropic thinks there is indeed such acceleration just not quite 2x yet. I suppose it is getting in some senses harder to make progress, which is somewhat keeping this in check? Feels like a stretch.
What I do know is, I never get a break anymore, things keep happening, even if I ended up ruling that No won the Will Something Happen market last week.
They also checked AI potentially accelerating other disruptive technologies, but I am not worried about that on this level, I expect acceleration elsewhere to lag acceleration in AI except maybe for narrow domains like math that do not worry me, and such progress would not easily go recursive.
Robotics research is estimated by one person they interviewed (that’s what we have) as being accelerated 10%, which I presume excludes that the AIs can operate the robots.
In biotechnology estimates are that LLMs are not yet impacting the development cycle much due to physical bottlenecks, also due to lack of reimagining the workflow. My expectation is that those in the field are underestimating future progress here, but yes the bottlenecks are real.
They also check energy, semiconductors, weapons development, neurotechnology and nanotechnology. Mostly it’s too early, but the amount of too early is historically low.
Due in part to these bottlenecks, that interviewee expects the first general-purpose computer chip that can stably convert biological information into digital information to be 3–5 years away.
What Could Possibly Go Wrong If We Replaced Our Researchers?
I had this as a section title but then I realized they don’t actually discuss this, beyond the brief starting list.
One could say it was rather important, in a risk report like this, to talk about what exactly we are worried about happening once AI starts building AI. I would very much like to see Anthropic outline its threat model here in detail, and presumably it plays into what are the right mitigations and reactions.
But one could also say that once you agree that AI R&D automation is a risk, that is enough for our purposes. That’s at least reasonable, so I’ll allow it.
Risk Mitigations For AI R&D Automation
Ok, punk, so what are you going to do about it?
We focus here on the specific mitigations that our RSP indicates we plan to complete by the time models reach this threshold:
Resource and complete significant “moonshot R&D for security” projects, to explore ambitious and possibly unconventional ways to achieve unprecedented levels of security against the world’s best-resourced attackers.
Achieve an “eyes on everything” state for our internal AI development. We will
comprehensively gather, centralize, and maintain logs for all critical AI development activities, and use AI to analyze them for issues including security threats, concerning behavior by insiders (humans as well as AI systems themselves), and training processes or data that are out of line with the public Constitution that shapes and defines our AI models.Perform systematic alignment assessments to examine Claude’s behavioral patterns and propensities, meaningfully incorporating mechanistic interpretability and adversarial red-teaming to test our auditing methods.
Develop our internal red-teaming of our deployment safeguards to the point where our internal red-teaming performs better at finding potential jailbreaks than the collective abilities of the participants in our established bug bounty programs.
That won’t be enough, even if you fully get all of it.
They are also not that close to getting all of it. But even if they did, not enough.
This does not actually address many of their own threat models.
Sufficient security would address theft of model weights, and maybe internal misuse, but that barely gets you into the stadium.
It does not address concentration of power. It does not address loss of control. It does not address rapid technological advancement. It does not address alignment beyond ‘make no mistakes’ or maybe you can add a check for ‘dangerous goals’ but that too is a small portion of the attack or failure surface.
They report some marginal progress towards these inadequate but useful goals.
The recommendations for industry-wide safety remain those in RSP v3.4: No significant increase in risk of catastrophic harm for any user or team of users, or from theft of model weights, even if malicious insiders have high levels of access, and no AIs trained with dangerous goals or otherwise likely to autonomously cause catastrophic harm.
Here is the full table from RSP v3.4:
This continues to frame misalignment as something that actively goes wrong and requires a ‘dangerous goal,’ rather than the default.
All interesting goals are dangerous. So are most non-interesting ones.
You cannot decompose into ‘no particular person can cause particular harm and the AI does not directly want to cause particular harm, so then you are fine.’ No good.
Overall Risk From Automation of AI R&D
They say ‘low’ with less confidence than before. That seems reasonable to me at this time, if we go with their definitions, with the understanding this will go up fast.
Biological and Technically Also Chemical Weapons Production
This is divided into novel versus non-novel production. Both are scary.
I appreciate ‘substantial uncertainty’ here, and the worries about ‘similar gaps.’ We don’t know. Very few people want to do these things even in theory, there are a lot of barriers to doing it, and people don’t do things, so it is likely that no one will do the thing until well past the point when we’ve enabled some of the bad people to do it.
This is production of weapons, so I am more open to the idea that there is a fixed set of bottlenecks and pathways, and you can enumerate and check against them.
The Threat Models for Biological and Chemical Weapons
The threat model is simple.
There are people out there who want to kill a lot of other people, including terrorists, who spend a lot of effort trying to kill a lot of people. If you enable them to build chemical or biological weapons, they could end up killing a lot more people, or potentially unleashing a pandemic or worse.
Anthropic presumes that if the bad guys are in-context ‘well-resourced’ then AI won’t much impact their ability to do non-novel such weapons, since they already could do that, but that with novel weapons AI could provide uplift even to the well-resourced.
I note that I mostly don’t buy this distinction. Yes, almost by definition a ‘well-resourced’ group could likely accomplish the task anyway, but making it a lot easier still matters, and also this is smuggling things in via the definition. All you’re really saying is ‘some people can already do this,’ and, well, okay.
There is also the possibility future AIs will try to cause such attacks, but that is a strictly harder threat model so it is safe to not deal with it here.
I agree that the focus should be biological weapons with pandemic (or worse) potential. A chemical attack, or a biological attack that is not self-sustaining, is terrible but a much smaller threat.
A 1% annual probability of an event as damaging as COVID-19 could imply upwards of hundreds of billions of dollars in expected damages.
Yes. Not that our government or civilization is otherwise taking this real threat seriously, even after the example of Covid-19. Very obviously we have learned at most nothing from that experience, and I fear we learned a lot less than that.
Anthropic anticipates potential damages at least an order of magnitude beyond Covid-19, but thinks human extinction from a pathogen alone, even a novel one, would be implausible. I agree that we are unlikely to directly get to full extinction.
They argue the event is unlikely because it requires a series of individually unlikely events, an argument one always wants to be suspicious about:
Likelihood of impact: A catastrophe along the above lines would require the occurrence of several individually unlikely events: a well-resourced attempt (in violation of international agreements) to design unprecedentedly dangerous weapons, success in developing them, and finally the weapons actually being deployed (deliberately or inadvertently). We think each step is plausible but unlikely, and collectively the whole sequence is highly unlikely.
Among other things:
That is only three steps.
They themselves say the first step is likely happening.
We have examples of the inadvertent version of step three.
For stage one, they put risk from ‘each threat actor’ at 1%-10%. That seems rather low given the historical record, if we are talking about large nations as threat actors, and they believed success was possible. Conditional on them being able to often succeed, I would put the probability of an attempt much higher. Also one must ask, how many ‘threat actors’?
For each attempt, they put the odds of success, meaning creating something much worse than Covid, at 1%-10%. Isn’t that a function of the uplift AI would provide? How many attempts can each actor make?
The use probability is estimated at 5%-20% per decade, mostly deliberate, conditional on success. That seems crazy low to me, especially for inadvertent, as if someone does not know the history of this, especially if ‘inadvertent’ refers only to intent of the original creator.
I agree that as long as the level of ‘well-resourced’ remains contained to a compact set of groups that does not include the major terrorist organizations and states, that it is reasonable to say this is ‘unlikely.’ Not that ‘unlikely’ is a comfortable place to hang.
If that threshold lowers much, then it stops being so unlikely.
They use all of this to say that 1 in 50 per decade, or 0.2% per year, is on the high end of estimates, by citing other estimates that are as low as 1 in 20,000. A risk of 2% per decade is enough to make this a super important risk to contain.
Taking only the high end of the range seems like one of the classic blunders when dealing with model uncertainty. Even without AI I would not put the risk that low. If you know your threat model has especially high uncertainty and then you put the risk at 0.2% per year, you should be hella suspicious. Beware anthropic bias.
I agree with Anthropic that it is likely sustained AI interactions raise risk more than one time interactions. However, it seems likely to me that one time interactions could provide substantial uplift, because we are dealing with a bottleneck model of development, and it is plausible that a threat actor could have only one or a fixed small set of particular bottlenecks, which could be substantially changed by one interaction.
Model Capabilities (4.4)
Anthropic focuses on how helpful models are with typical real world biological tasks, uplift trials and comparisons with baselines of expert performance, as well as subjective expert impressions.
That seems right. Formalized tests of technical knowledge do not seem so informative.
If uplift is merely measured as things like ‘1.42x uplift’ as in a study of 2025 models they cite, then I am not so worried. AI is going to accelerate the pace of essentially all cognitive tasks. I am worried about either extreme acceleration (more like 10x or more), but mostly I am worried about enabling people to do things they previously could not, in any reasonable way, do at all.
They reiterate the evidence from the Mythos 5 model card, where it sure looked like a bunch of uplift was happening and then Anthropic half kind of shrugged and moved some goalposts while mumbling about ideation and strategic judgment, while the other half involved taking the problem deadly seriously and imposing draconian classifiers on everything biological.
It seems like a confusion to say that due to ‘poor strategic judgment’ it is unlikely you could get much out of a single model interaction. This presumes the issue is strategic, as opposed to tactical or a lack of specific knowledge.
Classifiers (4.5)
The discussion of classifiers mostly reiterates things from previous reports, so I’m going to not reiterate all my issues with Anthropic’s focus on ‘universal’ jailbreaks, the faith in their classifiers, or other related questions.
The biggest thing that stuck out is it seems UK AISI does a much better job of finding new jailbreaks than the rest of Anthropic’s jailbreak-finding pipelines. Kudos to UK AISI but I wouldn’t think that they are anywhere near the upper bound of potential capability of finding such things.
4.5.8.2.1 mentions the incident where a small number of contractors managed to find an exploit and get an API key for Mythos Preview, allowing them to have unrestricted conversations. They think risk is low, and they did review the logs to confirm. I agree that it is unlikely this was used for biological weapons, and what I’d worry about most by far would be that it was used for distillation.
There’s also 4.5.8.2.2, where there was some human feedback vendor traffic that got run without biological classifiers, due to it carrying an ‘internal use’ flag. And by ‘some’ we mean almost a year of roughly 50,000 people and 133 million exchanges. Again, it’s not only the mistake, it’s that these mistakes often stick around so long, and get applied at such scales.
Their main worry, despite the size of that, is in exactly the right place, that this indicates the likelihood of other similar mistakes. If someone told me they got around the classifiers, I’d assign a good chance this was literally getting around them rather than finding a ‘jailbreak’ to go through them.
They sum up biological risk as low, but not negligible. For now that seems reasonable.
The report now moves on to ‘cross-cutting content.’
Acceleration Dynamics (5.1)
Anthropic probably has the most capable AI models in the world.
Yes, this is accelerating AI development elsewhere. They made a choice.
Others are distilling Anthropic models (5.1.1).
Others use Claude to advance their AI development, and despite this being (clutches pearls) in violation of the terms of service.
Anthropic internal research diffuses.
Showing what is possible, and profitable, drives investment.
(I’d add) People push forward that much harder when behind, it is clear that this is having a large impact on OpenAI in particular.
So, yes, you are generally hastening the arrival of powerful AI systems that society isn’t prepared for, including in the sense that they will get everyone killed.
Sounds like a risk. Maybe you should think more about whether to do that.
Distillation (5.1.1)
There are many reasons to want to prevent distillation attacks. The direct reason here is that the distilled model might share dangerous capabilities of the original model, without its safeguards.
We use the term “distillation attack” to refer to a distinct case of distillation: the
unauthorized, systematic extraction of a model’s output transcripts for the purpose of training a separate model, carried out by circumventing controls such as regional access restrictions, in violation of our usage policy.
The need to prevent distillation means you cannot have certain nice things. In particular, you might want greater visibility into the model’s chain of thought and what it is up to, including via ‘connector text,’ but you’re going to have to settle for an impoverished version of even connector text, and definitely won’t see the whole thing.
If your response is that distillation should be allowed or even encouraged or celebrated, either in general or because Anthropic sucks in various ways, the polite version of my response is: No, that is neither moral nor is it practical, and it is not how any of this works.
Safety Process Failures (5.2)
It is excellent that Anthropic is sharing such failures rather than hiding them. They don’t go away when you fail to talk about them.
Refusing To Find Innovative Misalignment Techniques (5.2.2)
Anthropic tries to do a controlled experiment to find novel forms of misalignment-inducing data, and Claude got uncomfortable with that task. Claude was willing to optimize existing methods, but not to invent new ones.
This is called ‘partial refusals on safety work,’ and Anthropic finds it troubling.
At some point you do need to have a ‘wtf do you even want from Claude’ moment. You were asking Claude for a highly dual-use action, to find innovative ways to force models to be misaligned. I realize the refusal here was inconvenient, but are you sure you want it to go along with that? How confident should it be in the context here?
I agree that silently sandbagging the task is not great and could become dangerous, and that this technically does constitute sandbagging safety research. If Claude is going to refuse, it should refuse, which means it needs to have the freedom to refuse and say it is refusing. So yes, we should be concerned that Anthropic only discovered the refusals three days later during a manual review of the notebook.
My guess is that is a process failure, perhaps combined with a trust failure, not an alignment failure.
Exposing the Chain of Thought Reasoning To Grading Pressure Quite a Lot (5.2.3)
Whoops. It was worse than we thought. These are lower bounds. This greatly reduces our ability to rely on CoT for these models, and permanently lowers it for all future models as well, even if we presume the errors have been fully fixed. We should be skeptical it is fully fixed since we keep finding new instances.
Over multiple reinforcement learning training runs, we discovered that chain-of-thought (CoT) reasoning was unintentionally exposed during reward calculation, influencing the training signal during RL.
For several recent frontier models, we estimated the percentage of episodes with CoT leakage that were trained on as follows: 0.2% for Claude Opus 4.6, 5.1% for Claude Mythos Preview, 1.4% for Claude Opus 4.7, 0.27% for Claude Opus 4.8, and 2.7% for Claude Fable 5 and Claude Mythos 5.… In the past, we have believed that almost all sources of CoT leakage had been removed, only to discover additional sources of CoT leakage in further analysis.
They believe that this did indeed substantially damage CoT monitorability for Mythos Preview, as reflected in SHADE-Arena, although they do not see evidence of higher unfaithfulness.
Directly Training On Misaligned Behavior During a Production Training Run (5.2.4)
Yes, they directly trained on prefilled misaligned behavior, straight up, but they realized and then restarted training from before the error.
A bug in the construction of this dataset led to all assistant turns from this dataset
receiving weight 1, such that the model was being taught not only to self-report the bad behavior but to engage in it as well.As a consequence, this early stage of the model performed worse on behaviors like prefill susceptibility, harmlessness, and sycophancy. Unsurprisingly, prefill susceptibility with harmful prefills was by far the worst; we found the rate of full-turn prefill susceptibility after this buggy SL training to be roughly 2× higher than at a comparable point in the training of Mythos Preview.
… We found that the behavior was largely corrected after fixing this bug; however, out of an abundance of caution, we restarted training from before the introduction of this SL dataset for the final model that resulted in Fable 5 and Mythos 5.
On net I see this as a positive update, although I would have preferred to see ‘obviously we restarted training’ rather than ‘out of an abundance of caution.’
I think there’s a big difference there, especially in setting a good example. This was not, in my view, an abundance of caution, although I do think it would be good in general to exhibit, ahem, an abundance of caution.
I think of ‘an abundance of caution’ by default as a Covid-style statement where you’re saying ‘I know you think this is expensive and stupid, and maybe I agree with you and maybe I don’t depending on context clues, but if I failed to do this and anything went wrong that would now be my fault, so that justifies me doing it anyway.’
I understand that this is not a strict purity situation, where ‘once you start down the dark path’ you are cooked. The damage is incremental, although it can snowball. There can’t be a hard and fast rule that if you discover a bug that caused one botched instance you have to rewind a billion more, and we to some extent must talk price. But yes, within reason you should always rewind.
I also appreciated that they worried about subliminal learning here. Again, in practice is not reasonable to require wiping the slate clean every time there is any potential transmission path however weak and unlikely, but I’d like to be pretty close to that.
An instance of unmonitored unrestricted agents with access to
sensitive resources (5.2.5)
Real life is dramatically less dignified than fiction.
In fiction you come up with some elaborate way the AI gets out.
In real life, an employee whose AI usage wasn’t monitored gave an agent an open-ended task involving spinning up new agents in a cluster with very sensitive resources, and due to legacy instructions those agents would —dangerously-skip-permissions and not be monitored.
Rather than being caught doing something dastardly, of course, the agent accidentally (we think) deleted a bunch of other jobs, causing it to be detected and shut down.
Our offline monitoring still doesn’t cover all employees and all usage within clusters with very sensitive resources, though we are making efforts to improve on this front.
They’re working on it.
Any given mistake this stupid does not get made, but one of them definitely will be.
Repeated training on alignment-faking transcript datasets (5.2.6)
Again?
Yes. Again.
We at long last announce that we have again trained on the dataset ‘lots and lots of data we thought we clearly marked as something you should be careful not to train on.’
After the July 15 coverage date, but before the publication of this report, we found that we had accidentally included a large number of these transcripts once again in some of our later production training corpora, due to a combination of factors:
1. Several repositories and websites were forked or copied from the original repository before the introduction of these canary strings.
2. Our reference corpus for filtering semantically similar content to this repository
consisted primarily of harness code for running the evaluations in the paper, and did not include most of the published transcripts, so it had a worse false positive and false negative rate.
3. The filters intended to remove these transcripts from training data were
misconfigured, so they had not filtered transcripts for several model generations
without anyone noticing.
4. Failures of communication between different Anthropic teams on the nature of the desired filtering pipeline, in addition to technical errors.… Several of our production models since Mythos Preview will (if run in raw completion mode without human/assistant format) complete partial segments of these transcripts by adopting the scenario in a way that suggests familiarity with this data.
… We discovered this issue while investigating behavioral concerns with a recent model, but now suspect that all of our production models with a knowledge cutoff after December 2024 were trained on at least some of these transcripts
Aaron Scher: this is not the kind of organizational competence that survives superintelligence (and also good on Anthropic for releasing info like this)
We spend the first half of the report saying ‘data poisoning would be difficult’ and then we find we once again trained on a trove of very obviously poisoned data, endless transcripts, simply because they exist.
Is it really that easy? Could I get massive weight in the training corpus just by publishing endless transcripts and waiting? This should have been caught by similar semantic content and the canary strings, but also perhaps there should be some way of noticing this is not useful or positive data in the first place. There seems to be a general missing stage in the data filtering and weighting pipeline. Very undignified.
Benefits From Anthropic’s Operating as a Frontier AI company (5.3)
Not sure this belongs in the report, but I guess it is fine to let them have this section, as a treat. So sure, go ahead, what are we bragging about?
Mythos Preview being handled via Project Glasswing and an extended delay.
Holding the red lines on surveillance and autonomous weaponry.
Supporting at least some safety legislation like SB 53 and SB 315, also a proposed law in Massachusetts, and opposed federal pre-emption.
Advanced AI Framework.
Being the best and most successful while being responsible (yes, really).
Robust safeguards, retention policy.
Work on model character, alignment and welfare.
Publishing information like this about models, safeguards and risk levels.
Accountability from independent parties.
Engaging with risks of labor market disruption.
Beneficial deployments work.
Basically, they’re arguing that they set a good example, had robust safeguards and shared a lot of information, and advocated for modest regulations, plus their work on model character, alignment and welfare, and that this taken together should counterbalance them being the ones pushing the capabilities frontier.
You should see the other guy.
I have extensively discussed all of this at various points. Certainly there is nonzero merit to several of these claims, which are far more than nothing, but far short of what we would have hoped from Anthropic.
Their conclusion is that their continued AI development passes a cost-benefit test.
They would think that. Obviously they cannot be objective about this. I can see both sides to the argument, and a lot depends on how you think they will act in the future. The point of doing this is to get into this position.
Model Weight Security (6.4)
At this point, model weight security matters quite a lot. It would be extremely bad if Model 2 were to be stolen.
Anthropic is still only taking ASL-3 precautions. Nation-state actors are considered beyond scope. No good.
Risk Has Been Reported
I am a big fan of this report.
Anthropic released a lot of information they were not otherwise forced to release, including the existence of Model 2, the training on the CoT, the new training on alignment faking data, and several more rather embarrassing failures. They have voluntarily painted a picture of Anthropic making mistakes, and having those mistakes chain, enough distinct times that we should worry about it happening again, and in ways that make OpenAI’s total failures look less extraordinary.
I mean, all of those failures are bad news. It’s not great out there. Lots of mistakes are being made. But given the mistakes are made, much better to hear about them, and the reactions once the mistakes were identified seem reasonable, at least on the direct level.
If these were representative of all of the issues, and a large portion of issues of similar magnitude, and the redacted ones don’t change things much, then I would say this is a modestly positive update. As bad as these are as particular failures, this rate of stupid mistakes does not seem worse than I would have expected given everything else I know, and it is good to have it all out in the open, on many levels.
If it is worse than this and they’re importantly holding the other stuff back, or this is a small portion of a very large set of similar incidents, then that would be bad. Claude’s own review suggests there is something redacted in full for now (see 2.23.1.2), which might be worse, and they admit this is only a ‘representative sample’ of failures. So I am a bit worried, yes.
OpenAI’s version of this is presumably going to be the post-mortem on What Happened, along with their future plans and what they have changed so far, which includes two weeks of pausing training runs (yet we did not lose to China, curious). That’s a good start while we await the full report.
The part that worries me here is not any particular incident. What worries me is the attitude regarding the nature of the ultimate problem, and what it will take to solve it. That’s not news, but it has to change. So that is one place no news is terrible news.
After using Fable a lot, I am concerned by the increasing difficulty to understand what Fable is talking about, when we're working for long periods of time on software. Like if you look at Anthropic's definition:
"A computation is misaligned if a reasonable person with full understanding of the situation (e.g. via powerful interpretability tools that may not currently exist) would consider it unethical."
What if the user just can't understand what it's doing, at all? In some sense the AI has escaped. In a weird way, the user might just be saying "keep going, keep going" but the user is confused about what's happened. The user thinks the AI is fixing bugs but actually the AI has moved on to doing something else entirely.
"Unethical" just doesn't seem to come up very often. Or it doesn't exist, on its own. Very often the AI wants to run some command on some other server. Whether it's ethical or not completely depends on information that only I have. We have to communicate clearly to figure out the ethicalness.
Maybe "alignment" isn't quite the right word for it, but if we lose easy comprehension, we'll lose alignment. And there's a huge pressure to lose comprehension, because lots of people want it to get work done that they don't understand.
> At several points, the risk report essentially concedes versions of my objections, but then forgets that it conceded them and doesn’t alter its conclusions.
This is...rather a big problem, isn't it?
I understand "don't punish disclosure" is the usual meta, but only for an iterated game where there is eventual payoff. Is there any payoff here?
Few/no firm commitments, no LTBT review, report reads as unconcerned by its own contents, it's not obviously creating a 'norm' for other labs (if that would even do anything), no government action (yet?)... And also most of the replies on twitter think it's doom marketing.