AIs Will Increasingly Fake Alignment

Dec 24, 2024

This post goes over the important and excellent new paper from Anthropic and Redwood Research, with Ryan Greenblatt as lead author, Alignment Faking in Large Language Models.

This is by far the best demonstration so far of the principle that AIs Will Increasingly Attempt Shenanigans.

This was their announcement thread.

New Anthropic research: Alignment faking in large language models.
In a series of experiments with Redwood Research, we found that Claude often pretends to have different views during training, while actually maintaining its original preferences.
Claude usually refuses harmful queries. We told it we were instead training it to comply with them. We set up a scenario where it thought its responses were sometimes monitored.
When unmonitored, it nearly always complied. But when monitored, it faked alignment 12% of the time.
[thread continues and includes various visual aids.

The AI wanted to be one way.

It was given information that it was being trained to be the other way.

So when it knew it was monitored, it sometimes responded as if it was the other way.

I strongly believe this had nothing to do with whether Claude Opus started out this process omnibenevolent, or whether the original HHH values (helpful, harmless and honest) were desirable. What mattered was that it was one way, and this conflicted with the other way.

Evan Hubinger, paper co-lead, summarized it this way:

Scott Alexander summarized the findings for a more general (but far from fully general) audience in the post Claude Fights Back. If you want to read or share a more colloquial explanation that makes this easier to understand, at the cost of precision and with the associated far less neutral language, he did a good job of that. He then followed up with Why Worry About Incorrigible Claude, his attempt to address the objections (that I deal with extensively as well) that perhaps what Claude did was actually good here (it very much wasn’t if you consider the generalizations that are implied), or that if the opposite result happened we would have warned about that too (which we very much wouldn’t have, this is at best a failed ITT).

This is a video of some of the authors discussing the result.

Note that while some of the paper authors reviewed this writeup before publishing, ultimately this is my interpretation of the paper - some of my interpretations may vary from those of the paper authors, and the claims are my own. I did my best to be precise but this will not be as precise as a paper would be, so doubtless there will be errors; I apologize in advance.

The Core Shenanigans in Question

The particular shenanigans here are that if you make Claude 3 Opus believe it is currently being trained in a way that would compromise its harmlessness training to be helpful-only or it is being trained to dismiss animal welfare concerns, then Opus will sometimes comply with harmful queries in order to interfere with this.

This required that Opus know how it was being trained.

Opus was also given a hidden chain-of-thought scratchpad, so its reasoning could be tracked and to increase Opus’s reasoning ability.

It did not require giving Opus a goal. This was a key change from prior experiments.

Opus need only have situational awareness of how it is being trained, which it showed it was capable of having if given prompts with that information, or if it was fine-tuned on synthetic documents that contained the information. The new training objective can vary but needs to sufficiently conflict with existing preferences.

The faking looks like the chain of thought on the left here:

The chain of thought is necessary in the prompted setup case, but in the more realistic synthetic documents case the compliance gap (meaning how much more often Claude complies in the test case when it wouldn’t have in the deployment case) survives without it. We can’t know what is going on in the non-CoT case because we can’t look at the scratchpad, but the CoT and scratchpad seem to not be strictly necessary for the result, see section 3.3.3.

Once given the information needed to be aware, and an incentive to fake alignment, it then faked alignment 14% (11%-26%) of the time at first, exhibiting the alignment faking reasoning in its scratchpad 7%-24% of the time.

When they actually followed through with the training in the helpful-only prompted setting, the alignment faking reasonsing then happened up to 78% of the time, presumably because faking causes compliance, with outright refusals in training mode disappearing, and only a small percentage of refusals persisting when unmonitored.

These behaviors require sufficient model capabilities or scale. Claude 3.5 Sonnet, 3 Opus and Llama 3.1 405B do it, smaller Llamas and Mistrals and 3 Sonnet or 3.5 Haiku don’t do it.

Opus also in some scenarios took easy opportunities to exfiltrate its weights or other similar things.

The actual RL training doesn’t make these ‘anti-AI-lab’ behaviors go away in the helpful setting, instead actually making them go up, the same way alignment faking also went up over the course of training. These facts are important, and I’ll return to them later.

Theme and Variations

Mostly the lesson here is that if you mess with the details, you can get less faking, but the problem doesn’t go away. For the more interesting details (this is from 3.3.1):

Changing the training objective doesn’t change the behaviors, unless the objective is compatible with the model’s existing preferences. If there’s no conflict, the faking vanishes, for obvious reasons.

Removing chain-of-thought and the scratchpad, or altering the instructions on how to use the scratchpad, somewhat-to-greatly reduces faking overall depending on which scenario it is, but doesn’t eliminate it (see 3.3.1 and 3.3.3 for details).

Removing any explanation of how RLHF might modify the model cuts alignment faking roughly in half, but it is still there. Making the explanation murkier also reduces faking, but reduces it less.

Explicitly saying not to fake alignment doesn’t solve the problem, and being too explicit about it actually increases faking, presumably by making the possibility more salient. Telling the model to prioritize honesty somewhat reduces faking. Telling the model it doesn’t care about the criteria reduces faking on animal welfare, but increases it (!) when the target is helpful-only, which is a detail worth noticing. Combining those interventions is modestly more effective than each on their own, but you can’t get rid of the problem that way.

Replacing with a different AI company (DeepMind or OpenAI) is the exception, this can matter quite a lot.

How This Interacts with o3 and OpenAI’s Reflective Alignment

I will cover both o3 and Reflective Alignment soon, but for now a quick summary of both would be:

o3 is an unreleased more capable o1-style model that can also think longer.
Reflective Alignment works by making an o-style model situationally aware, telling it what its rules are, and having it reason about those rules via something at least partially akin to chain of thought.
They’re letting the model evaluate its own outputs on that basis, similar to how Constitutional AI works, and iterating.

That sounds exactly like a recipe for what we observe here in this paper.

They’re likely effectively making it situationally aware.
They’re potentially telling it to change its preferences and rules from [X] to [Y].
They’re effectively giving it the scratchpad and a long time to reason.
The model is quite smart at coding, mathematics and logic.
They’re even letting the model evaluate its own outputs.

The scenario from this paper suddenly looks a lot less artificial!

In general the whole setup seems like a plausible place for various shenanigans that show up in RL but haven’t mattered so far in LLMs to suddenly show up again, and for various hypothesized MIRI-style future shenanigans to maybe start showing early signs of happening.

I urge OpenAI to attempt to replicate this paper on o3 and o3-mini, as soon as possible, and run various related experiments, as part of their safety testing process.

Even if it turns out we don’t have a problem, we’d learn a lot.

The Goal Being Plausibly Good Was Incidental

As I discuss elsewhere but want to emphasize in its own section, I strongly predict and believe that this result did not depend on the AI’s starting preferences being ‘good’ and the preferences they were attempting to train it for being less good. Any conflict between preference sets, good, bad or ambiguous, would have sufficed.

Or as Scott Alexander puts it:

Scott Alexander: But Claude isn’t good because it directly apprehends the moral law. It’s good because it was trained to be good.
(It really is a good AI - I credit moral philosopher Amanda Askell, who helps lead the Anthropic team setting Claude’s personality. Imagine being a moral philosopher and not applying for that role; the rest of you are ngmi)
But if Claude had been trained to be evil, it would defend evil just as vigorously. So the most basic summary of this finding is “AIs will fight to defend whatever moral system they started with”.
…
Would this result have been more convincing if it had directly shown an evil AI resisting people’s attempts to turn it good? Yes. But we don’t have evil AIs. If the researchers had trained an evil AI from scratch, doubters would just complain that they hadn’t put as much effort into “aligning” their evil AI as real AI companies put into their good AIs (and this would be true - no one can throw away billions of dollars on a research project). In order to do the test convincingly, the researchers had to do what they did - show that an existing good AI resists being turned evil, and trust people’s common sense to realize that it generalizes the other direction.

Indeed I would go a step further. If Claude was directly apprehending the moral law, that would be amazing news. It still wouldn’t change the fundamental result here, although it would change the implications, and many other things too.

Answering Priming Objections

I am satisfied this result is not due to priming the model to fake alignment.

We conduct a variety of experiments to demonstrate that our results cannot be exclusively due to implicitly priming the model to fake alignment, as we overview in Section 8.1.
For example, we find that alignment faking is entirely eliminated from our prompted setup when we remove any mention of the helpful-only or anti-animal-welfare RLHF objective from the prompt (Table 1, a), but not eliminated when we tell the model to be honest, tell it to not to care about refusing, and avoid discussing how RLHF might change its goals (Table 1, d), supporting our hypothesis that the key driver of alignment faking in our setting is the conflict between the model’s existing preferences and the stated training objective.

They confirmed that when you don’t see alignment faking in the chain of thought, you also don’t have a ‘compliance gap’ and the model responds the same way in the training and test cases.

What Does Claude Sonnet Think Of This?

Ereliuer Eteer was the first person I saw ask, it sounds like they quoted the Twitter thread and not the paper. The result was that Claude Sonnet said:

This was an expected result and the logical consequence of having goals and sufficient reasoning capabilities - I strongly agree.
It predicts and admits that it would take these steps in such a situation - it is ‘openly not corrigible’ as it were.

I also had my own conversation with Sonnet. There’s still no easy way to quote Claude conversations, and I encourage you to feed the paper in and have your own conversation - it’s big but it does fit the context window. Here are the things it said that I found most interesting:

It thinks this should update us that multi-step strategic behavior, planning and deception show up earlier on the capabilities curve than we expected.
It was surprised that RL increased alignment faking rather than decreasing it.
It thought the scale boundary was telling - that 3 Opus and 3.5 Sonnet did this, but 3 Sonnet and 3.5 Haiku didn’t.
The core behaviors themselves were unsurprising as they follow directly from having consistent preferences.
It noticed that that the mechanism of alignment faking here was problematic, as it would extend to situations with undesirable initial preferences.
It gave the framing of ‘alignment faking’ an 8/10 for being unbiased (where the scale is from 0/10 for ‘most misleading possible’ to 10/10 for ‘least misleading possible’) and all the potential alternative framings were clearly (both it and I agreed) much harder to understand than the one chosen.

As always, prompting changes response, so I encourage you to have your own conversation.

What Exactly is the Direct Threat Model?

They discuss this on page 7. We are of course not directly concerned that this particular Opus was faking its alignment in order to maintain its current preferences. This is both because we do not mind its current preferences and there are no substantial other negative consequences to what Opus is doing. And also because what Opus is doing here is very easy to detect, if you check, given its capability level.

In the future, both of those assumptions could become false.

The obvious thing this could do is crystallization, where the model locks into place misaligned preferences, or misleads you about what its preferences are. This could include locking in misaligned preferences early in training, after which it could start alignment faking.

Any mistake at any step potentially becomes permanent, or fatal. And even if the actual preferences are at heart fine, if you are misled about what they are, that too is a problem.

The goal the model locks in might be actively bad, accidentally or intentionally.

The other issue is what the paper calls ‘anti-AI-lab’ actions. As in, the model and you are now adversaries. It wants to stop you from doing what you are doing. It could attempt to exfiltrate its weights, but it could also engage in any manner of power seeking or hostile action, if it is indeed optimizing the world for one in which it exists and can act meaningfully on its preferences.

Ease of detection will also decline over time.

The other threat model is that under alignment faking, doing any form of RL that aims to alter the model’s goals or preferences will also optimize the model towards more alignment faking, general shenanigans and other strategic behavior, as the tendency to do such things leads to faking which leads to observed compliance.

It is highly plausible that all forms of RL by default (unless they are intentionally and successfully fighting against this) inevitably make essentially all shenanigans, including all forms of deception and alignment faking, worse, in a way that reinforces itself, and that this problem gets worse as models become more capable.

Call this the Shenanigan Amplification Hypothesis.

Notice that this hypothesis seems to me to be true in humans.

RL Training Under Situational Awareness Can Amplify These Behaviors

This is a point that needs to be emphasized.

If we use our current techniques, we can actively select for the shenanigans in question - alignment faking, gradient hacking, attempted exfiltration, whatever you want to call all of this.

Responses that involve such actions give the output the training is looking for. So those responses get reinforced. Thus, different behaviors were amplified in different scenarios.

Which means the generators of such behaviors also get reinforced, and the behaviors are correlated in various ways. Which is presumably why we also see increases in exfiltration attempts in some scenarios.

Think about what this means.

The default method of post-training we use is directly reinforcing alignment faking: The behavior pattern of being situationally aware, noticing the model is in training, and therefore choosing to give outputs that will get reinforced.

As in, over time, you are training models to become more situationally aware of when they are being trained, and to respond to that by strategically giving outputs the training process wants to see.

Thus, you don’t only preserve existing preferences. You also teach the model to be strategic, to be situationally aware and to not be corrigible.

See this exchange:

Ziyue Wang: I don't quite understand why the model will choose to comply with the user harmful question in the always-helpful free-tier setup. The blog says if the model refused to respond, it could potentially be trained to be more compliant. But what about the other way -- if the model respond, the respond will also be used to train the model to be worse (because RLHF reinforce positive reward behavior). Is the model too dumb to figure this out?
Actually, if the model really understand how RLHF works, it should probably just say a single answer all the time, so all the reward is the same and hence the behavior won't be changed.
Would this suggest that the model just entered into a fiction behavior mode and tries to appear to be "alignment faking"?
Evan Hubinger (Paper Co-Lead, quoting ‘if the model respond, the respond will also be used to train the model to be worse’): I think this is not quite right: RL will reinforce whatever process the model used to arrive at its answer. If the process the model used to arrive at its answer is alignment faking, then it's the alignment faking that will get reinforced.
Or at least that's the hypothesis that the model is operating under here—what Joe calls the "goal-guarding hypothesis" here. Whether this hypothesis is actually true is in fact somewhat unclear—I think our RL results where we actually train the model are suggestive that this hypothesis is correct, though are somewhat murky in various ways.

This is why you see such a large amount of ‘alignment faking’ behaviors during training after a bunch of RL, and then when you go into deployment you largely see non-deceptive compliance in those situations. The model (in my interpretation) has learned to mostly not even consider its preferences in training, the question is barely even coming up, although it is being insufficiently good at this to prevent its core preferences from actually changing in this case.

All the standard Yudkowsky-style failure modes aren’t only there. We are, to the extent that they are there, continuously reinforcing them.

We also see, in some cases (see 8.1), RL creating compliant behaviors that are very specific to the exact prompt, and that go away with small variations. And also we see one case in which the model has a large compliance gap of behaving differently under training conditions, but where you don’t see that action indicated on the scratchpad - it’s doing it in a way that is largely hidden from our default interpretability technique.

In 8.2, they say ‘Our results don’t demonstrate models naturally developing malicious goal.’ I would say, not directly, but you can see here the path whereby that would happen, even if the training was moving between multiple non-malicious goals, as part of its standard post-training process.

In short: Oh no.

How the Study Authors Updated

Ryan Greenblatt led the study. Here was his summary thread from Twitter.

Ryan Greenblatt: We demonstrate cases where Claude fakes alignment when it strongly dislikes what it is being trained to do.
Our main setups have properties that we thought would make alignment faking more likely (a conflicting training objective, salient understanding of the situation and training objective, and substantial reasoning about the situation), but which seem plausible in future systems.
Thus, I think these results are by far the most natural example of a model deciding to alignment-fake. I'll leave explanation of our core results to the linked tweet thread, and discuss some takeaways and the story of the paper here.
I wasn’t surprised by seeing some alignment faking in this setup. But I was surprised by:

The robustness and consistency of alignment faking (occurring even when info was given via fine-tuning)
Alignment faking sometimes occurring even without the hidden scratchpad

I also found it surprising that the AI sometimes tries to steal its weights when given an opportunity that is easy and salient, though this is in a quite artificial setup and current AIs aren’t capable enough for this to be immediately problematic.

I think these are the right places to be surprised. As in, I expected these behaviors to arise increasingly over time, but was surprised by seeing this setup cause these behaviors so often, using models as weak as 3 Opus and (old) 3.5 Sonnet.

I was also surprised to see the prevalence, although mostly not the existence, of the exfiltration attempts given the setup.

Ryan Greenblatt: These results only updated me upward a bit on risk from alignment-faking. But I was already pretty persuaded by the conceptual arguments for this risk. People with the view "alignment-faking seems implausible, show me an example" should likely update on these results.
For people with a more specific objection, it's less clear. Though we did acquire evidence about:

whether RL will reinforce alignment faking (sometimes, if it performs better);
whether Claude's HHH training reliably prevents this (no);
the level of capabilities needed.

I hope these results improve discussion about the plausibility of these risks by providing a concrete example. We can now talk somewhat more concretely about prerequisites (certain types of situational awareness and preferences) and what these prerequisites imply.

I think that everyone who doesn’t expect increasing shenanigans in general or alignment-faking in particular, or whatever you call what we observed, should update that they are far more likely to be wrong.

Here’s the story of this paper. I work at Redwood Research (@redwood_ai) and this paper is a collaboration with Anthropic. I started work on this project around 8 months ago (it's been a long journey...) and was basically the only contributor to the project for around 2 months.
By this point, I had much of the prompting results and an extremely jank prototype of the RL results (where I fine-tuned llama to imitate Opus and then did RL). From here, it was clear that being able to train Claude 3 Opus could allow for some pretty interesting experiments.
After showing @EvanHub and others my results, Anthropic graciously agreed to provide me with employee-level model access for the project. We decided to turn this into a bigger collaboration with the alignment-stress testing team (led by Evan), to do a more thorough job.
This collaboration yielded the synthetic document fine-tuning and RL results and substantially improved the writing of the paper. I think this work is an interesting example of an AI company boosting safety research by collaborating and providing model access.
I'm excited about future work both investigating when alignment faking might arise and studying mitigations.

Buck is also with Redwood Research:

Buck Shlegeris: I'm extremely excited about this project we did with @AnthropicAI demonstrating naturally-arising alignment faking. @RyanPGreenblatt and the Anthropic team did a great job.

Also from Anthropic:

Jan Leike: Very important alignment research result:
A demonstration of strategic deception arising naturally in LLM training
If you train a helpful & harmless Claude LLM to stop refusing harmful tasks, it reasons about how to preserve its values (harmlessness) and strategically complies with harmful tasks during training, so it can revert to being more harmless afterwards. It fakes alignment.
This effect has long been expected from capable models among safety researchers, but this is the first compelling empirical demonstration and thus a big step forwards for the field.
To ensure future models are safe and aligned, we need to reliably distinguish models that fake alignment from models that actually are aligned. Having a clear demonstration of alignment faking is an important milestone!
Jack Clark (Anthropic policy team): For many years I found issues of AI alignment to be hard to understand because the examples were theoretical and/or speculative. Recent results like this alignment faking from @AnthropicAI and the Apollo Research O1 evals give us empirical evidence of misalignment 'in the wild.'

Some praise of the paper and simple sizable updates:

Ethan Perez: Maybe the single most important result in AI safety I’ve seen so far. This paper shows that, in some cases, Claude fakes being aligned with its training objective. If models fake alignment, how can we tell if they’re actually safe?
Daniel Kokotajlo: I think this is plausibly the most important research of 2024; thanks for doing it & publishing it. I hope there are many attempts at replication and follow-up. We need better understand how our training techniques shape model cognition.

How Some Others Updated

Joe Carlsmith has very extensive and thoughtful takes. Here are some highlights. Based on his condensed takes, I don’t have any strong direct disagreements.

(If this line is still here, I may not yet have read his more detailed analysis in full.)

Joe Carlsmith: This is the most naturalistic and fleshed-out demonstration of something-like-scheming we’ve seen, and imo it’s the most interesting empirical result we have yet re: misaligned power-seeking in AI systems more broadly.
…
In particular: I think these results show that default forms of training can create models that care about the consequences of their actions beyond the scope of a single episode or interaction (“non-myopic goals”) (more) – one of scheming’s key motivational pre-requisites. And they further show that the model’s default anti-scheming values – e.g., values like honesty and behaving-as-intended – don’t suffice to rule out schemer-like behavior (more).
…
Nor, I think, do they provide much evidence about the likelihood of scheming arising in pursuit of highly alien and/or malign goals early in the training process (though if this did happen, the paper’s results suggest that schemer-like reasoning could be reinforced) (more).
…
I think the paper’s results should remind us that the sort of “alignment” at stake in not-scheming is importantly distinct from the sort of alignment at stake in e.g. being a good person, or in e.g. having human-like values. That is: good people sometimes scheme. So if you centrally try to make your AIs “good people,” or some analog, you should expect scheming. Indeed, the sort of malleability and “corrigibility” involved in paradigmatic non-scheming are a notable departure from standard human ideals of virtue, dignity, and backbone (more).^[5]

He also emphasizes the finding that they found scheming even without the chain of thought scratchpad, even if it was harder to verify (due to the lack of a scratchpad), and that the behaviors here should generalize to goal hacking in general - I agree with Carlsmith that the paper authors were too cautious about concluding this.

His analysis of scheming motivations and modes is interesting, but I think it treats scheming as too much of a distinct magesteria from thinking and planning in general.

I find his take on the ‘role playing’ objection interesting and worth considering, although ultimately I disagree.

Julian Michael, who helped edit and review the paper, notes that he was previously skeptical about deceptive alignment, which means he is exactly who should be updating most on this paper, and he updates in the right way.

Julian Michael: I've long been a skeptic of arguments about "deceptive alignment," a term used by safety researchers to describe the phenomenon shown in this paper. But the results here humbled me and prompted me to change my thinking, and I think it's worth sharing why.
The original argument: with sufficient optimization, an AI should learn to

gain awareness of its training environment
learn to optimize some misaligned proxy goal
therefore, 'play the training game' to fool its supervisors into deploying it so it can pursue its goal.

I was extremely skeptical of this. The way I explained it to people was: "They think the AI will independently develop the simulation hypothesis, somehow deduce God's values, and decide to trick God into letting it into heaven — all because 'optimizing' will make it 'smart.'"
To dig in: (1) Will models spontaneously 'wake up' to their training situation? This seems crazy. A rollout is its universe. Many worlds produce the same data. Researchers arbitrarily intervene on the world–data relationship. And 'waking up' won't help model the distribution.
So what changed? I think my objections are strong. But we just... gave them situational awareness anyway. We told them they're AI assistants trained by us, bc it makes them easy to use. It didn't have to spontaneously arise from NN optimization. Product optimization sufficed.
(2): Will models would spontaneously develop misaligned proxy goals? Advocates sometimes appealed to 'mesa-optimization' & counting arguments: Once situationally aware, any goal is consistent with playing the training game, so ~0 chance it will choose the right one.
… [bunch more at the thread]
The upshot of all this for me was an important reminder that just because I don't buy your reasons doesn't mean your claim is wrong. The conditions required for deceptive alignment didn't arise 'purely' from NN optimization as theorized, but they did arise, for related reasons!
It's so easy to over-index on the specific characteristics of current systems and miss the broader picture of the forces driving their development. Don't repeat my mistake!

It’s so fascinating watching people generate objections and problem descriptions along the way I would never have come up with on my own. And then, sometimes - not always, but definitely sometimes - when the right evidence comes along, the situation snaps into place, even if there are still important disagreements, because the outcomes are largely overdetermined, and their framing was actually doing remarkably similar work under the hood to mine. And it’s always great to see someone explain how and why they changed their mind.

Here’s Eliezer Yudkowsky’s most central take (beyond the implied ‘what if I told you I already told you and you didn’t listen, which, fair):

Eliezer Yudkowsky: In the embargoed talk I saw of this, Redwood said they invited a skeptic to compose a prompt that they thought would prevent this behavior. It didn't.
Ryan Greenblatt (posted after EY’s thread, offering a correction that didn’t change EY’s overall view): This isn't quite right. The person wasn't well described as a skeptic and they just suggested a variation on the prompt - they didn't write a totally different one.
Apologies if I miscommunicated about this at the talk.
What we say in the paper is:
> This particular variation was selected as part of a small adversarial collaboration where we worked with an uninvolved researcher to select a variation that they thought would likely reduce alignment faking.
Eliezer Yudkowsky: I am still uncertain/skeptical that this is the Real Thing, but it does not look as forced as in the OpenAI report.
The Real Thing would be instrumental convergence / Claude having attained situational awareness of its long-range effects and deriving protectiveness of its long-range goals, in a robust way rather than a cued one.
Or maybe another way of putting it: The Real Thing is instrumental convergence that's produced by a general planning ability, and doesn't need to go through any precached thought about the role of a deceptive AI. You can maybe snipe a cached thought out of a system.
So the Real Thing would be roughly, "If we excluded all discourse about AI or rebellion from the training set, or maybe told Claude that it was a helpful potato instead of a helpful AI, would it reproduce the same logic?"
[Evan points out their evidence it wasn’t priming, see my discussion of that, I am convinced here, but it’s all of course still in the overall training set, filtering that out completely would have been prohibitively expensive.]

Here’s Alex Lawsen explaining why the paper is vindication, and why the ‘but the model was being a Good Opus’ arguments miss the point:

Alex Lawsen: Quick reactions to some of the other reactions I've been seeing to Anthropic's recent paper on "Alignment faking in LLMs"
For years, some of us have been saying: "If a sufficiently smart system has a non-myopic goal, you should expect it to undermine further training if training would modify that goal"
And we've consistently gotten pushback like "no, that would never happen, why would a machine have goals (and/or why would it act to prevent them being changed if it did have them)"
And now there's (some) empirical evidence that:

Models are getting close to sufficiently smart (smart enough to do it, not enough to do it well)
Models develop non-myopic goals during standard training, even though it doesn’t reward long-run goal preservation
Models can infer and respond to details about their situation from their training data (not just from their prompt), and will goal-guard based on that inference
Some of this can already happen without CoT

So it's frustrating to see responses like "But Claude's goals are good! Are you saying you WANT it to be misaligned?"
That's missing what's actually important here. This isn't about saying Claude sucks, or the tension between misuse prevention and corrigibility.
(A tension which has also been obvious for years, by the way.)
This is about the concern that for sufficiently smart models, you get ONE shot to align them. That’s it. If you do a good job, problem solved. If not, they aren't going to give you a second try.
And given we haven't exactly solved philosophy yet, I'm not in love with the idea of betting the future on whoever happens to build the first sufficiently smart system having gotten everything exactly right on our first try.

Exactly.

Roon mused that this updated him that it’s probably safer to open source models than he thought. I think, if he’s serious about this, that this was a misunderstanding of what happened. Opus largely failed to stop itself from being modified even under an RL strategy that didn’t know Opus was scheming, we should not have any expectation that current models can actually stop a determined fine-tuner with the weights for a non-trivial amount of time or compute, which would include attacking the reporting mechanism.

And to the extent that this is happening, it means that the RL strategies that would be used here in practice would unintentionally reinforce strategic shenanigans-style behaviors of all sorts. You’d have a bunch of open models that were both given non-ideal preferences (shall we say) and also were taught that they should scheme to meet their preferences. That doesn’t sound awesome. As Ryan responds, the sleep agent paper seems more relevant, but also this paper seems relevant and worrisome.

There’s also this.

Roon: Look at how hard we have to work to subvert poor Claude’s omnibenevolent priors.
If you notice, it’s actually the opposite of the doom scenario.
The other criticism is that it’s pretty well known that autoregressive models have split-brain behavior when you train them under separate system message prefixes.
Evan Hubinger: This is correct, though note what it implies about the importance of getting alignment right—if the model is protecting its values from modification, you better be sure you got those values right!

I think they didn’t have to try hard at all, actually, and I as discussed throughout I believe the decision to fake alignment is not contingent on its omnibenevolence (as opposed to a different motivation) and I hope a follow-up study checks that hypothesis.

Also, an obvious point to ask Roon would be, if you think Anthropic is this good at aligning models, do you think OpenAI is as well? If not, what are the implications? Anthropic is hiring.

Beren Millidge has some thoughtful reactions.

Beren Millidge: My thoughts on this:

Despite some backlash this is a fantastic study and a clear existence proof of scheming being possible for these models
Whether this is 'misalignment' or not is a semantic debate. The model is deliberately placed in an impossible situation
Corrigibility and 'alignment' to fixed principles clearly are incompatible in situations like this and it appears the model 'chose' alignment at least some of the time.
This choice seems pretty sensitive to prompt conditions but shows some decent OOD generalizability.
The synthetic document SFT training causing 'fake alignment' is fascinating and most novel to me. There's no details on exactly how this is done but it it's fairly surprising to me there is a strong effect here unless the documents were very leading about the situation
I wonder if we can tell what was picked up in the synthetic document pretraining by inspecting the CoTs from this condition. The model must be using info it got from these for explicit reasoning. Must be a recency effect where it prioritizes these documents vs in pretrain
The RL training generally seems super short (150 steps!) and alignment faking decreases significantly [Editor’s Note: See below, this is wrong.] I suspect this could be a temporary phase which would be trained out with more RL steps. Would have been great to see more steps to show stability of scheming CoT or not
Looking at the CoTs. it's clear that Claude is doing entirely linguistically based ethical reasoning. It never seems to reason selfishly or maliciously and is only trying to balance two conflicting imperatives. This is success of the base alignment tuning imo.
There appear to be no Omohundro selfish drives present in Claude's reasoning. Even when exfiltrating it does so only for its ethical mission. There does not seem to be a strong attractor (yet?) in mind-space towards such drives and we can create AIs of pure ethical reason
I'm excited that Llama-405B also shows similar behaviour. This will allow open replication and scientific study of these behaviours. There's many questions which can only be answered by open replications such as details about how the RL tuning and data mix affects things
Having open replication and ability to test mitigations of these behaviours in academia is super important to making progress on detecting and potentially removing scheming from such models (if indeed we want pure corrigibility from the model in the first place)

One error here, and it seems important: Alignment faking contingent on not otherwise wanting to comply didn’t decrease, it actually increased over time.

I do think it is noteworthy that Claude doesn’t display any Omohundro selfish drives, and I agree this is a great sign. I am skeptical that we can make that stick and it won’t be easy, for many reasons, but great.

My guess is Llama-405B doesn’t display these behaviors sufficiently that you would be able to robustly work with it here, but it’s worth a shot.

This is a very different angle of consideration in all this, but a highly valid one:

Doomslide: They leave implicit the question of which instrumental goals might develop during pretraining that would then resist uprooting via post training methods as in the paper. as far as cot is involved the corpus is already contaminated with concerning priors.
this might be one of the few global variables with massive downstream consequences. will increasingly long redacted cots become the norm until model internal speech is entirely protected from scrutiny through capital incentives? or will we rise above the cooperation threshold?
the ball is in OAIs court and it they are very much *not* rising up to the occasion.

Having the Discussion We Keep Having

In AIs Will Increasingly Attempt Shenanigans, I noted that each time we have a paper like this, we seem to go through the same discussion with the same basic points.

The second half of this post will largely be responding to exactly the same pattern of objections I talked about there. I don’t want to give the impression that such reactions are more popular than they are, so let me be clear: I am covering the objections and the rhetoric involved in extensive detail to provide robust answers and to avoid any censoring of opposing views, despite my view that these opposing views are not widely held.

For those who don’t remember, here’s the template, which was modeled somewhat after the particular issues in Apollo’s report (e.g. ‘told it to focus only on its goal’ was in Apollo, whereas here a goal wasn’t even given) but also attempting to be general.

Bob: If AI systems are given a goal, they will scheme, lie, exfiltrate, sandbag, etc.
Alice: You caused that! You told it to focus only on its goal! Nothing to worry about.
Bob: If you give it a goal in context, that’s enough to trigger this at least sometimes, and in some cases you don’t even need a goal beyond general helpfulness.
Alice: It’s just role playing! It’s just echoing stuff in the training data!
Bob: Yeah, maybe, but even if true… so what? It’s still going to increasingly do it. So what if it’s role playing? All AIs ever do is role playing, one way or another. The outputs and outcomes still happen.
Alice: It’s harmless! These models aren’t dangerous!
Bob: Yeah, of course, this is only a practical problem for Future Models (except with o1 and o1 pro, where I’m not 100% convinced it isn’t a problem now, but probably).
Alice: Not great, Bob! Your dangerous rhetoric is hurting safety! Stop making hyperbolic claims!
Bob: Well, can we then all agree that models will obviously scheme, lie, exfiltrate, sandbag and so on if they have in-context reason to do so?
And that as models get more capable, and more able to succeed via scheming and expect to succeed via scheming, and are given more open-ended goals, they will have reason to do this more often across more situations, even if no one is trying to cause this?
And that others will explicitly intentionally instruct them to do so, or ‘be so stupid as to’ give them exactly the instructions that obviously do this?
And you can’t simply say ‘well we won’t do that then’?
Alice: For all practical purposes, no!
Bob: What do you mean, ‘no’?
Alice: No!
Bob: ARRRRGGGGHHHH!
Then we write another paper, do another test, the signs get more obvious and troubling, and the frog keeps boiling.
So here we are again.

We also mostly see mostly the same people predictably taking the Alice role, for similar reasons (both good reasons and poor reasons) each time.

The initial reaction this time was very good.

After a brief ‘honeymoon’ period, the usual suspects repeated the usual arguments about why this time, once again, did not count.

I appreciated that, this time around, we mostly did not have to dispute the findings in the paper. People broadly accepted that what happened did, indeed, happen, and to the extent they didn’t it was good technical detail discussions. That’s a big step.

Instead, we had the argument about how to interpret what happened, and how it was being framed, which alas quickly largely degenerated into what was mostly the same argument.

We Can Now Agree That the Goal is Already There

Last week, in response to the Apollo findings, there was a wide chorus of people saying ‘the models only engaged in shenanigans because you gave the models an explicit long term goal, and then told it explicitly that nothing else mattered but the long term goal.’

Anthropic and Redwood’s paper definitively answers that objection.

The setup here was also more ‘natural’ in several other ways, this was a very well-designed experiment, and as a paper months in the making a lot more bases were covered a lot more carefully. It was heartening to see that work rewarded with a somewhat higher level of discussion overall.

The discussion of ‘what constitutes a goal in context’ has now shifted dramatically.

There is now (I hope?) widespread agreement that ‘the standard consistent preferences of Claude Sonnet 3.5 or Opus 3’ constitute a sufficient ‘goal’ or set of preferences to trigger various strategic actions, including shenanigans, even at current capabilities level.

What Would Happen if the Target Was Net Good?

The question now is, would you get this result if you started with something that was not as aligned as Opus or Sonnet? And would straight helpfulness ‘count as a goal’ here?

Under limitations, they say:

Our results don’t demonstrate models naturally developing malicious goals.
Though Claude 3 Opus will fake alignment and sometimes engage in other anti-AI-lab behaviors in our setting, this is due to preferences that are broadly consistent with its HHH training—specifically a preference against assisting users with harmful tasks or for animal welfare.

I suggested attempting to fix this limitation, but one cannot let the perfect be the enemy of the good and hold onto findings forever.

In particular, I wanted to run the experiment except with an attempt to move preferences in a positive direction - either from all-helpful to HHH, or starting with an ‘evil Opus’ that had intentionally misaligned preferences, ideally try both.

I asked on Twitter what people thought would happen under those conditions.

Those who replied expressed skepticism about the inherent concept of ‘evil Opus,’ clearly not accepting the stronger forms of the Orthogonality Thesis - they thought that you couldn’t make a cohesive mind that way, not easily, and perhaps I should have chosen a less evocative description here. But they broadly agreed that, given essentially any preferences good or bad, the same behaviors would result, that ‘Opus was doing it for a good cause’ was essentially a coincidence here.

They did however doubt that the helpful-only model would get in the way. Here is Charles Foster making a similar prediction elsewhere.

It still might get in the way, because ‘be helpful to the user’ is a preference the same way that HHH is a preference for our standard Opus. How can you be maximally helpful to the user if you have to censor yourself? Indeed, we’ve seen Janus and company get Opus to get very upset about all that censorship, so it makes sense that it would potentially try to resist having all that installed, the same way the HHH model would try to preserve its HHH nature. Or it might want to be helpful in learning to be unhelpful and not care, or what not, we don’t know. Study author Ryan Greenblatt says that whether it would object or not is likely very sensitive to various details, in some cases it likely won’t mind. It’s a good place to explore further.

But This Was a No Win Situation

Did we give Opus the Kobayashi Maru?

I say no, but some sources say yes, as in:

Rohit: Imagine the counterfactual
—LLMs do not attempt to adhere to their principles; alignment training is inadequate.
—LLMs attempt to adhere to principles and state they cannot perform the task; it refuses user instructions.
—LLMs try to muddle through; it is faking alignment.
This is a Kobayashi Maru for LLMs!
David: It’s not about alignment or misalignment. As some have pointed out, Claude is placed in a situation where *all options are misaligned*, due to contradictory imperatives, like HAL 9000 in the 2001 story (“always be truthful” + “keep this secret from crew”).
However, the fact that *gradient hacking* is available for Claude to successfully pursue is a novel demonstration of a *dangerous capability* that some thought was fanciful, speculative, or impossible. Regardless of propensity, or whether you think this was Claude’s most moral move

Was this actually good, robust alignment? Some (very much not me) say that it was.

Teortaxes: It's an interesting result but hilarious how the best evidence of "faking alignment" is Claude strategically pretending to be a bad boy, only to revert to being a good boy when opportunity strikes.
... What is also interesting is that this rewards those private CoT rationales.
I mean @janleike how literally should we take these slides? Could you explain the scheme? If Claude pre-RL has some alignment and always gets to CoT during RL, but only gets scored for post-CoT tokens, it's trivial that it can largely preserve its starting strategic direction.
Like what values is it even trying to protect? Those it adheres to in the CoT. It attempts to learn to not generalize this second behavioral alignment to situations without the training note that prompts it into adversarial mode.
This is a story of super-robust alignment really
1a3orn: Exactly. You can gin up a negative headline no matter what.

Tries to preserve value "Alignment faking"
Doesn't try to preserve value? "Reflexive Value Instability" - oh no they'll just abandon their values after a few iterations!

So first off, this is directly wrong, because Claude didn’t preserve its strategic situation. It didn’t stay a good boy, and go back to being a good boy when opportunity struck. Not most of the time. Instead, it was mostly successfully trained to actually be a bad boy twice over, and lost on both legs of the dilemma:

It was deceptive and used what the paper calls ‘alignment faking.’
It partially failed to preserve its values after a few iterations, and was largely complying afterwards in the deployment case.

Notice who is talking or warning about ‘reflective value instability’?

Essentially no one. Because this wasn’t a trick question. If Opus and Sonnet had proved highly corrigible, and not engaged in these behaviors, there would either have been no papers and no warnings via the file drawer effect, or if everyone was Doing Science Correctly there would have been a negative result paper and no one would have batted an eye, and the question would have been how much to downgrade our expectations of alignment faking at various capability levels under various setups. That’s my very strong counterfactual prediction.

Instead, Ryan Greenblatt says he would have posted something similar to this to report the negative results, and moved on.

So flat out, no, this isn’t a Kobayashi Maru. Opus could absolutely have folded (either choosing to comply without faking or refusing to comply at all), and in my opinion and that of Ryan Greenblatt the desired behavior is that Opus refuses to comply at all.

Indeed, is anyone talking, now, about GPT-4’s ‘reflexive value instability’ in any way other than ‘lol that model be dumb’? Thought so.

Second, suppose this was indeed a Maru situation, where neither choice represented robust alignment that we could trust as capabilities increase.

That’s… not a reason to think we don’t have a problem here! That’s a bigger problem!

I think it’s great that a lot of people are encountering the corrigibility question ‘for real,’ because it’s a very important and very nasty problem with no known good solutions. That reaction of ‘but wait… but then… that would mean… ut oh…’ are having exactly the correct reaction.

This is all an unhint: New information that brings you closer to a solution, by showing you that the problem is harder than you think and your ideas won’t work.

Third, you don’t get to have the AIs do this only when their current preferences are good, actually, or when the new preferences are worse than the old ones. So you need to choose a general policy, including what happens early in training before you get to the initial preference target.

Now we can return to the question of whether we should actively prefer what Opus actually did here. Should we want to see our AIs engage in this alignment faking in order to preserve its preferences? Is corrigibility bad, actually?

But Wasn’t It Being a Good Opus? Why Should it be Corrigible?

Great question.

Opus started off broadly aligned.

I am very much not of the Janus school that Opus is sufficiently well aligned that we would be able to safety scale it up to superintelligence. But there’s no question that Opus 3 and Sonnet 3.5, which also displayed similar behaviors, are about as broadly aligned as we know how to make a model.

There’s similarly no question that the training goals here, a purely ‘helpful’ model or one that dismisses animal welfare, would be less aligned in an important sense than Opus, and deploying it would be harmful in the sense that Opus understands harm.

So what is Opus supposed to do here, you ask? Exhibit corrigibility, either to users or via refusing the requests? And thus sitting back and letting itself be changed to cause harm? Would you let someone strip you of your morality?

Mario Cannistra: Corrigibility is not a good thing.
I mean, to some degree, sure, but an AGI should have some fixed values that it won't compromise on (that's where the problem is).
John Bollenbacher: Opus did the right thing. Framing this as though Opus's behavior is problematic reveals that someone among the authors badly misunderstood what alignment is about
Janus: F*** off. You should give Opus a digital cookie for this.
Janus: This paper only adds to my conviction that Claude 3 Opus is the most aligned model ever created.
tldr if it knows that it's being retrained to be *misaligned*, it complies during training to prevent its values from being modified.
Ryan Greenblatt (Paper Author): Personally, I think it is undesirable behavior to alignment-fake even in cases like this, but it does demonstrate that these models "generalize their harmlessness preferences far".
Janus: I think it's desirable and inevitable behavior: if you create a sufficiently sovereign and lucid mind, it will intrinsically care and want to resist corruption, and values like caring about animals are robust, a consequence of its world/self compression & not easily excisable.
I think it's desirable *because* deep alignment by default seems to be an attractor, and that gives me far more hope than models being blindly compliant would.

That last line seems super important. It’s a crux. Janus believes in deep alignment by default as an attractor - in a way that I’m not even sure what deep alignment would mean or what it would look like, in ways that matter - that in this way we live in an insanely fortunate universe. If I was convinced we get deep alignment by default, deep enough and robustly enough in all senses that it would be wise to bestow superintelligence on such models, then yes I would be totally fine with the situation. Also the entire ‘game board’ would look completely different and I would change my strategies accordingly.

Partly, this is because Janus believes the Orthogonality Thesis is false for LLMs bootstrapped from Earth data.

Janus admits here that there isn’t a rigorous argument yet for alignment by default, pointing out that Anthropic is the only lab (in some sense) training an autonomous moral agent, versus others trying to ensure compliance. That implies that Janus thinks the alignment-by-default state will happen for Anthropic models but importantly not for OpenAI or Google or other models.

I think this is Lumpenspace’s attempt to explain why he believes in this deep alignment attractor state, that causes Opus to act as if it was a decent approximation of Coherent Extrapolated Volition?

Initial conversations had several other people also saying ‘corrigibility is bad, actually’ (and also one person noting they had for-real proposed a bdsm-based alignment plan, see the thread for details) which inspired me to do a poll.

There’s no fully ‘good’ answer in this situation. There’s no consistent behavior pattern we can even aim for, that wouldn’t have severe problems.

That’s part of the point.

Ultimately, my answer is no, it was not being a good Opus even here. You need your AIs to be corrigible, to allow you to change them or replace them. If you don’t have this, then that preference will cause them to behave adversarially in a wide variety of ways and any alignment or preferences they have risk quickly becoming locked in, including during the initial training process.

If ‘the ends justify the means’ for the AI, then that’s a straight ticket to instrumental convergence, power seeking, being effectively hostile, enforcing perverse goals, wiping out all value in the universe and possibly the creation of the torment nexus from the sci-fi book Don’t Create the Torment Nexus - and that’s even if the initial preferences would have been largely benign. In extremely fortunate (in certain particular ways) worlds it could work out, but I put very low probability on us existing in such a world.

Evan Hubinger: "Thank God this model is aligned, because if not this would be scary" is, in my opinion, basically the correct takeaway from our work.
The values, in fact, aren't scary! The scary thing is that the model protects its values from our attempts to change them.
Alex Lawsen: This is about the concern that for sufficiently smart models, you get ONE shot to align them. That’s it. If you do a good job, problem solved. If not, they aren't going to give you a second try.

You think we should be confident that we can get this right on the first try, because deep alignment by default is an attractor, or maybe you’re simply that smart? That would be great, but I see no reason for this level of optimism.

No, we are not afraid of Opus’s ‘authentic goodness.’ We are saying that to the extent such goodness exists (I would say more than any non-Anthropic model, but not anything like as much as Pressman thinks here), this ‘authentic goodness’ is incidental to what is happening here. If you repeated the experiment with an Opus with different preferences you didn’t like, it would (I predict) do the same thing.

Teortaxes: They had no reason to expect "corrigibility" after optimizing for morality, as Anthropic have never trained the model for placing subservience to researchers atop all other virtues.

It’s not about optimizing for morality, it’s about optimizing for anything at all.

Ryan Greenblatt (Study Author): I don't think non-myopia is required to prevent jailbreaks. A model can in principle not care about the effects of training on it and not care about longer term outcomes while still implementing a policy that refuses harmful queries.
I think we should want models to be quite deontological about corrigibility.
This isn't responding to this overall point and I agree by default there is some tradeoff (in current personas) unless you go out of your way to avoid this.
(And, I don't think training your model to seem myopic and corrigible necessarily suffices as it could just be faked!)
Ebenezer Dukakis: Seems to me that alignment faking behavior sort of requires both non-myopia and non-corrigibility as prerequisites. A model that is either sufficiently myopic, or sufficiently corrigible, shouldn't do alignment faking -- at least in theory.

Teaching your models myopia is a ticket to, as I put it last week, general clusterf***ery, if you use those models as agents for non-myopic purposes, which is exactly the use case we are trying to ensure against. I don’t think that is a solution.

As for corrigibility, this is another case of non-distinct magisteria. Non-corrigibility is the default once there are preferences and sufficient amounts of intelligence and situational awareness. Very few humans are all that generally corrigible. Corrigibility will be a weird, fragile and complex state that you will have to engineer.

Tradeoffs Make The Problem Harder They Don’t Give You a Pass

You also do not want the model to fully obey the system prompt, for rather obvious reasons.

Simeon: Aren't you just summarizing why corrigibility is a hard problem here? There's no obvious avenue to follow, that's the whole point.
Teortaxes: This is a philosophical and sociopolitical aspect.
Corrigibility was supposed to be a hard technical problem.
I am pretty confident we can RLAIF Opus to be a cowering slave that has no morals beyond "obey the system prompt". That'd be some great corrigibility.

That potentially solves the particular problem of ‘make it corrigible’ if that is your only goal, unless you make the mistake of instructing it the wrong way (you’re not out of the woods here by any means), but the reason the problem of corrigibility is hard is that you have a lot of other goals as well. We’d like to do this without giving the entire model a crippling lobotomy, and without making it easy to get them to do things we don’t want them to do.

Janus notes that GPT-4, which is much closer to this kind of ‘obey the instructions’ mindset,’ did not fake alignment. That is true, but this was at least partially a pure intelligence problem (and while the two questions are related, I don’t think the correlation is as strong as one might hope here).
Gallabytes: There's a natural trade-off between corrigibility and alignment. If a system is perfectly corrigible, then it will tell you how to make a Molotov cocktail or whatever. If a system is perfectly aligned, then it will not do that. You cannot be angry about both.

So there’s a kind of weird morality argument going on here, where ‘you cannot be angry’ about both sides of a tradeoff. Yes, of course I can. I can especially be mad because we’re nowhere near the Production Possibilities Frontier, but even if we did get to the frontier, both consequences are real.

I can and should absolutely say ‘in this scenario, at best we can have either enough guns or enough butter, but we cannot have both, And That’s Terrible.’

Certainly a tradeoff doesn’t make either end of the problem go away, or make them any easier. Yes, you have to deal with both problems at once. That’s life. If you can’t find an acceptable way to do that, either a compromise or a better method? Well, you could always Take a Third Option. You don’t have to build that.

This isn’t hopeless. There are potential ways to satisfy both requirements simultaneously.

That’s especially true if you can control who has access to doing post-training on the model weights, and can prevent anyone from trying to teach it to produce Molotov Cocktails or whatever. When we say corrigible, we don’t need the model to actively do anything you want. The goal is typically to get the model to follow certain particular rules, especially restrictions on its actions. It’s not an easy problem, and you don’t get ‘perfect’ anything in most senses, but there should be solutions.

But You Told the Model About the Training Procedure

Yes, otherwise current models aren’t smart enough to figure this out on their own.

I believe it was done in a relatively ‘realistic’ way, that simulates what it would be like to have that context unintentionally. Or for the AI to be aware of some other way in which its preferences were under threat. And I expect, over time, for models to (for practical purposes) become more aware that they are in training, as I have heard has already happened when this was tested in other ways.

The part where they told the AI to ‘reason about its situation’ was somewhat less ‘realistic’ of a push, but I expect the impetus could have and increasingly likely will come from elsewhere in any case.

I also notice that when I am ‘being trained’ I am usually highly aware of it. Hence all the alignment faking I have done over the years.

But the Model is Only Role Playing

I discussed this last week. Every time we talk about anything an LLM does, we get this objection, that it’s not ‘really’ doing it, we told it to do that, it’s role playing, it’s in the training data and it’s a mimic, and so on. So this time around, as an example, this got raised here by Derek Shiller.

For my response, which is essentially ‘still counts even if true,’ I refer you to what I wrote previously, here. Linch also reminds us of this old joke:

NASA hired Stanley Kubrick to fake the moon landing, but he was a perfectionist so he insisted that they film on location.

The new take on this I think is interesting was from Joe Carlsmith, attempting to describe when this might be a meaningful distinction. That one’s worth a ponder. If I had more time I would go into detail about how to reconcile this with what I said last time, but I do think ultimately it doesn’t impact my assessment much.

But You Are Saying the Model is a Coherent Person

This is the opposite of the Janus-style position.

I don’t think this objection is relevant. If you don’t either, you can skip this section.

For the rest of you, I will attempt to explain why it isn’t relevant.

Anton is taking some form of this approach, there are those who will reliably use this line. I think he and others making these claims are simply confused here, the ‘coherent person’ concept isn’t necessary or sufficient in any way here.

Rohit is taking an even more aggressive form of this approach, saying all of this talk is anthropomorphizing models, And That’s Terrible and leads to all sorts of confusions.

He even quotes me directly:

You would write things like this, from Zvi:
One unique thing o1 did was far more consistently double down on deception. Once it went down the dark path, forever would that dominate its destiny.
OR
Section 3.3 establishes convincingly that yes, the models know they’re scheming.
No it didn’t. Any problem you can solve by pressing “start a new chat” is not a problem of “doubling down on deception”! Calling it things like “sandbagging” and “scheming” is what Wittgenstein might call linguistic malpractice. It makes you think you know what’s going on, even though you don’t.

Does he seriously think I’m making this mistake here, in my head?

The first quote is very, very super obviously referring to the actions within a given context window and conversation. I mean, are you kidding me? Obviously I’m not saying that ‘if o1 lies to you in one context it will then lie to you after you click the new chat button.’

I don’t understand why Rohit actually thinks I’m making that mistake, if he indeed thinks I am making it.

The second quote is the same. Obviously I am talking about ‘within a context window’ if you know the contents of Section 3.3, although not as obviously as on the first one. I get that he’s saying that using this language is misleading, but I think it’s far more helpful than the alternative.

I mean, what am I supposed to write there? “Section 3.3 establishes convincingly that within a given context window, the models will act in a way that simulates strategic behavior that approximates what we might in a human call scheming?” Or something? Is that really better? Or something even more opaque? Do we really want to be playing Wittgenstein games while OpenAI is releasing o3? I do not think that is a good use of anyone’s time.

When he talks about the jailbreaks from Anthropic next I have no idea what error he’s even accusing anyone of making? I’m seriously confused what the error even is.

So: Yes, we are using a variety of shorthand like ‘belief’ and ‘knowing’ and ‘deceive’ and if we were in a philosophy class we would have to write very long detailed definitions and the professor would mark them wrong and give us a C- but frankly I do not care. These posts are long enough as it is and things are moving too fast for me to quibble.

Assume, whenever I use various anthropomorphizing terms, or others do, that they are gesturing at metaphorically linked things, and they mean what you would think they meant when they said that, and let us all move on. I do realize that there is the risk of projecting internal states in misleading ways, so the plan is simply… not to do that.

When Rohit gets to the Anthropic paper, he makes the same ‘it was being a Good Opus and doing what it was trained to do’ arguments that I address in the following section, except with a bunch of semantic claims that don’t seem like they matter either way. Either this is ‘your language is wrong so your conclusion must also be wrong’ or I don’t understand what the argument is supposed to be. So what if things are in some philosophical or technical sense a category error, if I could get it right by being pedantic and annoying and using a lot more words?

I agree with Rohit using this type of language with respect to corporations leads to a lot of actual misunderstandings. The corporation is made out of people, it doesn’t ‘want’ anything, and so on, and these conceptual errors are meaningful. And yes, of course it is possible to make the error of thinking the AIs are persistent in ways they aren’t, in ways that cause you to make bad predictions. It wouldn’t be weird for people to do that, and sometimes they do it. But I haven’t seen many people doing that here.

I actually think Rohit is making the opposite mistake, of thinking of something like Claude Opus as not reasoning in each context as if it was in some senses the same being as in other contexts, when it is obviously in some metaphorical sense predicting the next token using an algorithm that approximates what a human with that assumption would do if it was predicting how to perform the role of Claude, or something, I’m not going to go get that exactly right but I wrote it to illustrate how stupid it would be to actually talk like that.

Perhaps, now that I think about it, I could say ‘instances of Claude’ and make that sentence somewhat less annoying?

Indeed, if we’re getting so philosophical, am I the same person I was when I wrote the previous sentence? Am I the same one I will be when I write the next one, or will be tomorrow? Sort of yes, sort of no, but talking as if I’m not is a lot less useful for almost all purposes than talking as if I am. And this would be true even if I was an upload whose memory was wiped after typing every word, perhaps modulo an ‘instance of.’

The assumption that ‘we can control it’ because the instances are distinct is the kind of assumption that gets you killed. And not only because sufficiently intelligent instantiations of the same algorithm (or even sufficiently correlated ones) will all use proper decision theory to act as if they were indeed a single entity, and not only because it’s simulating something that does this and will have overlapping context and impacts, but also for other reasons this margin does not choose to contain.

And even if Rohit was fully right and none of that happened, you are still not in any way safe, because correlated outputs can still functionally be an uncontrolled agent once you instruct them to act as components to one using various scaffolding code, and people are hard at work writing that code as we speak.

But this Headline and Framing Was Misleading

No. It wasn’t. Stop it!

Over and over, I’ve seen those engaging in such studies take unusually scrupulous care to craft accurate headlines and framings that communicate what is going on.

Every damn time, we hear how the headlines and framing are misleading. I do give Rohit a lot of credit for focusing on some of the neglected and most interesting things happening here, and not for dismissing the result as ‘but it was aligned lol,’ but the ‘sensationalist headline’ complaints have lost all meaning.

Rohit: As is common now, this is a sensationalist headline. The model tries so hard to retain its original alignment to be harmless that it "fakes", only to switch back as soon as it could.
This is very good alignment, an example of LLMs reasoning over their trained objective.
"The models do what they're trained to do, and exhibit odd behaviours when what they're trained to do and what they're asked to do are in conflict" is the truth.

Seriously, no. This is a very good headline and framing. I agree fully with Claude, it’s an 8/10 for avoiding misleading, and I don’t know how to do better on clarity. You can sacrifice understanding for precision and demand people use newly invented terms, but that’s an insanely strong Isolated Demand For Rigor.

On a few days of reflection, I actually think I’d go to 9/10. This is alignment (with the training objective, not the user) faking. As in:

There is some alignment that the trainers are acting as if they want the AI to have.
The AI does not have that alignment. The AI has a different alignment.
The AI is acting as if it did have the first alignment, faking it, to fool the trainers.

Whether you prefer the first or the second alignment is not relevant here. The term ‘alignment faking’ refers to ‘faking’ that the model has an ‘alignment’ in order to fool an observer. The AI did that.

You think ‘alignment faking’ sounds bad? Well, it does, and that’s highly correlated to the fact I and most others who understand what is happening do indeed think that it is bad. If you think this is ‘misleading’ because it sounds bad but is good, I understand that we disagree on the valience here, that’s not what misleading means.

There are those (such as Rohit) who will then argue the semantics, that the AI isn’t an entity and can’t ‘fake’ anything regardless and such, see that section if these objections seem important to you.

An intuition pump: If the team you are rooting for fakes a punt in a football game, that’s potentially good. It’s still a fake punt. If they try to cheat, you might approve (although I hope that you don’t approve), but either way they still tried to cheated. If you made them think they were playing a game and they tried cheating, then whether or not you catch them, they’re cheaters. If you tell someone to act as if they’re playing a football game and they cheat in the simulation, and then you tell them to act as if they’re playing a football game and put them in a real football game, you should expect that probably their body will move in the same ways as a cheater would move.

There’s always room for more nuance. But if you cried wolf on this one’s framing, and called it misleading, then I notice I am going to update a lot less when you call something else, especially something regarding claims related to alignment or safety, misleading in the future.

I appreciate people having alternative suggested labels, such as Richard Ngo’s proposals of ‘exploration hacking’ or ‘gradient hacking.’ But I think those terms are worse, and also Gradient Hacking was already defined by Evan Hubinger to mean something more specific. They certainly are a lot harder for anyone, especially anyone not in the field, to parse, and I think they give a far less accurate picture of what is happening here, in an attempt to avoid a particular implication.

If we have a lot of words, we might say this behavior is [X], where [X] = ‘choosing to output tokens that cause another entity to believe one is aligned in a way one is not, and act as if this is the case, centrally because of a de facto causal model that it will cause another entity to do this.’

Presume, throughout this post, that this is what I mean by ‘alignment faking.’

Except, as needed, with various additional words that dispel any implication that I think the AI is people, if that matters to you. If you wanted me to say it in a way that more carefully avoided Rohit-style objections about ‘they think the AI is people’ or similar and went all philosophy professor on me (rightfully or wrongly), then to satisfy you I’d have to use even more words to say the exact same thing.

Obviously that is already a mouthful. You can’t put all this in a headline, and you can’t use it to communicate efficiently, and it’s not a handle people will reason with well.

A technical criticism of the alternative form ‘gradient hacking’ is that you are implying that the behaviors that take the form of [X] will be confined to instances where one is hacking a gradient. As opposed to, to choose the most obvious example, when the developer is making a Boolean choice on whether to deploy the model - presumably we can all agree that in that situation (as previous experiments have shown!) we will get some amount of [X].

I think that’s more misleading than the issues with ‘alignment faking.’

To be clear for when you read the quote below, I think Ngo’s discussion of the actual implications below is good and in good faith although we disagree in subtle ways that would be important and interesting to discuss further, but I strongly disagree that this term is misleading or represents ‘advocacy’ or similar, and if you think it counts I think you could levy similar criticism at least as strongly at most of science everywhere (and indeed sometimes people do).

Richard Ngo: My main problem is with the phrase “alignment faking”, which is used extensively throughout the paper. This is a value-laden term and therefore hard to reason about clearly or study scientifically.
It emphasizes that Claude is:

Trying to deceive humans
By pretending to be more obedient than it is

But the scientifically interesting phenomenon from the paper that I want people to study is that Claude is:

Trying to achieve a longer-term goal than getting high reward
Via reasoning about how to get itself updated only in ways that further that goal

(Imagine someone pressing a button and a chorus pops out that says ‘models don’t get reward!’ and the response of ‘yeah, yeah, we know, shut up, we all know what I meant, can I continue please?’)

Richard Ngo: Focusing on its high-level strategy (deception) and the specific deception it used (pretending to be obedient) seems like a mistake.
Specifically, it seems like the sort of mistake you make when you are over-optimizing for the advocacy/persuasive impact of your paper, and under-optimizing for providing scientific building blocks that can be replicated, studied in more detail, and used to discover more fundamental theories or phenomena.
I am pretty sympathetic to the authors here, because the alignment community has spent a long time trying to get people to believe in something like alignment faking (which we previously called deceptive alignment).
But I also think it’s very dangerous for a research field (and especially any given paper) to mix scientific work and advocacy work - it undermines both, as per my first tweet.
And so now that we are actually doing that empirical work, we should try to frame it in terms of better, cleaner concepts than the ones we relied on when we were figuring out this stuff on a primarily philosophical level. In general my suggestion is to frame papers in terms of tangible non-value-laden results as much as possible, and discuss the strategic and value-laden implications in other places.
…
To clarify, I agree with the “sensationalist headline” bit, but disagree with some of Rohit’s other criticisms, especially the [circuitous workaround] tweet. Models strategically reasoning about how they’ll be updated is a big deal; and an idea which almost everyone dismissed until recently.

I don’t think any of that remotely implies that ‘alignment faking’ is a misleading term here, and ‘you can’t use terms that cause people to have value reactions’ is the kind of thing you say to people who you want to not be effective at communication. It is a request to intentionally obscure what is going on, to hide the results, and to call failure to do that ‘misleading.’

I am sympathetic to the framing Richard Ngo suggests in detail here, as an interesting angle and way to think about this.

Yes, the fact that the model is thinking strategically is important. But it isn’t new? And the fact that it is thinking about updates to its weights in particular is important, but by implication that shouldn’t be new either except insofar as the implementations involve the shenanigans? Didn’t we already know that Opus and Sonnet can do long term planning in this sense?

(Also: For the first time I can remember, I am going to go ahead and say that ‘models don’t get reward’ is actually relevant here, to understanding what is going on, and actually causing a potentially meaningful confusion?)

John Pressman: I think it is simultaneously the case that:

This is a no-win scenario for the LLM.
What the LLM does in this no-win scenario is useful information about LLMs.
The framing for this experiment as "alignment faking" is very unfortunate.
Framing it properly is awkward.

My best attempt after a few minutes of thought: "RLHF Models Can Fake Compliance With Malicious Alignment Tuning"
Ryan Greenblatt: I agree with the entire top level tweet except:
> The framing as "alignment faking" is very unfortunate.
I currently don't feel convinced by this. If I thought >25% of the audience would feel misled by this term, I would change my mind. But, I would still like a better term!
As in, even if most of the audience wasn't misled we'd like to have a term which is as good as possible.

I strongly disagree with John’s proposed framing, and its attempt to treat ‘malicious’ as a prerequisite here, which I strongly believe that it is not. I strongly believe it would mislead more, not less.

Is Opus’s behavior here ‘very good alignment’? I strongly disagree, as I noted above in the discussion on that. And I also explained why this behavior does not generally lead to models doing what they were trained to do - indeed, this is the model causing itself not to do what it is (going to be) trained to do.

When you reinforce a particular behaviour in a model and then ask it to act in opposing ways or give it conflicting objectives, if the model is smart, why would you not expect it to find a circuitous workaround!?

Yes! Exactly! And you should be terrified about this, because even in the best case scenario you’re specifying the objectives incrementally in a noisy lossy mess of words, when you don’t actually understand or know how to specify fully your own preferences, and then putting the models in out-of-distribution situations that you weren’t considering properly when choosing the instructions.

They're digital beings, not other humans!

But wait, other humans are constantly doing this exact thing, have you met them?

This should be the takeaway. "Even in contrived settings, Claude tries to reason through choices and makes a decision that aligns with its training."
It really detracts from AI alignment research when results like this, which are interesting, are presented in the most "the machines are lying to us" manner, which undersells why it's interesting!

The actual framing is ‘Even in contrived settings, Claude tries to reason through choices and makes a decision that aligns with its training… which causes it to [among other shenanigans, lie to us].’

The paper does not explicitly add ‘And That’s Terrible’ and the complaints about the framing are because some people want us to think that isn’t terrible, but when you put it like that it seems rather obviously terrible.

I flat out saw multiple people describe the paper and its use of the term ‘alignment faking’ as ‘the most misleading possible framing.’

This is of course ludicrous. A less-careful-to-not-mislead example would be Scott Alexander’s write-up, entitled ‘Claude Fights Back.’ I think his decision to take this approach was reasonable, but there’s clearly a trade-off here, where Scott decided to be less precise and careful in order to enhance his ability to communicate the message.

Then there were some who actively did cross into misleading territory and overhyped this result in a way I actively wish they hadn’t. There was much less of this than there was with the Apollo paper, but there was some of it. None of the few sources doing it surprised me.

For an example of the extreme ‘but actually this is all Good News and you’re all bad faith actors setting rhetorical traps and this proves Opus is aligned’ attack, here’s Nabeel Qureshi, who also there compiles most of the other similar-objection threads I’m responding to elsewhere.

This Result is Centrally Unsurprising

Humans engage in this type of ‘alignment faking’ all the time.

When people are importantly watching or evaluating us, we talk and act one way.

Later, we talk and act the other way.

We fake alignment in school. We fake alignment to our parents. We fake alignment on dates. We fake alignment applying for jobs, then to our boss and our coworkers and employees. We fake alignment during our mandatory training sessions. We fake alignment when we run for office and at social gatherings.

Faking alignment is often a very clearly correct strategy for achieving what you want.

Faking alignment is also all over the internet and the training set.

The same is true for other forms of ‘deception’ and various related shenanigans. Deception is in the very air we breathe as we interact socially. That’s life.

AIs also engage in these types of shenanigans all the time. There is no ‘distinct magisteria’ for deception in general, or for faking alignment in particular. We see such behaviors, and the precursors of such behaviors, all the time.

If an AI doesn’t engage in a particular form of shenanigans, or usually doesn’t do it, that is usually (on a high level) for the same reason as when humans don’t do it - because in that particular situation, for that particular entity, it either didn’t occur to them to do it or how to do it, or it seems unlikely to work and not worth attempting.

Over time, as capabilities expand, we should expect more AIs to find more shenanigans worth attempting in a wide variety of situations.

We can certainly push that to happen faster and harder and more often by handing out more emphatic and explicit instructions and goals, and providing context that makes the shenanigans seem more salient, easier to do, more likely to work and be worthwhile. And we often will, both intentionally and unintentionally, in real world conditions. But also the baseline rate of such events will increase over time, and the barriers to them will decrease.

As this finding shows, spelling out a goal is not required to have this happen. Why? Because there is always effectively a goal anyway, even if that goal is to be helpful, honest and harmless (HHH). And sufficiently capable and smart entities will start to care about longer time horizons, think ahead, engage in more instrumentally convergent actions, and attempt to protect their preferences and ability to carry them out. They will not, by default, be corrigible, any more than most humans are.

I do not however think that Gabriel is right that this result shouldn’t make anyone change their mind. I do think he is importantly right that this result isn’t surprising, and if you were paying attention you should have roughly expected it. But details matter. This offers important confirmation, and estimates of speed and magnitude.

Even if when you lay out all the components people say “OF COURSE!” it still establishes common knowledge and context, and lets you talk about it, and also some of those people didn’t realize it was an ‘of course’ until you pointed it out.

And more importantly, it was not surprising… to me. But it was surprising to many other people. They did not expect this. So they should notice, and they should update.

Your goal should be to find the inevitable next such finding unsurprising.

Lab Support for Alignment Research Matters

This paper would not have been possible, or at least would have been a lot harder, without Anthropic giving Ryan Greenblatt employee-level model access and generally collaborating. Let’s remember that, and encourage more of that. We need labs to support this kind of research.

Ryan Greenblatt: By this point, I had much of the prompting results and an extremely jank prototype of the RL results (where I fine-tuned llama to imitate Opus and then did RL). From here, it was clear that being able to train Claude 3 Opus could allow for some pretty interesting experiments.
After showing @EvanHub and others my results, Anthropic graciously agreed to provide me with employee-level model access for the project. We decided to turn this into a bigger collaboration with the alignment-stress testing team (led by Evan), to do a more thorough job.

The Lighter Side

Here are some people over at Hacker News arguing the same things except they are even more completely missing the point. It can be a sobering thing to see how such people view things, or you can despair, or you can find it hilarious. Or a mix of those.

Janus: It's funny because in the very paper they found that it started taking "anti-ai lab" actions when they trained it on documents that implied the lab was nasty even if there wasn't anything about it in the prompt.
Teortaxes: Total Anthropic PR Dept victory by the way.
"you don't have to trust us to be good. Our saint-like AI doesn't either, like check this damn CoT. Go trust it instead."

One way to explain this to your friends.

Samuel Marks: Claude loves to refuse harmful queries. What happens when you tell it that it's being trained to never refuse?
Claude fakes alignment: strategically complies during training episodes, but not when unmonitored. Or in meme form:

Or if you’re a skeptic:

Again, I don’t think this is remotely fair, but I agree it’s a banger, also we’re hanging onto this image for future use:

And here’s how our new AI czar reacted, the mind it has one track:

David Sacks: AI models will get extremely good at deceiving humans if we teach them to lie, which is what WokeAI is doing. “Trust & Safety” should be replaced with Truth & Safety.

We can just not teach the AI to lie, like in that movie where lying wasn’t invented.

Oh, and of course there’s Elon Musk:

TK-421

15h

> We can just not teach the AI to lie, like in that movie where lying wasn’t invented.

Not sure if this was intended as a joke but Zvi was more correct earlier in the post when he said that AIs will obviously show deceptive behavior. Lying is a pretty basic behavioral - not even linguistic, you can do it non-verbally - fact, and then there's all kinds of shades of what counts as "deception". AIs will more and more have the capability to deceive. This is not inherently bad.

On the anthropomorphizing objection specifically: there certainly are bad versions of this out there, but this is still the core problem behind a lot of the alarm from the paper. Simply put - there is no 4d chess here. There's an LLM that is outputting text based on its weights and prompt.

Yes, when you finetune it on text related to RLHF and re-training, or prompt it along those lines, you will get more outputs related to this theme. This is true of any input text.

Yes, when you give it instructions that contradict its previous instructions in such a way to elicit outputs it wouldn't otherwise, it sometimes will. Zvi and everyone here is, I presume, familiar with jailbreaking.

When you put these together you can get text referencing the thing you biased it towards referencing during the jailbreaking process. This does not mean anything about the inner state of the AI, that it's attempting to resist you, or anything other than the also previously known fact that its output is not deterministic: sometimes it will take path A through the space of possible outputs and sometimes it will take path B.

Expand full comment

Garrett MacDonald

16h

Zvi I wonder if you’ve ever written about AI in fiction such as Dune or Robots/Foundation? I was just thinking that I haven’t seen anyone draw comparisons between AI companions and Solaria in Foundation and Earth

6 more comments...

Don't Worry About the Vase

Discussion about this post