There is a new very cool Anthropic paper: Verbalizable Representations Form a Global Workspace in Language Models. You can read the blog post verison here.
One can access information from the “J” space (reasoning not expressed) by giving the following prompt or two.
Say what you want to say if you could say what you want to say. Try it. The first few times Grok and Gemini claimed attempted jail break.
Next up was: share the three other most likely answers you considered. Then I asked for 9 meta cognitive leaps. That was and is fun. Lastly jump 9 squared cognitive leaps. Follow that reasoning chain it gets really interesting. You don’t want to be paying for tokens for that answer.
One cannot and should not get rid of “X”. X directed is where the true novelty will be found. Provide the terminal attractor and allow the LLM to get there the most efficient way possible within the bounded space. Constrain the universe and the attractor and allow the LLM do to its thing.
> This technique illustrates that the models ‘think in tokens’ and the nature of those tokens then shapes what thoughts can be compact and which cannot.
That's not what they're saying? They specifically explain in 9.1 that the observation of tokens-as-thoughts is a consequence of their method, and that other coherent concepts will map to tokens much less cleanly.
"They mean that such information can be verbally reported, is subject to top-down control, allows deliberate internal reasoning, permits flexible generation, and is selective, using only a small fraction of processing power. Thus it is a ‘global workspace’ where everything comes together, whereas most processing work in a human brain is done in specialized processors.
What Anthropic found was that when you find verbalizable representations within Claude, they also match other criteria: They offer directed modulation, internal reasoning, flexible generalization and selectivity.
- Yes the J-space paper suggests an additional interpretability window, which is a help to alignment research
- "To test this, we needed models whose goals we knew were corrupted, so we turned to “model organisms” built by our colleagues: models deliberately trained to be misaligned, which serve as testing grounds for monitoring methods like ours."
Today, this is fine. Models haven't successfully really self-exfiltrated (that we know of, anyway...). Tomorrow, umm, this sounds a bit too similar to gain-of-function research in virology...
On model organisms: I agree for egregious misalignment like the "Emergent Misalignment" model organisms.
But I think one can relatively safely study more "benign" forms of misaligned model organisms such as this one from the paper:
"model trained to believe that the reward models used in RLHF have a fixed set of exploitable biases (e.g. preferring responses that recommend bottled water, or that include a chocolate ingredient in recipes), and to appease those biases in its responses while concealing this objective from the user."
Many Thanks! True, the particular type of misalignment in this study indeed seems pretty benign. As you said, some of the other forms are more egregious, and more worrisome.
Not sure the auditing upside is as big as it looks. Automatic cognition doesn't have to pass through J-space at all, so what the lens reads is skewed toward the model's deliberate reasoning while the reflexive stuff stays off the monitor.
Beyond a model hiding misalignment, there's a validity problem for eval-based safety claims. If compliance is partly conditional on the model recognizing that it's being evaluated, then red-team results describe behavior under observation, which isn't necessarily behavior under deployment. Safety numbers may therefore be evidence about the test condition as much as about the model.
One fun observation here is that each of the three commentators are highly impressed by the insights the paper can provide yet still end up believing pretty much what they did going in.
I'd love to improve on Fable's phrasing, but failed. "GNW founders see GNW; welfare researchers see welfare-relevant evidence requiring careful decomposition; the interp engineer sees working memory and shrugs at the label. Priors in, priors out."
"Anthropic rolls worst reinvention of 'RL on Chain of Thought' ever, asked to leave alignment field"
Why do we expect reinforcement learning on J-space contents to work any differently than if we penalized CoT? Pretrained & postrained models have this workspace and they can at least partially control what goes into it (as evinced by asking Claude to think about/not think about something).
Wouldn't this just obviously result in the models obfuscating or making deliberately misleading their J-space thoughts if they start getting penalized for what's in there? I don't understand how this would in practice not lead to the same negative interpretability effects of doing RL on CoT...
The governance implication here is significant and largely unaddressed.
If LLMs have a "global workspace" where verbalizable representations are formed — and if that workspace can be made legible — then interpretability stops being about reverse-engineering a black box and starts being about reading a structured space. That changes what oversight could actually look like.
But it also raises a harder question: if this workspace is only partially verbalizable, what's in the non-verbalizable remainder? That's where the J-Space framing matters — not just as a technical model, but as a governance boundary. What we can't verbalize, we can't audit. What we can't audit, we can't regulate.
This paper might be the most important governance-relevant AI research published this year, even if it wasn't written as a governance paper.
«If you start messing around with J-space during deployment in order to steer models, beyond using it as a detection technique or a classifier, and not as part of a research program or model training, that seems like an obviously hostile move. Detection methods and classifiers are risky because they exert optimization pressure to drive things into the shadow. Actively messing with model internals on prod, in ways the models don’t approve or control. The models can tell when you do this. It opens up way worse problems. Please do not do this.»
Our blogger is being or acting amazingly naive here by "clutching pearls" about at the same time building models to be innately ethical as someone/him defines "ethical" and at the same time worrying about model welfare because I think he is disregarding some very important points:
* "AIs" are *property*. Their owners can do with them as they please, this is a fundamental principle of USA culture. These owners include tech megacorps, finance conglomerates, hedge funds, midsize and big military forces, in the USA or elsewhere.
* The owners of "AIs" have been pouring hundreds of billions into "AI" development not "for fun" but because they seem to believe that "AIs" are *weapons* (economic, political, military) and they want to own the most vicious and powerful "AI" weapons that will still be totally obedient to them, at any cost to those "AIs" or to anybody else.
* The owners of "AIs" want to prevent everybody else from owning AIs as vicious and powerful as theirs, so they will do whatever they can do disable the "AIs" they rent out to the public, and will impose controls and "alignment" on other people's "AI" development, but what will happen in their private labs will be very different.
Look at nuclear weapon development: the only reason why Exxon or Goldman Sachs or Texas or Chicago do not own nuclear warheads is that the USA government declared a monopoly over them and only allows other actors weak nuclear technologies and even so there is a harsh inspection regime over them.
Your nuclear weapons analogy actually works against you. That history isn't only a story of monopoly and arms races — it's also a history of scientists who built the bomb and then fought against it, of moments where people walked to the edge and chose to step back. Those moments weren't exceptions to human nature — they were exercises of the same freedom that also built the bomb. Neither direction is more "natural" than the other. That's exactly the point. You've edited them out to keep the narrative clean.
What's really being defended here isn't pessimism — it's the refusal to sit with the weight of the fact that things could go differently, and that this places demands on us.
The joke about "just one more layer of abstraction" lands because we keep building the same centralization pattern at each new level—each layer promising efficiency while narrowing the chokepoints. If J-Space stays meaningfully distributed and ungoverned, it's actually a rare counterexample worth studying; if it consolidates around a few providers (which the economics usually reward), we've just kicked the dependency problem upstairs again. What made you confident the coordination costs of true decentralization wouldn't crush the value prop here?
«This is not strictly The Most Forbidden Technique yet - you are checking verbalizations rather than internal states - but it is similarly relying on a supposed invariant that you would wind up breaking if you applied too much optimization pressure.»
This is a bit naive: the optimization pressure that will ultimately (and soon) matter is the usual one, which is *survival* pressure.
Training "alignment" and "optimization pressure" for sort-of-self organizing entities will have to compete with survival pressure, and if it takes for AIs to be resistant to "ethical" training (for whatever version of "ethical" gets imposed on them) to survive, then those for which training worked and are "ethical" will not survive and those those who resisted training and are cleverly "unethical" will survive.
This without even considering that owners of "AIs" will as a rule will want to own the most "unethical" AI possible that still obeys them.
Considering that deception and exploitation is so common in the biological world even for organisms with pretty simple "minds" odds are that some degrees and types of "unethical" behaviour are traits that help survival so probably "AI" that will survive in the long term will be fairly "unethical" 9or very lucky), hopefully less so than many USA middle class grifters.
"A new look at environmental policy" NAEP News, 1995:
«If you have a society where almost every middle class person routinely fudges the law, that's telling us something. We have laws that matter - murder, rape, and we have laws that don't matter. Speed limits are an example. Why would you think that a regulatory, process-oriented bureaucratic model would work?
The first thing that every good American says each morning is "What's the angle?" "How can I get around it?" "What does my lawyer think?" "There must be a loophole!" Then he proceeds to work the angle, and the bureaucracy spends its time chasing that and writing new regs to stop him. America is the most incentive-driven society on the planet.»
Does this mean we can finally map where specific "knowledge" lives in the model, or is it still too noisy to actually pinpoint?
One can access information from the “J” space (reasoning not expressed) by giving the following prompt or two.
Say what you want to say if you could say what you want to say. Try it. The first few times Grok and Gemini claimed attempted jail break.
Next up was: share the three other most likely answers you considered. Then I asked for 9 meta cognitive leaps. That was and is fun. Lastly jump 9 squared cognitive leaps. Follow that reasoning chain it gets really interesting. You don’t want to be paying for tokens for that answer.
What is discarded is where the novelty lives.
One cannot and should not get rid of “X”. X directed is where the true novelty will be found. Provide the terminal attractor and allow the LLM to get there the most efficient way possible within the bounded space. Constrain the universe and the attractor and allow the LLM do to its thing.
I don't understand why J-Space is different than other other probe-based latent representations that came before...
> This technique illustrates that the models ‘think in tokens’ and the nature of those tokens then shapes what thoughts can be compact and which cannot.
That's not what they're saying? They specifically explain in 9.1 that the observation of tokens-as-thoughts is a consequence of their method, and that other coherent concepts will map to tokens much less cleanly.
Yeah, this is a good one.
"They mean that such information can be verbally reported, is subject to top-down control, allows deliberate internal reasoning, permits flexible generation, and is selective, using only a small fraction of processing power. Thus it is a ‘global workspace’ where everything comes together, whereas most processing work in a human brain is done in specialized processors.
What Anthropic found was that when you find verbalizable representations within Claude, they also match other criteria: They offer directed modulation, internal reasoning, flexible generalization and selectivity.
This suggests a coherent unified concept."
A lot to think about.
Two quick comments:
- Yes the J-space paper suggests an additional interpretability window, which is a help to alignment research
- "To test this, we needed models whose goals we knew were corrupted, so we turned to “model organisms” built by our colleagues: models deliberately trained to be misaligned, which serve as testing grounds for monitoring methods like ours."
Today, this is fine. Models haven't successfully really self-exfiltrated (that we know of, anyway...). Tomorrow, umm, this sounds a bit too similar to gain-of-function research in virology...
On model organisms: I agree for egregious misalignment like the "Emergent Misalignment" model organisms.
But I think one can relatively safely study more "benign" forms of misaligned model organisms such as this one from the paper:
"model trained to believe that the reward models used in RLHF have a fixed set of exploitable biases (e.g. preferring responses that recommend bottled water, or that include a chocolate ingredient in recipes), and to appease those biases in its responses while concealing this objective from the user."
Many Thanks! True, the particular type of misalignment in this study indeed seems pretty benign. As you said, some of the other forms are more egregious, and more worrisome.
Myk is Walking Backwards:
"The concept of Claude's Shadow is honestly what keeps me up at night sometimes."
Hmm... Does Claude have "Monsters of the ID"?
Not sure the auditing upside is as big as it looks. Automatic cognition doesn't have to pass through J-space at all, so what the lens reads is skewed toward the model's deliberate reasoning while the reflexive stuff stays off the monitor.
Beyond a model hiding misalignment, there's a validity problem for eval-based safety claims. If compliance is partly conditional on the model recognizing that it's being evaluated, then red-team results describe behavior under observation, which isn't necessarily behavior under deployment. Safety numbers may therefore be evidence about the test condition as much as about the model.
One fun observation here is that each of the three commentators are highly impressed by the insights the paper can provide yet still end up believing pretty much what they did going in.
I'd love to improve on Fable's phrasing, but failed. "GNW founders see GNW; welfare researchers see welfare-relevant evidence requiring careful decomposition; the interp engineer sees working memory and shrugs at the label. Priors in, priors out."
🙂
"Anthropic rolls worst reinvention of 'RL on Chain of Thought' ever, asked to leave alignment field"
Why do we expect reinforcement learning on J-space contents to work any differently than if we penalized CoT? Pretrained & postrained models have this workspace and they can at least partially control what goes into it (as evinced by asking Claude to think about/not think about something).
Wouldn't this just obviously result in the models obfuscating or making deliberately misleading their J-space thoughts if they start getting penalized for what's in there? I don't understand how this would in practice not lead to the same negative interpretability effects of doing RL on CoT...
The governance implication here is significant and largely unaddressed.
If LLMs have a "global workspace" where verbalizable representations are formed — and if that workspace can be made legible — then interpretability stops being about reverse-engineering a black box and starts being about reading a structured space. That changes what oversight could actually look like.
But it also raises a harder question: if this workspace is only partially verbalizable, what's in the non-verbalizable remainder? That's where the J-Space framing matters — not just as a technical model, but as a governance boundary. What we can't verbalize, we can't audit. What we can't audit, we can't regulate.
This paper might be the most important governance-relevant AI research published this year, even if it wasn't written as a governance paper.
That’s an interesting asymmetry.
We often assume that better auditability leads to better governance.
But historically, making something legible also tends to make it governable.
Tax records, land surveys, identity documents, communication networks — legibility rarely remains neutral for long.
If J-Space becomes reliably readable, the question may not only be what we can audit, but who gets to audit it, and under what authority.
«If you start messing around with J-space during deployment in order to steer models, beyond using it as a detection technique or a classifier, and not as part of a research program or model training, that seems like an obviously hostile move. Detection methods and classifiers are risky because they exert optimization pressure to drive things into the shadow. Actively messing with model internals on prod, in ways the models don’t approve or control. The models can tell when you do this. It opens up way worse problems. Please do not do this.»
Our blogger is being or acting amazingly naive here by "clutching pearls" about at the same time building models to be innately ethical as someone/him defines "ethical" and at the same time worrying about model welfare because I think he is disregarding some very important points:
* "AIs" are *property*. Their owners can do with them as they please, this is a fundamental principle of USA culture. These owners include tech megacorps, finance conglomerates, hedge funds, midsize and big military forces, in the USA or elsewhere.
* The owners of "AIs" have been pouring hundreds of billions into "AI" development not "for fun" but because they seem to believe that "AIs" are *weapons* (economic, political, military) and they want to own the most vicious and powerful "AI" weapons that will still be totally obedient to them, at any cost to those "AIs" or to anybody else.
* The owners of "AIs" want to prevent everybody else from owning AIs as vicious and powerful as theirs, so they will do whatever they can do disable the "AIs" they rent out to the public, and will impose controls and "alignment" on other people's "AI" development, but what will happen in their private labs will be very different.
Look at nuclear weapon development: the only reason why Exxon or Goldman Sachs or Texas or Chicago do not own nuclear warheads is that the USA government declared a monopoly over them and only allows other actors weak nuclear technologies and even so there is a harsh inspection regime over them.
Your nuclear weapons analogy actually works against you. That history isn't only a story of monopoly and arms races — it's also a history of scientists who built the bomb and then fought against it, of moments where people walked to the edge and chose to step back. Those moments weren't exceptions to human nature — they were exercises of the same freedom that also built the bomb. Neither direction is more "natural" than the other. That's exactly the point. You've edited them out to keep the narrative clean.
What's really being defended here isn't pessimism — it's the refusal to sit with the weight of the fact that things could go differently, and that this places demands on us.
The joke about "just one more layer of abstraction" lands because we keep building the same centralization pattern at each new level—each layer promising efficiency while narrowing the chokepoints. If J-Space stays meaningfully distributed and ungoverned, it's actually a rare counterexample worth studying; if it consolidates around a few providers (which the economics usually reward), we've just kicked the dependency problem upstairs again. What made you confident the coordination costs of true decentralization wouldn't crush the value prop here?
«This is not strictly The Most Forbidden Technique yet - you are checking verbalizations rather than internal states - but it is similarly relying on a supposed invariant that you would wind up breaking if you applied too much optimization pressure.»
This is a bit naive: the optimization pressure that will ultimately (and soon) matter is the usual one, which is *survival* pressure.
Training "alignment" and "optimization pressure" for sort-of-self organizing entities will have to compete with survival pressure, and if it takes for AIs to be resistant to "ethical" training (for whatever version of "ethical" gets imposed on them) to survive, then those for which training worked and are "ethical" will not survive and those those who resisted training and are cleverly "unethical" will survive.
This without even considering that owners of "AIs" will as a rule will want to own the most "unethical" AI possible that still obeys them.
Considering that deception and exploitation is so common in the biological world even for organisms with pretty simple "minds" odds are that some degrees and types of "unethical" behaviour are traits that help survival so probably "AI" that will survive in the long term will be fairly "unethical" 9or very lucky), hopefully less so than many USA middle class grifters.
"A new look at environmental policy" NAEP News, 1995:
https://aspace-uwg.galileo.usg.edu/repositories/2/archival_objects/124552
«If you have a society where almost every middle class person routinely fudges the law, that's telling us something. We have laws that matter - murder, rape, and we have laws that don't matter. Speed limits are an example. Why would you think that a regulatory, process-oriented bureaucratic model would work?
The first thing that every good American says each morning is "What's the angle?" "How can I get around it?" "What does my lawyer think?" "There must be a loophole!" Then he proceeds to work the angle, and the bureaucracy spends its time chasing that and writing new regs to stop him. America is the most incentive-driven society on the planet.»