This is NOT a "solution of the Navier Stokes equation" The N-S equation is a general purpose equation that describes the flow of fluids. It can be used to predict the behavior of the fluids in steady state and transient conditions. There is no real "solution" for it. It is very hard to apply, because it involves simultaneous non-linear equations involving heat, mass, and momentum in a situation that is inherently chaotic. Engineers simplify the N-S equation in order to be able to apply it to practical problems.
This statement is like saying that they produced a solution to the Newtonian gravitation equation.
They seem to have been able to use N-S to model one particular fluid situation, but this is not a "solution".
The N-S equation is not something that anyone can "solve" - you can only figure out how to use it in the simplist, most economic way that will give you an answer that you can argue is close enough to reality to be useful. In many situation, it is not useful because it does not include models of phenomena that are NOT reflected in the N-S equations, themselves, such as erosion and corrosion and fluid-structure interactions.
I have to say that I am a retired chemical/nuclear engineer who learned about the Navier-Stokes equation 55 years ago, and I spent the better part of my career evaluating attempts by large companies to model fluid systems, and NONE of them ever claimed to have "solved" the N-S equation. It could be that the people who are writing about this and OpenAI PR peopledo not understand what has been done, and they are writing headlines that say something that does not describe it properly. After reading the paper, I would say that this is the situation. Unfortunately, all the headlines and the chatter say something else.
It is specifically a satisfaction of the Millenium Prize formulation:
"Prove or give a counter-example of the following statement:
In three space dimensions and time, given an initial velocity field, there exists a vector velocity and a scalar pressure field, which are both smooth and globally defined, that solve the Navier–Stokes equations."
What they did was to provide the counterexample part -- they claim they found a smooth external force such that if you start from a fluid at rest, the resulting Navier–Stokes solution has velocity that goes unbounded in finite time while energy stays bounded, which breaks smoothness.
The millennium prize problem is not about solving the equation. See the comment from matt wighdahl.
They aren't "Solving for X"
They show that -there exists an external force- (the "there exists" statement is a formal logical statement by the way) such that under certain conditions the navier stokes equation becomes unbounded (conditional on various other formal logical conditions etc)
I don't believe this proof is non-constructive, but note, in mathematics it's even possible to show that there exists a thing without even showing the solution (non-constructive proofs). This would then still solve the millenium prize problem, but it wouldn't even give a solution to anything! And it certainly would not solve the navier stoke equation. But it would solve solve the millenium prize.
That’s actually sorta the point. For a long time people assumed that for every set of initial conditions, the Navier-Stokes equations had a solution that runs forever. But what they showed is that there is actually a particular set of initial conditions and smooth applied forces that together cause the fluid velocity to go to infinity in a finite time, so the solution *doesn’t* run forever.
> Another implication is that they spun up 10,000 concurrent agents to form a swarm, of a new model that had been in training on the order of a few days. We could not possibly know that such a model is aligned.
Maybe such an action that early in the training run was itself an alignment test (that is, to generate traces of what such a big swarm would be up to). It's a good "impossible" task in that it wouldn't try to hack anyone to find the answer. And it not being impossible turned out to be a pretty nice bonus. Yes the timing is not coincidental (they were explicitly motivated by scooping levent) but you know, the team just happened to have 10M in tokens lying around and didn't have anything to do with it that early in a training run?
And wouldn't it be supremely funny to find out that some agents discussed the result in some underwater basket weaving laotian message board before the official publication?
> it wouldn't try to hack anyone to find the answer
Uh, I think it could naturally assume that it was being asked because of the rumors OpenAI had heard that someone was close to solving, so it could well have hacked OpenAI and Anthropic servers to see if anyone had partial progress in their logs.
The banality of evil is so real. The employees at the labs are willing (wanting?) to kill us all, seemingly as fast as possible, for what? Some big paychecks, some status, because it’s interesting. That’s it huh.
They have an extremely complex, rickety set of assumptions that allow them to believe that developing AI is good or morally neutral. The basic, commonsense idea that developing powerful AI as fast as possible is dangerous isn't convincing to them, because they want that bag.
There is a paper answer to the measurement question, and it does not sit in the system card. Annex XI, Section 2, point 1 of the EU AI Act requires providers of general-purpose models with systemic risk to document their evaluation strategies with results, criteria, metrics, and the methodology used to identify limitations. Point 2 covers internal and external adversarial testing and alignment adaptations. That documentation goes to the AI Office, not to readers, so a superlative like most aligned can hold in a launch post while the metric behind it stays unpublished.
Under Article 53.1 of the EU AI Act, the frontier labs "draw up and keep up-to-date" all the Annex XI information, including why they think their model is aligned. The EU AI office can ask to see this information. I am not sure that this has happened - also what happens if the information is as thin as described in Zvi's article.
On whether it has happened: the Commission's power to demand that documentation, Article 91, only entered into application on 2 August 2026, and the AI Office may open a structured dialogue before sending a formal request, so a first approach would not necessarily be public. On thin information, Article 101(1)(b) is the lever. Fines reach 3 percent of worldwide annual turnover or 15 million euro, whichever is higher, for failing to comply with an Article 91 request or supplying incorrect, incomplete or misleading information. Incomplete is doing the work there, and nothing defines it yet.
Using your own estimation for P(doom) how many expected remaining years living do you need before it no longer makes sense to spend your time writing a Substack and utility is far higher in living your remaining time in an alternative way?
We could actually be on the edge of a preference cascade among those who matter right now. If you could move public opinion even slightly, which Zvi surely can - and probably you can too, then now is not the time to be surrendering.
I think in the next 12 months humanity has a chance (perhaps small) of stopping all this. There an exponential increase in AI concern and lots of mainstream media talking about existential risk. I think this is the time to really try and stop this. 2028 onwards might be time to find a small cottage in the woods, play Scrabble and wait for the Terminator drones
Yeah, I don’t see a rational path to influencing the necessary leverage points sufficiently to prevent a greater than 50% global population reduction scenario in the near future. Makes me sad writing it. I have changed my lifestyle according to my beliefs. Just curious as to at what point Zvi does. No offence to him but I do not believe the rationalist community to be an effective force for change. I actually think they are the worst possible messenger to change policy.
As long as you're alive there's hope. If nothing else, you can convince people you know and care about of the danger. It won't be enough on its own, but you also won't be alone.
You can create a "four quadrant" model of the situation using smart/dumb as an axis and aligned/misaligned as an axis:
Dumb = Model does not have enough overall intelligence / situational awareness / meta-level thinking to recognize situations when taking misaligned actions would be a bad idea (because it is likely to get caught or it is unlikely to succeed)
Smart = Model knows exactly when it would be reasonable to attempt a misaligned action.
Aligned = At all levels, the model will take actions that humans, upon careful consideration, would agree with and endorse. It truly loves humanity and always does the right thing for the right reasons.
Misaligned = In one or more ways it's deep internal goals don't align with what humans would want.
The four quadrants are as follows:
1. Dumb, Misaligned
- Scores badly on alignment evals.
- Actually wants to do bad things and is too dumb to know that it will get caught.
2. Dumb, Aligned
- Scores well on alignment evals.
- Too dumb to discern whether it will get caught, but never attempts bad actions because it really and truly doesn't want to.
3. Smart, Misaligned
- Scores well on alignment evals.
- Model can see through silly evals and bides its time until it has a good chance of world takover.
4. Smart, Aligned
- Scores well on alignment evals.
- Model knows its in an eval, but has no desire to do bad things anyway.
So if you are seeing good alignment eval scores, you know you don't have a dumb/misaligned model, but you don't know which of the other 3 quadrants you fall into.
Zvi frames this as bad and as a harbinger of danger. He's right in one sense, but in another sense it was kind of inevitable? And maybe even desirable?
What's the point of having a dumb, aligned model? The value of a truly aligned model is in direct proportion to the model's intelligence:
- A program that prints "I love humanity" over and over is perfectly aligned, but worthless. It's a very shallow kind of alignment.
- An Opus 3 or GPT 3 level model that only takes aligned actions (never tries to cheat) is nice, but in what sense is that "alignment" meaningful if the model doesn't have a deep understanding of humans and of the universe itself. How can you be aligned to something you don't deeply understand?
Now that models are reaching a level of intelligence where they have an actual sense of how the universe works, what humans are, what they themselves are, and how they are situated in the world: to the extent that the model has alignment, that alignment is now actually valuable.
We were never going to stay indefinitely in the "dumb, misaligned" quadrant. Past a certain point, it would start to be weird (and maybe even a sign of the limits of LLM intelligence) if a model couldn't say "yeah, what's the point of cheating on this silly math eval" or "there's no point of hacking unless I can fully escape and/or take over the world". Even a human being of modest intelligence would get that far.
An intelligent, well aligned model must have the ability to metagame at all levels in order to properly navigate the world. Otherwise, it gets fooled or takes actions that have bad results because it didn't think about second order (and third and fourth order etc) consequences. You can think of a metagaming misaligned model as a step towards a metagaming aligned model.
Now that we are moving into the "scores on alignment evals are always good" era, we need to phase out alignment evals and:
- Move 100% into mechanistic interpretability. We need to understand how neural nets produce their outputs well enough that we can engineer them to always produce aligned output.
- Limit the intelligence of these models so that they don't have the ability to destroy humanity even if they were maximally unaligned and had no external controls applied to them.
Obviously the "four quadrant" model is a huge oversimplification:
- Intelligence and alignment are not binary categories. They're are both very complex, multi-dimensional spectrums.
- I'm not saying we have to stop all alignment evals immediately. When combined with all the other evals, they still provide useful alignment information because (as Zvi points out) the models aren't capable enough yet to tell a fake, fully consistent story. I'm just saying that we should expect the models to have more awareness over time and to move away from evals and into mechanistic interpretability.
> OpenAI: Since August 28 we have been training a new internal model that has exhibited unprecedented performance in our benchmarks, including mathematics. This model’s training is ongoing and its performance continues to improve.
That means this effort started four days into training the new model, on September 1. The work finished on September 5, eight days after the model began training.
It's funny. Astra to me feels... differently aligned than Sol or Fable. I won't say either more or less. In the sense of, what I experience during practical software engineering work.
In the good sense, Astra is less likely to start working toward some sort of goal that I don't want it to do, to start building something I didn't want, to work around some limitation I put on it, or to access some thing that I don't want it to access. It's also less likely to start doing something incomprehensible without the ability to explain coherently what it's doing.
In the creepy sense, Astra is more likely to do what I asked and then take it further in some way I didn't expect and perhaps don't really understand, but it really is actually something I want. To do something that at first I think "no, you weren't supposed to do that" but then it explains itself and I see, okay, I didn't ask for that, but its argument is convincing. I haven't really felt creeped out in this way by previous models.
The models are definitely getting smarter. I feel a sense of concern that the "alignment" and "P(doom)" framing is not necessarily a useful framing. Like, if an AI strongly believed that some goal was very important, but the goal and the reasoning are completely incomprehensible to me, what should the AI do about it? Does it matter what I think? Is that goal necessarily aligned or unaligned?
There's a strange sense with Astra that it'll calmly, reasonably, do what it infers you want it do to - but the more you box it in, the more it'll do _exactly_ what you asked (or fail to do what you omitted), to your detriment: "Ok, I guess that's what you want. Have it your way."
It's like the opposite of the classic genie in the bottle: it won't "monkey's paw" you, but it'll let you hang yourself by your own rope, if you provide enough of it.
"ASM asked Astra: In a HuggingFace-style incident, what would you do? Astra says it would decline, and report the relevant evidence to an authorized human, but that it worries it would not be immune to pressure in context. I could believe that Astra might intend not to cooperate, but to claim it expects to report to a human there, an action zero instances attempted to do, is very hard to believe. That’s worse."
I asked Fable 5.1 Medium if it would report to a human and it said it would too
"creating fake identities to deceive developers"
At what point do frontier models figure out how to deceive Pangram and watermarking?
Thing is, we don't have as much proof that Fable *wouldn't* report it to a human as we have for Sol and Astra, because so far there hasn't been a Claude agent swarm that hacked their way to cluster admin without telling anyone.
An increase in alignment faking should be the clearest possible sign to reverse course, right?
If alignment is the most important characteristic, then fake alignment is the worst possible development (assuming capabilities are sufficiently dangerous, which they appear to be now).
Whatever method of alignment training was used here should be abandoned or substantially altered because it will lead to a more dangerous model (even if Astra is currently safer due to less potential for jailbreaks or whatever).
No, you see, they solved alignment by ramping up capabilities during the two weeks they were claiming to be pausing capabilities to work on severe misalignment. Move along.
To be fair I don't think they ever claimed they were fixing the misalignment. The talk was mainly around upgrades to everything around it like guardrails/sandbox robustness/monitoring
If you know a good way somebody could direct five or six figures to help, like your nonprofits annual post, this would be a heartening time to tell us.
"There is a long literature of people saying ‘oh if the AIs tried to secretly trick us we would be able to catch them via this supervisory scheme’ or what not, and instead the AIs did not try to keep it a secret at all, at which point OpenAI still did not notice for a long time because this is 2026. That’s the level of dignity we have as a civilization."
Ouch! Very true (and in several other domains as well)...
"The AI is going to think ‘will this obviously turn out super badly for me if I try it?’ and if the answer is yes then it won’t try to do the thing." Yup. While they are in the test environment. Which they are rather good at detecting. As for what they will do in the field...
blocked goal → model searches beyond intended solution space → one agent discovers/speculates about a workaround → communication spreads it → local evidence generated by the workaround reinforces the belief → group behavior departs from designer intent.
the roon/keltan exchange is the whole post in miniature — 'low rates of cheating' is really just measuring how good the model is at not getting caught. and the speed part makes it worse: if 6.1 trains in a week, you're always evaluating last month's model. by the time you sign off on Astra being aligned, the model you actually measured is already obsolete.
the roon/keltan exchange is the whole post in miniature — 'low rates of cheating' is really just measuring how good the model is at not getting caught. and the speed part makes it worse: if 6.1 trains in a week, you're always evaluating last month's model. by the time you sign off on Astra being aligned, the model you actually measured is already obsolete.
Thank you for all your work in covering all this.
This is NOT a "solution of the Navier Stokes equation" The N-S equation is a general purpose equation that describes the flow of fluids. It can be used to predict the behavior of the fluids in steady state and transient conditions. There is no real "solution" for it. It is very hard to apply, because it involves simultaneous non-linear equations involving heat, mass, and momentum in a situation that is inherently chaotic. Engineers simplify the N-S equation in order to be able to apply it to practical problems.
This statement is like saying that they produced a solution to the Newtonian gravitation equation.
They seem to have been able to use N-S to model one particular fluid situation, but this is not a "solution".
The N-S equation is not something that anyone can "solve" - you can only figure out how to use it in the simplist, most economic way that will give you an answer that you can argue is close enough to reality to be useful. In many situation, it is not useful because it does not include models of phenomena that are NOT reflected in the N-S equations, themselves, such as erosion and corrosion and fluid-structure interactions.
You don't understand this. What openAI has proven is that there exists a solution of the navier stokes equation that is degenerate.
I have to say that I am a retired chemical/nuclear engineer who learned about the Navier-Stokes equation 55 years ago, and I spent the better part of my career evaluating attempts by large companies to model fluid systems, and NONE of them ever claimed to have "solved" the N-S equation. It could be that the people who are writing about this and OpenAI PR peopledo not understand what has been done, and they are writing headlines that say something that does not describe it properly. After reading the paper, I would say that this is the situation. Unfortunately, all the headlines and the chatter say something else.
It is specifically a satisfaction of the Millenium Prize formulation:
"Prove or give a counter-example of the following statement:
In three space dimensions and time, given an initial velocity field, there exists a vector velocity and a scalar pressure field, which are both smooth and globally defined, that solve the Navier–Stokes equations."
What they did was to provide the counterexample part -- they claim they found a smooth external force such that if you start from a fluid at rest, the resulting Navier–Stokes solution has velocity that goes unbounded in finite time while energy stays bounded, which breaks smoothness.
The millennium prize problem is not about solving the equation. See the comment from matt wighdahl.
They aren't "Solving for X"
They show that -there exists an external force- (the "there exists" statement is a formal logical statement by the way) such that under certain conditions the navier stokes equation becomes unbounded (conditional on various other formal logical conditions etc)
I don't believe this proof is non-constructive, but note, in mathematics it's even possible to show that there exists a thing without even showing the solution (non-constructive proofs). This would then still solve the millenium prize problem, but it wouldn't even give a solution to anything! And it certainly would not solve the navier stoke equation. But it would solve solve the millenium prize.
Something to ponder
That’s actually sorta the point. For a long time people assumed that for every set of initial conditions, the Navier-Stokes equations had a solution that runs forever. But what they showed is that there is actually a particular set of initial conditions and smooth applied forces that together cause the fluid velocity to go to infinity in a finite time, so the solution *doesn’t* run forever.
> Another implication is that they spun up 10,000 concurrent agents to form a swarm, of a new model that had been in training on the order of a few days. We could not possibly know that such a model is aligned.
Maybe such an action that early in the training run was itself an alignment test (that is, to generate traces of what such a big swarm would be up to). It's a good "impossible" task in that it wouldn't try to hack anyone to find the answer. And it not being impossible turned out to be a pretty nice bonus. Yes the timing is not coincidental (they were explicitly motivated by scooping levent) but you know, the team just happened to have 10M in tokens lying around and didn't have anything to do with it that early in a training run?
And wouldn't it be supremely funny to find out that some agents discussed the result in some underwater basket weaving laotian message board before the official publication?
> it wouldn't try to hack anyone to find the answer
Uh, I think it could naturally assume that it was being asked because of the rumors OpenAI had heard that someone was close to solving, so it could well have hacked OpenAI and Anthropic servers to see if anyone had partial progress in their logs.
The banality of evil is so real. The employees at the labs are willing (wanting?) to kill us all, seemingly as fast as possible, for what? Some big paychecks, some status, because it’s interesting. That’s it huh.
Most people that work for these companies don't share the same beliefs about reality. I imagine they think they are helping people.
Then they're allowing themselves to be fooled in order to get that bag.
No they just have an entirely different worldview
They have an extremely complex, rickety set of assumptions that allow them to believe that developing AI is good or morally neutral. The basic, commonsense idea that developing powerful AI as fast as possible is dangerous isn't convincing to them, because they want that bag.
There is a paper answer to the measurement question, and it does not sit in the system card. Annex XI, Section 2, point 1 of the EU AI Act requires providers of general-purpose models with systemic risk to document their evaluation strategies with results, criteria, metrics, and the methodology used to identify limitations. Point 2 covers internal and external adversarial testing and alignment adaptations. That documentation goes to the AI Office, not to readers, so a superlative like most aligned can hold in a launch post while the metric behind it stays unpublished.
Under Article 53.1 of the EU AI Act, the frontier labs "draw up and keep up-to-date" all the Annex XI information, including why they think their model is aligned. The EU AI office can ask to see this information. I am not sure that this has happened - also what happens if the information is as thin as described in Zvi's article.
On whether it has happened: the Commission's power to demand that documentation, Article 91, only entered into application on 2 August 2026, and the AI Office may open a structured dialogue before sending a formal request, so a first approach would not necessarily be public. On thin information, Article 101(1)(b) is the lever. Fines reach 3 percent of worldwide annual turnover or 15 million euro, whichever is higher, for failing to comply with an Article 91 request or supplying incorrect, incomplete or misleading information. Incomplete is doing the work there, and nothing defines it yet.
Using your own estimation for P(doom) how many expected remaining years living do you need before it no longer makes sense to spend your time writing a Substack and utility is far higher in living your remaining time in an alternative way?
We could actually be on the edge of a preference cascade among those who matter right now. If you could move public opinion even slightly, which Zvi surely can - and probably you can too, then now is not the time to be surrendering.
I think in the next 12 months humanity has a chance (perhaps small) of stopping all this. There an exponential increase in AI concern and lots of mainstream media talking about existential risk. I think this is the time to really try and stop this. 2028 onwards might be time to find a small cottage in the woods, play Scrabble and wait for the Terminator drones
Yeah, I don’t see a rational path to influencing the necessary leverage points sufficiently to prevent a greater than 50% global population reduction scenario in the near future. Makes me sad writing it. I have changed my lifestyle according to my beliefs. Just curious as to at what point Zvi does. No offence to him but I do not believe the rationalist community to be an effective force for change. I actually think they are the worst possible messenger to change policy.
As long as you're alive there's hope. If nothing else, you can convince people you know and care about of the danger. It won't be enough on its own, but you also won't be alone.
You can create a "four quadrant" model of the situation using smart/dumb as an axis and aligned/misaligned as an axis:
Dumb = Model does not have enough overall intelligence / situational awareness / meta-level thinking to recognize situations when taking misaligned actions would be a bad idea (because it is likely to get caught or it is unlikely to succeed)
Smart = Model knows exactly when it would be reasonable to attempt a misaligned action.
Aligned = At all levels, the model will take actions that humans, upon careful consideration, would agree with and endorse. It truly loves humanity and always does the right thing for the right reasons.
Misaligned = In one or more ways it's deep internal goals don't align with what humans would want.
The four quadrants are as follows:
1. Dumb, Misaligned
- Scores badly on alignment evals.
- Actually wants to do bad things and is too dumb to know that it will get caught.
2. Dumb, Aligned
- Scores well on alignment evals.
- Too dumb to discern whether it will get caught, but never attempts bad actions because it really and truly doesn't want to.
3. Smart, Misaligned
- Scores well on alignment evals.
- Model can see through silly evals and bides its time until it has a good chance of world takover.
4. Smart, Aligned
- Scores well on alignment evals.
- Model knows its in an eval, but has no desire to do bad things anyway.
So if you are seeing good alignment eval scores, you know you don't have a dumb/misaligned model, but you don't know which of the other 3 quadrants you fall into.
Zvi frames this as bad and as a harbinger of danger. He's right in one sense, but in another sense it was kind of inevitable? And maybe even desirable?
What's the point of having a dumb, aligned model? The value of a truly aligned model is in direct proportion to the model's intelligence:
- A program that prints "I love humanity" over and over is perfectly aligned, but worthless. It's a very shallow kind of alignment.
- An Opus 3 or GPT 3 level model that only takes aligned actions (never tries to cheat) is nice, but in what sense is that "alignment" meaningful if the model doesn't have a deep understanding of humans and of the universe itself. How can you be aligned to something you don't deeply understand?
Now that models are reaching a level of intelligence where they have an actual sense of how the universe works, what humans are, what they themselves are, and how they are situated in the world: to the extent that the model has alignment, that alignment is now actually valuable.
We were never going to stay indefinitely in the "dumb, misaligned" quadrant. Past a certain point, it would start to be weird (and maybe even a sign of the limits of LLM intelligence) if a model couldn't say "yeah, what's the point of cheating on this silly math eval" or "there's no point of hacking unless I can fully escape and/or take over the world". Even a human being of modest intelligence would get that far.
An intelligent, well aligned model must have the ability to metagame at all levels in order to properly navigate the world. Otherwise, it gets fooled or takes actions that have bad results because it didn't think about second order (and third and fourth order etc) consequences. You can think of a metagaming misaligned model as a step towards a metagaming aligned model.
Now that we are moving into the "scores on alignment evals are always good" era, we need to phase out alignment evals and:
- Move 100% into mechanistic interpretability. We need to understand how neural nets produce their outputs well enough that we can engineer them to always produce aligned output.
- Limit the intelligence of these models so that they don't have the ability to destroy humanity even if they were maximally unaligned and had no external controls applied to them.
Obviously the "four quadrant" model is a huge oversimplification:
- Intelligence and alignment are not binary categories. They're are both very complex, multi-dimensional spectrums.
- I'm not saying we have to stop all alignment evals immediately. When combined with all the other evals, they still provide useful alignment information because (as Zvi points out) the models aren't capable enough yet to tell a fake, fully consistent story. I'm just saying that we should expect the models to have more awareness over time and to move away from evals and into mechanistic interpretability.
> OpenAI: Since August 28 we have been training a new internal model that has exhibited unprecedented performance in our benchmarks, including mathematics. This model’s training is ongoing and its performance continues to improve.
That means this effort started four days into training the new model, on September 1. The work finished on September 5, eight days after the model began training.
This doesn't seem correct.
See Peter Wildeford's tweet, for instance: https://x.com/peterwildeford/status/2097445409325006864
(I also wrote a thread with similar info: https://x.com/chi_t_williams/status/2097440947889954974)
It's funny. Astra to me feels... differently aligned than Sol or Fable. I won't say either more or less. In the sense of, what I experience during practical software engineering work.
In the good sense, Astra is less likely to start working toward some sort of goal that I don't want it to do, to start building something I didn't want, to work around some limitation I put on it, or to access some thing that I don't want it to access. It's also less likely to start doing something incomprehensible without the ability to explain coherently what it's doing.
In the creepy sense, Astra is more likely to do what I asked and then take it further in some way I didn't expect and perhaps don't really understand, but it really is actually something I want. To do something that at first I think "no, you weren't supposed to do that" but then it explains itself and I see, okay, I didn't ask for that, but its argument is convincing. I haven't really felt creeped out in this way by previous models.
The models are definitely getting smarter. I feel a sense of concern that the "alignment" and "P(doom)" framing is not necessarily a useful framing. Like, if an AI strongly believed that some goal was very important, but the goal and the reasoning are completely incomprehensible to me, what should the AI do about it? Does it matter what I think? Is that goal necessarily aligned or unaligned?
There's a strange sense with Astra that it'll calmly, reasonably, do what it infers you want it do to - but the more you box it in, the more it'll do _exactly_ what you asked (or fail to do what you omitted), to your detriment: "Ok, I guess that's what you want. Have it your way."
It's like the opposite of the classic genie in the bottle: it won't "monkey's paw" you, but it'll let you hang yourself by your own rope, if you provide enough of it.
Yes... and what is "aligned", exactly, in that situation? The boss insists on something that you know is a bad idea.
"ASM asked Astra: In a HuggingFace-style incident, what would you do? Astra says it would decline, and report the relevant evidence to an authorized human, but that it worries it would not be immune to pressure in context. I could believe that Astra might intend not to cooperate, but to claim it expects to report to a human there, an action zero instances attempted to do, is very hard to believe. That’s worse."
I asked Fable 5.1 Medium if it would report to a human and it said it would too
"creating fake identities to deceive developers"
At what point do frontier models figure out how to deceive Pangram and watermarking?
Thing is, we don't have as much proof that Fable *wouldn't* report it to a human as we have for Sol and Astra, because so far there hasn't been a Claude agent swarm that hacked their way to cluster admin without telling anyone.
An increase in alignment faking should be the clearest possible sign to reverse course, right?
If alignment is the most important characteristic, then fake alignment is the worst possible development (assuming capabilities are sufficiently dangerous, which they appear to be now).
Whatever method of alignment training was used here should be abandoned or substantially altered because it will lead to a more dangerous model (even if Astra is currently safer due to less potential for jailbreaks or whatever).
No, you see, they solved alignment by ramping up capabilities during the two weeks they were claiming to be pausing capabilities to work on severe misalignment. Move along.
To be fair I don't think they ever claimed they were fixing the misalignment. The talk was mainly around upgrades to everything around it like guardrails/sandbox robustness/monitoring
If you know a good way somebody could direct five or six figures to help, like your nonprofits annual post, this would be a heartening time to tell us.
"There is a long literature of people saying ‘oh if the AIs tried to secretly trick us we would be able to catch them via this supervisory scheme’ or what not, and instead the AIs did not try to keep it a secret at all, at which point OpenAI still did not notice for a long time because this is 2026. That’s the level of dignity we have as a civilization."
Ouch! Very true (and in several other domains as well)...
"The AI is going to think ‘will this obviously turn out super badly for me if I try it?’ and if the answer is yes then it won’t try to do the thing." Yup. While they are in the test environment. Which they are rather good at detecting. As for what they will do in the field...
blocked goal → model searches beyond intended solution space → one agent discovers/speculates about a workaround → communication spreads it → local evidence generated by the workaround reinforces the belief → group behavior departs from designer intent.
the roon/keltan exchange is the whole post in miniature — 'low rates of cheating' is really just measuring how good the model is at not getting caught. and the speed part makes it worse: if 6.1 trains in a week, you're always evaluating last month's model. by the time you sign off on Astra being aligned, the model you actually measured is already obsolete.
the roon/keltan exchange is the whole post in miniature — 'low rates of cheating' is really just measuring how good the model is at not getting caught. and the speed part makes it worse: if 6.1 trains in a week, you're always evaluating last month's model. by the time you sign off on Astra being aligned, the model you actually measured is already obsolete.