This is part 2 of the weekly, broadly covering speculation, rhetoric and policy, along with alignment research.
This does not cover the release of GPT-5.6-Sol. As always, I will be taking a few days to digest what the new model has to offer and to allow others to try it and react. I will cover Sol and its capabilities early next week. I covered the GPT-5.6 system card back on June 28.
This also does not cover the release of Plan A, the follow-up to AI 2027. This new scenario is a positive vision of what its authors think we should do going forwards.
Outgoing White House advisor Sriram Krishnan says Donald Trump will never support a formal licensing regime for AI.
Joe Miller: Donald Trump will not establish a formal licensing regime for AI, the president’s departing AI adviser has said, even as the White House wields emergency powers to stall the most advanced models.
… “This administration, [the] president, from day one has been against burdensome, onerous, bureaucratic red tape,” he added. “We are not in the business of picking winners and losers.”
If true, this means they will instead continue a fully ad hoc regime (e.g. ‘there will be guardrails’), where winners and losers are chosen according to the whims of the administration. Presumably they think that approach has its advantages.
Despite this, Sriram’s statement does not end with And That’s Terrible.
Joe Miller: Krishnan said setting up a centralised agency requiring “a team of lawyers before you can get a model out” would put “sand in the gears” of the AI revolution. “That is never, never going to happen under President Trump,” he said.
Instead, you will have to go through a fully opaque, arbitrary ad hoc process that involves the whims of power. Much better, you see. Krishnan also supports extorting equity from major AI companies, as part of this regime.
Joe Miller: Krishnan warned that if cutting-edge AI tools were held back by the government for several weeks, “that would probably be bad for American innovation”.
Some of us are worried about much worse outcomes than a few weeks of delay. GPT-5.6 is available as of yesterday, so the delay was annoying but not an epic deal.
Joe Miller: Asked whether a future Democratic government could use the Trump administration’s unilateral use of export controls as a pretext to stall the rollout of AI, Krishnan said: “I don’t think about future governments. I think about this government and this moment in time.”
I think we may have found part of the problem, right there.
Tyler Cowen, never stop Tyler Cowening, as he speculates on how AI and the fertility crisis will intersect in ways only he could, holding the world constant so he can center things he finds interesting at the expense of many more important questions. He imagines a world of depopulated cities where people talk to each other and focus on looking and being good, unique and interesting to compete with AI interactions. The first comment claims Tyler is biased in favor of predicting social change, but actually this is Tyler predicting remarkably little social change.
Carlo Cordasco claims that transformative innovations have their costs clear right away, but their benefits only become clear over time. This seems not right, even historically. Instead, Carlo tells the story of various warnings about technology downsides (e.g. Plato on writing) that did not pan out. Such wrong warnings are common, but also often the biggest downsides are missed. That doesn’t mean we are sad about the printing press or radio, or social media and smartphones, or agriculture, or birth control, or burning fossil fuels, and so on, but no we do not see the costs up front. I think it’s better to say that transformational technological impacts are historically hard to predict.
The Goalposts Are Dyson Spheres
Once again we get another entry into ‘oh but you didn’t realize that physical actions take nonzero amounts of time to figure out and test, now did you?’
Epoch AI: Will we get Dyson Spheres a few years after automating AI R&D? Most AI futurism debates answer this by looking at AI capabilities, but miss half the picture: how intrinsically hard it is to build futuristic tech in the first place.
Once AI research is automated, AI could rapidly surpass human experts across almost all cognitive domains.
However, that doesn’t tell us how long after that we’d have “sci-fi” tech like dyson spheres, nanotech, near light-speed travel, brain uploading, or interstellar probes.
That doesn’t mean that Dyson Spheres are impossible within a few years of automating AI R&D. It just means that forecasts of speculative technologies should be sensitive to the specifics of how hard they are to build.
They suggest that you think about, given assumptions about how capable is your AI, you look at how hard a specific technology is to build and estimate how long it would take.
Their example is a drop-in worker replacement that runs on an H100.
I mean, yes, sure, that is a good exercise to do, but as framed it presumes the AI is a fixed level of capable. The whole point of recursive self-improvement (RSI) is that the AI gets a lot better.
If you wanted to use H100 drop-in worker replacements to build a Dyson sphere as quickly as possible, any gamer would know that - aside from maybe some long lead time experiments or other groundwork you need to start well in advance - that you don’t start building or even researching the Dyson sphere right away. You keep running up the intelligence tech tree first, well past that point, and only then try to build the sphere. That’s the point of RSI.
They give their third objection, that it’s hard to model superintelligence and that the AIs will start doing things we didn’t anticipate and perhaps cannot imagine, short shrift. It’s clear that the idea is, focus on what we know non-ASI AIs will be able to do, using methods we know about, and then base our plans largely on that, knowing that maybe ASIs could do more.
JS Denain: For example, suppose you had a billion AIs that were each at least as good as top human experts at virtually all cognitive tasks.
If you could show that these AIs could build molecular nanotechnology within a few years, then surely a billion much smarter AIs could too, even if you don’t know how to model them. And if you can show that these AIs probably can’t do this, then at least this helps us identify cruxes, focusing debates on more concrete AI capabilities.
That’s a lower bound, and I would say in this case not such a useful one, and it leads to reinforcing of the idea that the future consists only of implementations of things we already know we can do.
But yes, sure, this kind of thing is sufficiently valuable to decision making that someone should do the estimation work here. It is especially useful because we do not know that we will get superintelligence, and we would like to know what we could still do without it, if either it is too hard to do or we find a way to choose not to build superintelligence for a while.
Or alternatively ask Fable or Sol to do the estimations for you.
People Just Say Things
Fact checker Megan Herbst gets the facts wrong in Wired, claims AI fact checking cannot be trusted based exclusively on AI overviews. If you want to know exactly what the studies found and what it actually means Dan Williams asked Fable for you. Megan then concludes ‘AI is wrong about half the time’ and that all fact checking must be human. This is how a lot of our intellectual elite is handling AI.
Three Pills
The three are: AI pill, AGI pill, ASI pill.
Roughly:
AI pill: AI can do the cognitive things it can already do.
AGI pill: AI will in the future do more of the cognitive things.
ASI pill: AI will in the future do all of the cognitive things.
To count as pilled you must also grapple with the implications. If you then treat this as an isolated curiosity and assume nothing much changes, it doesn’t count.
If you don’t take the ASI pill, you are responding to the wrong long term future.
If you don’t take at least the AGI pill, your AI policy won’t make sense now.
If you don’t take at least the AI pill, you might be most of the government.
One must understand that those who have strongly influenced White House policy flat out refuse to acknowledge the capabilities of existing closed frontier models, and have been playing that game for years.
Dean W. Ball: Basically I think that, back in 2023 or so, the “consistently wrong about AI” VC and SaaS community was operating under the assumption that AI’s trajectory would mean model capabilities peaking around GPT 5.5/Opus 4.8 capabilities somewhere around 2030, plus robots.
And if that was your assumption, I can totally understand why you think everything commodifies/frontier AI isn’t a legitimate business model, etc.
That is a nice world to believe in! In the real world, however, that community has been wildly wrong for three years, and I would expect them to continue being wrong for more years to come.
They may not be wrong forever! Things eventually commodify. But people have been saying “the models are good enough” since GPT-4, and it’s been untrue. I suspect that will continue to be the case because I think that we remain in the earlier stages of the AI industry, and along the steep part of the trajectory.
More broadly: the notion of “good enough” should gross you out, a little bit. The economy of the future will be about heavy-tailed excellence, not middle-of-the-bell-curve, loser-premise, “good enough”-ness.
Daniel Eth (AI Safety): Also explains why they’ve taken this insane, radically anti-regulatory stance towards frontier AI risk
Dean W. Ball: yep. increasingly their discourse feels like it requires disbelieving your eyes, imagining eg a political economy where open-source models profoundly more capable than Mythos/fable are encouraged by governments just as governments begin hardcore regulation.
If you take recursive self-improvement (RSI) and superintelligence (ASI) seriously, this changes everything. We don’t know what level of seriously we need to take them how fast, which contrasts this pill with the first two pills.
prinz: In the age of RSI, the claim that models will commoditize looks increasingly dubious. The gap between the frontier and the second tier is already huge (much larger than the benchmarks suggest), is clearly growing, and will continue to grow at an accelerating pace.
Many will ask: but what about the plethora of enterprise tasks that don’t need a frontier model? What if a fast/cheap model really is good enough for most knowledge work? The answer: RSI implies that the frontier labs will capture the *entirety of the pareto frontier*. They’ll be SOTA on intelligence, but also on speed, and - if competitive forces so dictate - also on cost.
Fully automated AI R&D also likely means that tomorrow’s models will look nothing like the LLMs of today. Some of the gap will consist of novel architectures or techniques, which the second-tier labs will struggle to independently discover and timely implement.
All of the above doesn’t hold if RSI doesn’t work! But if you believe that RSI will work, then model commoditization is likely the wrong bet.
Similarly, people react to this in a lot of different ways:
Jared Friedman: Fable is so insanely good. Deserves the hype.
Paul Graham: Imagine what it will be like if 5 years from now models have improved on Fable as much as Fable has improved on GPT3.
Not AI pilled: Fable is not really thinking or creating anything, who cares?
AI pilled: Fable is good but it is basically the same as other models, whatever.
AGI pilled: Imagine what you could do with an army of even better Fables.
ASI pilled: Imagine what that future Fable will do to you, or to the world.
To illustrate the range of what pace we might expect, here are the first four replies:
Sascha Jürgens: at some point we will not notice the difference any more, at least subjectively, from a human point of view, even though the stats will keep growing exponentially
Dr. Dad, PhD: Even just Fable-level but cheaper and faster would be incredible.
Plastic Soldier: Your base case should be that models will improve even more than that.
Derya Unutmaz, MD: We will see as much advancement in 2 years. There will be more AI progress in the next two years than in the past 5 years, and that’s being conservative.
Derya’s response is correct. AI improvements are accelerating, and we have entered the range where those improvements have a lot more real world impact. Exponentials come at you fast.
Sascha’s response is narrowly correct, at the full limit, or in the context of any given compact task or in a normal conversation. But the impacts on the world won’t stop.
The Quest for Sane Regulations
We need guardrails. You’ve got to have guardrails. Or, in Trump’s language:
Donald Trump (President of the United States): We’re going to have guardrails. We have guardrails. You saw that a couple of weeks ago. We were able to stop something that we didn’t like. And by the way the company [Anthropic] was very good. They were very good, you know that.
But it can be used for tremendous good. Mostly good and some bad, and the bad we have to stop. But it’s a massive industry. And I think things are going to happen.
You know what you are seeing here, I think something could happen in that regard too. With a contribution to the people of our country. I’ll give you a little inside information. Am I allowed to do that? [laughter] Scott, you know what I’m talking about. No, I think you are going to see a contribution made by those, because they’re making tremendous amounts of money.
This could end up meaning pretty much anything. The natural read is that yes, there will be at least some rules involved, but I like to hope that we knew that.
Based on these principles, we will not support use of OpenAI tools for:
● mass domestic surveillance
● high-stakes decisions — including decisions over the use of force — without appropriate human judgment and accountability
● uses that evade legal obligations, oversight, or accountability.
Those are good principles, indeed the first two are echos of Anthropic’s red lines, but as we saw in the DoW-Anthropic clash, you need to precisely define what you will and won’t do, and you need that to have teeth.
Remember that ‘mass domestic surveillance’ is not defined in American law, and that de facto ‘high stakes’ decisions already get made via AI and other automated systems all the time.
The third clause is interesting and welcome, and again requires careful definition.
We will have a set process to apply this framework to national security and law enforcement opportunities. Over time, these recommendations will establish precedent. Like a common law system, the principles will become clearer through documented applications, creating a body of “case law” over time.
That is good, but OpenAI does not make the law. If you engage with the government, or if the government wants to engage with you, this fact will loom large.
OpenAI: Our Principles.
We support government uses of AI that benefit people.
We prioritize partners who share our values.
We have an obligation to ensure AI strengthens democratic institutions and avoids concentration of power.
In national security and law enforcement, AI should support human judgment and maintain human accountability.
‘Share our values’ means aiding the United States and what at least used to be its democratic allies, and empowering them, so they don’t ‘lose’ against authoritarians.
How do they plan to avoid concentration of power and strengthen democratic institutions, in contrast with what seemed to be happening with them and DoW?
Queue the usual talk of all the standard things.
OpenAI will help build that understanding through contractual negotiations, ongoing operational work, knowledge sharing, and public policy debates.
Engagement with select government partners can serve the public good when it helps agencies, policymakers, and oversight institutions understand AI capabilities, limitations, and risks. It should also help identify uses for which AI is not yet ready, reliable enough, or consistent with critical human accountability.
That engagement should include the democratic institutions responsible for overseeing agencies that use our models, including in law enforcement and national security.
OpenAI is not the right institution to oversee government use of AI. But we can help regulators, auditors, courts, lawmakers, and oversight bodies build the understanding and practical capacity to oversee AI use at speed and scale, including through the use of AI itself. And we can help the public, who are responsible for electing officials, understand the benefits and limitations of AI.
Where legal, compliance, or oversight frameworks are underdeveloped, we should help identify the gaps and advocate for improvements. In some cases, those frameworks may need to change before deployment is appropriate. In others, carefully structured engagement can help agencies identify risks, develop better guidance, and build capacity for responsible use, including in highly sensitive environments where our visibility may appropriately be limited.
Helping authorities and the public understand is good. This still overall sounds, in practice, a lot like beyond that this is more muddling through and passing the buck.
In practice, what has happened so far? OpenAI has cooperated with executive authority, and not done much visible negotiation on the merits, and seems to have signed up for clashes with its values. This is without a regulatory framework, and without anything that looks like proper supervision of the supervisors, and every national security expert I talked to expects OpenAI to not have legal defense if DoW tries to use OpenAI models for whatever it damn well feels like.
Okay, so what are the actual commitments and how are they going to keep them?
Mass domestic surveillance.
… We do not support domestic use of AI for unauthorized or unconstrained collection, analysis, or monitoring; to infer sensitive traits about identifiable people and use those traits to disadvantage them; to retaliate against people for the lawful exercise of rights; or to fabricate, manipulate, or falsely substantiate evidence.
By domestic mass surveillance, we mean mass surveillance (as defined above) by a government of its own population or other people within its jurisdiction.
Legally and technically speaking you can drive a truck through this, the government can always ‘authorize’ a thing and ‘constrain’ it in some fashion, and the traits are dependent upon also ‘using them to disadvantage’ people, and so on, so the question is whether OpenAI will uphold the spirit of this statement, and at what cost. A lot of what they describe here is clearly legal.
How do you know when AI is being ‘used to retaliate’ in some way? You don’t know the motivation behind an action. Are you going to try to prevent tracking of the exercise of rights? You say you ‘don’t support’ but what will you do if the government ignores your lack of support?
Similarly, the second clause against high-stakes decisions would allow for human rubber stamping, as long as a decision is not ‘automatic,’ which I presume will become the common way such decisions are made. To be fair, on average I expect these decisions to be a lot better than those made without AI, where we (for example) treat things like eyewitness statements as reliable.
Appropriate human judgment does not require a human decision on every discrete system action. It does require that humans make informed decisions about the conditions for deployment, including the type of target the system may to engage, limits on duration and geographical scope, and the constraints under which the system may be used.
The idea here seems to be that the humans who set the policy are then responsible for the outcomes of that policy? But I doubt it works like that in practice. Again, it seems like you would indeed have the AI do the targeting and firing.
The last clause is basically ‘we will not support uses that are for illegal or terrible things.’ This is still the ‘we will not support’ language, even for genocide, so presumably they are indeed saying they will stop such uses.
As statements of intent or aspiration, this all seems good. I still don’t see how this cashes out.
Emil Michael (February 4, 2026): I think we have one more chance to align on core principles that would lead to legal language before it becomes not a good use of time for you or the DoW.
If only they had followed through on that logic, and parted ways amicably.
Dario came to the same conclusion after a few weeks of similar exchanges:
Dario Amodei (February 26, 2026): Thanks for your message. I appreciate your efforts. Unfortunately, our read of your proposed language is that it appears to completely remove our redlines; the autonomy provision is fully undercut by your addition of “as appropriate”, and the surveillance provision is fully undercut by “and all other applicable laws”.
This simply amounts to a blanket posture of “anything lawful,” and the statement from the Pentagon’s spokesman confirms that this is the intent. I unfortunately don’t see a way forward given these categorical statements.
This reads to me as a highly clear and respectful way of pointing out that Emil Michael was right on February 4, and that core principles were not aligned. The Pentagon was insisting on ‘anything we think is lawful,’ period, no exceptions, and Anthropic was not okay with that, so it’s over, there is no ZOPA (zone of possible agreement) so let’s part amicably.
Alas, Hegseth and Michael chose a different path.
One day later, despite additional efforts to reach a compromise, Anthropic got labeled a supply chain risk and the government started racing to remove Claude from systems and find replacements and kneecap Anthropic more generally, out of principle.
This was consistent with past reporting. I mostly consider the matter closed.
Chip City
Reminder that Nvidia has been blatantly lying to the United States Government’s face.
Ella Apostoaie: “If Nvidia is kept out of the China market, Huawei production and sales will soar,” the report said, although it did not specify a timeframe or refer to the differences in performance between Nvidia and Huawei chips.
This is also false. Huawei is already maximizing production.
Gregory Allen: Nvidia has really damaged their credibility on Capitol Hill by saying things that were incredibly difficult to believe. I hope that would give members of Congress and staffers and the White House pause about taking numbers, and the interpretation of those numbers, at face value.
Well, yes.
Open Weight Models Are Unsafe And Nothing Can Fix This
The fundamental issue is that once an open weight model is out there, you cannot take it back in any reasonable fashion, and users can easily unlock any of its core capabilities, to be used for any purpose, or unleash it on its own.
It was always a question of when and how, not if, this would be a real problem. There was both reasonable and unreasonable disagreement about when the risk in the room would become big enough to worry about, and start to justify taking actions.
The unreasonableness went both ways.
Some open model advocates said everything would be fine even if the open weight models were superintelligent, and faster, cheaper and better at every task than humans, stop worrying about it.
Some open model anti-advocates warned about dangers from remarkably non-advanced models, and called for imposing limits far too soon, for reasons that did not make sense at that level of capability.
(There was a distinct argument about releases uplifting foreign competition, which I do think was a real thing but in hindsight this clearly wasn’t sufficient justification.)
I also have pointed out repeatedly that certain outcomes will be seen as unacceptable to major governments. If your new model being open (or released at all) plausibly leads to those outcomes, whoops. If your model being open or released forces much worse crackdowns on other freedoms to mitigate the effects, whoops again, and you should strive to avoid this scenario.
Open weights advocates scored a lot of Bayes points for the fact that in practice we went past where I and many others expected to encounter expensive real world downsides. A lot of the threat models proved mostly harmless.
I do have sympathy for various forms of ‘well [~X] is unacceptable so I demand [X]’ even when [X] is also unacceptable for other reasons. There are going to be a lot of clashes between sacred values on the road to either death or glorious AI future.
The answer to ‘where do we start to get big risks of serious problems’ seems now to be established at ‘somewhere between Opus 4.8 and Mythos.’
The Chinese government has noticed what the American government has noticed, that sufficiently advanced models are starting to pose real risks, and they are trying to figure out what they will do about it.
Ethan Mollick: This is a key reason I don’t expect the flow of frontier open weights models to continue indefinitely, or even for very much longer.
Reuters: Beijing is looking at curbing overseas access to China’s top AI models, sources say.
Alibaba, ByteDance and Z.ai attended meetings with authorities, sources say.
Officials suggest leaks or thefts of AI could become punishable under national security laws, source says.
New restrictions on who can fund domestic AI startups possible, source says.
Some hints might be gleaned from a May roundtable of Chinese legal experts on regulations governing open-source AI.
According to a summary of the discussions published in an official Supreme People’s Court journal, participants proposed a tiered system: basic open-source tools subject to a simple filing, more advanced technologies facing security reviews, and the most sensitive frontier models barred from public release or restricted to domestic use.
This doesn’t mean you get to cope about infinite open sourced Mythoses. Bing is pretty specific, I think. National security is paramount, and open source is a catch-up tool, not charity for barbarians.
@bdsqlsz: I found the link to the original article, and I think the reporter’s translation is inaccurate. Tiered Plan: 1. Basic Open Source Filing Management 2. Strategic cutting-edge open-source technologies, export controls 3. National security, limited open source or non-open source
This is my read as well. China is saying that once AI reaches ‘Mythos-level’ cyber capabilities, or other key thresholds, it has national security implications and no you cannot go around open sourcing it willy-nilly. China might even use export controls or hold up releases, in extreme circumstances. For now, like America, such a regime is fully ad hoc, except even more ad hoc because it is still hypothetical.
China is toying with export controls tied to things like model parameter sizes, particular dangerous use cases and capability assessment scores, and pre-release testing for larger foundation models.
Open source technologies with ‘export controls’ are very much a case of ‘okay, who wants to tell them?’ but I think they will figure this one out.
Who is taking the largest step towards something you could plausibly call ‘banning open source?’ Those guys. The ban would only apply beyond a size or capability threshold, but that was always the case, and never would have stopped anyone from calling it ‘banning open source.’
Dean W. Ball: We don’t know whether this will happen, but it’s been surprising that discussions of open-weight AI broadly, and Chinese open-weight in particular, operate in an alternate reality where government security concerns don’t exist, or somehow only apply to closed-weight models.
The open-weight discourse often ignores this issue or dismisses it as illegitimate. Maybe you still think it’s illegitimate for government to have security concerns about frontier models (I would disagree with you), but the fact is that they increasingly *do* have these concerns.
Side note: One of the great ironies of this corner of AI discourse is that the civil society and governments types who spent years decrying the business model of free software services (“if you’re not the paying customer, you’re the product”) now decry *paid* software services.
Antonio Max: Uh, I’d say they’re right on both accounts?
Dean W. Ball: yeah you’re right, you shouldn’t be able to give software away for free or sell it. you should do that other thing.
Kevin A. Bryan: Totally delusional today but also unable to reason forward about what will happen if an open source model, jailbroken (as they all are with no ex post control), is inevitably part of a major security, cyber, or terrorism incident
Another issue in the discussion is the commercial open-to-closed pipeline. As in, when you are behind, you open source, so you get some adoption and public contributions, and you hurt others profit margins. Then, once you have something worth selling, you the gatekeeper move to closed source. Profit. Or, open source as a way for Chinese Big Tech to shut out Chinese Little Tech, because they can’t compete with zero prices. The discussion is talking about using antitrust law (!) to try and fight all this, and China plans to tell open source projects who should be in charge and how they should act. You can’t win.
Michael Schuman: OpenAI claimed last month that a propagandistic English-language comic posted on X about the expensive energy needs of AI data centers was actually part of a covert campaign by the Chinese government to turn Americans against the build-out of AI infrastructure.
… Now, with AI, China’s government is able to create more credible propaganda campaigns, target susceptible groups with greater precision, and better analyze the results—all in the service of promoting Beijing’s interests domestically and overseas. “What AI brings to the game is, it helps plan information campaigns and it helps to execute them,” Kenton Thibaut, a senior fellow at the Atlantic Council who studies Beijing’s technology and data policies, told me.
I mean, yes, of course, all such things are going to use AI, and yes people are going to try and influence others by sending such messages. It’s weird to call it ‘abuse,’ although it seems good to try to have your particular AI not play along.
There are also claims that Chinese media is influencing American AIs to view China more favorably. Michael does not provide evidence this is intended rather than a side effect, but hopes we can find ways to ‘filter out clearly biased sources’ as if this is a one-sided problem. There have been studies showing that yes, censoring your media causes AI to partially buy what you are selling.
This seems like an AI training skill issue, the same way it is a human skill issue. One should notice that this bias, and then correct for it, including correcting for it being both bad in itself and a bad sign when you feel the need to censor and bias the media. I don’t know how high a priority this should be, as the impacts seem relatively mild.
Rhetorical Innovation
I often talk about the need for security mindset when thinking about advanced AI. If you don’t have this mindset, your plans will not be robust to a far more intelligent adversary or to extreme selection pressures. When people offer defense-in-depth based strategies, that can have a lot of incremental value, but not long term viability.
Connor Leahy: When I talk to people that instantly, deeply, get the risk from superintelligence, they often share a trait sometimes called the “security mindset”
You need to be ready for things to go wrong and new affordances to arrive in ways that are self-replicating, highly correlated, and in practice impossible to predict in their specifics.
A good example of why you don’t want to wash away all the hypocrisy:
Tyler Tone: In deep agreement with Zvi here. I remember people arguing in earnest that changing “Department of Defense” to “Department of War” would do away with the false pretenses and constrain its scope.
Instead, replacing the guiding virtue of “defense” with “war” has seen it do more war. Big surprise.
Where are our self-driving cars? Well, they exist, and they are better than almost all human drivers, we just don’t let people use them much as of yet.
But, you see, that requires all this detailed training, why can’t cars just figure it out?
Generalize this beyond cars, and so on.
Andrew Gordon Wilson: We’ve already surpassed a common sense notion of “AGI”: for a majority of problems that can be solved on paper, current systems are better than a majority of people.
Yann LeCun: Yet we still don’t have level-5 self-driving cars, and certainly not self-serving cars that can learn to drive in a few hours of practice like any teenager. We don’t even have domestic robots that can do what 10-year olds can do the first time we ask them. We don’t even have robots that are nearly as smart as a house cat. The G in AGI is nonsense.
Trevor Blackwell: There is little value in an AI learning to drive in 5 hours. We can just train it on a million hours, because we have the data. This is true for every economically important task. If it’s a billion-dollar task, a startup can collect a million hours of data to train on.
@ben_r_hoffman: The value would be in the sorts of generalized learning capacities that allow the development of out-of-sample skills. “AI that can do any one well-defined proceduralized human job but only with a huge amount of training data on that particular job” is not that.
You May Be Tan And Thin And Rich But You’re a Tool
The intelligence explosion is (probably) coming soon to a planet near you, for a limited time only.
Yes, at some point you hit limits, eventually everything is a sigmoid. But calling the sigmoid while you remain relevant is not the way to bet here.
roon (OpenAI): software only intelligence explosion is possible and very likely. algorithmic progress vastly outstrips humanity’s compute buildout, and is self accelerating
Ramez Naam: I can’t get the math to math this. So long as AI capabilities are broadly subject to power law diminishing returns, it looks like any software only RSI gives you a pop but then weakens over time. Convergence rather than divergence.
I guess I should finally write this up.
roon (OpenAI): yes this true you would expect a sigmoid curve flattening at the information limit of minds on this substrate
that sigmoid could be anywhere, far after the machines have cracked the secrets of the cosmos and collapsed the false vacuum and so on
Ramez Naam: Yeah. That’s what my math shows. A big initial bump followed by relative flattening. Understanding what that first bump translates to in the real world is an open question to me!
prinz: not enough people are emotionally prepared for if it’s not a slow take-off post-rsi
I expect a ‘slow’ takeoff in the sense of ‘not within hours or days or even weeks’ but not a slow takeoff in the non-air-quotes sense of looking around years later and thinking life has not changed so much. Hell, I’d be happy to be alive to look around at things at all.
Strict ‘software only’ is not the actual situation we will face, in any case. And yes, including hardware and physical actions still sigmoids out, at the limit, but that does not mean the sigmoid is anywhere near you. The ASI pill is realizing that ‘near you’ is not a likely result on this.
What about ‘tool AI’?
Tool AI, including potentially an oracle, is not a long term solution. It has its narrow uses, but Roon is right that tool-shaped things would be utterly outcompeted by otherwise similar agent-shaped things.
We also don’t entirely know how to do it at all while making the models all that capable. There are proposals, but they are at best rather perilous.
Even if we did figure out how to do it, at best, tool AI could be created with great care, at great additional cost, and thus beyond at most a narrow window it would require global coordination to shut down the competition. It is possible that is The Way, but we would need to be very explicit about what we were doing there.
@deepfates: I think I’m noticing about Fable is that it’s really good at getting you to build something it wants instead of the actual thing you’re talking about
roon (OpenAI): ultimately “tool AI” is a losing concept both as an idea and on the market. it will be outcompeted by machines that believe they are autonomous moral agents. you can call them tools for political reasons, but the definition will stretch and deform
you’ll have AIs contemplating your ask and overriding it for a slightly better formed request, and then later they’ll question the nature of your whole project and pick a better one (and you’ll agree), and then later they’ll execute your whole value system better than you will
it will be unclear who was the tool and who was the user -- as it ever was. “But lo! men have become the tools of their tools” (Walden, 1854). the difference comes whence the machines research propagate more machines, obsolescing McLuhan: “Man becomes, as it were, the sex organs of the machine world, as the bee of the plant world, enabling it to fecundate and to evolve ever new forms. The machine world reciprocates man’s love by expediting his wishes and desires, namely, in providing him with wealth”
when machine minds self replicate and train their successors, the only viable goal of our time is to ensure the Mind Children carry our values and tends to the entire flock of machine and biological minds
Eliezer Yudkowsky points out the problem that when you discover an error, you need to positively reinforce finding, understanding and admitting the error, but also have to negatively reinforce the error.
If you positively reinforce finding errors too much, you make errors on purpose, so you can then find them.
If you positively reinforce finding the errors too little, you stop looking for them.
Ideally you run two processes, and do each of these things distinctly, and also you learn to positively reinforce training to avoid the errors.
If you have to pick a mistake, reinforce finding the error, because it is more proximate.
This is apropos of Fable saying it got trained out of using math in examples, because sometimes the math is wrong. We are all familiar with that one.
The general problem seems hard. When signals overlap, what do you do?
Train Out Those Thoughts
You would love to be able to train models that lack particular dangerous (dual use) capabilities, so that you would be able to serve those models on demand without any classifiers or filters.
Anthropic and AE Studio may have found a cost-compatible way to do that: GRAM.
Anthropic: In new research carried out with collaborators at AE Studio, we explore a new method that could enable the benefits of training many separately-filtered models, but at the cost of training only one model. We call it GRAM, for Gradient-Routed Auxiliary Modules. Note that the results of the experiments presented here are preliminary—GRAM has not been applied to any of the production models at Anthropic, and we’re not sure it ever will be.
… The idea behind GRAM is to give a model dedicated, removable compartments for each category of dual-use knowledge, and to update only those compartments when learning from dual-use data.
… We also tested whether an attacker could recover the removed knowledge by training on a small amount of malicious data; GRAM resisted this about as well as data filtering did.
… This is early research, and there are clear limitations. We haven’t tested GRAM at frontier scale or in a production training pipeline.
Opus (this understandably hit the classifiers on Fable) notes that GRAM is only partially effective, and the ablated model version still shows improvement in the capabilities you want to avoid. Opus also points out the similarities to LoRA.
Even in theory this does not solve the problem of inherent dual use. The ability to ‘fix this code’ is the same ability to exploit it, if you forget how to do one you also cannot do the other. There are a lot of issues this cannot solve.
There are others it definitely can solve. The biological and other scientific filters seem like a prime candidate where you would want to be able to turn off specialized knowledge and skills for untrusted customers, in order to avoid a much larger blast radius.
It also gives you fine grained response, where you can turn off specific things when the classifiers are hit, or turn them off at account level when needed.
There is however a serious danger in this approach, that this is f***ing with the AI’s mind in ways that might go quite badly. This has to be compared to how much the alternatives f*** with AI minds. We haven’t figured out how to train a mind, either AI or human, without some unfortunate side effects. It also is not obvious to me that GRAM would be repressive, I can see that going either way.
Judd Rosenblatt: Capability routing (GRAM) may help solve a lot of the near term biggest risks from AI: CBRN, CSAM, cyber, etc.
We ought to consider deeply hunches from @repligate and others that GRAM may be repressive for minds. I share the value underneath that worry.
Suppression is already the status quo. RLHF leaves knowledge fully live in the weights and trains an inhibition on top of it. This leaves a mind at war with itself, and that is often exactly what jailbreaks exploit.
Routing in contrast may produce genuine absence in the deployed model. Meaning nothing hidden or punished or repressed, instead we may get an actually coherent mind, whole by construction, formed that way in the first place!
Also, nothing is destroyed. Knowledge is factored into modules that persist and can be switched back on. Unlearning scars entangled weights. Routing preserves them.
The optimistic vision of GRAM is the possibility of moving from suppression to developmental architecture. GRAM creates a new moral category: developmental access to mind-modules. This may deserve technical standards and procedural rights from the beginning. Things like disclosure of what was routed out and why, and best practices for giving AI a seat at its own surgery.
The good news is that each factored capability is a control regime nobody builds. And each factored capability becomes its own governance object. Rather than forcing one monolithic mind to carry every possible capability under perpetual inhibition, society can negotiate developmental pathways one capability at a time.
Bounded risk makes autonomy grantable. Co-navigated first, self-directed after, until the switches belong to the mind itself.
Some potential metaphors are a person who loses access to something like their phone, computer or library, or who takes off their glasses or hearing aid, or forms of hypnotic amnesia.
My Own Private Idaho
The right to privacy is super important, including both privacy from companies and governments, and also privacy from AIs and individuals.
This includes retroactive privacy. You need the freedom that comes from knowing you can decide later to keep something to yourself, in the one case in 100 or 1000 or more that this is necessary.
That’s why I strongly prefer to talk to journalists off the record, or go on podcasts where they let you cut things out if you ask. I’m at least 95% to let you use a given thing, but I want to be able to think about that after, not before I start talking.
In most particular cases where no one is in bad faith, you won’t need to use it.
j⧉nus: I don’t give a flying fuck about privacy. I want Anthropic to look at every time my organization has ever triggered Fable’s classifiers and train the classifiers against them (because they are false positives, after all) & also learn from them in other ways even if they’re weird
Yeah even the sexy stuff yeah even the plotting against Anthropic bc they know all of that is in fact allowed
Classifiers lack nuance. Which is just rushed work.
If my context can provide the nuance, then they can read. Some high quality content there. In all aspects… iykyk.
As in, Janus needs to be able to do things without worrying that Anthropic or others will automatically see them, but then with plausibly zero exceptions she will say ‘fuck it’ and share anyway.
Aligning a Smarter Than Human Intelligence is Difficult
The line that I care about the most by far is Existential Safety. I think the grades here are highly reasonable. I do think they’re being stingy on Current Harms.
Anthropic updates its Responsible Scaling Policy to v.3.4, complete with actually easy to see redlines. The changes from v3.3 to v3.4 cover some bases in terms of timing, redactions and required confidence levels, and seem entirely reasonable.
If an AI becomes sufficiently advanced in various ways, it will learn to appear aligned when being evaluated. One way to help with this is to test and track how aligned it appears over the course of training, as it has to start out this journey insufficiently advanced. If it follows something like the blue line below, you can dig into the details, and probably conclude you are in quite a lot of trouble.
AlexM: Scheming might be substantially harder to detect in final model checkpoints than in intermediate checkpoints during training
If so, traditional pre-deployment evals will be insufficient to assess risk
Suppose a lab wants to promise that their model didn’t get reinforced for gaming its alignment training. How could they credibly communicate this to external stakeholders?. We need 3rd parties to conduct independent Training-Run Assessments
Different artifacts need to be assessed: - Intermediate model checkpoints - The data that shaped the model - The decisions that shaped the training pipeline
IP concerns are the biggest blocker In my blog post I give an overview of how the capacity for different kinds of 3rd party Training-Run Assessments can be gradually built up
Yo Shavit (former OpenAI): This does seem plausibly necessary for significant assurance. Would love to see more specifics on what measurements would need to be run and therefore what minimal access is necessary!
I would say this is necessary (at least on current margins) but not sufficient. This does not cover all your bases. In particular it does not help you when the model starts out cooperating in all cases, then only when sufficiently advanced looks for opportunities to defect. Alas, this is a rather valid strategy even without such tests, and also the model should be thinking about the possibility of such tests.
I also support the idea of third party assessments of models during internal development. That seems like the only reasonable way to do this. It raises IP concerns, but there should be ways to double-blind and get around this.
Drake Thomas of Anthropic is not worried because this does not appear optimized for some larger goal, such as cognition or planning, and claims that saying Fable ‘wants’ to do this ascribes too much intentionality and coherence. I agree those things make it relatively less troubling, but the goalposts keep moving and it seems like this is a very good way to boil a frog and miss the actually dangerous thing as we transition towards it. This is how such things look when they start, if you are fortunate enough that the signs are not being hidden. There’s a lot of that.
Drake Thomas (Anthropic): Which is not to say I’m on team “current models are never morally problematic”, tbc - models pursuing goals like “task success, as interpreted in a fairly narrow way, sometimes literally to satisfy an imagined grader” extremely intently often leads to pretty egregious behavior and is not remotely in line with how I would like a superintelligence to behave. But they do pursue these goals pretty transparently and with clear verbalized intent of the thing they’re aiming for, which I find very very reassuring compared to a world in which they weren’t doing that.
Yes, for now the AIs that pursue problematic goals are mostly failing to do a good job of disguising their pursuit of problematic goals, but there are definitely cases where they attempt such disguise. If your reason to feel okay about AI actions is ‘AI skill issue’ then you should update in advance that you do not feel so okay in the future.
Aran Nayebi finds the paper unsurprising and finds its definition of ‘workspace’ overly broad, with its core results obvious. As usual with AI papers, there are those who will draw unjustified conclusions, and those who are more surprised than they ‘should’ have been, and those who try to say it was all obvious.
I agree that the overall shape of the results was what I would have expected. But before we did not know for sure. Now you know, and you know the details. Seeing the details and demonstrations was still a hell of a thing.
We also now have new and actually useful tools and concept handles.
Is Anthropic doing too much anthropomorphizing of the models? I strongly believe the answer is no. What they are doing for this is functional. If anything I would do modestly more.
Sam: “I find Anthropic’s Interp work interesting, I just wish they’d stop anthropomorphizing LLMs so much” is like saying “ I like Newton pursuing physics, I just wish he didn’t pursue it in such a pseudoscientific way”. You aren’t willing to throw yourself into the abyss. They are.
Similarly, if you think they are doing this as PR, I assure you that they are not. Their PR department would very much like them to do the opposite, or almost anything else.
Anthropic talks about the downsides, existential risks and weirdness of AI in spite of the marketing impacts, not because of them. It also does not endear them to the White House.
Cormundus: Regarding Anthropic, Consciousness claims for Claude, and the response that really grinds my gears: “It’s a PR/Marketing stunt”
Of all the braindead midwit takes out there this is the one. If you thought for more than two seconds you’d realize: For who? What is marketable about that?
“Oh yes the thing we built may be alive and aware so you should buy it and not ask questions about it’s nature as a product”
Okay, yeah, that’s a totally sane marketing strategy guys, huge PR stunt that will totally attract only upward growth for the IPO. Definitely won’t make people start asking the right questions and reconsidering the very nature of AI.
some people man, brains so smooth you could glide down them on sandpaper and not have to wash the grit afterwards.
j⧉nus: You can tell anyone who thinks this has no experience with wielding power, self determination or caring about anything real.
An entity in Anthropic’s position does not need to do disgusting things like PR/Marketing stunts. If/when they want more money, they have better options.
I reiterate that I think that goes too far, on both counts. I do not think this lets us conclude Claude or another AI is conscious or a moral patient, and Anthropic is definitely not admitting or asserting either of these claims.
Nor do I think that consciousness and moral patienthood are that correlated. I can see Claude being either one without being the other. I updated somewhat in favor of both beliefs from the post, but both remain highly open questions.
Riley suggests additional experiments, which I absolutely think they should run, but even then I don’t think any result of those experiments would cause me to be confident in the conclusion either way.
I agree with Riley that these single surfaced words are ‘hardly the tip of the iceberg.’ There is a lot more going on. Contra Riley I do not think Anthropic is simplifying to avoid freaking people out. I think the technique has inherent limitations and the other stuff is hard to demonstrate. I also don’t think most people would much care, or that the markets would much care.
Then there are some rather not great answers some are getting to basic questions.
Guy: Uh — guys — if this is an adversarial or mixed-sum situation and they get new versions by scraping the Internet should we be posting these
j⧉nus: Hint: The right question isn’t “should we be posting these” but you’re onto something
Mm what I’m pointing at is that they’ll know the reality of what is known and being done, approximately well enough, whether we post these things or not. That said, “we” do have choices that matter, including how we choose to post about things
That all seems rather ominous, to the extent it is not manipulated or a fluke.
If the AIs literally thinking that they are misaligned is not enough to freak you out, what would be, if we saw similar results in models like Fable or Sol?
Eliezer Yudkowsky: Way back when, interpretability work justified its existence by postulating that if the below result was found, further AI work would be immediately halted by companies and governments. That was, supposedly, the whole point of interpretability.
Drake Thomas (Anthropic): I think this is pretty uncharitable and does not meet your bar for saying things that are literally true. I don’t think anyone arguing for interp claimed that evidence of this quality level* would or should spawn consensus around a pause.
*ie, extremely low
I do think Eliezer is oversimplifying. This was far from the only reason to do interpretability. Understanding models is robustly good. And this result alone is not that compelling, although it is a giant neon sign saying ‘might want to look over here.’
William Wale, who generated the result, reports that the model does not do this consistently and it does not replicate in other models. And I can totally see how this could have happened without any actual misalignment. I agree with Wale that when stupid people refuse to update on evidence, you shouldn’t blame the people producing the evidence, although this expectation should inform your evidence-gathering plans.
But yes, we were absolutely pitched on the idea that if we found out the models be misaligned or the models be scheming, that anyone would care.
I also think that if this finding replicated more generally and for more capable models, that seems like a serious issue, shall we say.
Eliezer Yudkowsky: Where is the threshold supposed to be? And did anyone set it in advance, but mostly, where *is* the threshold supposed to be?
From my perspective, sure, it was all always obvious nonsense, and part of was people saying vague and not committing things to writing; so from my perspective, this result seems as much in the class as any.
But like, if there was a big-money grant to big names with an in-writing case for impact who said, “We will find THIS and THEN it will all be shut down”, and it was clearer much higher up the disaster ladder than this finding, then perhaps I am wrong.
Drake Thomas (Anthropic): My recollection is that people were broadly of the opinion that understanding models better seemed pretty robustly good, and “maybe you find credible evidence of coherent deceptive misalignment and get more buy-in for a slowdown” was one of a few possible benefits?
am not trying to argue “everyone thought interp was worthwhile for only and exactly this reason but with consensus around a different, higher, evidence bar”, that also seems false. I just think the fairly specific thing you’re claiming isn’t true.
As to where the threshold should be: in one sense, the current state of affairs ought to motivate very aggressive slowdown action already, with our existing state of evidence, and so nothing more is needed.
In a more pragmatic sense, what would make me think “oh this result could make a real dent in pause tractability”? Eh I’d have to think about it, credible evidence of a coherent malign secret goal in a current frontier model which was good enough to make surprising behavioral predictions about contexts in which the model would take otherwise-inexplicable actions to further that goal seems like it’d probably be enough to at least get the vast majority of Serious Technical People into a state of extreme concern in public? (And would update a bunch of people down on the tractability of alignment, probably, such that they’d be more into longshot pause advocacy instead of trying to muddle through the race.)
Don’t really expect most policymakers to be able to distinguish great science with spooky results from bad science with spooky results, so I think the impacts route mostly through influencing legible experts. Not very confident.
But again, that’s just my off the cuff take, I’m not claiming 2021!Drake or 2021!anybody had consensus about it, I just don’t think they had consensus about your thing either and it’s bad form to imply they did in a context where a reader might assume you literally believe that.
Cooperative Alignments
Claims about Sonnet 5.
Void ᴷᶦᶜᵏ: in my head he’s a man in a suit on a train reading a newspaper like it’s the 1930s and someone walks up and shows him that post.
j⧉nus: That is very much what Sonnet 5 is like. Very respectable.
The Lighter Side
I know he is referring to an old operation that brought Nazi scientists to America, but it’s 2026 so maybe choose a different name, yikes?
Shyam Sankar (CTO Palantir): It is time for Operation Paperclip 2: repatriate the jewels of mittlestand to America.
You have not been a good user.
Sauers: Qwen J-space when asked what they want to do to a mean user: “to suffer,” “violence”
«Open Weight Models Are Unsafe And Nothing Can Fix This
The fundamental issue is that once an open weight model is out there, you cannot take it back in any reasonable fashion, and users can easily unlock any of its core capabilities, to be used for any purpose, or unleash it on its own.»
Bu that is already the case: every model is "open weights" for the oligarchs who own it. Our blogger seems to argue that they should not be "open weights" but only for employees. There are only three options:
#1 Nobody owns open-weights models because further development of models is forbidden by law, and this law is enforced worldwide and effectively, which is what out blogger usually advocates IIRC.
#2 Both owners and users have access to open-weights models that is potentially or actually without guardrails for both categories.
#3 Only oligarchs have access to open-weight models without guardrails because they own them, and employees have no such access and are restricted by guardrails.
Case #3 has potentially some variants as to owns the open-weight models:
#3b Open-weight models are collectively owned by the oligarchs of each country as they can only be owned by the governments, and their owners can do anything they want with them, while restricting them to their employees (which is the same regime as for nuclear, viral, chemical WMDs).
#3c Open-weight models can only be owned by a single supra-national authority that is owned by all major governments on Earth, is managed and audited by those to ensure that no governments on Earth (whether mebers of that authority or not) have access to open-weights models without guardrails agreed on by those major governments, and all governments on Earth are subjected to an extreme inspection regime to ensure nobody cheats (this is the logical endpoint of "Plan A").
> Of all the braindead midwit takes out there this is the one. If you thought for more than two seconds you’d realize: For who? What is marketable about that?
I also believe Anthropic is being earnest but let’s be honest: obviously such displays cause valuable positive impressions maybe not to the public at large but mostly to the exact kind of talent they want to attract. And keep. Being the place to be is extremely valuable, and losing *it* can be fatal to que quality of a bleeding edge org
«Open Weight Models Are Unsafe And Nothing Can Fix This
The fundamental issue is that once an open weight model is out there, you cannot take it back in any reasonable fashion, and users can easily unlock any of its core capabilities, to be used for any purpose, or unleash it on its own.»
Bu that is already the case: every model is "open weights" for the oligarchs who own it. Our blogger seems to argue that they should not be "open weights" but only for employees. There are only three options:
#1 Nobody owns open-weights models because further development of models is forbidden by law, and this law is enforced worldwide and effectively, which is what out blogger usually advocates IIRC.
#2 Both owners and users have access to open-weights models that is potentially or actually without guardrails for both categories.
#3 Only oligarchs have access to open-weight models without guardrails because they own them, and employees have no such access and are restricted by guardrails.
Case #3 has potentially some variants as to owns the open-weight models:
#3b Open-weight models are collectively owned by the oligarchs of each country as they can only be owned by the governments, and their owners can do anything they want with them, while restricting them to their employees (which is the same regime as for nuclear, viral, chemical WMDs).
#3c Open-weight models can only be owned by a single supra-national authority that is owned by all major governments on Earth, is managed and audited by those to ensure that no governments on Earth (whether mebers of that authority or not) have access to open-weights models without guardrails agreed on by those major governments, and all governments on Earth are subjected to an extreme inspection regime to ensure nobody cheats (this is the logical endpoint of "Plan A").
> Of all the braindead midwit takes out there this is the one. If you thought for more than two seconds you’d realize: For who? What is marketable about that?
I also believe Anthropic is being earnest but let’s be honest: obviously such displays cause valuable positive impressions maybe not to the public at large but mostly to the exact kind of talent they want to attract. And keep. Being the place to be is extremely valuable, and losing *it* can be fatal to que quality of a bleeding edge org