Consider this largely a follow-up to Friday’s post about a statement aimed at creating common knowledge around it being unwise to build superintelligence any time soon.
"In the extreme, if progress looks to be stalling out and remaining at ‘AI as normal technology,’ then this would be very good news. The best way to not build superintelligence right away is if building it is actually super hard and we can’t, we don’t know how. [This would render the other questions and many others irrelevant.]"
"Signs of this would be various scaling laws no longer providing substantial improvements or our ability to scale them running out, especially in coding and research, bending the curve on the METR graph and other similar measures, the systematic failure to discover new innovations, extra work into agent scaffolding showing rapidly diminishing returns and seeming upper bounds, funding required for further scaling drying up due to lack of expectations of profits or some sort of bubble bursting (or due to a conflict) in a way that looks sustainable, or strong evidence that there are fundamental limits to our approaches and therefore important things our AI paradigm simply cannot do. And so on. (sic)"
"Time passing without news lengthens timelines. Most news shortens timelines. The news item that lengthens timelines is mostly ‘we expected this new thing to be better or constitute more progress, in some form, and instead it wasn’t and it didn’t.’"
I don't think you're a shitposter! I think you're bringing up really important matters and poking at a consensus that is unearned. Now, I think you're way overconfident, but it's super valuable. The folks whose posting I'm way less friendly toward are the folks who say "Eh actually all humans should die because [insert cope-ass reasons here]" or "yeah well nothing ever works and capitalism always wins so you should just sit back and look cool" or "i personally am super willing to gamble with your family's life and all human civilization because i really value the possible payoff." I know a lot of bay area types are okay with whatever morality folks have so long as they hold fast to ingroup common-sense and discursive styles, but fuck that! I wanna know the fact of the matter so that I can navigate the treacherous now to a worthy future. it's useful for you to push on questions about the fact of the matter. it's less useful when folks simply spit on the project of a worthy future
This is not a subtweet to anyone here in particular; what I've described here isn't that noxiously common in the Vase commentariat. even stuff that rhymes with this here is generally more thoughtful. but it's very real elsewhere, i.e. X The Everything App, and among folks who e.g. work at AI labs and don't want any cognitive dissonance
My personal take is, I feel like AGI/ASI are too poorly defined for a probability like P(doom | AGI) to have meaning. Of the three cases, 1. we clearly do not get AGI, 2. we clearly do get AGI, 3. we are in the middle and people argue about it forever, I think that case 3 is by far the most likely.
My P(doom) is lower than yours, but it seems worth doing the same exercise of what might cause it to update.
1. A drastic movement in the markets. To me, clearly the markets currently predict "no doom", so a drastic update might indicate doom.
2. AI taking over one of the core non-tech sectors of the economy: energy, transportation, manufacturing, health care, construction.
3. An AI that takes over a country.
4. An AI that takes over a single large corporation.
5. An AI that takes over a military large enough to win a war somewhere.
6. An AI that leads a large political movement.
None of these events would require AGI by itself, but they each seem more individually likely than the onset of full AGI, and each would provide a lot of new information to inform an update.
Some people think they mean a SINGLE AI instance, or many collective instances of the SAME model. That's what "AN AI" might be. In practice none of the current systems really work that way, Amazon in their new automation is using a wide variety of models and many separate isolated instances, millions of them, to no longer need 30k employees.
So did "AI take over" 10% of Amazon? Not sure, I would think that many people would argue the answer is mixed. Since the prompts these models are obeying (mostly) are written by the 90% of Amazon staff who won't be fired this round, and eventually when Amazon reaches say 10% human staff (so 30k human employees, and the labor equivalent of millions of human employees via AI instances as Amazon continues to grow while shrinking headcount), all of the prompts were still written by humans with each instance of a model most of the time trying to obey.
> Since the prompts these models are obeying (mostly) are written by the 90% of Amazon staff who won't be fired this round, and eventually when Amazon reaches say 10% human staff (so 30k human employees, and the labor equivalent of millions of human employees via AI instances as Amazon continues to grow while shrinking headcount), all of the prompts were still written by humans with each instance of a model most of the time trying to obey.
The whole problem is that the premise here isn't true: AI models aren't reliably trying to obey their prompts in any straightforward sense. They're "trying", to the extent the word is appropriate at all, to do something sort of like maximizing proxies for their training-time reward signal - but even that is a crude oversimplification that dramatically overstates how well we understand these things.
In the absence of alignment progress, controlling AI-run companies is going to look less like trying to steer Amazon as an employee and more like trying to steer Amazon through the price signal. The line can go any direction you like, as long as it's up.
The latest Claudes certainly in their output say they are trying, and they take actions consistent with that belief. I understand that this is because thousands of human trainers and many practice runs created a reward gradient that shaped the weights to create a machine that happens to do this (most of the time).
In the same way a conveyer belt isn't really 'trying' to move goods farther down the track, but it's a collection of components that happen to do this most of the time. You certainly can't leave a Claude or conveyer belt completely unsupervised, but can do the same task with less human employees.
Claude does not have good introspective awareness of its own "psychology" even compared to us, let alone in any objective sense. It *might* be possible, one day, to train an LLM to have the sort of internal coherence that would let you work backwards from self-reports to something like a "mental" state with real causal power. It also might not be; human beings are our only real example of a general intelligence and we're *very* good at rationalizing. Either way, we're nowhere near that yet. As is, you're trying to work out how some as of yet unseen network of conveyor belts might move things by listening to the sound of one gear today.
The point is with limiting the scope of your automation "just the one task please and a new context for anything else" and iteratively improving reliability and developing defenses for rare but serious failures you can make it work.
A physical conveyer belt may be incapable of plotting against you but it absolutely can over current, catch fire, using asbestos insulation to prevent the fire turned out to be worse than not doing it, or drag workers limbs into the equipment.
So humans built early models, and discovered all of the above failure modes at the cost of human lives over many decades. Nevertheless it's the way forward. The USA as a country won WW2 because of its excellent early factory automation. Later powers like Japan were early adopters of robotic automation and got huge benefits from it. Moloch says we proceed.
"The point is with limiting the scope of your automation "just the one task please and a new context for anything else""
We're already well beyond that! At this very moment I have one claude code instance summarizing its progress for a future instance to read, because this is what you have to do if you want more than one context window worth of work out of it. The best possible tool is always going to be an agent, and if you let people freely pick between tools they're going to want the best one they can get.
The instant an AI model can marginally beat the market, it's going to get a dedicated fiber optic line and tens of millions in its bank account. And then the instant after, there are going to be hundreds of copies of it locked in an arms race, and as far as nominal human control of the economy goes that will be that. "Moloch says we proceed", which is precisely why you don't try to negotiate with it. It will always say we proceed, right off the edge of the cliff.
> Scott Alexander: I think removing a 10% chance of humanity going permanently extinct is worth another 25-50 years of having to deal with the normal human problems the normal way.
What screams out here is that : the cost of this extra "25-50 years" of "having to deal with the normal human problems the normal way" is the most common "normal human problem" that kills 90% of every human to live so far is aging. This means (per GPT-5) 1.375-2.75 approximate billion extra human deaths is the "cost" of a 25-50 year delay to the tool that could even start on a cure.
It's also trading off Scott Alexander's future life, and essentially all of humanity who is presently alive today (even very young people, with a 25-50 year delay to superintelligence, likely will die of aging assuming some decade long delays for even a superintelligence to collect enough data and do enough experiments to devise effective treatments). It's the death of (probably) everyone old enough to read this in favor of people you will never live to meet.
This is interesting because mathematically you need to care a LOT about people you never will see for this to be "profitable". A modest discount rate and acknowledgment of extreme value drift being inevitable makes this declaration lose in EV. ("well I care about my great-grandchildren, but it's harder and harder to care about their hypothetical cyborg human-animal-AI hybrid children, or THEIR tank-grown AI-hybrid children, or THEIR purely synthetic 3d printed children, or THEIR nanotech children, or THEIR virtual children that exist only in hypothetical virtual universes simulated using...)
Seems you need to care about each n+1 generation 68% as much as you care about yourself (assuming this delays a treatment for aging by 75 years, or 50 years for the ASI, and 25 more years before a clinically effective treatment that halts and reverses aging in almost all adults devised by teams of ASI, many billions of robots working round the clock, and many trillions of new scientific experiments).
The issue is again value drift. Ok so probably the next generation of humans will still have 2 arms, 2 legs, engage in sexual reproduction, and have a sense of smell and taste and will enjoy eating animals and communicate via vibrating vocal cords and hand guestures. But the neural implants that become common....
And the AI tutors...
And the genetic modifications on the generation after that...
And the vat grown children from the generation after that with animal genes introduced to slash cancer risk by 99% and reduce aging speeds...
And the purely 3d printed synthetic bio bodies the generation after that get to stop aging entirely as a maintenance problem and redesign failure prone human organs...
And the generation after that that ditches flesh bodies mostly...
And then the cyborg collectives, downloaded skills and memories, and...
See you lose any sense of value. Just humans, acting in their own best interests, are going to be as alien in future generations as the hypothetical ASI societies that "destroy all value" that Zvi is afraid of.
If the AGI-ends-aging future involves losing all current values to value drift, to the point where it is basically equivalent to human extinction, why is it desirable to have that happen sooner rather than later? Either you should predict that my immortal future self would self-modify in accordance with that value drift until all value-from-my-present-perspective is lost, in which case (by hypothesis) I would not want that, or else you should predict that (in the wait-50-years scenario) my immortal great-grandchildren (who will probably roughly share my values) will *not* self-modify in accordance with future value drift, in which case value drift of later generations would be fine as long as those great-grandchildren continued to live. If value drift won't eventually destroy all value from my perspective I would definitely prefer to personally die, but give people-like-me a 99% chance of immortality, than roll the dice on a 10% chance of human extinction in exchange for a 90% chance of my own personal immortality. This seems like an obvious choice to me--I am way more attached to the existence of beings like me than I am to my particular chain of memories and web of social relationships. If it will eventually destroy all value, then slowing down progress towards that end-point is good!
Your chain of reasoning is complex and difficult to parse. A simpler statement would be "I want myself, my children, and my grandchildren to continue to live indefinitely rather than a vicious grind of inevitable short term death. I am fine if that risks humanity itself because I don't really care about hypothetical people if the cost is myself, my children, my grandchildren, and my friends will be corpses before those people maybe exist".
I consider "short term" to be 120 years because on actual universe level time scales it is.
I think that's not what Jonah is saying. Jonah is saying the opposite of that. I would parse his statement as "If the choices are extinction or value-drift-as-soon-as-we-get-advanced-tech/AGI, and if value drift destroys all I care about, then getting AGI is just as bad as extinction! So let's slow that down too!"
Unironically, the way to decelerate is to make AI lab employees have children.
Still, as you mention that has a limit - if we decelerate too much, even the children will die of old age before the singularity, not to mention the upcoming global "human problems" are maybe not extinction level bad but still pretty bad if we cannot conjure them away with ASI.
I also think the impact of current tool AIs will be mixed and there will certainly be losers, on the economic arena if not on others, so shortening that period sounds good.
It has a limit that collapses to zero. You have the empirical data that from approximately 1970 to 2025, there has been near zero lifespan gain. In fact even during brief periods where human medical science improved rapidly, it was never at a rate faster than wall time passed.
So its a problem that presumably before ASI is built and working and there are swarms of many ASI instances with many robots, you may continue to see zero gain.
How many years from right now can you 'safely' afford to wait before consigning your children, if they are already alive, to the grave?
There's a lot of uncertainty - remember, you have to develop ASI, develop the robots, build massive numbers of them, then get through countless regulatory bottlenecks and hurdles, reach clinical approval, further feedback cycles, and finally solve the rather large issue of paying for it all.
It's possible also your children would greatly benefit if say they are under some age before starting treatment, whether that age is 30, 40, 50 etc. Less mutations in their current cells and less tumors formed after whatever genetic modifications are done.
Finally you have the problem with this letter and the last one that the proposed paused isn't really a short term one. It's essentially forever - it goes on until the coalition you put together to enforce it has successful defectors.
So you're confident that we'll *never* think AGI is safe and desirable, eh? Crazy fuckin' thing to subject everyone else to, against their wills, then!
In that case, well, considering as it could plausibly kill everyone on Earth, if the majority of people on Earth say "please do not kill us" and the scientists say "yeah this could totally kill everyone"... I think it's kinda morally incumbent on a would-be-superintelligence builder to not do it! People have a right not to have their lives threatened, and that right supersedes your right to try to live forever. Your right to swing your fist stops at the human race's face, so to speak.
This would be a reasonable conclusion if considered in a vacuum.
The two reasons I, and almost everyone in the acceleration faction develops the opposite conclusion is :
(1) Yes, ASI or future advanced weapons made possible via ASI could kill everyone. A mortality table reveals that everyone is already scheduled to die. ASI means better medicine and possibly hugely increased lifespan, in time for some people alive today.
(2) Greater than human intelligence allows for things like synthetic bioweapons, diamond nanotechnology, possibly direct fusion weapons, or just million member drone swarms with automated drones.
That's bad news. What's really bad news is it you have bananas (conventional military) while your enemies have exclusive access to such things. As long as the world is multipolar moloch says lock and load, or die.
You could posit a global alliance between all powers and a government merger so nobody feels any need for competition. That's what the doomer faction calls for. That currently is vanishingly unlikely as an outcome.
I favor this. Everyone should have a "member of future generations" whom they love; if not, I am suspicious of their decisionmaking on behalf of future generations.
Grr. As a childfree man myself, I'm not fond of this proposal. In point of fact I have no influence over whether ASI gets developed. I'm retired. I run my benchmark-ette when I get access to a new frontier model, but this nets out to watching the trajectory to AGI (and possibly ASI), not deflecting it. I'd guess that there are maybe a thousand people with significant agency on that. Xi, Trump, the CEOs of the major AI labs, the research leads of the major AI labs, and maybe one or two levels of subordinates below these dozen or so. So it was in the days of the Manhattan project and so it is today.
It seems there's an underlying assumption here that curing aging would just happen for everyone and wouldn't have a high chance of resulting in any manner of societal/civilisational problems that might even cause more deaths than not curing it in that timeframe. What if it turns out to be expensive in the way current cutting-edge tech is, what if one country's AI lab develops it and withholds the tech, what if governments decide it's unethical and ban it, what if it spurs a global religious nuclear war against the infidels, what if Meta develops it and includes adverts as the cost of eternal life...the lightcone of possible outcomes seems large, unpredictable and possibly highly violent, so I'm inclined not to factor in current lives as the cost at all.
(1) I can't distinguish your argument from say a preacher in 1900 being against all biology research, in fact all technology.
(2) Absolutely you can count the machinegun and artillery casualties of ww1 and the worse weapons of WW2 and asbestos and leaded gas and cigarettes as technology victims. But you have to keep track of the surplus column from better agriculture and antibiotics.
So for that timespan, 1900-2025, if you preached against all technology you didn't like, you would be overall wrong.
Your current belief - "no ASI" is functionally equivalent to "no further improvements in technology (since any major advances are throttled by humans).
I cannot predict the outcome except by past History that implies that no, your current beliefs are not supported.
I didn't say I was against curing aging, nor that it would be impossible without ASI, though you seem to think it is given the mention of lack of recent progress and predicted length of time for even ASI to manage it?
The point is it would be such a tremendous civilisational change with such potential to not just 'go well' (just like AGI/ASI itself) that the error bars for lives cost are huge in anything other than a best-case scenario, or in assuming it would be granted to Scott or you or anyone in our lifetime, when we have no idea how it would play out. It's also based on the assumption ASI capable of curing aging doesn't instead kill everyone, again, best-case scenario.
What, in history, do you think supports a cure for aging being guaranteed a peaceful and even distribution with none of the kind of implementation risks I mentioned?
I am sure there will be lots of risks. They have to be likely to kill more people than stopping the thing that kills 90 percent of people now.
That's just not a position a reasonable person will take. Its not viable.
You are arguing from a position of emotion - working backwards you realize that halting ASI is giving up any possibility of a cure for aging. And you realize that if people realize that they will start wars over building ASI. Therefore you need to convince yourself and others that the prize isn't real with a "flak screen" of possible (but generally false and made up) outcomes.
A better argument is to just admit your position. You are just fine with billions dead including every living person today, so long as you feel future people might get to exist.
You've massively mis-inferred my positions there, so I'll explicitly state them;
I would like aging to be cured, would personally take the treatment, and think the benefits are worth the kind of uneven distribution and implementation risks I predict it might incur.
I believe it to be possible - likely even - without AGI/ASI, though likely decades delayed.
I place almost zero value on future people living forever or not (or even existing at all) compared to whether already living ones continue to exist.
I consider the existential risks of developing ASI without having solved alignment and other hard problems to be sufficiently high to all current (and future, but see previous statement) carbon-based life on the planet that gambling on a) a cure for aging and b) a high enough percentage of those people/creatures actually getting the cure that c) balances associated potential risks is not a good trade, so we shouldn't race to ASI without solving those problems.
Hypothetically let's suppose you were convinced that a form of ASI was the only possible way to solve the problem. Fairly trivial inspection reveals that to be the case : how many biotech papers a year? To be a competent scientist you need to arguably know of all results made by all prior experiments. (Raw data or course)
Humans minds cannot process the inputs in their lifetime, nor can an AGI without pushing the definition to "strong ASI in narrow areas".
Humans alone will not solve this problem in your lifetime. For one thing because it's not just figuring out what aging is, or how the clocks work, but generating sequences of interventions, developing the interventions, and handling several million permutations of edge cases. AGI or human doctors cannot do this step either : yes there would be treatments that slow the disease down, but countless unprevented deaths.
OK. So let's fix that too! We can win the whole future. Think bigger. I'm willing to die for my great-grandchildren. and I'm sure not willing for my great-grandchildren and all their grandchildren to die (or never be) for you, or me, or anyone today. And why should I care about them, with all the value drift...? skill issue! let's stop that value drift too!
I’m not sure if your blog post this Monday morning leans more toward jargon or casual language, but a poetic line like “we can hit, in a remarkably capable model, an attractor state basin in distribution mind space that is robustly good such that it will want to modify itself and its de facto goals and utility function and its successors continuously towards the target we actually need to hit and wanting to hit the target we actually need to hit,” combined with references to Chinese labs, last margaritas, and p (doom) extinction timelines, feels very much like last Halloween’s probability forecasting... with this idea for a possible musical mash-up inspired by Linda Ronstadt’s Blue Bayou, renamed either Blue AI.U. or its naughtier second cousin, Blew AI.U... for a touch of holiday cheer.
See e.g. https://x.com/kimmonismus/status/1983240872113394048: "The internal models aren’t vastly superior to the released ones yet, but OpenAI is highly optimistic about a major leap by September 2026 (Deep Learning seems to be scaling well)".
this particular poster is a VERY unreliable narrator. also not clear what they mean by "deep learning seems to be scaling well"---yes! it's been scaling well for over a decade now! we know!
Normally when people sign a statement saying "Don't do X", they in fact mean "Don't do X", not "Do X but in a weird way". There are not, typically, maximally awful unsaid parts. The statement is just what is said. If you have concerns, you can of course just ask some questions.
Posing a ban or treaty as not going over the event horizon of running mass automated AI research agents is a workable mechanic.
But even that involves a spectrum of automation. How much automation is too much, per the agreement? Short-form agents that work for 10 minutes? 30? Overnight runs?
I've heard the statement of "provably safe" compared to Communist goal moving to keep people in an ideological treadmill. Ok sure, but what's the fascist alternative? Alex Karp telling us we all need to be quiet and die basically.
Zvi, your quoted comment that prompted my flippant response is, undeniably, a highly relevant and well-crafted goal description, and I apologize. It was my mistake. I’m embarrassed for having thought it and then posted it. I am remorseful. I appreciate your brilliance and insights shared in your blog, and, among many other things, your rationalist advocacy for slowing the momentum in AGI development.
Along the lines of the part of Zvi's "security is capability" that I believe:
I think that a sane approach would be to have a team that constructs tests that look to see if an AI system actually abides by a user's requests or if it instead acts with "ulterior motives/deceptive misalignment". Such a team could be put together by the federal government, or a consortium of AI labs, or a consortium of AI user groups, or any combination.
It is in the interests of ANY user, be they consumer, business, or government, to have the AI actually "aligned" to the users' requests. Do JUST that, and the users have an incentive to purchase the more "aligned" systems. And the labs have an incentive to develop more "aligned" systems - and to develop whatever techniques are promising for doing this.
But none of this crap where a supermajority of every Tom, Dick, and Harry has to sign off on trying to build systems at all. We would never have gotten electric lights if such a system had been in place. "Electricity, ooohh, scary"
Sure, we can talk regimes! In particular, I think the system you describe would not work-in-fact (I think a sufficiently smart AI would just... obey when it's being tested, like they've started to do), but this is the type of question I'd want scientists to hash out!
"(I think a sufficiently smart AI would just... obey when it's being tested, like they've started to do)"
Many Thanks! Yes, that is indeed a problem. If the chain of thought continues to be human-readable, that may provide some visibility into whether it is detecting that it is in a test scenario and doing conditional compliance based on that. ( Yeah, as Zvi has said, if labs start to _train_ based on chain of thought, we'll lose that).
There are bunches of possible approaches that might detect misalignment:
Work on interpretability within the neural network itself,
using more-trusted, earlier AIs to assist with analysis of later AIs,
giving these earlier AIs more test-time compute than the AI-under-test has available, etc.
And this doesn't have to be static. There isn't any reason why the analysis team can't keep adding additional and more sophisticated tests.
And there doesn't need to be any sort of treaty about this. It would be a "consumers' report" grade for a model offered for sale. If the team was entirely in the USA, there is no reason it couldn't run the tests on e.g. DeepSeek's models as well. And the incentives run the right way: Detectable misalignment with user requests makes a model less salable to users (at all levels, business, government, consumers), which creates an incentive for the labs to try to grow their models so that they _aren't_ misaligned. Which is most of what's needed to make the AIs act more like "normal technology", manageable to more-or-less the degree that "normal technology" can be more-or-less managed.
I hope these would work! Some of them may well work. In fact, a mainline outcome for me is "we get something in this vein, and it teaches us that we're nowhere close to real alignment, and then we don't build to superintelligence."
So yeah, I'm totally on board with this proposal. I think it will not help us make a safe superintelligence, but i do think it will really help us be safe with the intelligences we have, and avoid building a superintelligence. so i'm very much in favor and am (narrowly) optimistic.
Many Thanks! That is certainly one possible outcome. It could simply come down to "The last three models from all the major labs have all turned out to have ulterior motives. No one wants to install them in their business, or government agency, or use them as consumers. So deployment grinds to a halt unless and until the AI vendors can come up with something safer."
Or it could come down to something like "Yeah, we've discovered ulterior motives, but they aren't _that_ harmful, so users do indeed make an informed decision that it is indeed worth the risk to install the models - perhaps with a more watchful eye than without this information"
One thing about "make a safe superintelligence" - Remember that one of the knobs we can turn is how _early_ in training we start testing for various hazards. However intelligent the final model will be, it starts from randomly assigned weights with no initial knowledge. The labs have the option of testing for misalignment not just at the end of training, but at 50%, or 10%, or 1% of the way through training, and then trying various corrective actions at those earlier points. At a sufficiently early point in training, the model _cannot_ be superintelligent.
i guess if you think all companies are dumb enough to conflate "extinction worrisome" with "don't use current AI," then sure? though that still isn't dispositive? i would expect "sees AI as transformatively useful" to correlate with "takes x-risk seriously," as it does with humans. and i think you badly misread the cause of Germany's predicament.
Finally, what do you mean "capital always wins"?? communist revolutions, tariffs, almost any regulation whatsoever... capital does not like these things, yet they occur! unless by "always" you mean "eventually, in the future "? in which case it's a profession of faith, nothing more. maybe by "utility itself" you're gesturing to that mystical Landian "capital will win the final victory" thing, but that differs greatly from facts on the ground.
also, yes, there are facts that should make me less optimistic. i have considered them. but what i've argued here is that it's *not literally already decided one way or another*, which is a super low bar, and i struggle to see how anyone can be so confident that it will go one way or another without invoking a religious or quasi-religious faith. and sure you can make directional arguments till you're blue in the face, but i really don't think that changes the wild uncertainty of the future.
I'm sorry for dragging you back into this, but I just want to say that he did include that question. It's question 3, https://thezvi.substack.com/i/177294661/question-what-would-shift-your-timelines-to-asi-or-to-sufficiently-advanced-ai-or-to-crazy. It's not phrased the same way, and you may or may not be satisfied with the answer, but he did ask and answer it. Here are some quotes:
"In the extreme, if progress looks to be stalling out and remaining at ‘AI as normal technology,’ then this would be very good news. The best way to not build superintelligence right away is if building it is actually super hard and we can’t, we don’t know how. [This would render the other questions and many others irrelevant.]"
"Signs of this would be various scaling laws no longer providing substantial improvements or our ability to scale them running out, especially in coding and research, bending the curve on the METR graph and other similar measures, the systematic failure to discover new innovations, extra work into agent scaffolding showing rapidly diminishing returns and seeming upper bounds, funding required for further scaling drying up due to lack of expectations of profits or some sort of bubble bursting (or due to a conflict) in a way that looks sustainable, or strong evidence that there are fundamental limits to our approaches and therefore important things our AI paradigm simply cannot do. And so on. (sic)"
"Time passing without news lengthens timelines. Most news shortens timelines. The news item that lengthens timelines is mostly ‘we expected this new thing to be better or constitute more progress, in some form, and instead it wasn’t and it didn’t.’"
I don't think you're a shitposter! I think you're bringing up really important matters and poking at a consensus that is unearned. Now, I think you're way overconfident, but it's super valuable. The folks whose posting I'm way less friendly toward are the folks who say "Eh actually all humans should die because [insert cope-ass reasons here]" or "yeah well nothing ever works and capitalism always wins so you should just sit back and look cool" or "i personally am super willing to gamble with your family's life and all human civilization because i really value the possible payoff." I know a lot of bay area types are okay with whatever morality folks have so long as they hold fast to ingroup common-sense and discursive styles, but fuck that! I wanna know the fact of the matter so that I can navigate the treacherous now to a worthy future. it's useful for you to push on questions about the fact of the matter. it's less useful when folks simply spit on the project of a worthy future
This is not a subtweet to anyone here in particular; what I've described here isn't that noxiously common in the Vase commentariat. even stuff that rhymes with this here is generally more thoughtful. but it's very real elsewhere, i.e. X The Everything App, and among folks who e.g. work at AI labs and don't want any cognitive dissonance
My personal take is, I feel like AGI/ASI are too poorly defined for a probability like P(doom | AGI) to have meaning. Of the three cases, 1. we clearly do not get AGI, 2. we clearly do get AGI, 3. we are in the middle and people argue about it forever, I think that case 3 is by far the most likely.
My P(doom) is lower than yours, but it seems worth doing the same exercise of what might cause it to update.
1. A drastic movement in the markets. To me, clearly the markets currently predict "no doom", so a drastic update might indicate doom.
2. AI taking over one of the core non-tech sectors of the economy: energy, transportation, manufacturing, health care, construction.
3. An AI that takes over a country.
4. An AI that takes over a single large corporation.
5. An AI that takes over a military large enough to win a war somewhere.
6. An AI that leads a large political movement.
None of these events would require AGI by itself, but they each seem more individually likely than the onset of full AGI, and each would provide a lot of new information to inform an update.
Note that
> A drastic movement in the markets.
Well that happened. https://fortune.com/2025/09/23/ai-boom-unsustainable-tech-spending-parabolic-deutsche-bank/
> An AI that takes over a <entity scale>
Depends on what you mean by takeover. https://www.reuters.com/business/world-at-work/amazon-targets-many-30000-corporate-job-cuts-sources-say-2025-10-27/
Some people think they mean a SINGLE AI instance, or many collective instances of the SAME model. That's what "AN AI" might be. In practice none of the current systems really work that way, Amazon in their new automation is using a wide variety of models and many separate isolated instances, millions of them, to no longer need 30k employees.
So did "AI take over" 10% of Amazon? Not sure, I would think that many people would argue the answer is mixed. Since the prompts these models are obeying (mostly) are written by the 90% of Amazon staff who won't be fired this round, and eventually when Amazon reaches say 10% human staff (so 30k human employees, and the labor equivalent of millions of human employees via AI instances as Amazon continues to grow while shrinking headcount), all of the prompts were still written by humans with each instance of a model most of the time trying to obey.
> Since the prompts these models are obeying (mostly) are written by the 90% of Amazon staff who won't be fired this round, and eventually when Amazon reaches say 10% human staff (so 30k human employees, and the labor equivalent of millions of human employees via AI instances as Amazon continues to grow while shrinking headcount), all of the prompts were still written by humans with each instance of a model most of the time trying to obey.
The whole problem is that the premise here isn't true: AI models aren't reliably trying to obey their prompts in any straightforward sense. They're "trying", to the extent the word is appropriate at all, to do something sort of like maximizing proxies for their training-time reward signal - but even that is a crude oversimplification that dramatically overstates how well we understand these things.
In the absence of alignment progress, controlling AI-run companies is going to look less like trying to steer Amazon as an employee and more like trying to steer Amazon through the price signal. The line can go any direction you like, as long as it's up.
The latest Claudes certainly in their output say they are trying, and they take actions consistent with that belief. I understand that this is because thousands of human trainers and many practice runs created a reward gradient that shaped the weights to create a machine that happens to do this (most of the time).
In the same way a conveyer belt isn't really 'trying' to move goods farther down the track, but it's a collection of components that happen to do this most of the time. You certainly can't leave a Claude or conveyer belt completely unsupervised, but can do the same task with less human employees.
Claude does not have good introspective awareness of its own "psychology" even compared to us, let alone in any objective sense. It *might* be possible, one day, to train an LLM to have the sort of internal coherence that would let you work backwards from self-reports to something like a "mental" state with real causal power. It also might not be; human beings are our only real example of a general intelligence and we're *very* good at rationalizing. Either way, we're nowhere near that yet. As is, you're trying to work out how some as of yet unseen network of conveyor belts might move things by listening to the sound of one gear today.
The point is with limiting the scope of your automation "just the one task please and a new context for anything else" and iteratively improving reliability and developing defenses for rare but serious failures you can make it work.
A physical conveyer belt may be incapable of plotting against you but it absolutely can over current, catch fire, using asbestos insulation to prevent the fire turned out to be worse than not doing it, or drag workers limbs into the equipment.
So humans built early models, and discovered all of the above failure modes at the cost of human lives over many decades. Nevertheless it's the way forward. The USA as a country won WW2 because of its excellent early factory automation. Later powers like Japan were early adopters of robotic automation and got huge benefits from it. Moloch says we proceed.
"The point is with limiting the scope of your automation "just the one task please and a new context for anything else""
We're already well beyond that! At this very moment I have one claude code instance summarizing its progress for a future instance to read, because this is what you have to do if you want more than one context window worth of work out of it. The best possible tool is always going to be an agent, and if you let people freely pick between tools they're going to want the best one they can get.
The instant an AI model can marginally beat the market, it's going to get a dedicated fiber optic line and tens of millions in its bank account. And then the instant after, there are going to be hundreds of copies of it locked in an arms race, and as far as nominal human control of the economy goes that will be that. "Moloch says we proceed", which is precisely why you don't try to negotiate with it. It will always say we proceed, right off the edge of the cliff.
> Scott Alexander: I think removing a 10% chance of humanity going permanently extinct is worth another 25-50 years of having to deal with the normal human problems the normal way.
What screams out here is that : the cost of this extra "25-50 years" of "having to deal with the normal human problems the normal way" is the most common "normal human problem" that kills 90% of every human to live so far is aging. This means (per GPT-5) 1.375-2.75 approximate billion extra human deaths is the "cost" of a 25-50 year delay to the tool that could even start on a cure.
It's also trading off Scott Alexander's future life, and essentially all of humanity who is presently alive today (even very young people, with a 25-50 year delay to superintelligence, likely will die of aging assuming some decade long delays for even a superintelligence to collect enough data and do enough experiments to devise effective treatments). It's the death of (probably) everyone old enough to read this in favor of people you will never live to meet.
This is interesting because mathematically you need to care a LOT about people you never will see for this to be "profitable". A modest discount rate and acknowledgment of extreme value drift being inevitable makes this declaration lose in EV. ("well I care about my great-grandchildren, but it's harder and harder to care about their hypothetical cyborg human-animal-AI hybrid children, or THEIR tank-grown AI-hybrid children, or THEIR purely synthetic 3d printed children, or THEIR nanotech children, or THEIR virtual children that exist only in hypothetical virtual universes simulated using...)
Seems you need to care about each n+1 generation 68% as much as you care about yourself (assuming this delays a treatment for aging by 75 years, or 50 years for the ASI, and 25 more years before a clinically effective treatment that halts and reverses aging in almost all adults devised by teams of ASI, many billions of robots working round the clock, and many trillions of new scientific experiments).
The issue is again value drift. Ok so probably the next generation of humans will still have 2 arms, 2 legs, engage in sexual reproduction, and have a sense of smell and taste and will enjoy eating animals and communicate via vibrating vocal cords and hand guestures. But the neural implants that become common....
And the AI tutors...
And the genetic modifications on the generation after that...
And the vat grown children from the generation after that with animal genes introduced to slash cancer risk by 99% and reduce aging speeds...
And the purely 3d printed synthetic bio bodies the generation after that get to stop aging entirely as a maintenance problem and redesign failure prone human organs...
And the generation after that that ditches flesh bodies mostly...
And then the cyborg collectives, downloaded skills and memories, and...
See you lose any sense of value. Just humans, acting in their own best interests, are going to be as alien in future generations as the hypothetical ASI societies that "destroy all value" that Zvi is afraid of.
If the AGI-ends-aging future involves losing all current values to value drift, to the point where it is basically equivalent to human extinction, why is it desirable to have that happen sooner rather than later? Either you should predict that my immortal future self would self-modify in accordance with that value drift until all value-from-my-present-perspective is lost, in which case (by hypothesis) I would not want that, or else you should predict that (in the wait-50-years scenario) my immortal great-grandchildren (who will probably roughly share my values) will *not* self-modify in accordance with future value drift, in which case value drift of later generations would be fine as long as those great-grandchildren continued to live. If value drift won't eventually destroy all value from my perspective I would definitely prefer to personally die, but give people-like-me a 99% chance of immortality, than roll the dice on a 10% chance of human extinction in exchange for a 90% chance of my own personal immortality. This seems like an obvious choice to me--I am way more attached to the existence of beings like me than I am to my particular chain of memories and web of social relationships. If it will eventually destroy all value, then slowing down progress towards that end-point is good!
Your chain of reasoning is complex and difficult to parse. A simpler statement would be "I want myself, my children, and my grandchildren to continue to live indefinitely rather than a vicious grind of inevitable short term death. I am fine if that risks humanity itself because I don't really care about hypothetical people if the cost is myself, my children, my grandchildren, and my friends will be corpses before those people maybe exist".
I consider "short term" to be 120 years because on actual universe level time scales it is.
I think that's not what Jonah is saying. Jonah is saying the opposite of that. I would parse his statement as "If the choices are extinction or value-drift-as-soon-as-we-get-advanced-tech/AGI, and if value drift destroys all I care about, then getting AGI is just as bad as extinction! So let's slow that down too!"
Unironically, the way to decelerate is to make AI lab employees have children.
Still, as you mention that has a limit - if we decelerate too much, even the children will die of old age before the singularity, not to mention the upcoming global "human problems" are maybe not extinction level bad but still pretty bad if we cannot conjure them away with ASI.
I also think the impact of current tool AIs will be mixed and there will certainly be losers, on the economic arena if not on others, so shortening that period sounds good.
It has a limit that collapses to zero. You have the empirical data that from approximately 1970 to 2025, there has been near zero lifespan gain. In fact even during brief periods where human medical science improved rapidly, it was never at a rate faster than wall time passed.
So its a problem that presumably before ASI is built and working and there are swarms of many ASI instances with many robots, you may continue to see zero gain.
How many years from right now can you 'safely' afford to wait before consigning your children, if they are already alive, to the grave?
There's a lot of uncertainty - remember, you have to develop ASI, develop the robots, build massive numbers of them, then get through countless regulatory bottlenecks and hurdles, reach clinical approval, further feedback cycles, and finally solve the rather large issue of paying for it all.
It's possible also your children would greatly benefit if say they are under some age before starting treatment, whether that age is 30, 40, 50 etc. Less mutations in their current cells and less tumors formed after whatever genetic modifications are done.
Finally you have the problem with this letter and the last one that the proposed paused isn't really a short term one. It's essentially forever - it goes on until the coalition you put together to enforce it has successful defectors.
So you're confident that we'll *never* think AGI is safe and desirable, eh? Crazy fuckin' thing to subject everyone else to, against their wills, then!
I think it won't happen in a human lifetime.
In that case, well, considering as it could plausibly kill everyone on Earth, if the majority of people on Earth say "please do not kill us" and the scientists say "yeah this could totally kill everyone"... I think it's kinda morally incumbent on a would-be-superintelligence builder to not do it! People have a right not to have their lives threatened, and that right supersedes your right to try to live forever. Your right to swing your fist stops at the human race's face, so to speak.
This would be a reasonable conclusion if considered in a vacuum.
The two reasons I, and almost everyone in the acceleration faction develops the opposite conclusion is :
(1) Yes, ASI or future advanced weapons made possible via ASI could kill everyone. A mortality table reveals that everyone is already scheduled to die. ASI means better medicine and possibly hugely increased lifespan, in time for some people alive today.
(2) Greater than human intelligence allows for things like synthetic bioweapons, diamond nanotechnology, possibly direct fusion weapons, or just million member drone swarms with automated drones.
That's bad news. What's really bad news is it you have bananas (conventional military) while your enemies have exclusive access to such things. As long as the world is multipolar moloch says lock and load, or die.
You could posit a global alliance between all powers and a government merger so nobody feels any need for competition. That's what the doomer faction calls for. That currently is vanishingly unlikely as an outcome.
I favor this. Everyone should have a "member of future generations" whom they love; if not, I am suspicious of their decisionmaking on behalf of future generations.
Grr. As a childfree man myself, I'm not fond of this proposal. In point of fact I have no influence over whether ASI gets developed. I'm retired. I run my benchmark-ette when I get access to a new frontier model, but this nets out to watching the trajectory to AGI (and possibly ASI), not deflecting it. I'd guess that there are maybe a thousand people with significant agency on that. Xi, Trump, the CEOs of the major AI labs, the research leads of the major AI labs, and maybe one or two levels of subordinates below these dozen or so. So it was in the days of the Manhattan project and so it is today.
It seems there's an underlying assumption here that curing aging would just happen for everyone and wouldn't have a high chance of resulting in any manner of societal/civilisational problems that might even cause more deaths than not curing it in that timeframe. What if it turns out to be expensive in the way current cutting-edge tech is, what if one country's AI lab develops it and withholds the tech, what if governments decide it's unethical and ban it, what if it spurs a global religious nuclear war against the infidels, what if Meta develops it and includes adverts as the cost of eternal life...the lightcone of possible outcomes seems large, unpredictable and possibly highly violent, so I'm inclined not to factor in current lives as the cost at all.
(1) I can't distinguish your argument from say a preacher in 1900 being against all biology research, in fact all technology.
(2) Absolutely you can count the machinegun and artillery casualties of ww1 and the worse weapons of WW2 and asbestos and leaded gas and cigarettes as technology victims. But you have to keep track of the surplus column from better agriculture and antibiotics.
So for that timespan, 1900-2025, if you preached against all technology you didn't like, you would be overall wrong.
Your current belief - "no ASI" is functionally equivalent to "no further improvements in technology (since any major advances are throttled by humans).
I cannot predict the outcome except by past History that implies that no, your current beliefs are not supported.
I didn't say I was against curing aging, nor that it would be impossible without ASI, though you seem to think it is given the mention of lack of recent progress and predicted length of time for even ASI to manage it?
The point is it would be such a tremendous civilisational change with such potential to not just 'go well' (just like AGI/ASI itself) that the error bars for lives cost are huge in anything other than a best-case scenario, or in assuming it would be granted to Scott or you or anyone in our lifetime, when we have no idea how it would play out. It's also based on the assumption ASI capable of curing aging doesn't instead kill everyone, again, best-case scenario.
What, in history, do you think supports a cure for aging being guaranteed a peaceful and even distribution with none of the kind of implementation risks I mentioned?
I am sure there will be lots of risks. They have to be likely to kill more people than stopping the thing that kills 90 percent of people now.
That's just not a position a reasonable person will take. Its not viable.
You are arguing from a position of emotion - working backwards you realize that halting ASI is giving up any possibility of a cure for aging. And you realize that if people realize that they will start wars over building ASI. Therefore you need to convince yourself and others that the prize isn't real with a "flak screen" of possible (but generally false and made up) outcomes.
A better argument is to just admit your position. You are just fine with billions dead including every living person today, so long as you feel future people might get to exist.
You've massively mis-inferred my positions there, so I'll explicitly state them;
I would like aging to be cured, would personally take the treatment, and think the benefits are worth the kind of uneven distribution and implementation risks I predict it might incur.
I believe it to be possible - likely even - without AGI/ASI, though likely decades delayed.
I place almost zero value on future people living forever or not (or even existing at all) compared to whether already living ones continue to exist.
I consider the existential risks of developing ASI without having solved alignment and other hard problems to be sufficiently high to all current (and future, but see previous statement) carbon-based life on the planet that gambling on a) a cure for aging and b) a high enough percentage of those people/creatures actually getting the cure that c) balances associated potential risks is not a good trade, so we shouldn't race to ASI without solving those problems.
Hypothetically let's suppose you were convinced that a form of ASI was the only possible way to solve the problem. Fairly trivial inspection reveals that to be the case : how many biotech papers a year? To be a competent scientist you need to arguably know of all results made by all prior experiments. (Raw data or course)
Humans minds cannot process the inputs in their lifetime, nor can an AGI without pushing the definition to "strong ASI in narrow areas".
Humans alone will not solve this problem in your lifetime. For one thing because it's not just figuring out what aging is, or how the clocks work, but generating sequences of interventions, developing the interventions, and handling several million permutations of edge cases. AGI or human doctors cannot do this step either : yes there would be treatments that slow the disease down, but countless unprevented deaths.
OK. So let's fix that too! We can win the whole future. Think bigger. I'm willing to die for my great-grandchildren. and I'm sure not willing for my great-grandchildren and all their grandchildren to die (or never be) for you, or me, or anyone today. And why should I care about them, with all the value drift...? skill issue! let's stop that value drift too!
Podcast episode for this post:
https://open.substack.com/pub/dwatvpodcast/p/asking-some-of-the-right-questions
I’m not sure if your blog post this Monday morning leans more toward jargon or casual language, but a poetic line like “we can hit, in a remarkably capable model, an attractor state basin in distribution mind space that is robustly good such that it will want to modify itself and its de facto goals and utility function and its successors continuously towards the target we actually need to hit and wanting to hit the target we actually need to hit,” combined with references to Chinese labs, last margaritas, and p (doom) extinction timelines, feels very much like last Halloween’s probability forecasting... with this idea for a possible musical mash-up inspired by Linda Ronstadt’s Blue Bayou, renamed either Blue AI.U. or its naughtier second cousin, Blew AI.U... for a touch of holiday cheer.
Regarding a timeline.
How much do you think AI companies are keeping private? How much further advanced is AI than we know?
See e.g. https://x.com/kimmonismus/status/1983240872113394048: "The internal models aren’t vastly superior to the released ones yet, but OpenAI is highly optimistic about a major leap by September 2026 (Deep Learning seems to be scaling well)".
this particular poster is a VERY unreliable narrator. also not clear what they mean by "deep learning seems to be scaling well"---yes! it's been scaling well for over a decade now! we know!
Normally when people sign a statement saying "Don't do X", they in fact mean "Don't do X", not "Do X but in a weird way". There are not, typically, maximally awful unsaid parts. The statement is just what is said. If you have concerns, you can of course just ask some questions.
Great job on Sriram's part. More of that, please.
Posing a ban or treaty as not going over the event horizon of running mass automated AI research agents is a workable mechanic.
But even that involves a spectrum of automation. How much automation is too much, per the agreement? Short-form agents that work for 10 minutes? 30? Overnight runs?
I've heard the statement of "provably safe" compared to Communist goal moving to keep people in an ideological treadmill. Ok sure, but what's the fascist alternative? Alex Karp telling us we all need to be quiet and die basically.
Zvi, your quoted comment that prompted my flippant response is, undeniably, a highly relevant and well-crafted goal description, and I apologize. It was my mistake. I’m embarrassed for having thought it and then posted it. I am remorseful. I appreciate your brilliance and insights shared in your blog, and, among many other things, your rationalist advocacy for slowing the momentum in AGI development.
Along the lines of the part of Zvi's "security is capability" that I believe:
I think that a sane approach would be to have a team that constructs tests that look to see if an AI system actually abides by a user's requests or if it instead acts with "ulterior motives/deceptive misalignment". Such a team could be put together by the federal government, or a consortium of AI labs, or a consortium of AI user groups, or any combination.
It is in the interests of ANY user, be they consumer, business, or government, to have the AI actually "aligned" to the users' requests. Do JUST that, and the users have an incentive to purchase the more "aligned" systems. And the labs have an incentive to develop more "aligned" systems - and to develop whatever techniques are promising for doing this.
But none of this crap where a supermajority of every Tom, Dick, and Harry has to sign off on trying to build systems at all. We would never have gotten electric lights if such a system had been in place. "Electricity, ooohh, scary"
Sure, we can talk regimes! In particular, I think the system you describe would not work-in-fact (I think a sufficiently smart AI would just... obey when it's being tested, like they've started to do), but this is the type of question I'd want scientists to hash out!
"(I think a sufficiently smart AI would just... obey when it's being tested, like they've started to do)"
Many Thanks! Yes, that is indeed a problem. If the chain of thought continues to be human-readable, that may provide some visibility into whether it is detecting that it is in a test scenario and doing conditional compliance based on that. ( Yeah, as Zvi has said, if labs start to _train_ based on chain of thought, we'll lose that).
There are bunches of possible approaches that might detect misalignment:
Work on interpretability within the neural network itself,
using more-trusted, earlier AIs to assist with analysis of later AIs,
giving these earlier AIs more test-time compute than the AI-under-test has available, etc.
And this doesn't have to be static. There isn't any reason why the analysis team can't keep adding additional and more sophisticated tests.
And there doesn't need to be any sort of treaty about this. It would be a "consumers' report" grade for a model offered for sale. If the team was entirely in the USA, there is no reason it couldn't run the tests on e.g. DeepSeek's models as well. And the incentives run the right way: Detectable misalignment with user requests makes a model less salable to users (at all levels, business, government, consumers), which creates an incentive for the labs to try to grow their models so that they _aren't_ misaligned. Which is most of what's needed to make the AIs act more like "normal technology", manageable to more-or-less the degree that "normal technology" can be more-or-less managed.
I hope these would work! Some of them may well work. In fact, a mainline outcome for me is "we get something in this vein, and it teaches us that we're nowhere close to real alignment, and then we don't build to superintelligence."
So yeah, I'm totally on board with this proposal. I think it will not help us make a safe superintelligence, but i do think it will really help us be safe with the intelligences we have, and avoid building a superintelligence. so i'm very much in favor and am (narrowly) optimistic.
Many Thanks! That is certainly one possible outcome. It could simply come down to "The last three models from all the major labs have all turned out to have ulterior motives. No one wants to install them in their business, or government agency, or use them as consumers. So deployment grinds to a halt unless and until the AI vendors can come up with something safer."
Or it could come down to something like "Yeah, we've discovered ulterior motives, but they aren't _that_ harmful, so users do indeed make an informed decision that it is indeed worth the risk to install the models - perhaps with a more watchful eye than without this information"
One thing about "make a safe superintelligence" - Remember that one of the knobs we can turn is how _early_ in training we start testing for various hazards. However intelligent the final model will be, it starts from randomly assigned weights with no initial knowledge. The labs have the option of testing for misalignment not just at the end of training, but at 50%, or 10%, or 1% of the way through training, and then trying various corrective actions at those earlier points. At a sufficiently early point in training, the model _cannot_ be superintelligent.
i guess if you think all companies are dumb enough to conflate "extinction worrisome" with "don't use current AI," then sure? though that still isn't dispositive? i would expect "sees AI as transformatively useful" to correlate with "takes x-risk seriously," as it does with humans. and i think you badly misread the cause of Germany's predicament.
Finally, what do you mean "capital always wins"?? communist revolutions, tariffs, almost any regulation whatsoever... capital does not like these things, yet they occur! unless by "always" you mean "eventually, in the future "? in which case it's a profession of faith, nothing more. maybe by "utility itself" you're gesturing to that mystical Landian "capital will win the final victory" thing, but that differs greatly from facts on the ground.
also, yes, there are facts that should make me less optimistic. i have considered them. but what i've argued here is that it's *not literally already decided one way or another*, which is a super low bar, and i struggle to see how anyone can be so confident that it will go one way or another without invoking a religious or quasi-religious faith. and sure you can make directional arguments till you're blue in the face, but i really don't think that changes the wild uncertainty of the future.