The folks who brought you AI 2027, a so far remarkably accurate set of predictions despite those predictions having seemed freaky to many at the time, now bring you their positive vision that involves more freaky predictions: Plan A.
These guys have rather strong prediction track records. In addition to AI 2027, among other things, Daniel Kokotajlo has What 2026 Looks Like (which is remarkably similar to what 2026 looks like) and Ryan Greenblatt, who is also the chief scientist at Redwood Research, was the #2 most accurate AI forecaster in 2025 out of 413 entries. Past performance is as always no guarantee of future success.
If you’re the type to read at least some of my posts, or if you thought AI 2027 was worth reading, I recommend reading Plan A.
To be clear up front: I am not endorsing Plan A. I am not suggesting we should go off and try to enact Plan A as written. There is a lot more work to do and a lot of potential problems and downsides to grapple with.
I do think we should do that work, and give it, and its details, serious consideration.
The bulk of this post is engaging with various objections, in progressively more detail. If you only need the highlights, you can safely stop after Thus Selective Optimism.
As always, most people who interact with Plan A will not read it.
They will condense it into a few key sentences.
There is clear agreement on which sentences survive and which do not.
The top 5 things people will discuss will largely be, in descending order of focus:
We should slow down AI development.
To do that, we should make a deal with China.
We should monitor the world’s major sources of compute.
We should use mutually assured compute destruction (a version of MAIM).
Things will still happen super fast and feel like it, e.g. ASI by 2040, with vast economic growth before this.
Axios’s Ashley Gold compacts the plan thus:
Ashley Gold: The group behind a 2025 report predicting dire outcomes from AI development is out with a new prescription: To avoid dangerous outcomes from superintelligent AI, slow everything down.
That’s not technically wrong, but there is an obvious misinterpretation if you allow that much compression.
Daniel Kokotajlo: We think it’s still good to recommend what would actually be good, even if you think that your audience is probably not going to listen.
The strongest and loudest objection, and in some ways the best one, is some form of:
Plan A Detractors: Superintelligence is not coming any time soon. The threat is not real, so we shouldn’t be paying high costs to deal with it. There is no reason to slow down that which is already slow enough on its own.
At the extreme you get people like Joshua Saxe wondering why people didn’t learn their lesson from what didn’t happen to radiologists, and so on. Le sigh, but I appreciate saying it straight, and I appreciated Timothy Lee’s reaction even more:
Timothy B. Lee: I struggle with what to say about the new AI 2040: Plan A website. It all seems so implausible to me that I'm not sure where to start. There's an epistemic chasm between those who think superintelligence implies near-omnipotence and those (like me) who don't.
I've found that people believe it at such a deeply intuitive level that it's hard to have a meaningful discussion about it. Each side finds it baffling to encounter people with the opposite intuition, and on some level can't believe they're being serious.
I would hope that with enough time I could get Timothy Lee to come around, since he has established he takes arguments seriously, but so far I’ve been unable to find a compelling argument to convince such folks that for practical purposes yes sufficiently advanced AIs could and would do all the things.
I think the premise in the Detractors argument, as stated above, is wrong. I think superintelligence is likely to be coming soon, as do the labs. Many do not agree. If you are one of those who do not agree, then you should absolutely not want to implement Plan A, or anything like Plan A, and you should say so plainly.
This is true whether or not you want to further engage with the scenario anyway, to consider the hypothetical where you are wrong. That’s up to you, and ‘no’ is a respectable response.
Top 10 criticisms other are, translated into my language (not intended to pass ITTs):
America would never do it. You don’t understand America (or the government).
China would never do it. You don’t understand China (or its government).
This scenario doesn’t understand that this is a race. Or, this scenario places too much emphasis on the framing that this is now being seen and treated as a race.
This is still too fast, or this is far too slow. Unacceptable.
This is unnecessary, market can handle alignment, it is all easy, stop worrying. You warned things might eventually be not fine but so far everything is fine. I am opposed to anything vibing with the words ‘slowdown’ or ‘pause’ and will try to incept that any such action is impossible and discredited or ‘naive.’
This still would not work, either it does not work technically for various reasons or alignment is too hard. We need a full pause. Or, in the better version: There aren’t enough worlds where this turns losses into wins.
The economic projections are way too optimistic.
The scenario is confusing: It conflates realistic prediction with aspiration. Also people will focus only on particular key points, so you should obfuscate those.
The scenario involves unlikely things happening. Nothing ever happens. This is something happening. Also you made it up. Never gonna happen.
A lot of these being symmetrical is a sign that the scenario is doing something right.
My basic responses to these objections:
Not with that attitude. If true, get cracking on figuring out an alternative.
Not with that attitude. If true, get cracking on figuring out an alternative.
I think this is roughly the right amount of talking like this is a race.
Slower would be better if possible, it’s a question of what is achievable.
If you think this, then you should oppose things like Plan A, but you’re wrong.
Yes, to deal with big things you often have to pay real costs, but do not equate such actions with authoritarianism let alone totalitarianism. This plan tries hard to minimize this downside. If you have better implementation ideas, speak up.
I think this is a topic for healthy debate and a strong objection.
I think they are overly optimistic, but not crazy, and I think the plan and scenario survive having much less dramatic medium-term economic impacts.
This is the nature of such a scenario. I think they did the best they could.
Some seemingly unlikely things do happen. A scenario without any unlikely events is itself even more unlikely. And in AI, the one thing I know for sure is that a lot of big somethings are going to happen. Results literally range from maximally bad to maximally good, and ~zero movement is not that likely.
Extended versions of most of these are included later in the post.
Proactive Response To Objections
There is a long tail of other objections as well. Even after listing the top 10 I found that a large percentage of responses to my open thread were not covered.
AI 2040, like AI 2027, tries to proactively answer many of the concerns and objections people have, both common and rare, including via supplementary material.
Whereas here are some shorter pitches for why you should read it and it matters:
Romeo Dean: At some point soon, humanity will be forced to reckon with the creation of AIs that are smarter than humans in every way. This is a terrifying prospect. Plan A is our vision of how we can make this go well if we rise to the occasion.
Eli Lifland: Excited to finally publish AI 2040: Plan A, our plan for international coordination to get to a great AI future. We’ve put a lot of work into this and I hope that it sparks alternative plans that bring us closer to treating superintelligence with the seriousness it deserves.
Buck Shlegeris: I’m really excited this is out; I think it’s the best summary of the spectrum of possible plans for preventing catastrophe from AI.
Ryan Greenblatt: Plan A seems like a good plan for handling powerful AI, or at least the best plan anyone’s written up. Many choices initially seem crazy, but are actually pretty carefully considered. Plan A isn’t likely to happen, but pushing for something like this seems worthwhile.
A nice property of Plan A is that it’s reasonably robust to partial or lower-quality implementation. So nearby proposals still seem good (and worse-done versions can bootstrap into better ones). The core—verification and (very strong) transparency—suffices to go pretty far.
Eli Tyre: Plan A is the single most thorough and thoughtful plan that I’m aware of for what the world should do as we approach Superintelligence. If we want to get through the singularity alive, this is my current best guess of the playbook we should be aiming to implement. It’s my current best guess mainly because there are so few serious contenders that actually engage with the details of the situation.
Nevin Freeman: As you try to make sense of what *should* happen with AI, this is an absolute must read (or listen). It’s very hard to reason about potential catastrophe and how to ensure we avoid it, and it’s even harder to turn ideas into concrete stories many will understand. Huge thanks to [everyone involved].
Jeffrey Ladish: We’re in a pretty dire situation, but we can change course! I highly recommend checking out @DKokotajlo Plan A write-up & AI 2040 scenario.
Steven Adler: Very grateful that the AI 2027 team has a plan for the global calamity that is superintelligence [that arrives fast and before we are ready]. It is by no means perfect, but I’d love to see critics lay out a stronger competing vision; seems pretty hard.
Jonny Miller: Someone should fund turning this into a Netflix-quality production to give normies a heads-up of what is coming down the pike.
Andy Masley: AI 2027 now has a follow up called AI 2040: Plan A, which is a scenario where things go well. I got to be an early reviewer on it. I have extremely agnostic takes on the difficulty of alignment and the economics of advanced AI, but this scenario is the clearest concrete description of what the world could like under the specific worldview of:
-Extremely rapid AI progress -Alignment right now is very difficult -AI very able to substitute for human inputs in the economy -Assuming in Plan A that things are governed well at the global level
Each of those I need to throw my hands up and say “Man I have zero clue at all” but I’m always happy when different sides in the AI debate make their assumptions clear, and the consequences of the assumptions here are really wild.
Giving these assumptions I’m a complete agnostic on, I was surprised at how difficult it was to find holes in the conclusions. The way the global energy system is shaped by data centers in a super high growth scenario sent me down a ton of rabbit holes among other things.
Charlie Bullock: The Bay Area rationalist preference for communicating important policy proposals via structurally postmodern medium-form fiction writing will never cease to amuse me.
It would be a kinder and gentler world if this were the norm everywhere. I want to start my next draft bill with a 25,000-word Choose Your Own Adventure novella set in late Victorian London.
To be clear I like the piece and think it’s good that people are trying to think about the future in a very detailed way and take things out. But it is just descriptively true that the publication is a work of fiction. The authors would not disagree with that claim.
Boaz Barak (OpenAI): I don’t agree with everything in AI 2040 “Plan A” but it is very thoughtful. One element I love: push for increased transparency and diffusion.
Instead of safety meaning locked down information and restrict frontier models only to labs, government, and chosen partners, a key component in their plan is to maximize sharing information and distribution.
One could also call it an optimistic scenario, as per Richard Ngo.
One strong criticism of AI 2027 was that it laid out two scenarios, but did not tell us what we should do or offer us a practical path forward. In the default scenario, which lays out what the authors expect to happen, we all die.
In AI 2027’s alternative scenario we do a hard reset at a crucial moment, which is less a strategy or plan or positive vision and more like a hail mary pass, where the authors make things turn out well to show that it is in theory possible.
Plan A updates to start from our present situation. Rather than being primarily a prediction, it lays out a positive vision of a possible future where we coordinate (including using various enforcement and verification mechanisms) to slow down the development of superintelligence (ASI), and give it the best chance to go well. Ultimately in 2040 the torch is passed to the AIs, the singularity proceeds in earnest, and hopefully we got it right on the first try and things work out.
The proposed implementation of Plan A is that America and China reach a mutually beneficial deal to slow down AI development and share research information. Joint control is established over existing and new chip supply, with common knowledge of the location of existing concentrations of chips, and universal auditing of data centers. Both sides can verify that the deal is being upheld.
This can sound rather fantastical, but if a few months ago you had described how the US government has responded to the whole Mythos situation, you would have sounded rather fantastical then too. Something has to give. The authors really did invest a lot of work on seeing which paths forward were viable here.
Things go well, including economically along the way, as we make a series of good decisions, and we get to a happy ending, although not without speed bumps as even a slower transition is rather sudden and even if the technical issues are handled there remain lots of real problems. They consider alternative ways this could go, such as China cheating aggressively or flawed safety cases being approved.
During the transition, America experiences explosive economic growth, and uses some of that to pay out a rapidly increasing Citizen’s Dividend, as an extremely generous form of UBI, and also a small fraction is devoted to various forms of defensive acceleration. Various issues are discussed along the way.
If things going vertical in 2040 still sounds super fast, that is because in the authors’ current baseline or default scenario this happens in 2030.
They briefly also describe the scenarios where we instead choose Plans B, C, D or S:
Plan A: Coordinated Slowdown, make a deal with China.
Plan B: Fight China, as in try to sabotage other efforts to buy a time buffer).
Plan C: Burn the Lead, roughly the AI 2027 scenario, with a bit more awareness.
Plan D: Race to ASI, the path we were on before Mythos and mostly are still on.
Plan S: Shut It All Down, which is what it sounds like. They are sympathetic but don’t recommend because it would be unstable and hard to get buy-in for.
Roughly speaking, they predict the presidential election in 2028 is all about AI because by that point it is obvious AI is the Main Thing, with candidates throwing around bold agendas. Then in 2029 one gets implemented, which they are hoping is Plan A.
There are a ton of details and explorations of different topics in Plan A. Those involved put a lot of effort into parts almost no one will see. I could easily write a response post at least as long as the scenario to go into them. This is not that post.
A lot of those details sound like science fiction, because they are serious attempts to predict the future. Best start believing in science fiction stories, because you’re living in one. There are any number of ways that can go, and this might be wrong, but some very science fiction things are happening and a lot more will be happening soon.
Plan S for Shutdown
This perspective, a form of Objection #7 above, is negative on Plan A’s chances of success, but positive on the chances it causes us to gather enough impetus to shift to Plan S, which is a full shutdown.
Nate Soares (MIRI): The AI futures folk and I agree on quite a lot. This is the best concrete vision of a positive future I’ve ever seen spelled out, bar none.
I doubt their Plan A would work as written, but it’d have a chance of producing evidence that convinced world leaders to switch to Plan S, which I think has a shot.
Why not Plan A? Humans working with AIs on alignment are likely to converge on wrong answers. The honest answer is likely “pushing to superintelligence using anything remotely like modern methods is fucked; back off”; humans are unlikely to listen rather than push through.
But if humanity was especially competent and especially lucky while pursuing Plan A, and was able to heed the warning signs, I could imagine us taking the off-ramp to Plan S and surviving. So attempting Plan A doesn’t look completely fatal.
Or alternatively:
Damian Tatum: Again, still reading the alternative paths, but
If you are of the camp that the odds are against us and the situation is grim, as I am and Nate Soares is even more so, then every winning scenario involves a lot of things going unexpectedly well.
The reason they give for not choosing Plan S is that they believe that an agreement would be unstable over time, so you have to keep moving forward and also give people a dividend of sorts. The agreement breaking down would potentially be quite bad.
Raymond Arnold thinks out loud here about the importance of ‘how long would such a deal hold?’ in terms of choosing the best plan within the class of As and Ss.
Something (Unexpectedly Good) Ever Happens
Every scenario, including whatever turns out to happen in real life, is going to involve some rather unlikely things happening. For any given thing, ‘nothing ever happens.’ For all things combined, surprising individual things happen all the time and each might be good. If this is not intuitive, look back on the last year, or any other period.
If you have a good plan to change the world, it is going to involve causing a bunch of otherwise low-probability events to happen, either intentionally or via good luck.
Of course, any good plan involves not relying on too many specific good things happening. You need to be as robust as possible, but no more robust than that.
Marius Hobbhahn: When I read it for the first time my intuitive reaction was like “too many low probability things have to go right for this to work out”
Which my brain somehow mapped to “therefore unrealistic” instead of the obvious “it’s a really hard problem and this is actually the sota plan”
I think it’s a good plan and I hope we see much more work on this, especially from the people who are deadset on building AGI.
Tyler John: I had the same reaction and I still do want more nimble plans. (I am also working on building the verification stack, walk and chew gum, etc.)
Plan A here definitely has some flexibility and room to recover from failures or to reroute, but not unlimited room. It involves good fortune, but not maximal amounts.
There’s no good answer, and no one best answer. If you tell a story, your story is going to have to choose one path out of many. It must balance realism with your hopes, in addition a bunch of individually unlikely things.
The alternative is to not tell a story and not have a scenario at all, to never tell stories or offer particular scenarios at all. I think that’s clearly far worse.
Most centrally, Richard would prefer a slower handoff than described in the scenario. I would as well, as would the authors of Plan A, and this objection is not uncommon.
Richard Ngo recommended taking all the dates out to avoid people mainly focusing on the date 2040, since what is valuable are the details.
I think this echoes a lot of similar criticisms of AI 2027, and I see the argument but I rejected it then as well. You can’t tell a story like this properly without dates, you have to pick some point on the curve of potentials to lay out in a potential future.
If you try, guess what the first criticism would be? They didn’t put dates on. What a joke, what do we even do with that? And those people would be right.
In general I don’t like Isolated Demands For Anti-Virality lest someone somewhere accuse you of giving people the wrong idea or focus on the wrong aspect.
Yes, some people will stop at the headline, or latch onto only the date, and you can change the frequency of that somewhat, but mostly that cannot be helped, and those people were never going to engage with the real content anyway, or if anything this makes them more likely to then seriously dig in.
Race Conditions
Richard also challenges the ‘race with China’ framing, expecting other disruptions, but I predict that any such move towards Richard’s framing would cause most to dismiss the scenario as unrealistic, and also risks further politicizing the whole thing. Any failure to acknowledge that America and China really don’t trust each other at this time would make the whole thing land with a thud or worse.
I too of course have always hated the race framing, and tried to fight against it for years, but as Larsen says that’s how people think and talk now, especially in DC, and you have to acknowledge this.
For the opposite view on that, here is Poplicola saying that Plan A fails as a scenario because it doesn’t treat American hostility to China, and its view of ‘authoritarianism’ as the real existential threat (at least when it comes from outside the house) seriously enough.
Also things like this:
Kyle Corbitt: The tech development seems ~plausible but the politics feel quite naive. Like I just can't see Canadian voters letting China construct thousands of datacenters in 2030, no matter how good the economics are. Esp if the pitch is "we chose you since you're a soft target for the US"
I don’t see that proposal as an especially hard part of the problem, as long as the economics work out.
I think in general the way politics works is until the reasons are compelling things look hopelessly naive until suddenly things change, the atmosphere shifts and then they happen. Peace treaties (and other agreements) often involve concessions that looked impossible, right until the deal got signed.
And I think Richard is conflating the ‘race with China’ we are currently in, as in allowing OpenAI and Anthropic to push full speed ahead while we make at least some attempt to deny China chips and otherwise tip the balance, while any attempt to do anything to help is by default dismissed as ‘if we do that we Lose To China’ with a theoretical full-on planned Race With China that I agree we are not doing and are not likely to do any time soon.
This Is A Lot Of Diffusion And Economic Growth
Richard ends with a criticism of the expected fast pace of both AI progress and the diffusion and impact of AI in the broader economy, saying he doesn’t expect AI progress to generalize. You certainly can argue that the forecast here is overly optimistic, even as things spend years going along exactly the path that these styles of prediction would predict. I just don’t think a strong form of that is the way to bet.
In terms of the diffusion and real world impact, which I think is the strongest critique, I agree with Shakeel Hashim here that this does not impact the recommendations much. The economic impacts during this scenario might go a lot slower, but that doesn’t change the central path.
Would the scenario have been memetically more fit if it involved less economic growth in the 2030s? Probably, and I do think the authors are being optimistic here, but the authors are telling us how they think such a scenario would actually play out, and they explain why they think this.
At the limit, with sufficiently advanced AI, diffusion stops being an issue for long, and those who pretend otherwise are denying the premise or being silly.
Topynate also has some good criticisms. We could use more Chinese perspectives, and more concern about how willing people would be to make a deal. I am 100% fine with trading away half the lightcone, but others may not be. I agree that justifying Plan A requires that alignment be hard enough that we need Plan A, and also doable enough that Plan A can work before it unravels.
Bryce Nyeggen: It's pure fantasy. The critical analysis of US / China relations is completely glossed over with "surely they will both recognize they must defer to our superior understanding of forum lore"
Yishan suggests you could do a Plan A1 that acknowledges this reality, but I think you basically couldn’t. If he is right, then there is probably no ZOPA (Zone of Possible Agreement), as in even a maximally wise and cooperative American government cannot do that without getting a lot of other things in return, especially if this is asked for up front, but desperate times could change things. The Plan A scenario involves rebalancing things like share of robots, so they’re thinking about this aspect.
I do agree that finding a ZOPA will at best be difficult, as the two sides have very different views. But there is indeed a long history of similar deals looking impossible for this reason, until suddenly a deal does get made when there is sufficient impetus.
A lot of this is for me that yes, in the world of this level of AI capabilities, there is quite a lot to be gained from a deal. When there is a lot of gains from trade, trade often finds a way.
Planning For Shifting Overton Windows Is Essential
dave kasten: A huge portion of all reactions to this (on several sides) are about to be as overcome by events as opining on the likelihood of the Trump COVID unemployment benefits in January 2020.
You should have been able to predict the Trump COVID unemployment benefits in January 2020. I didn’t, but that’s on me. If you can get to ‘extended lockdowns that last months or more’ then something like that likely follows. The part that I did not expect in advance was where the lockdowns would reach an equilibrium indefinitely, rather than either solving the problem or being overwhelmed.
Covid also illustrates that the impossible suddenly becomes possible.
Shutting down all ‘non-essential’ activities and services and almost not letting people go outside? Trillions in cash handouts? Vaccine mandates?
It’s all a crazy violation of liberty until the alternative is worse, and that’s with only a low single digit chance of dying. And yes, more and better planning in January and February, even as almost everyone dismissed your premises as crazy and your proposals as authoritarian, would have been extremely valuable.
The Standard Handwave
There are of course those who will pattern match anything that involves any form of ‘slowing down’ or ‘pause’ or ‘buying time’ or any strategy that involves paying a nontrivial price, and queue up their standard responses, without engaging further.
You might think if nothing else that Mythos, and the White House response to this, would make people rethink such dismissals. Well, maybe it sometimes did.
Many simply call those involved an unkind name and move on, feeling superior.
The first example of the polite version of this I saw was here from Maxwell Tabarrok, as retweeted by Tyler Cowen. This is basically ‘oh well you see if we had responded to this exponential too early that would have been bad and looked foolish, so we can only respond too late’ and ‘don’t worry markets solve everything until proven otherwise.’
We also get a side helping of ‘until you know exactly how to solve the problem you’re not allowed to do anything about it’ and also a failure to actually engage with the details of the proposal.
And also ‘the idea we should slow down AI to buy time has no credibility given events, despite the actual slow downs that happened in highly recent events in response to Claude Mythos.’
If you think that’s not a fair characterization, okay then, here’s his full comment, so you can judge for yourself.
Maxwell Tabarrok: The best plan is still “pause and let us figure out what to do”?
Any credibility this policy might have had has been overtaken by events.
What could we possibly have figured out about how to make Mythos safe if we had paused AI in 2023?
So far, we’ve made lots of progress on AI alignment and safety coincident with improvements in capability. Alignment and control are very useful properties for products to have so I expect this to continue on the default path.
We should not make big sacrifices of property rights and cede lots of power to new institutions unless there’s an extremely clear vision of the risks and solutions at hand. Not knowing what to do and wanting time to figure it out is not enough to justify or induce world-wide buy-in on slowing down an important technology.
Thomas Larsen: That’s not the plan we recommend. Have you read the scenario? We recommend scaling quickly to AIs that are as capable as top human experts, and only then pause as the world is being widely transformed.
Maxwell Tabarrok: The number one recommendation of Plan A is “Buy Time”
The major feature of the plan is more time to organize research to perhaps someday figure out how to solve the risks you see from AI.
Except of course they don’t phrase it like that. But yes, the general view is this, from June 26:
Maxwell Tabarrok: yeah yeah there’s externalities and property rights issues
but still, a good heuristic is that if no one is willing to pay for something, it’s probably not that important
pick your nonprofit cause carefully! a lack of market incentive is not a good signal of importance!
A lack of market incentive is not a good signal of importance. An understanding of what externality or other issue is mechanistically causing a lack of market incentive, however, very much is a good signal that there is an opportunity. Demand of a sort is unable to induce supply. A different kind of profit is plausibly available to be made, or utility is available to be created.
Some Equate Any Controls Over Compute To Authoritarian Dystopia And Those Same People Mostly Think Superintelligence Won’t Happen
We’ve been through many years of this idea, that somehow any controls on chips or compute or models means authoritarian dystopia. This is objection #6.
There is a remarkably high correlation between those who equate such things, and those who deny that superintelligence will be developed any time soon.
Séb Krier (AGI Policy Dev Lead, Google DeepMind): The reactions to prescriptions about AGI have less to do with being 'AGI pilled' or not, and more about whether you're more concerned with AIs taking over (xrisk), companies taking over (anti-capitalism), or the abuse of power by empowered governments (anti-authoritarianism).
Very obviously this should mostly depend on whether you are what I call ASI pilled.
You can be what I call ‘AGI pilled’ but expect returns to rapidly drop off, and thus reject scenarios like AI 2027 or Plan A as unrealistic, and thus you are evaluating any proposal largely on the basis of risk of abuse or concentration of power. If you are what I call ‘ASI pilled,’ where you see the tech side of the scenario as highly possible, then that gets a lot harder, although you can still reject this style of proposal on the grounds that it doesn’t buy you enough to be worth the price.
I am not referring to Krier in particular, but in general it seems like those who want to be concerned about governments therefore choose to not be ASI pilled. Or at minimum there is a common generator causing both responses, that everything must forever be some form of ‘economic normal’ and why aren’t idiots understanding that.
Vitalik Buterin Is Right, The Crux Is Future AI Capability Levels
I respect the hell out of ‘superintelligence won’t happen any time soon, so stop proposing expensive ways to deal with it.’
Except the part where you’re wrong about that, but if that is your True Objection then yes please do state it that way. Vitalik is characteristically very strong on this.
vitalik.eth: One thing I find striking in the discourse between AI 2040 and its detractors is that the two seem to be locked in to totally incompatible worldviews of how fast and how much of a big deal AI progress is:
* In AI 2040, every scenario sees superintelligence of some kind emerging by 2040, unless a herculean effort is made to completely stop it * Detractors say things like "AI 2040 is naive about human coordination ability and a threat to freedom", but don't seem to see any naivety in assuming that the ASI transition will just go well by default, don't seem to see ASI itself as a massive power concentrator risk, and don't seem to feel fear of humanity's "hard power" dropping to zero if ASIs can do literally every task better than we can. This stance makes total sense in a "AI is normal technology" world, zero sense in a world where superintelligence is possible by 2030 and almost guaranteed by 2040.
Exactly. If you buy the premise you do not have to buy the particular plan. You do need to take this seriously and not dismiss it out of hand as naive or authoritarian, and if you don’t like the plan you should be trying to think of better plans.
I think my beliefs are:
- If I was confident that (present-day-style) AI is normal technology, I would be in the detractor camp - If I was confident that superintelligence is coming in 2030 by default, I would be closer to the AI 2040 camp - it's naive, but every other option is naive squared?
But my problem is that I feel great uncertainty and have no idea which of the two worlds (or some other third thing) we're living in?
This seems totally fair. The labs think this is happening but that does not mean that you have to agree with them.
It would be a very large mistake to do Plan A, or another similarly disruptive plan, if superintelligence was never coming before 2050, or AI will indefinitely remain a ‘normal technology’ no matter what.
Hence why I continue to be open-minded about slowdowns/pauses, but also I feel very uncomfortable with the "open source bad, the good outcome is the one where our guys have controlling global dominance" push coming from some major AI companies and intellectuals - in a "normal" world that's the sort of thing that triggers every political alarm bell at the same time.
A big reason why I have been advocating and trying my best to support the d/acc platform (rapid up-skilling in formal verification, cryptography, secure and open hardware, pandemic resistance and other defensive biotech, food and basic resource security, public epistemics, non-power-concentrating versions of physical security) is that these things are clearly worth doing in both worlds.
D/acc is a way to ‘play it safe,’ as in work on things that are clearly good. Those things are good things to do, but in the superintelligence scenario that’s not going to get it done. Vitalik realizes this.
The 2040 plan is already much more open source friendly (even mandating it! yay). It also includes "mutually assured compute destruction" ideas which (if they work) effectively give one of 2-5 actors the ability to trigger a global compute winter - as opposed to giving 1-5 actors the ability to selectively disenfranchise people they consider baddies while exempting themselves. This is also a big improvement. So I can see the earnest attempts to improve along the dimensions detractors criticize on ("does this concentrate power in big AI labs and superpower governments?"), and I appreciate this. I think many people don't appreciate enough the differences between different "kinds" of pause buttons, and how some concentrate power far more than others. Probably we can think harder and improve even more here.
But on the "slowdown/pause or not" topic, there isn't a magic "escape the tradeoff" button.
Again, yes, exactly. There are a bunch of places where there is no Secret Third Thing you can do, and the tradeoffs are real, and there is no safe play. But you can work to improve your options.
The Hansonian in me says: the winning deal is a deal which, from the perspective of both sides' present-day beliefs and knowledge, both sides would accept, though for different reasons. If the crux is AI progress speed, then identify a set of pre-agreed triggers for "okay, serious shit is happening" [super-pandemics? >25% unemployment? something involving slaughterbots?], and pre-agree that we become much more open-minded to the slowdown or pause thing if enough triggers come to pass within some timeframe.
2040 detractors (who clearly implicitly think that we'll see amazing speedup of progress from AI but think that what I call the "serious shit" category is overhyped) will accept expecting that the triggers don't come to pass, and AI worriers will accept expecting that they will. Pre-agreeing on the specific triggers means that once the triggers either hit or don't hit, there is stronger legitimacy around the idea that one side's worldview turned out more correct and we should be more inclined toward their program.
In theory this would be great. If such a deal were credible, I would love to say ‘well, let’s agree that if [X] happened then we would do [costly intervention basket Y].’
The problem, as always, is who is going to make that agreement? How can we expect it to be honored? There is a long history of various things, that used to be considered obvious red lines or alarm bells, being rushed by almost without a second thought.
If I were @elonmusk (or zuck, or...) I would re-tool twitter much more heavily into being a platform for helping to identify and make these kinds of grand win-win deals, so that we can bypass big-country governments and big-company CEOs and big nonprofit intellectuals and give more people a voice in the discussion. It's possibly one of the best things that social media _could_ do for humanity if it wanted to.
But again, maybe this is also naive. Actually, probably it's naive.
But currently, I see zero plans for how to deal with an ASI transition that are not naive. Perhaps humanity is stuck with a choice between naive and naive squared (or maybe even naive squared and naive cubed), so I feel inclined to cut some slack to people who are trying.
I, too, see no ‘non-naive’ options, in the sense that people object that Plan A is naive. You have to be naive from at least some point of view.
One can argue that the costs of Plan A and similar interventions are small compared to the benefits, so you don’t need that much confidence you live in a world with future superintelligence before such a path makes sense, if you think Plan A is good in ASI worlds. I don’t think this is obvious. The costs in non-ASI worlds are quite real here. As are the costs in ASI worlds, especially if some things go sideways.
And also for acknowledging the good intent, and that the plan is designed to minimize the amount of restriction and authoritarianism, given what must be done.
I am less thrilled with his characterization here of this being about ‘a made up threat.’ If that’s how you view all this talk of superintelligence or sufficiently advanced AI, and are reasoning on that basis, then that’s the crux we need to be discussing, since otherwise nothing else matters.
Ramez Naam: If I saw evidence of fast takeoff + loss of control risk + plausible anti social volition, yes, I'd change my mind.
I think we should do more on AI safety and transparency. I favor third party audits, for example.
…
I would need to see some believable model that a software only singularity or near singularity is even possible. The math says that the steep power law diminishing returns in AI make that improbable. I'd need to be convinced otherwise.
I think he believes what he’s saying here, but I basically don’t believe this can be both good faith and a self-aware response. There are plenty of good models saying this is possible, and Naam is rejecting those models of superintelligence as ‘not believable.’
My experience is that such requests are impossible to meet, and the goalposts will be adjusted as needed. Also, there is no reason to presume that it would need to be software-only in order to require Plan A levels of response. In AI 2027, and in Plan A, the takeoff very much is not software-only.
Anyway, back to the good part of the objection:
Ramez Naam (after his summary of key restrictions in Plan A): These proposals violate at minimum the spirit of the 1st and 4th Amendments.
This is well intentioned, from people who sincerely believe that runaway AI is a greater risk than, say, China's oppression of millions of people today, or the ideologies that killed tens of millions in the 20th century. I profoundly disagree. This is a recipe for authoritarianism.
AI safety must be conducted in a manner that preserves a resilient, decentralized, private, and free society with checks, balances, competition, and a distribution of power between private actors, each other, and governments.
This is not it.
FlawlessContext: the plan is anti-authoritarian and has a lot of protocols for reducing authoritarianism. I see surveillance increasing overall as likely to happen given AI capabilities increases. Safeguards against dangerous tech are very likely net positive, including watching the watchers.
Ramez Naam: It has anti-authoritarian intent. I agree there. And yet its actual proposals are profoundly authoritarian.
Long history of that sort of thing. Big chunk of the 20th century.
If you think that China’s current oppression of people is a greater danger than the dangers posed by highly capable AI, then yes, obviously you should oppose all of this.
Ramez Naam seems to be:
Equating restrictions on use of AI and compute with authoritarianism, including the 20th century’s fascism and communism.
Comparing the Plan A scenario to a scenario where we go full speed ahead and do not place any major restrictions on AI, at the kind of pace they predict we would get, but nothing goes seriously wrong.
Doing the standard political thing, when there is a proposal you dislike on the table, of dismissing the proposal as having big downsides, then calling for a mysterious alternative proposal that works without downsides.
Denying the idea that there importantly exists AI existential or catastrophic risk, or doing a rather literal version of ‘better dead than red.’ At minimum, saying that the dangers from the current actions of the CCP exceed the dangers from future more advanced AIs.
Denying (in a response) that people can meaningfully predict future events.
I find it hard to engage with people who state Claim #5 so strongly, especially given people have actual track records. I disagree.
Claim #1 seems very clearly hyperbolic and unreasonable. Can we please stop equating restrictions on compute use to the worst governments in history?
Claim #4 would be enough on its own, if true. Obviously, if you think AI will work out fine without major interventions, or other things are ultimately far more important, then you shouldn’t want to implement Plan A.
Obviously, I disagree here in the strongest possible terms. Claim #2 is not a plausible alternative world, for the same reason, and you can’t use it as the comparison point.
That brings you to point #3. We are, if you accept the technological premise of those who wrote Plan A, which to a large extent I absolutely do, in quite the pickle. By default we all probably die. All known light touch solutions do not help much. Any solution will involve the sacrifice of sacred values, since among other things one of those sacred values is ‘humans remain alive.’
Concepts Of A Plan
Yes, if we can find a way to do it, we all want the various desiderata that Naam lists, which are commonly expressed by many including Altman and Amodei. We want to be able to rely on liberal democracy and free markets while creating minds increasingly more capable than humans at increasingly many tasks, and have that work out for humans. Yay applause lights.
But there is no proposal there. How would this work, even in outline form, under the types of AI capabilities described in Plan A / AI 2040, even if alignment efforts were successful? Is there any proposal that is not, mostly, ‘This Is Fine?’
Indeed, there are some like Seb Krier who reject the very idea of having a plan. I would like to say this entirely misses the point. Even when plans are worthless, planning is essential. Having a concrete idea of what you want to do is a good idea, even if no one has the authority to do it. If you are dismissing the idea of planning to steer the future at all, on principle, as dangerous or wrongheaded or worthless, then you lose.
The other thing you can do in response to a plan is propose a better plan, or a better model of what might happen. Bleys Goodson is trying to figure out what he calls a less authoritarian way to a safe open-research ramp-up, and offers enia.cc to help with modeling what things will look like. I appreciate that Bleys is open about not yet having such a plan to present.
Seb is actually saying, no, we should just muddle through, that this is the best strategy. And I very strongly think no, that’s a really dumb strategy, but even if we do muddle through that is a good reason to figure out our next options.
The Kitchen Sink
Seb Krier also throws the kitchen sink of usual other objections out there, in what I read as the disappointed professor tone, most of which also apply to AI 2027, and to basically any proposal that we collectively try to solve the problem.
One interesting note, that Krier frames as part of an objection (in his #9) is that ‘we already have a lot of Plan A stuff going on, that people thought was never going to happen because normies would not wake up.’ That is not an argument against Plan A, or an argument ‘against alarmism,’ as he puts it, and also I take strong issue with the term ‘alarmism’ when presented like this as a clear negative. Alarms and warnings exist for a reason.
Seb Krier has precommitted to not engaging further, and as I said this feels like reiteration of past arguments and throwing the kitchen sink more than particular new points, so I won’t engage further with him in particular unless he requests it.
I also think that the same objections raised here to Plan A can be made, in far stronger form, against past and present government policies across the board even in normal times, and definitely under crisis or wartime conditions.
Consider our response to Covid as per the discussion above, or World War I, or World War II, or the way we collect and report on income taxes, or the war on drugs, or the rules on who can do work or provide healthcare or do research or build which houses or the way we implicitly regulate online speech already. Or look at the way Europe and the UK currently restrict speech and the use of computers and phones, or labor, or air conditioning, or lots of other things.
Do I want to roll a whole lot of that back? Oh yes, absolutely.
Still, none of that means that America, the UK or EU is an authoritarian state, and certainly I hope one would not equate them with 20th century fascism or communism. The same would apply to these kinds of compute and other restrictions.
This is not that high on the list of dangers, but to be as ‘normal’ sounding as possible: Compare the restrictions under Plan A to what happens if a lack of restrictions leads to a Covid-level pandemic. Which restrictions would you prefer?
I welcome a discussion about the particular interventions and restrictions, and whether they would actually be necessary in such scenarios. I am absolutely not endorsing this exact list of responses.
But oh man is ‘the spirit of the 1st and 4th amendments’ being violated a lot worse than this, and for far worse reasons, as it is, all the time, in a currently largely free country. Plan A raises obvious dangers from centralization and expansion of government power, but it is not a centrally authoritarian vision. Indeed, in many ways it is designed to try and head off authoritarianism, including what one might call ‘authoritarianism by the AIs.’
You Either Can Steer The Future Or You Cannot
A lot of these objections have little to do with the particulars of this proposal.
Instead, two more global objections loom largest.
The first is not believing in superintelligence. If you don’t think superintelligence is coming by 2040 either way, then Plan A does not make sense. Fair enough.
The other objection is, essentially, that if humans have the ability to importantly and collectively steer the future, that this is bad, because it would be concentration of power, and would inevitably be abused, and that would be so much worse.
Or the objection is that the humans steering the future cannot be governments, because that in particular is really terrible.
This has little to do with the particular steering that Plan A has in mind.
So, one more time:
The central problem is either humans can collectively steer the future, or they can’t.
(And also, if you don’t give the humans good tools or let them plan and prepare, for fear someone might use those good tools, they will still grasp around for whatever is on hand, and use bad tools, and this will go worse, as we have recently seen.)
The entities steering are either governments, or they are private individuals and organizations that are not government, or no one is steering.
You either can do and actually do the things that actually influence the path of AI development and diffusion, or else you can’t or don’t.
Historically, we have been very fortunate that a mostly hands-off, unsteered version of things has been the winning strategy on almost all levels, yet even that required quite a lot of particular steering. As discussed above, we steer a lot of things quite a lot.
If we get technological developments at the magnitude or speed of those envisioned in AI 2027 or even the slowed vision of Plan A, then the alternative to humans steering those outcomes is that we solve for the equilibrium.
The equilibrium is increasingly many things are turned over to the AIs. The equilibrium is that the AIs rapidly end up in charge because the alternative is being outcompeted. This happens without AIs tricking anyone or firing a shot. The AIs, and their deals or competitions among themselves, and their goals, steer the future. And that’s the good version.
Once that happens, there is no undoing or fixing it, and by default I do not expect humans to long survive, or the things we value to long endure.
Cooperative Alignment
The approach in Plan A, as presented, treats alignment as a technical problem and a control problem, of needing to create AIs that are trustworthy.
They only consider one form of cooperative alignment, or the idea that the AIs will or won’t cooperate with humans largely based on game theory and decision theory, and based on how we choose to treat them. They focus only on how to make deals with misaligned AIs, rather than how to align with AIs in the first place.
I think within the context of writing the scenario and not getting too distracted basically This Is Fine. If on consideration we would be better off doing alignment this other way, then we would choose to do it this other way, and I don’t think it changes the central strategies so much.
John Wittle: there's no attempt to give the AI real standing, they just take it for themselves *without* conflict? this seems very unlikely to me. no moral agent cooperates against defection, that is fantasy
tons of talk about making sure the AI are trustworthy, zero talk about the humans being trustworthy to the AI
i just keep getting the feeling that... idk. daniel and eli should take one week to read some Wikipedia articles about changing ecologies in reality.
it's a very simple change! it would not be costly! it wouldn't even take away from control!
yeah, after carefully reading it (instead of skimming as i did before)
the humans always treat the AIs as a hostile adversary, all the way up to the very end. I just don't know how to convey the game theory here, it seems like it should be obvious to people in our community. you do not get mutual cooperation via programming it into the minds of your adversaries by force, because that is what we call DEFECTION.
I also don’t view the game theory the same way here. Once you hand off to the kind of AIs that exist in 2040 in Plan A, the humans don’t have leverage, and that’s why this is far more decision theory than game theory: Human cooperation from that point forward is irrelevant. I won’t get into my view of the decision theory involved here, except to say that yes it should be something we spend a lot of attention on, and I hope to see further engagement on this point.
I still need to read it carefully but I'm not sure that pausing (more generally, not necessarily the Plan A way) only makes sense if you think superintelligence is likely? Even big-but-not-extinction harms like cyberattacks could benefit from a pause before some kind of int'l agreement to make sure we don't release models that lead to mass cyberattacks in the interim.
Re “you cannot forecast this”: this is my position. By its very nature, superintelligence will act in unknowable ways, just like your dog won’t be able to predict when you’ll eg have to stay late at work because of a given phone call. If its actions are predictable, then it isn’t super intelligent.
I still need to read it carefully but I'm not sure that pausing (more generally, not necessarily the Plan A way) only makes sense if you think superintelligence is likely? Even big-but-not-extinction harms like cyberattacks could benefit from a pause before some kind of int'l agreement to make sure we don't release models that lead to mass cyberattacks in the interim.
Re “you cannot forecast this”: this is my position. By its very nature, superintelligence will act in unknowable ways, just like your dog won’t be able to predict when you’ll eg have to stay late at work because of a given phone call. If its actions are predictable, then it isn’t super intelligent.