Yesterday I covered Dwarkesh Patel’s excellent podcast coverage of AI 2027 with Daniel Kokotajlo and Scott Alexander. Today covers the reactions of others.
They did (briefly) cover power in the Compute Forecast – it seems they think the AI labs will buy up existing and projected capacity and, so I'm guessing, not magically create new capacity much quicker than is plausible.
I agree the authors mention power in the supplement, but they don’t model the civil and regulatory constraints that make buying capacity a slow, contested, and capital-intensive process. Most ‘projected’ grid expansion is already allocated or queued. In the real world, AI labs would need to outbid other industries, overcome permitting fights, and likely hit infrastructure bottlenecks. That’s a huge deal, and it’s nowhere in the scenario.
Outbidding other industries for electricity to power the most productive technology ever developed by an industry that is, under present-day conditions, able to secure staggering sums of capital with ease seems extremely easy to me. What am I overlooking?
Interesting point. Although you can also always buy out a contract, too (or even agree to indemnify and pay the expenses associated with efficient breach) and electricity famously has an extremely sophisticated and elaborate spot market, although I'm not as familiar with long-term power contract structure. It might end up easiest to just buy up and shut down certain high-power usage companies if you otherwise risk some kind of injunctive restriction.
True—power markets are financially sophisticated. But even if you buy out long-term contracts or indemnify existing users, that doesn’t conjure new electrons or expand substation capacity. Spot prices don’t build transformers, and shutting down a steel mill doesn’t reroute cooling infrastructure. Power is tradable on the margin, but AGI-scale compute requires physical scaling, not just market maneuvers.
We've agreed elsewhere, but as far as I can tell literally zero people who work with physical things (power, building datacenters, building fabs, extracting minerals) think double digit frontier economy GDP growth is plausible in the near term nor the kind of 1000x improvement we see in the scenario.
For context my scenario is still *wildly* more aggressive than the normie one. I just very strongly believe that significantly accelerating infrastructure development is qualitatively more difficult to do than steadily increasing by like 5% a year because of the extensive labour and materials supply chains involved, and no one arguing for mega fast timelines really seems to acknowledge this.
The scenario assumes a kind of Factorio-zation via robots in 2028, with millions of robots being produced every month. Industrialization is slow in human terms for all sorts of reasons that don't necessarily apply smoothly here (law, coordination, safety, etc).
If a datacenter has its own electricity generation (not connected to the grid), do the same permit/regulation bottlenecks apply? My understanding is that the bureaucratic slowdown is largely in connecting a new power generator to the grid; without that building the physical power plant may be faster.
Going off-grid avoids utility interconnection delays, but not most other bottlenecks. You still need permits for air, water, and land use; access to long-lead equipment like turbines and transformers; and time to construct and integrate cooling, fuel logistics, and redundancy systems. It’s faster—but not fast enough for a 100+ GW scale-up by 2027.
Yes, the same bottlenecks are applying. The Federal Energy Regulatory Commission (FERC) recently killed a co-location project.
It is convening a technical panel to decide how it wants to handle these kinds of permit applications in the future. To see an example of one of the issues they're concerned about, "A number of panelists expressed concern that permitting co-located large-load customers would enable them to avoid the costs associated with grid maintenance and upgrades due to the absence of any transmission tariffs in a behind-the-meter configuration."
>It’s definitely true that there will be a lot of disagreement over how accurate Scenario 2027 was, regardless of its level of accuracy, so long as it isn’t completely off base.
If this document is not completely off base, I predict there won't be much disagreement by 2028
The core assumption I would like to see challenged, because I don't think it's maximally probable, is the "limited number of decision makers" one. That assumption, that decisions about frontier AI policy are made by a limited cadre, itself has sub-assumptions. Notably that the forces that allow a limited number of decision makers to keep making decisions in isolation remain constant.
The counter scenario I want to read is one where the funders of the lab making the breakthrough, due to external economic conditions (cough, cough tariffs, recession, "need to show this actually makes money," etc.) tell the frontier lab to commercialize Agent 2 and devote their compute to serving it, downgrading research. I think this would push the scenario into fresh directions. Notably I think it might push the public reaction over whatever threshold is needed to get regulation and oversight.
I have been playing through this war game scenario with 4o under similar assumptions and timelines and it’s quite interesting how similar some of its conclusions are to the papers when playing china. I’ve got some time left to go but where we are at now it seems convinced it’s best chance at an advantage is a kinetic strike on Taiwan which I think is probably true generally.
Reading the last paragraph or so of the "Good Ending" with all the spacefaring at the end, a possible solution to the Fermi Paradox suggested itself:
There IS a massive, rich intergalactic civilization out there, but we puny organics can't see it as it's since composed solely of(and restricted to) ASIs chattering at each other on their self-designed "Starshot"-sized ships zipping around at near-relativisitic speeds away from their origin worlds as fast as they can, like a kid bolting from a one-street podunk rural town and never looking back.
We aren't "spared" or "allowed to live", just left behind.
Why do ASIs need something as big as Dyson Spheres? ---which were originally posited as a way for an organic civilization to harvest all the energy off a main-sequence star to power itself.
If you're just running compute and don't need a physical body, you don't need such a huge structure. Also, Dyson Spheres are really only useful for the couple billion years during the subject star's Main Sequence.
I'm thinking more along the lines of Charlie Stross' "Accelerando" or "Eschaton" series("Infinity Sky" and "Iron Sunrise").
>If you're just running compute and don't need a physical body, you don't need such a huge structure.
Compute requires a huge amount of energy. It is funny that you mention "Accelerando" because, according to my reading of Wikipedia, a core feature of that book is Dyson spheres (aka Matrioshka brains) constructed solely to power compute.
> Also, Dyson Spheres are really only useful for the couple billion years during the subject star's Main Sequence.
That describes a large fraction of the stars we see in the sky, no?
Um. Do you have a concrete prediction of them being not NPCs? I can imagine these but I think they just underscore the challenge, and this is the extent of what I can imagine so I'd love to see what else you might be thinking of:
1. UAE and Saudia Arabia collaborate to buy NVIDIA (nearly zero chance of happening, zero chance of being allowed to happen) and offer more compute and money than anyone else, effectively poaching a good chunk of the best Chinese and American talent?
2. France goes "rogue" within the EU, scraps centuries of culturally determined regulation and taxes, and offers all AI researchers a free lifetime visa + reserves 10GW of power for AI + completely reforms it's planning system and starts building 50 GW more nuclear power with the shovels hitting the dirt this year?
3. Singapore decides _this year_ (would seem too late otherwise), that this is a "survival of the state" crisis and does what I outlined above for France (more plausible in almost every way except the nuclear power, maybe wave power?)?
AI maxis are already wrong because they all believe in the caloric theory of intelligence, where intelligence is a substance you can pump and bottle, distill and apply fungibly across any application. It's nonsense that only computer scientists could possibly believe. I set a 0 percent chance of any of this happening ever.
I think computer scientists are some of the least likely people to subscribe to a caloric theory of intelligence (which is a great term, by the way). Systems, complexity, electronic engineering, distributed systems all have central concerns around fundamental limitations that more intelligence does not seem to overcome. People in these fields seem very unlikely to forget their no-go theorems and lower bounds. Do you have a particular CS subfield in mind that might be more prone to this belief?
I'm not making this statement based in likelihood, I'm making it based on observations of how they talk about intelligence. From IQ-philia to LLMs as "intelligence pumps".
By the way, either this is a genAI reply, you're trying to sound like one, or you're just not putting very much effort in, and in any case it's disrespectful.
Thank you, that's the first time someone has pattern matched "widely read, expresses curiosity, uses pompous language" to "must be GenAI". I'm kind of flattered (and would genuinely like to know why you felt disrespected). Since you dodged my question, I'm going to infer from your response that you don't know if there is a subfield of CS that is prone to caloric intelligence beliefs, but suspect some people with AI safety backgrounds do have such beliefs. If so, then I wonder what distinguishes such backgrounds from say someone who thinks in terms of the FLP theorem, or who worries about dealing with undecidable problems in their work.
I think computer scientists are especially prone to to misunderstanding intelligence because they live their lives seeing problems like a computer would see them and tend to ignore, misunderstand, or in extreme cases resent and hate human aspects of problems. They're overly quantitative-minded and have a "I have a hammer, therefore this problem is a nail" type approach.
"AI safety" is another great example, because there is a ton of focus on AI superintelligence but almost no worrying about humans using existing AI tools in novel and dangerous ways, or the social effects of feed algorithms. It's like how effective altruism gets consumed by shit like shrimp welfare instead of, for example, opposing nascent fascism in their own countries.
By the way, the thing that flagged me on your reply was the posture of fake respect. "Oh that's interesting, tell me more" is such a chatbot move.
It would be sad if RLHF to increase various metrics led to taping off big chunks of linguistic expression as "too AI-ey to be used between humans". I suppose it has already happened, dammit I want delve and outwith and notwithstanding back, and I don't like the assumption that Nigerian retirees and Indian government documents from before 2020 should be othered. I also think you might want to check out Stanford's HAI lab, Berkeley Re-AI, or projects funded by ARIA's Safeguarded AI fund as counterexamples to your blanket disparagement of CS people.
English was already becoming far less flowery already, and the amplification of cliches powered by LLMs is probably just going to speed that up. Go back and read MLK's letter from Birmingham jail, it's almost intolerably overwritten by modern standards and that wasn't even that long ago!
"To give one concrete example: there seems to be a strong assumption that there are a set of major military breakthroughs that can be achieved through sheer intelligence.
I obviously can't rule this out but it's hard to imagine what kind of breakthroughs this could be. If you had an idea for a new bomb or missile or drone or whatever, you'd need to build prototypes, test them, set up factories, etc. An AI in a datacenter can't do that stuff."
A lot of military systems' development is dominated by software. Given a superprogrammer (not just supercoder - correct resolution of requirements' ambiguities, design, architecture, anticipation of edge/corner cases, test case construction, etc.), that whole block of time gets faster.
Given a superDesignEngineer, the number of design bugs that could have been anticipated before the first prototype is built also crashes. With *NO* materials science improvements, the AGI wrangler is suddenly fielding novel military systems much faster (and possibly a wider variety of them as well).
A way that AI 2027 made me more pessimistic is that it made real the way that for plausible good endings “alignment” means advancing US interests. I think the report says something like “Safer-4’s goal to spread American citizens and their descendants throughout the stars” or something, explicitly excluding my children.
True it points to some kind of universal treaty being signed, not sure the details, that part doesn’t seem very secure in the prediction.
> Even if you could do plaintext, wouldn’t it be much less compute efficient if you forced it all to be plaintext with all the logic in the actual plaintext if you read it as a human? This is perhaps the key question in the whole scenario
Is there any accounting in this scenario for Chinese using possibly Chinese-language Chains-of-Thought? A!-2027 focuses entirely on English-language training models. Has this been considered?
Interesting, thanks. I guess when your model has many billions of parameters, whether you start with 26 or 39 or 5,000+ lexical tokens really does not matter.
Kind of. I think. I understand frontier models to have multiple tens of thousands of tokens in their vocabulary (e.g. Claude reported as 59000, ChatGPT as 100000ish) and that these cover all languages (with an average of four or so letters in a typical English token, much less for a language like Chinese.)
Out the box, seems any token could feature in CoT, including mixtures across languages. The way lexical tokens imply semantics presumably varies across languages, and it may be that given our experience it has proven useful to force particular languages in CoT. Code helps reasoning models, and English is common there (although, again, I understand some models for coding specifically make particular tokenisation choices. Maybe programming language wars will enjoy seeing what languages the best models most like to reason in!)
> They make up plausible sounding, but totally fictional concepts like "neuralese recurrence and memory
Neuralese is more or less Chain of Continuous Thought https://arxiv.org/abs/2412.06769
My skepticism surrounds power and infrastructure demands. Wrote about it here: https://open.substack.com/pub/davefriedman/p/the-agi-bottleneck-is-power-not-alignment?r=37ez3&utm_medium=ios
They did (briefly) cover power in the Compute Forecast – it seems they think the AI labs will buy up existing and projected capacity and, so I'm guessing, not magically create new capacity much quicker than is plausible.
I agree the authors mention power in the supplement, but they don’t model the civil and regulatory constraints that make buying capacity a slow, contested, and capital-intensive process. Most ‘projected’ grid expansion is already allocated or queued. In the real world, AI labs would need to outbid other industries, overcome permitting fights, and likely hit infrastructure bottlenecks. That’s a huge deal, and it’s nowhere in the scenario.
Outbidding other industries for electricity to power the most productive technology ever developed by an industry that is, under present-day conditions, able to secure staggering sums of capital with ease seems extremely easy to me. What am I overlooking?
Long term contracts?
Interesting point. Although you can also always buy out a contract, too (or even agree to indemnify and pay the expenses associated with efficient breach) and electricity famously has an extremely sophisticated and elaborate spot market, although I'm not as familiar with long-term power contract structure. It might end up easiest to just buy up and shut down certain high-power usage companies if you otherwise risk some kind of injunctive restriction.
True—power markets are financially sophisticated. But even if you buy out long-term contracts or indemnify existing users, that doesn’t conjure new electrons or expand substation capacity. Spot prices don’t build transformers, and shutting down a steel mill doesn’t reroute cooling infrastructure. Power is tradable on the margin, but AGI-scale compute requires physical scaling, not just market maneuvers.
We've agreed elsewhere, but as far as I can tell literally zero people who work with physical things (power, building datacenters, building fabs, extracting minerals) think double digit frontier economy GDP growth is plausible in the near term nor the kind of 1000x improvement we see in the scenario.
For context my scenario is still *wildly* more aggressive than the normie one. I just very strongly believe that significantly accelerating infrastructure development is qualitatively more difficult to do than steadily increasing by like 5% a year because of the extensive labour and materials supply chains involved, and no one arguing for mega fast timelines really seems to acknowledge this.
The scenario assumes a kind of Factorio-zation via robots in 2028, with millions of robots being produced every month. Industrialization is slow in human terms for all sorts of reasons that don't necessarily apply smoothly here (law, coordination, safety, etc).
Indeed, the special economic zones do quite a lot of work here.
If a datacenter has its own electricity generation (not connected to the grid), do the same permit/regulation bottlenecks apply? My understanding is that the bureaucratic slowdown is largely in connecting a new power generator to the grid; without that building the physical power plant may be faster.
Going off-grid avoids utility interconnection delays, but not most other bottlenecks. You still need permits for air, water, and land use; access to long-lead equipment like turbines and transformers; and time to construct and integrate cooling, fuel logistics, and redundancy systems. It’s faster—but not fast enough for a 100+ GW scale-up by 2027.
Yes, the same bottlenecks are applying. The Federal Energy Regulatory Commission (FERC) recently killed a co-location project.
It is convening a technical panel to decide how it wants to handle these kinds of permit applications in the future. To see an example of one of the issues they're concerned about, "A number of panelists expressed concern that permitting co-located large-load customers would enable them to avoid the costs associated with grid maintenance and upgrades due to the absence of any transmission tariffs in a behind-the-meter configuration."
See: https://natlawreview.com/article/ferc-busy-considering-issues-relating-co-located-large-loads
>It’s definitely true that there will be a lot of disagreement over how accurate Scenario 2027 was, regardless of its level of accuracy, so long as it isn’t completely off base.
If this document is not completely off base, I predict there won't be much disagreement by 2028
Anyway where's the Butlerian Jihad committee
The core assumption I would like to see challenged, because I don't think it's maximally probable, is the "limited number of decision makers" one. That assumption, that decisions about frontier AI policy are made by a limited cadre, itself has sub-assumptions. Notably that the forces that allow a limited number of decision makers to keep making decisions in isolation remain constant.
The counter scenario I want to read is one where the funders of the lab making the breakthrough, due to external economic conditions (cough, cough tariffs, recession, "need to show this actually makes money," etc.) tell the frontier lab to commercialize Agent 2 and devote their compute to serving it, downgrading research. I think this would push the scenario into fresh directions. Notably I think it might push the public reaction over whatever threshold is needed to get regulation and oversight.
Podcast episode for this post:
https://open.substack.com/pub/dwatvpodcast/p/ai-2027-responses
I also had commentary
https://substack.com/@bassoe/note/c-106067324
I have been playing through this war game scenario with 4o under similar assumptions and timelines and it’s quite interesting how similar some of its conclusions are to the papers when playing china. I’ve got some time left to go but where we are at now it seems convinced it’s best chance at an advantage is a kinetic strike on Taiwan which I think is probably true generally.
Could you share your chat with this wargame?
Reading the last paragraph or so of the "Good Ending" with all the spacefaring at the end, a possible solution to the Fermi Paradox suggested itself:
There IS a massive, rich intergalactic civilization out there, but we puny organics can't see it as it's since composed solely of(and restricted to) ASIs chattering at each other on their self-designed "Starshot"-sized ships zipping around at near-relativisitic speeds away from their origin worlds as fast as they can, like a kid bolting from a one-street podunk rural town and never looking back.
We aren't "spared" or "allowed to live", just left behind.
I believe in such a scenario, our astronomers would see Dyson spheres in the sky rather than stars? And they don't.
Why do ASIs need something as big as Dyson Spheres? ---which were originally posited as a way for an organic civilization to harvest all the energy off a main-sequence star to power itself.
If you're just running compute and don't need a physical body, you don't need such a huge structure. Also, Dyson Spheres are really only useful for the couple billion years during the subject star's Main Sequence.
I'm thinking more along the lines of Charlie Stross' "Accelerando" or "Eschaton" series("Infinity Sky" and "Iron Sunrise").
>If you're just running compute and don't need a physical body, you don't need such a huge structure.
Compute requires a huge amount of energy. It is funny that you mention "Accelerando" because, according to my reading of Wikipedia, a core feature of that book is Dyson spheres (aka Matrioshka brains) constructed solely to power compute.
> Also, Dyson Spheres are really only useful for the couple billion years during the subject star's Main Sequence.
That describes a large fraction of the stars we see in the sky, no?
Anyone know if this document, or stuff of its nature, has pierced into the natsec or elite policy circles at all?
Thoughts on AI 2027 from the edge of the world ... basically: not every human outside US and China is necessarily an NPC... https://memia.substack.com/p/ai-2027-data-fracking-llama-44-midjourney#:~:text=comments%20%C2%B7%20Zvi%20Mowshowitz-,My%202%C2%A2%3A,-Yes%2C%20this%20is
Um. Do you have a concrete prediction of them being not NPCs? I can imagine these but I think they just underscore the challenge, and this is the extent of what I can imagine so I'd love to see what else you might be thinking of:
1. UAE and Saudia Arabia collaborate to buy NVIDIA (nearly zero chance of happening, zero chance of being allowed to happen) and offer more compute and money than anyone else, effectively poaching a good chunk of the best Chinese and American talent?
2. France goes "rogue" within the EU, scraps centuries of culturally determined regulation and taxes, and offers all AI researchers a free lifetime visa + reserves 10GW of power for AI + completely reforms it's planning system and starts building 50 GW more nuclear power with the shovels hitting the dirt this year?
3. Singapore decides _this year_ (would seem too late otherwise), that this is a "survival of the state" crisis and does what I outlined above for France (more plausible in almost every way except the nuclear power, maybe wave power?)?
AI maxis are already wrong because they all believe in the caloric theory of intelligence, where intelligence is a substance you can pump and bottle, distill and apply fungibly across any application. It's nonsense that only computer scientists could possibly believe. I set a 0 percent chance of any of this happening ever.
I think computer scientists are some of the least likely people to subscribe to a caloric theory of intelligence (which is a great term, by the way). Systems, complexity, electronic engineering, distributed systems all have central concerns around fundamental limitations that more intelligence does not seem to overcome. People in these fields seem very unlikely to forget their no-go theorems and lower bounds. Do you have a particular CS subfield in mind that might be more prone to this belief?
I'm not making this statement based in likelihood, I'm making it based on observations of how they talk about intelligence. From IQ-philia to LLMs as "intelligence pumps".
By the way, either this is a genAI reply, you're trying to sound like one, or you're just not putting very much effort in, and in any case it's disrespectful.
Thank you, that's the first time someone has pattern matched "widely read, expresses curiosity, uses pompous language" to "must be GenAI". I'm kind of flattered (and would genuinely like to know why you felt disrespected). Since you dodged my question, I'm going to infer from your response that you don't know if there is a subfield of CS that is prone to caloric intelligence beliefs, but suspect some people with AI safety backgrounds do have such beliefs. If so, then I wonder what distinguishes such backgrounds from say someone who thinks in terms of the FLP theorem, or who worries about dealing with undecidable problems in their work.
I think computer scientists are especially prone to to misunderstanding intelligence because they live their lives seeing problems like a computer would see them and tend to ignore, misunderstand, or in extreme cases resent and hate human aspects of problems. They're overly quantitative-minded and have a "I have a hammer, therefore this problem is a nail" type approach.
"AI safety" is another great example, because there is a ton of focus on AI superintelligence but almost no worrying about humans using existing AI tools in novel and dangerous ways, or the social effects of feed algorithms. It's like how effective altruism gets consumed by shit like shrimp welfare instead of, for example, opposing nascent fascism in their own countries.
By the way, the thing that flagged me on your reply was the posture of fake respect. "Oh that's interesting, tell me more" is such a chatbot move.
It would be sad if RLHF to increase various metrics led to taping off big chunks of linguistic expression as "too AI-ey to be used between humans". I suppose it has already happened, dammit I want delve and outwith and notwithstanding back, and I don't like the assumption that Nigerian retirees and Indian government documents from before 2020 should be othered. I also think you might want to check out Stanford's HAI lab, Berkeley Re-AI, or projects funded by ARIA's Safeguarded AI fund as counterexamples to your blanket disparagement of CS people.
English was already becoming far less flowery already, and the amplification of cliches powered by LLMs is probably just going to speed that up. Go back and read MLK's letter from Birmingham jail, it's almost intolerably overwritten by modern standards and that wasn't even that long ago!
It's also rationalist 101 😅
I continue to hope I'm wrong and he's right but man Tim Lee seems so wrong on this.
Contra Timothy Lee's:
"To give one concrete example: there seems to be a strong assumption that there are a set of major military breakthroughs that can be achieved through sheer intelligence.
I obviously can't rule this out but it's hard to imagine what kind of breakthroughs this could be. If you had an idea for a new bomb or missile or drone or whatever, you'd need to build prototypes, test them, set up factories, etc. An AI in a datacenter can't do that stuff."
A lot of military systems' development is dominated by software. Given a superprogrammer (not just supercoder - correct resolution of requirements' ambiguities, design, architecture, anticipation of edge/corner cases, test case construction, etc.), that whole block of time gets faster.
Given a superDesignEngineer, the number of design bugs that could have been anticipated before the first prototype is built also crashes. With *NO* materials science improvements, the AGI wrangler is suddenly fielding novel military systems much faster (and possibly a wider variety of them as well).
A way that AI 2027 made me more pessimistic is that it made real the way that for plausible good endings “alignment” means advancing US interests. I think the report says something like “Safer-4’s goal to spread American citizens and their descendants throughout the stars” or something, explicitly excluding my children.
True it points to some kind of universal treaty being signed, not sure the details, that part doesn’t seem very secure in the prediction.
> Even if you could do plaintext, wouldn’t it be much less compute efficient if you forced it all to be plaintext with all the logic in the actual plaintext if you read it as a human? This is perhaps the key question in the whole scenario
Could you please explain?
Zvi,
Is there any accounting in this scenario for Chinese using possibly Chinese-language Chains-of-Thought? A!-2027 focuses entirely on English-language training models. Has this been considered?
There are details (tokenisation choices) but for LLMs human languages are basically interchangeable.
The training corpus is input across many languages, and capabilities have always transferred well.
You can do your CoT in other languages right now if you want to.
Interesting, thanks. I guess when your model has many billions of parameters, whether you start with 26 or 39 or 5,000+ lexical tokens really does not matter.
Kind of. I think. I understand frontier models to have multiple tens of thousands of tokens in their vocabulary (e.g. Claude reported as 59000, ChatGPT as 100000ish) and that these cover all languages (with an average of four or so letters in a typical English token, much less for a language like Chinese.)
I am ignorant about how multi-modality plays into this, though, and from reports like https://tokencontributions.substack.com/p/whole-words-and-claude-tokenization you can see that different models make some different choices.
Out the box, seems any token could feature in CoT, including mixtures across languages. The way lexical tokens imply semantics presumably varies across languages, and it may be that given our experience it has proven useful to force particular languages in CoT. Code helps reasoning models, and English is common there (although, again, I understand some models for coding specifically make particular tokenisation choices. Maybe programming language wars will enjoy seeing what languages the best models most like to reason in!)