He raises, and dismisses, that idea in this paragraph from page 138 (which Zvi quoted in part in this post):
> Some hope for some sort of international treaty on safety. This seems fanciful to me. The world where both the CCP and USG are AGI-pilled enough to take safety risk seriously is also the world in which both realize that international economic and military predominance is at stake, that being months behind on AGI could mean being permanently left behind. If the race is tight, any arms control equilibrium, at least in the early phase around superintelligence, seems extremely unstable. In short, ”breakout” is too easy: the incentive (and the fear that others will act on this incentive) to race ahead with an intelligence explosion, to reach superintelligence and the decisive advantage, too great. At the very least, the odds we get something good-enough here seem slim. (How have those climate treaties gone? That seems like a dramatically easier problem compared to this.)
This ties into my biggest beef: Leopold makes very rosy assumptions regarding our ability to speedrun ASI and superalignment, but does not allow for similarly rosy assumptions regarding our ability to negotiate a slowdown. So of course he concludes that a sprint to ASI is possible, necessary, and safer than the alternative.
Yeah it's easy to think it's an arms race, and make the parallel to climate change, but for now it looks like unforced error...
And the problem with climate change is that nobody wants to give up their planes and their cars. But the voters won't riot if they don't have GPT-5 (even though yeah I want GPT-5)
I'm not sure how realistic a global treaty to pause AI would be, but I agree it deserves more thorough exploration. The stakes of AGI/ASI going wrong are way higher than for climate change. If we get warning shots (deception, self-replication, power-seeking in pre-takeoff AI models) that make it abundantly clear to world leaders that the power of ASI couldn't be harnessed, incentives to coordinate might be sufficiently strong for it to happen. Therefore, building the conditional infrastructure for slowing down/pausing so that *if* all major sides agree to slow down we are actually able to, seems very much worthwhile to me.
I like the "building the conditional infrastructure" framing. An idea I've been repeatedly encountering is that often the only opportunity to introduce a new approach is after a crisis, but that it's necessary to be prepared to seize the moment if and when that crisis occurs.
Yeah, he's very much, "There's no way to stop this, and we have to beat the Chinese, but don't worry, as long as we're careful it will turn out to be fine."
It would sure be nice if at minimum we do this tremendously risky and disruptive thing a lot slower, couldn't we at least be trying to talk to the Chinese about that?
The biggest problem I see with Aschenbrenner’s argument, which he acknowledges, is that we seem to be running out of data for training. It’s certainly a solvable problem but I don’t know whether it can be solved in an amount of time that conforms to his prediction of AGI by 2027-28.
Yep, that's a big one. I certainly see his arguments, and he's good about noticing uncertainty here and in many other places, although he seems not to multiply that uncertainty together in the bigger picture.
Beyond Rose's comment, I think Aschenbrenner is subtly implying why this isn't worrying the field, they are mostly banking on switching paradigms to RL, just like alphago did, where it gained maybe 90% of its interesting capabilities. AGI isn't compressing the internet it's interacting with texts and generating and checking knowledge continuously. If we stick the landing.
Yeah, I should revise my claim. I believe epoch and especially Amodei's comments here are consistent with having a few plausible but nontrivial strategies here, to include RL/self-play, synthetic data generation, trying to recognize and squeeze more out of higher quality data.
Some versions of these are all happening now, but not to the extent humans seem to be able to independently think through problems, generate new knowledge, or sift for and gain more from high quality data, by a wide margin, so there should be room to take a few halting steps toward the evolutionary solution here.
But that doesn't mean nobody is worried or that paradigm shifts are costless or without risk. They have several outs left. The frontier labs are probably better positioned to drive these directions and are relatively less worried. I'm sure it's nonzero but they'd have had a long time to come to terms with when we'd run out of human generated data. Even if you are supplementing with automated transcription and translation and ripping from everywhere.
So, step one: waive the Biden tariffs on Chinese photovoltaics *if* they're to be installed as captive power for AI datacenters in the United States.
Because otherwise China's surplus PV capacity gets bought up by the Abu Dhabi Investment Authority (assets under management: $1 trillion, easily leveraged severalfold with debt secured by power generation capex), which then paves the northern Empty Quarter with solar panels and sells the electricity to Sam Altman.
Assuming you put the photovoltaic _outside_ the datacenter, and ensure you smooth the power to avoid leaking information via spikes in current, it seems fairly safe to me.
I think where Aschenbrenner goes wrong is when he frames AGI as a fight of the West against China. I would prefer to see us as just one intelligent civilization in the universe. We should have an interest in the survival of human civilization on our planet Earth.
The fact is that we need to contemplate that the technical solutions for worldwide oversight of AGI(and pausing to get there) is likely easier than trying to align a superintelligence. Neither are easy, but the latter is completely intractable in comparison.
A world in which Xi Jinping fully controls the only locus of superintelligence is obviously a catastrophic failure mode, while one in which both sides have superintelligence is also really bad. I suppose even the first is better than everyone dying, but it’s very unclear how you get there without an unforced error by the US, so why should you?
*an unforced error that dramatically increases arms race dynamics, critically—just one country winning is the only route to there being a single reasonably controlled project rather than an out of control race.
What is more likely to lead to human extinction? An arms race between USA and China or AGI? If we are lucky, both can be prevented because both may have the same solution. For human civilization (and AI) to survive we need to come to a new understanding of what it means to be human. On the one hand, that will include a new understanding of how biological and human intelligence is different from artificial, non-human intelligence. On the other hand, that will include the idea that it is worth to preserve human intelligence in the wider universe. I see both projects, which are philosophical projects, as closely related.
Finally finished listening to it. This is not self consistent. There’s no superalignment solution, there probably will be superintelligence, but property rights will be respected. So we can’t install side constraints, but we can install side constraints
I'm not sure if I'm right, but this is what I've come to believe after much thought about AGI, super-intellegience and the like.
1. The real world is messy and probablistic, and collectively humans are close to the limit of making predictions about it.
2. You need to define intellegence. We don't spend enough time asking ourseleves what it is, without that you can't really understand the idea of super-intellegence.
3. Defining intellegience, or what we mean by intellegence helps contextualize what the AI can do better and what it can't, as well as the limits of better means. For example one part of intellegeince is reasoning. Humans make mistakes, but we use collective intellegence to fix individual reasoning mistakes. The AI should make fewer mistakes in formal reasoning, and fix them better, but few problems are pure reasoning problems anyway. Humans are close to the limit here. Plus there are other parts of intellegence.
4. One part of intellegence AI is good at is being able to keep track of more information to reason over. But do you think AI will beat human predicition markets in prediciting events? If so but how much? If not what the fuck are we talking about. Humans might be close to the limit.
Conclusion, humans are close to the limit of intellegence, Super intellegence might barely beat a Van Neuman. It won't be a god. It won't turn off all the enemies nukes. The real world is messy trying to make things happen in the real world is hard, and probablistic. Like any plan that involves more than 3 steps will fail for AI just like it does for people.
> (40) Page 81: Reports suggest OpenAI was at a $1B revenue run rate in August 2023, and a $2B revenue run rate in February 2024. That’s roughly a doubling every 6 months.
This is pretty crazy. Received wisdom in tue startup world is that in the realm of 8-9 figures a yearly 2x is best in class, mostly falling off to less than that near the top of that range. OpenAI did roughly an order of magnitude better and I have to ascribe most of that to its being the main character in a massive hype cycle which I don't see staying this intense. A more realistic (and still optimistic) projection from here would be superlinear growth hitting $10B sometime in 2027. Predicting 3x sooner than that isn't very believable, and reduces my confidence in the other naive exponential-growth predictions being made here.
Well-put: Aschenbrenner's many "naive exponential-growth predictions" are exactly what need critical attention and analysis. Just because we can identify long-term exponential curves doesn't mean that everything one can chart on an log graph (going up *or* down) will proceed along the same exponential path. Some things will! But many won't. At the risk of becoming the resident "Singularity is Near" guy, here is one example among many: "...we can project the exponentially declining cost of computation and the rapidly declining size and increasing effectiveness of both electronic and mechanical technologies. Based on these projections, we can conservatively [!] anticipate the requisite nanobot technology to implement these types of scenarios [of nanobot brain scanning] during the 2020s." Ray Kurzweil, The Singularity is Near (2005). He predicted not just nanobot brain scanners. Based on the same (naive) exponential extrapolations, RK predicted that within "twenty to twenty-five years" of 2004 we would see "[n]anobots [that] will be able to travel through the bloodstream, then go in and around our cells and perform various services, such as removing toxins, weeding out debris, correcting DNA errors, [etc.]." Nanobots correcting DNA errors! I would bet we'll have flying cars or AI doom before we have those kinds of bloodstream nanobots. To be clear, RK's work was very thought-provoking, and some of his predictions-based-on-exponential-curves were or might turn out to be roughly right. But a lot were wrong, and, especially with smart folks like Aschenbrenner giving us The Singularity is Near 2.0, it's worth pondering why.
I think this is one of those 'right for the wrong reasons' situations. I would not say 'well this is the doubling rate so that will continue' but I would take the over on $10B by 2027.
Anyone reading these lines has at most 50-60 years left on this planet, absent any accidents or early cancers. I hope Aschenbrenner is right and we get AGI sometime in the next decade or two, so that we could at least have a chance of finding a cure for ageing/death.
Probably not much. GPT-4 has been very locked into a certain a style and worldview. It really, really wants to believe that we'll all talk to each, work together, solve our problems, and reach a happy ending through the power of friendship. To see what I mean, try to get GPT-4 to write a long film noir scene, where people are self-destructive and untrustworthy, and where things will probably end badly. It resists writing this way.
Claude, now, I've seen Claude Opus get carried away with very different styles and deeply strange world views.
"In 2027, a leading AI lab will be able to train a GPT-4- level model in a minute"
It's hard to cherry pick quotes since the original is so good, and many of the most jarring bits rely on a scaffolding contained in the pages before it.
If you are reading this and thinking it's a fair summary, don't. If you want a nice snapshot of the current state of things go read the original. 165 pages but they're very short pages.
> A number of people have told me that in an hour of conversation they could relay enough information to a competitor such that their firm’s alpha would go to ~zero—similar to how many key AI algorithmic secrets could be relayed in a short conversation—and yet these firms manage to keep these secrets and retain their edge.
Wow. This was news to me.
> For years people claimed robots were a hardware problem—but robot hardware is well on its way to being solved.
To what extent do the markets believe this in particular? In Zvi's commentary of the Dwarkesh Patel-Leopold Aschenbrenner talk (https://thezvi.substack.com/p/on-dwarkeshs-podcast-with-leopold), there's discussion of EMH in general, but not on this in particular.
AI Narration of this post:
https://askwhocastsai.substack.com/p/quotes-from-leopold-aschenbrenners
Am I missing the paragraph where he explores the mood of "wait what if we didn't do that then, what if we stopped?"
Going to have to use numbers here, I don't know which one you mean.
No, exactly, I mean that I can't find such a paragraph in this post *or* in what I read from the original Situational Awareness doc.
So maybe I missed it while reading too fast / skipping some parts... But as far as I can tell, probably not.
Are you asking whether Aschenbrenner contemplates stopping AI research? If so, he doesn't contemplate that.
Or implies that stopping mid stream in an arms race is de facto losing to whoever doesn’t stop.
Yes, that too.
He raises, and dismisses, that idea in this paragraph from page 138 (which Zvi quoted in part in this post):
> Some hope for some sort of international treaty on safety. This seems fanciful to me. The world where both the CCP and USG are AGI-pilled enough to take safety risk seriously is also the world in which both realize that international economic and military predominance is at stake, that being months behind on AGI could mean being permanently left behind. If the race is tight, any arms control equilibrium, at least in the early phase around superintelligence, seems extremely unstable. In short, ”breakout” is too easy: the incentive (and the fear that others will act on this incentive) to race ahead with an intelligence explosion, to reach superintelligence and the decisive advantage, too great. At the very least, the odds we get something good-enough here seem slim. (How have those climate treaties gone? That seems like a dramatically easier problem compared to this.)
This ties into my biggest beef: Leopold makes very rosy assumptions regarding our ability to speedrun ASI and superalignment, but does not allow for similarly rosy assumptions regarding our ability to negotiate a slowdown. So of course he concludes that a sprint to ASI is possible, necessary, and safer than the alternative.
Thanks!
Yeah it's easy to think it's an arms race, and make the parallel to climate change, but for now it looks like unforced error...
And the problem with climate change is that nobody wants to give up their planes and their cars. But the voters won't riot if they don't have GPT-5 (even though yeah I want GPT-5)
I'm not sure how realistic a global treaty to pause AI would be, but I agree it deserves more thorough exploration. The stakes of AGI/ASI going wrong are way higher than for climate change. If we get warning shots (deception, self-replication, power-seeking in pre-takeoff AI models) that make it abundantly clear to world leaders that the power of ASI couldn't be harnessed, incentives to coordinate might be sufficiently strong for it to happen. Therefore, building the conditional infrastructure for slowing down/pausing so that *if* all major sides agree to slow down we are actually able to, seems very much worthwhile to me.
I like the "building the conditional infrastructure" framing. An idea I've been repeatedly encountering is that often the only opportunity to introduce a new approach is after a crisis, but that it's necessary to be prepared to seize the moment if and when that crisis occurs.
> If we get warning shots...
I currently suspect that a depressingly high percentage of good outcomes pass through something along the lines of, 'Oops, there goes Tokyo.'
Yeah, he's very much, "There's no way to stop this, and we have to beat the Chinese, but don't worry, as long as we're careful it will turn out to be fine."
It would sure be nice if at minimum we do this tremendously risky and disruptive thing a lot slower, couldn't we at least be trying to talk to the Chinese about that?
So, who are they running for president [of/to deploy] The Project then?
The biggest problem I see with Aschenbrenner’s argument, which he acknowledges, is that we seem to be running out of data for training. It’s certainly a solvable problem but I don’t know whether it can be solved in an amount of time that conforms to his prediction of AGI by 2027-28.
Yep, that's a big one. I certainly see his arguments, and he's good about noticing uncertainty here and in many other places, although he seems not to multiply that uncertainty together in the bigger picture.
I wrote a bit about it here: https://davefriedman.substack.com/p/thoughts-on-leopold-aschenbrenners . Note David Rose's comment though.
Beyond Rose's comment, I think Aschenbrenner is subtly implying why this isn't worrying the field, they are mostly banking on switching paradigms to RL, just like alphago did, where it gained maybe 90% of its interesting capabilities. AGI isn't compressing the internet it's interacting with texts and generating and checking knowledge continuously. If we stick the landing.
Hm, that's an interesting point. But if true why is there so much commentary from AI researchers about whether or not they are running out of data on the internet? For ex here's Epoch AI with recent research about the running out of data question: https://epochai.org/blog/will-we-run-out-of-data-limits-of-llm-scaling-based-on-human-generated-data
Yeah, I should revise my claim. I believe epoch and especially Amodei's comments here are consistent with having a few plausible but nontrivial strategies here, to include RL/self-play, synthetic data generation, trying to recognize and squeeze more out of higher quality data.
Some versions of these are all happening now, but not to the extent humans seem to be able to independently think through problems, generate new knowledge, or sift for and gain more from high quality data, by a wide margin, so there should be room to take a few halting steps toward the evolutionary solution here.
But that doesn't mean nobody is worried or that paradigm shifts are costless or without risk. They have several outs left. The frontier labs are probably better positioned to drive these directions and are relatively less worried. I'm sure it's nonzero but they'd have had a long time to come to terms with when we'd run out of human generated data. Even if you are supplementing with automated transcription and translation and ripping from everywhere.
So, step one: waive the Biden tariffs on Chinese photovoltaics *if* they're to be installed as captive power for AI datacenters in the United States.
Because otherwise China's surplus PV capacity gets bought up by the Abu Dhabi Investment Authority (assets under management: $1 trillion, easily leveraged severalfold with debt secured by power generation capex), which then paves the northern Empty Quarter with solar panels and sells the electricity to Sam Altman.
Sounds like a good way to get spyware into your datancenters.
Assuming you put the photovoltaic _outside_ the datacenter, and ensure you smooth the power to avoid leaking information via spikes in current, it seems fairly safe to me.
I think where Aschenbrenner goes wrong is when he frames AGI as a fight of the West against China. I would prefer to see us as just one intelligent civilization in the universe. We should have an interest in the survival of human civilization on our planet Earth.
The fact is that we need to contemplate that the technical solutions for worldwide oversight of AGI(and pausing to get there) is likely easier than trying to align a superintelligence. Neither are easy, but the latter is completely intractable in comparison.
A world in which Xi Jinping fully controls the only locus of superintelligence is obviously a catastrophic failure mode, while one in which both sides have superintelligence is also really bad. I suppose even the first is better than everyone dying, but it’s very unclear how you get there without an unforced error by the US, so why should you?
*an unforced error that dramatically increases arms race dynamics, critically—just one country winning is the only route to there being a single reasonably controlled project rather than an out of control race.
What is more likely to lead to human extinction? An arms race between USA and China or AGI? If we are lucky, both can be prevented because both may have the same solution. For human civilization (and AI) to survive we need to come to a new understanding of what it means to be human. On the one hand, that will include a new understanding of how biological and human intelligence is different from artificial, non-human intelligence. On the other hand, that will include the idea that it is worth to preserve human intelligence in the wider universe. I see both projects, which are philosophical projects, as closely related.
Why did you decide not critically engage with any of it?
I will do that later. For now I want to provide a resource without injecting my counterarguments.
Finally finished listening to it. This is not self consistent. There’s no superalignment solution, there probably will be superintelligence, but property rights will be respected. So we can’t install side constraints, but we can install side constraints
I'm not sure if I'm right, but this is what I've come to believe after much thought about AGI, super-intellegience and the like.
1. The real world is messy and probablistic, and collectively humans are close to the limit of making predictions about it.
2. You need to define intellegence. We don't spend enough time asking ourseleves what it is, without that you can't really understand the idea of super-intellegence.
3. Defining intellegience, or what we mean by intellegence helps contextualize what the AI can do better and what it can't, as well as the limits of better means. For example one part of intellegeince is reasoning. Humans make mistakes, but we use collective intellegence to fix individual reasoning mistakes. The AI should make fewer mistakes in formal reasoning, and fix them better, but few problems are pure reasoning problems anyway. Humans are close to the limit here. Plus there are other parts of intellegence.
4. One part of intellegence AI is good at is being able to keep track of more information to reason over. But do you think AI will beat human predicition markets in prediciting events? If so but how much? If not what the fuck are we talking about. Humans might be close to the limit.
Conclusion, humans are close to the limit of intellegence, Super intellegence might barely beat a Van Neuman. It won't be a god. It won't turn off all the enemies nukes. The real world is messy trying to make things happen in the real world is hard, and probablistic. Like any plan that involves more than 3 steps will fail for AI just like it does for people.
fascinating thorough analysis. Thank you ... I will need to study this in detail. SI When? and then what?
> (40) Page 81: Reports suggest OpenAI was at a $1B revenue run rate in August 2023, and a $2B revenue run rate in February 2024. That’s roughly a doubling every 6 months.
This is pretty crazy. Received wisdom in tue startup world is that in the realm of 8-9 figures a yearly 2x is best in class, mostly falling off to less than that near the top of that range. OpenAI did roughly an order of magnitude better and I have to ascribe most of that to its being the main character in a massive hype cycle which I don't see staying this intense. A more realistic (and still optimistic) projection from here would be superlinear growth hitting $10B sometime in 2027. Predicting 3x sooner than that isn't very believable, and reduces my confidence in the other naive exponential-growth predictions being made here.
Well-put: Aschenbrenner's many "naive exponential-growth predictions" are exactly what need critical attention and analysis. Just because we can identify long-term exponential curves doesn't mean that everything one can chart on an log graph (going up *or* down) will proceed along the same exponential path. Some things will! But many won't. At the risk of becoming the resident "Singularity is Near" guy, here is one example among many: "...we can project the exponentially declining cost of computation and the rapidly declining size and increasing effectiveness of both electronic and mechanical technologies. Based on these projections, we can conservatively [!] anticipate the requisite nanobot technology to implement these types of scenarios [of nanobot brain scanning] during the 2020s." Ray Kurzweil, The Singularity is Near (2005). He predicted not just nanobot brain scanners. Based on the same (naive) exponential extrapolations, RK predicted that within "twenty to twenty-five years" of 2004 we would see "[n]anobots [that] will be able to travel through the bloodstream, then go in and around our cells and perform various services, such as removing toxins, weeding out debris, correcting DNA errors, [etc.]." Nanobots correcting DNA errors! I would bet we'll have flying cars or AI doom before we have those kinds of bloodstream nanobots. To be clear, RK's work was very thought-provoking, and some of his predictions-based-on-exponential-curves were or might turn out to be roughly right. But a lot were wrong, and, especially with smart folks like Aschenbrenner giving us The Singularity is Near 2.0, it's worth pondering why.
I think this is one of those 'right for the wrong reasons' situations. I would not say 'well this is the doubling rate so that will continue' but I would take the over on $10B by 2027.
Anyone reading these lines has at most 50-60 years left on this planet, absent any accidents or early cancers. I hope Aschenbrenner is right and we get AGI sometime in the next decade or two, so that we could at least have a chance of finding a cure for ageing/death.
How much of his document was written by GPT-4.0 I wonder.
Probably not much. GPT-4 has been very locked into a certain a style and worldview. It really, really wants to believe that we'll all talk to each, work together, solve our problems, and reach a happy ending through the power of friendship. To see what I mean, try to get GPT-4 to write a long film noir scene, where people are self-destructive and untrustworthy, and where things will probably end badly. It resists writing this way.
Claude, now, I've seen Claude Opus get carried away with very different styles and deeply strange world views.
Great paper from LA. Really enjoyed reading it. Thank you for sharing it like that.
FWIW I’d take the bet that by 2030 we’ll still be muddling thru pretty much as usual.
Also have to say, that last paragraph…dude thinks he’s Dougie Howser in Starship Troopers 😄
Also:
"In 2027, a leading AI lab will be able to train a GPT-4- level model in a minute"
It's hard to cherry pick quotes since the original is so good, and many of the most jarring bits rely on a scaffolding contained in the pages before it.
If you are reading this and thinking it's a fair summary, don't. If you want a nice snapshot of the current state of things go read the original. 165 pages but they're very short pages.
> A number of people have told me that in an hour of conversation they could relay enough information to a competitor such that their firm’s alpha would go to ~zero—similar to how many key AI algorithmic secrets could be relayed in a short conversation—and yet these firms manage to keep these secrets and retain their edge.
Wow. This was news to me.
> For years people claimed robots were a hardware problem—but robot hardware is well on its way to being solved.
To what extent do the markets believe this in particular? In Zvi's commentary of the Dwarkesh Patel-Leopold Aschenbrenner talk (https://thezvi.substack.com/p/on-dwarkeshs-podcast-with-leopold), there's discussion of EMH in general, but not on this in particular.
I am very confident that the alpha story is NOT true for Jane Street, btw. You could zero out particular trades but the firm would be fine.
By the way, here is a comment - not related to AI - that Gwern posted last year: https://www.reddit.com/r/slatestarcodex/comments/16wdxzk/whats_the_deal_with_subtle_poisons/k2wlpvb/. I've been debating ever since whether I should follow his advice.
I thought it might interest you as both a healthcare issue in particular and an epistemic challenge in general.