"This is the peer! This is the review! That is how all of this works! This is it working!"
I'm not sure this is fair. Peer review comes before publishing, not after, for a good reason: most of the people who will ever read the AI 2027 post will have read it in its pre-review state.
There's also a bunch of subtle but important norms and practices that formal peer review enforces but a lesswrong back-and-forth does not. Consider things like preregistration of methodologies, establishment of data integrity procedures, conflict of interest statements, robust version control - all of those hard-won scientific table stakes (and more) are being lost by this choice of review strategy.
Agreed with this. It's concerning to me that Zvi doesn't seem at all aware of the obvious asymmetry in reach between titotal, random blogger, and the AI Futures team, which was literally in the New York Times about AI 2027. It's obviously not working!
The current peer review system is too slow for the pace at which AI is advancing. So in the absence of an accelerated review track, it still seems better to publish early, even if that means no formal peer review.
Then again, misinformation spreads more easily than corrections... so it’s hard to say what the best path really is.
From the perspective of those who believe existential risk is plausible and that things could go wrong fast, the case for early, detailed publication becomes stronger. They wanted to have an effect of the political landscape and I would gather they did. If short timelines are correct, then the cost of publishing too late is much greater than the cost of publishing too early, even if some parts of the model are wrong.
This is a fair point. However, my understanding is that AI 2027 received at least some amount of peer review – possibly a lot, I don't know – before it was released. Source: I was asked to provide feedback on an early draft, and I think it's very unlikely that I would have been the first person thus asked. (I don't recall whether the version shared with me included the detailed models or just the narrative; I primarily gave feedback on the presentation, not the model.)
On a related note, here's someone pointing out that public, post-publication peer review was kind of slow to catch on to the (apparently) fraudulent MIT paper about AI enhancing scientific productivity: https://moreisdifferent.blog/p/that-mit-paper-about-ai-enhancing. Though in fairness, my (amateur) understanding is that conventional peer review is also not great at detecting fraud.
You make a point early on that uncertainty can go two ways, maybe it's a lot slower, maybe it's a lot faster.
But can it actually? Like, AI by mid 2027 is pretty darn fast already! Can it get "a lot" faster than that? I mean, clearly on some level no. The absolute fastest it could've gotten is 2 years faster, and realistically maybe the very fastest it could be is one year faster? This isn't symmetric. AI could be a LOT slower than AI2027 suggests, but it can't really be a lot faster than it suggests.
I asked Daniel about this exact thing at lessonline, that predictions are left-bounded right-unbounded and so surely his protestations of "but it could just as well go even faster" didn't fully apply
he stood by them nonetheless in a way that impressed me
i'll just dump my entire memory of my interaction with him
when i asked during the Q&A it was sort of a two-parter. he'd said that the feedback he got after publishing the model had pushed his median further into the future. first i noted that it seems like it would have been a reasonable prediction to make *prior* to getting the feedback, that it would push back his estimate. i myself made that prediction, that 6 months after publication his model would have advanced by less than 6 months. and i'm sure many others thought so as well. since our prediction ended up coming true, i asked if he thought we were being reasonable. he hemmed a bit, so i asked him if he thought that, 6 months from now, his estimated time to SAI would have decreased by more or less than 6 months
i was pretty sure he would say "less", but instead he thought for a moment and said "fifty fifty". he really is running on pure inside view here. and i was a bit taken aback, and tbh, impressed. the fact that a layman would have predicted he would push his timeline back, and then that prediction was successful, did not sway him that such a prediction would be successful a second time... or else he's already baked that evidence into his model. probably the latter.
later, i was smoking outside the venue and he came out and was waiting for an uber, and I gently probed to see if he was open to talking, and he was. i brought up this point, that trying to predict a future event is left-bounded and right-unbounded, and doesn't this sorta automatically create an asymmetry? and doesn't this asymmetry matter more and more, the more uncertain about a prediction we are and the wider the error bars?
he was shaking his head before i finished talking, and while i didn't fully comprehend his response (he was in the middle of getting his bags all sorted), he sorta handwaved at the idea that he's not thinking about time linearly but logarithmically. will we get ASI in 10^6 seconds? 10^7 seconds? n*10^8 seconds?
this sorta makes sense to me, at least as a specific objection to the left-bound right-unbound thing. of course, it opened up a bunch more lines of questioning i wanted to ask him, but his ride showed up and i didn't get a chance
Just wanted to say that Opus4 completely blows Sonnet4 and Sonnet3.7 out of the water on “real life” coding tasks in my experience. Would be interesting to see these plots on this graph. Looks like we going exponential
Also, quibbling about exponential vs super exponential is dumb imo. Exponential is scary enough and doesn’t really change time horizons significantly based on how an exponent works…
1) What do you see as the effective time horizon for Opus 4 software tasks today?
2) I'm a bit uncomfortable with the whole "smooth curve of time horizon" view (albeit one sort-of _has_ to assume _something_ like this to do any extrapolation). For Agile development, what is supposed to happen (as I've read - I haven't actually been on a team run this way) is that there are daily meetings of 15 minutes to coordinate the daily work of developers. I'm viewing this as: Software development can be broken up into roughly 8 hour chunks that can be done independently, then there are kind-of team-lead-ish tasks that take 1/32 of the overall effort, but are at a higher level of abstraction. So I kind-of expect some kind of discontinuity at the 8 hour level, but don't know whether to expect progress to speed up or slow down at that point. What do you think?
The effective time horizon just depends so much on what you are trying to do. If you are trying to start up some new project, need endless boilerplate code and imports it can do something which might take you hour(s) especially if you forget something, would need to check the docs, etc.
I’ve had Opus4 do something in a single prompt that would probably take me a day (nice multiframe python data viz) since I’m not very familiar with the plotting library.
If you’re trying to make modifications to some algorithm or math equation then you might need to prompt it line by line, or change by change. Then it’s only saving you moderate time versus coding it yourself ofc
I’d say it’s made me a 3-5x better/faster developer, since I generally know what I want to do with a project, but pre-AI would spend hours digging thru documentation and stack exchange for stupid bugs and syntax for new libraries.
Re: 8 hour sprints, I just don’t think that’s a good way to look at software dev. So many sprints/all hands are “yep, still working on feature X, it’s a slog”
Many Thanks! Yeah, I see what you mean about the wide variety of software tasks preventing a clean "currently at X hours" way to assess Opus4 (or other software assistants).
>Re: 8 hour sprints, I just don’t think that’s a good way to look at software dev. So many sprints/all hands are “yep, still working on feature X, it’s a slog”
Man, that actually sounds like *exactly* what we want from AI (I mean, at current margins. The killing us all part I'm less enthusiastic about). Get rid of the tedious scutwork that invariably requires hours wading through poorly-documented project-specific idiosyncrasies (config files and imports all the way down) and a whole bunch of boilerplate code where at the end you're not a better person in any meaningful way, while retaining human input on the interesting whiteboard-centric parts.
It can't and surely won't last, but maybe this is a (very, very) brief golden age.
It’s definitely the beginnings of a golden age. Developers are way happier too as the satisfying parts of the job remain, and the tedious parts of the job are done.
I’m not confident in AI solving the entire stack of company formation and business plan execution any time soon. Things like coding and math you can RL the hell out of and get really good really quickly. How do you do reinforcement learning on project design and business strategy? You simulate a highly complex economy and highly variable consumer wants, and then RL based on how the AI-led project/business model did in this simulated economy? Seems… difficult
I think humans will be leading AIs for a long time, and capabilities will be super spiky (good at things you can RL and iterate/check quickly, don’t understand how this abstracts to things you can’t)
"The central criticism here seems to be that there isn’t enough uncertainty, that essentially all the parameters here should be uncertain. I think that’s correct. I think it’s also a correct general critique of most timeline predictions, that people are acting far more certain than they should be. Note that this goes both ways - it makes it more likely things could be a lot slower, but also they could be faster."
This isn't really true. Increasing uncertainty estimates in one parameter will often skew the results in one direction. The types of functions that AI 2027 uses will certaintly suffer from this: increasing uncertainty of an exponential parameter will *increase* the median timeline.
I think I was wrong with this general point. For an exponential y=e^(bt) with uncertainty in b, inverting time T it takes to hit some target Y gives T=ln(Y)/b.
The mean value of T is not ln(Y)/b. Qualitatively, higher variance in b increases probability mass at smaller denominators, so E[T] decreases when variance increases (if the mean is positive and finite).
"This seems like an argument that strictly exponential curves should have a very strong prior? So you need to argue hard if you want to claim more than that?"
Yes, exponential curves have more entropy, so they are a better choice of prior. In general, the fewer parameters the more entropy.
I haven't seen it discussed anywhere (though presumably it has been?), but is there any distinction made in the METR time horizon doublings between AIs being able to do "longer" tasks (in the sense that the task takes humans a long time to do) and the AI actually working for longer without going off the rails?
It always seems to be presented as the AIs being able to work for longer, but isn't the measure actually how long it would take a human to do the task?
It seems to me there is a difference between AI becoming able to do a 20-step proof (when it used to only be able to do a 10-step proof), and an AI becoming able to do a "20-step" proof in 10 steps just by being really smart.
Maybe the measure is intentionally smushing these things together, because the outcomes are enough the same, but I feel like they're not always the same, and failing to be able to operate over a long time horizon could still be an issue even if, on current tasks, increased intelligence is allowing them to solve harder (and so "longer") problems on short time horizons.
AI might have gone from being able to produce a painting that would have taken me a day, to producing a painting that would have taken me a month, but that's just it learning how to paint - not anything to do with actually working better over long time horizons.
Correction: You say "development of ‘superhuman coders’ (SC), that can do an AI researcher job (...)", but SC is just for coding tasks, from ai-2027.com: "an AI system that can do any coding tasks that the best AGI company engineer does (...)".
Podcast episode for this post:
https://open.substack.com/pub/dwatvpodcast/p/analyzing-a-critique-of-the-ai-2027?r=67y1h&utm_campaign=post&utm_medium=web&showWelcomeOnShare=true
Good critique.
"This is the peer! This is the review! That is how all of this works! This is it working!"
I'm not sure this is fair. Peer review comes before publishing, not after, for a good reason: most of the people who will ever read the AI 2027 post will have read it in its pre-review state.
There's also a bunch of subtle but important norms and practices that formal peer review enforces but a lesswrong back-and-forth does not. Consider things like preregistration of methodologies, establishment of data integrity procedures, conflict of interest statements, robust version control - all of those hard-won scientific table stakes (and more) are being lost by this choice of review strategy.
Agreed with this. It's concerning to me that Zvi doesn't seem at all aware of the obvious asymmetry in reach between titotal, random blogger, and the AI Futures team, which was literally in the New York Times about AI 2027. It's obviously not working!
The current peer review system is too slow for the pace at which AI is advancing. So in the absence of an accelerated review track, it still seems better to publish early, even if that means no formal peer review.
Then again, misinformation spreads more easily than corrections... so it’s hard to say what the best path really is.
From the perspective of those who believe existential risk is plausible and that things could go wrong fast, the case for early, detailed publication becomes stronger. They wanted to have an effect of the political landscape and I would gather they did. If short timelines are correct, then the cost of publishing too late is much greater than the cost of publishing too early, even if some parts of the model are wrong.
This is a fair point. However, my understanding is that AI 2027 received at least some amount of peer review – possibly a lot, I don't know – before it was released. Source: I was asked to provide feedback on an early draft, and I think it's very unlikely that I would have been the first person thus asked. (I don't recall whether the version shared with me included the detailed models or just the narrative; I primarily gave feedback on the presentation, not the model.)
On a related note, here's someone pointing out that public, post-publication peer review was kind of slow to catch on to the (apparently) fraudulent MIT paper about AI enhancing scientific productivity: https://moreisdifferent.blog/p/that-mit-paper-about-ai-enhancing. Though in fairness, my (amateur) understanding is that conventional peer review is also not great at detecting fraud.
You make a point early on that uncertainty can go two ways, maybe it's a lot slower, maybe it's a lot faster.
But can it actually? Like, AI by mid 2027 is pretty darn fast already! Can it get "a lot" faster than that? I mean, clearly on some level no. The absolute fastest it could've gotten is 2 years faster, and realistically maybe the very fastest it could be is one year faster? This isn't symmetric. AI could be a LOT slower than AI2027 suggests, but it can't really be a lot faster than it suggests.
I asked Daniel about this exact thing at lessonline, that predictions are left-bounded right-unbounded and so surely his protestations of "but it could just as well go even faster" didn't fully apply
he stood by them nonetheless in a way that impressed me
Uh, so, how did he answer that critique?
i'll just dump my entire memory of my interaction with him
when i asked during the Q&A it was sort of a two-parter. he'd said that the feedback he got after publishing the model had pushed his median further into the future. first i noted that it seems like it would have been a reasonable prediction to make *prior* to getting the feedback, that it would push back his estimate. i myself made that prediction, that 6 months after publication his model would have advanced by less than 6 months. and i'm sure many others thought so as well. since our prediction ended up coming true, i asked if he thought we were being reasonable. he hemmed a bit, so i asked him if he thought that, 6 months from now, his estimated time to SAI would have decreased by more or less than 6 months
i was pretty sure he would say "less", but instead he thought for a moment and said "fifty fifty". he really is running on pure inside view here. and i was a bit taken aback, and tbh, impressed. the fact that a layman would have predicted he would push his timeline back, and then that prediction was successful, did not sway him that such a prediction would be successful a second time... or else he's already baked that evidence into his model. probably the latter.
later, i was smoking outside the venue and he came out and was waiting for an uber, and I gently probed to see if he was open to talking, and he was. i brought up this point, that trying to predict a future event is left-bounded and right-unbounded, and doesn't this sorta automatically create an asymmetry? and doesn't this asymmetry matter more and more, the more uncertain about a prediction we are and the wider the error bars?
he was shaking his head before i finished talking, and while i didn't fully comprehend his response (he was in the middle of getting his bags all sorted), he sorta handwaved at the idea that he's not thinking about time linearly but logarithmically. will we get ASI in 10^6 seconds? 10^7 seconds? n*10^8 seconds?
this sorta makes sense to me, at least as a specific objection to the left-bound right-unbound thing. of course, it opened up a bunch more lines of questioning i wanted to ask him, but his ride showed up and i didn't get a chance
A small note: based on his methodology, 2027 was never Daniel's *median* -- it was his *mode.* (His median was later, even before shifting his mode to 2028.) https://intelligence.org/2025/04/09/thoughts-on-ai-2027/
“plans are worthless, planning is essential”
Not directly related to this post, but makes me think that now with AI, “code is worthless, coding is essential”
Just wanted to say that Opus4 completely blows Sonnet4 and Sonnet3.7 out of the water on “real life” coding tasks in my experience. Would be interesting to see these plots on this graph. Looks like we going exponential
Also, quibbling about exponential vs super exponential is dumb imo. Exponential is scary enough and doesn’t really change time horizons significantly based on how an exponent works…
Dumb questions:
1) What do you see as the effective time horizon for Opus 4 software tasks today?
2) I'm a bit uncomfortable with the whole "smooth curve of time horizon" view (albeit one sort-of _has_ to assume _something_ like this to do any extrapolation). For Agile development, what is supposed to happen (as I've read - I haven't actually been on a team run this way) is that there are daily meetings of 15 minutes to coordinate the daily work of developers. I'm viewing this as: Software development can be broken up into roughly 8 hour chunks that can be done independently, then there are kind-of team-lead-ish tasks that take 1/32 of the overall effort, but are at a higher level of abstraction. So I kind-of expect some kind of discontinuity at the 8 hour level, but don't know whether to expect progress to speed up or slow down at that point. What do you think?
The effective time horizon just depends so much on what you are trying to do. If you are trying to start up some new project, need endless boilerplate code and imports it can do something which might take you hour(s) especially if you forget something, would need to check the docs, etc.
I’ve had Opus4 do something in a single prompt that would probably take me a day (nice multiframe python data viz) since I’m not very familiar with the plotting library.
If you’re trying to make modifications to some algorithm or math equation then you might need to prompt it line by line, or change by change. Then it’s only saving you moderate time versus coding it yourself ofc
I’d say it’s made me a 3-5x better/faster developer, since I generally know what I want to do with a project, but pre-AI would spend hours digging thru documentation and stack exchange for stupid bugs and syntax for new libraries.
Re: 8 hour sprints, I just don’t think that’s a good way to look at software dev. So many sprints/all hands are “yep, still working on feature X, it’s a slog”
Many Thanks! Yeah, I see what you mean about the wide variety of software tasks preventing a clean "currently at X hours" way to assess Opus4 (or other software assistants).
>Re: 8 hour sprints, I just don’t think that’s a good way to look at software dev. So many sprints/all hands are “yep, still working on feature X, it’s a slog”
Ouch! Excellent point!
Man, that actually sounds like *exactly* what we want from AI (I mean, at current margins. The killing us all part I'm less enthusiastic about). Get rid of the tedious scutwork that invariably requires hours wading through poorly-documented project-specific idiosyncrasies (config files and imports all the way down) and a whole bunch of boilerplate code where at the end you're not a better person in any meaningful way, while retaining human input on the interesting whiteboard-centric parts.
It can't and surely won't last, but maybe this is a (very, very) brief golden age.
It’s definitely the beginnings of a golden age. Developers are way happier too as the satisfying parts of the job remain, and the tedious parts of the job are done.
I’m not confident in AI solving the entire stack of company formation and business plan execution any time soon. Things like coding and math you can RL the hell out of and get really good really quickly. How do you do reinforcement learning on project design and business strategy? You simulate a highly complex economy and highly variable consumer wants, and then RL based on how the AI-led project/business model did in this simulated economy? Seems… difficult
I think humans will be leading AIs for a long time, and capabilities will be super spiky (good at things you can RL and iterate/check quickly, don’t understand how this abstracts to things you can’t)
Easy peasy, once you're powerful enough you optimize away the messy (human) parts of the stack, and then the stack itself.
"The central criticism here seems to be that there isn’t enough uncertainty, that essentially all the parameters here should be uncertain. I think that’s correct. I think it’s also a correct general critique of most timeline predictions, that people are acting far more certain than they should be. Note that this goes both ways - it makes it more likely things could be a lot slower, but also they could be faster."
This isn't really true. Increasing uncertainty estimates in one parameter will often skew the results in one direction. The types of functions that AI 2027 uses will certaintly suffer from this: increasing uncertainty of an exponential parameter will *increase* the median timeline.
I think I was wrong with this general point. For an exponential y=e^(bt) with uncertainty in b, inverting time T it takes to hit some target Y gives T=ln(Y)/b.
The mean value of T is not ln(Y)/b. Qualitatively, higher variance in b increases probability mass at smaller denominators, so E[T] decreases when variance increases (if the mean is positive and finite).
"This seems like an argument that strictly exponential curves should have a very strong prior? So you need to argue hard if you want to claim more than that?"
Yes, exponential curves have more entropy, so they are a better choice of prior. In general, the fewer parameters the more entropy.
I haven't seen it discussed anywhere (though presumably it has been?), but is there any distinction made in the METR time horizon doublings between AIs being able to do "longer" tasks (in the sense that the task takes humans a long time to do) and the AI actually working for longer without going off the rails?
It always seems to be presented as the AIs being able to work for longer, but isn't the measure actually how long it would take a human to do the task?
It seems to me there is a difference between AI becoming able to do a 20-step proof (when it used to only be able to do a 10-step proof), and an AI becoming able to do a "20-step" proof in 10 steps just by being really smart.
Maybe the measure is intentionally smushing these things together, because the outcomes are enough the same, but I feel like they're not always the same, and failing to be able to operate over a long time horizon could still be an issue even if, on current tasks, increased intelligence is allowing them to solve harder (and so "longer") problems on short time horizons.
AI might have gone from being able to produce a painting that would have taken me a day, to producing a painting that would have taken me a month, but that's just it learning how to paint - not anything to do with actually working better over long time horizons.
Correction: You say "development of ‘superhuman coders’ (SC), that can do an AI researcher job (...)", but SC is just for coding tasks, from ai-2027.com: "an AI system that can do any coding tasks that the best AGI company engineer does (...)".
Humans : *argue with each other endlessly about timelines*
ASI: "Wow, Operation Hominid Stationary is going to be easier to execute than I expected. LOL."