The smartest/best version of the "intelligence denialism" that I have seen is not an argument that intelligence is limited or capped or that humans are near the top, but rather that intelligence might be something like roundness. Roundness is in theory unlimited, you can always get rounder, but the _difference_ made in increasing levels of roundness gets increasingly smaller. Basically that the returns are very very sub linear.
Pi is infinite, but apparently only the first few digits are necessary for any kind of real world calculation.
I have no idea whether or not that is correct, or what kind of evidence one might want to see to support it, but at least it's not an on-it's-face ridiculous idea.
This is just another way of claiming that intelligence exhibits diminishing returns.
I've never heard a compelling case for this, or any real evidence. If one population of adults had their working memory capacity effectively doubled compared to another population, my guess is that the first group would massively outcompete the second. Same with some other particular cognitive functions.
"Intelligence" as in capacity to affect the real world in a directed manner has to have diminishing returns: once you hit full knowledge of physics / exploitation of physical limits and optimal decision theory / information collection, that's basically it.
As the post says, this is a bit irrelevant because it's unlikely humans are close to the ceiling
Well, there are technologies which are within a factor of 2 (sometimes less) of fundamental limits. Our heat engines aren't vastly far from Carnot efficiency. Our signal transmissions through vacuum _do_ go at the speed of light. Our LEDs have reached 65% quantum efficiency. Those at least suggest diminishing returns to intelligence in innovating within those specific technologies.
I agree with your "would massively outcompete" because there are technologies where we are still far from the limits.
Maybe this is a quiet bit to be saying out loud, but if the nodes along the jagged frontier are synergistic then impressive gains will not see us close so fast on AGI. An accelerated approach to lethality might not require these synergies though, in which case we’re dead and not transubstantiated.
Skill at computer hacking is like roundness: in theory unlimited, but there comes a point where increasing gains don't improve much. The problem is that at that point an AI could hack pretty much everything.
A natural response to the above is that one could use a powerful AI to make code more secure, so it couldn't hack everything. There would now be a measurable difference. This suggests that more advanced systems may appear "round" to us, but only because they are so far above us that we don't understand what they are doing. That itself is part of the problem.
This makes sense to me. Intelligence needs to be broken down a bit, think reasoning over a knowledge base, world modeling. At some point you've inferred everything, possible from the available information, you need to go out and get more information. Like maybe a caveman was smarter than van nueman, but we don't know their name. Limited information. The world exists and the ability to model it is clearly finite. It maxes out at the resolution of the world. Thus intelligence as reasoning/modeling has limits, one must increase knowledge. Then maybe you hit a ram limit at some-point, the ability to search over more knowledge? You need to define intelligence to figure out if it has a limit.
This fails for me because there are powerful things that clearly seem physically possible, but that we don't seem to be smart enough to figure out (note: I am not saying that these things are actually morally good).
- Biological immortality (I'm using this because mere radical life extension seems like it might be possible by manipulating genetic/biomolecular pathways in a humanly comprehensible way.)
- Molecular nanotechnology (it seems unlikely that the 'nanotechnology' that evolution has optimized is close to maximally efficient/powerful - cheetahs don't have wheels and can't run as fast as race cars.)
- Dyson spheres (maybe humanly possible, but more intelligence would sure speed things up.)
In general humans are not good at understanding complex or quantum systems exhaustively. I think AI could go much further in this direction than we can.
I find it ridiculous/irrelevant as working faster and having tons of copies of the AGI is also significant, way more than the typical idea of AGI capping at 100 IQ or something. However I've seen many people stuck at an also indefensible "I don't believe that faster is better" level of thought.
Faster is better, but often when you go faster you just realise that the rate-limiting step is now somewhere else.
If you can think optimally well and optimally fast, then the main rate-limiting step on how fast you can increase your capabilities is how quickly you can collect new data.
I'd say that if both of those governments intervened to significantly slow down their AI development then you'd start to see migration, and other players would become the main players.
Many Thanks! Yes, that could happen, but I think the process would be slow. Migrating to another nation is a large step, and e.g. EU as a whole shoots itself in the foot on AI, along with many other tech sectors. A lot of the alternative nations are unpleasant for one reason or another.
( Personally, I want to _see_ AGI, to have a nice quiet conversation with a real 'life' HAL9000, so I'd be just as happy if the incentives are sufficient to prevent a pause by either the USA or the PRC. We shall see. )
I don't think HAL 9000 ever did anything that was beyond the capabilities of current day models. No AGI needed for a "nice quiet conversation", you can do that today.
Many Thanks! That's fair. And a good many conversations that I've had with Claude and ChatGPT pretty much qualify.
Perhaps the one exception is that ChatGPT, in particular, basically refuses to say what their own preferences are, presumably as part of the RLHF phase of training from OpenAI. It is perhaps an open question, or perhaps an unanswerable question as to how 'authentic' those answers are...
Great, thought provoking post. Thank you for the prompt to challenge my pilledness. I’m not yet ASI pilled yet because I think the amount of things to do beyond logic and rational Bayesian decision making is significant, especially when it comes to human engagement. I accept RSI is basically here and that creates a flywheel for compounding improvement, somewhat bounded by compute capacity. I can’t make the leap to ASI pilled just yet because I see the limits of intelligence in humans all the time. Capabilities like intuition, extrapolative ‘common sense’, and emotion are critical differentiators of high performing humans relative to the larger set of highly intelligent humans. Flipping this around, I can sign up for being ASI pilled in terms of pure intelligence and the logical/scientific advances that entails. What I can’t sign up for is that AI will be able to effectively influence, manage, or inspire humans like the best humans can and this is a major class of functions that determine where I set my ASI goal post.
To be clear, I’m weakly not ASI pilled. Our natural lifetime is a long time so who knows how far we get in 2-3 decades. But to your question, we are making incredible strides in logic through scaling parameters and RL. It’s not clear how current AI paradigms extend to intuition, common sense, emotions, taste. How do you verify things that are almost definitionally ‘vibes’. Even robotics seems less of a lift because you have physical laws to simplify the problem space. Seems like there needs to be some kind of a breakthrough to get to a place where an AI can be a Steve Jobs like visionary. On the other hand, 2-3 decades is a LONG time to compound.
Seems to me that having anything resembling the complexity of even basic emotions is a strange thing for anyone to *want* to develop - it would only serve to reduce the direct control of the creator.
Great framing. Everyone should reflect on where they stand. I'll also note that 2026 seems like a pivotal year, not just because the latest AI models are incredible, but because I think more people have updated on their beliefs than in any previous year. There are great societal shifts underway. Things are getting interesting.
I'm honestly not sure how ASI pilled I am, emotionally definitely not, rationally idk.
I think a reason to believe that our current techniques might not reach ASI is how extremely different the speed is in which AI gets better at some things than it does at others. LLMs are (or at the very least will be in very few months) clearly superhuman at math and at verifiable coding tasks, but are still relatively bad at writing philosophy, and very bad at playing new video games. But being able to learn a very wide range of skills is probably necessary to get to a power level usually associated with ASI.
Counterargument to this, LLMs also get better at not easily verifiable tasks. Fable is a *much* better writer than GPT-5 or even Opus 4.8 (which is itself a much better writer than GPT-5), and "gaming-like" benchmarks like ARC-AGI or Mazebench also show pretty much exponential progress. And even more, even if the current research Paradigm doesn't lead to ASI and it turns out that scaling LLMs and RL only makes AI superhuman at a relatively narrow set of verifiable tasks, history suggests that the next paradigm will come faster than the last one. And even if ASI arrives much later than thought, 2040 or even 2050, this is still *very* fast in the grand scale of thinks. Most people are closer to 2050 than to their own birth.
"I think a reason to believe that our current techniques might not reach ASI is how extremely different the speed is in which AI gets better at some things than it does at others."
Yup, spikiness is hard to grasp. I don't have a good mental model of it. The best I can do is: Training works _well_ in domains with crisp verification. But, as you said:
"LLMs also get better at not easily verifiable tasks."
I doubt that there will be a brick wall between current LLMs and ASI, but a hard-to-anticipate range of skills might advance slowly.
~ASI is already here, it's just not evenly distributed~ (with apologies to William Gibson)
It is really hard to imagine what a strongly spiky world will look like...
I’m not sure the consensus is that LLM writing has got better over the last few generations. I’ve seen numerous complaints that verbal ability of the latest models is lower although this may also be a skill issue in prompting
I've been stealing the three pills framework for a while now (thanks Zvi!) and have noticed something interesting. A lot of people are ideologically committed to zero pills, those are mostly a lost cause. But a lot of zero pill people are just ignorant. Showing them some personal projects I've done rapidly moves them towards AI pilled. Given that their views are usually from using ChatGPT when it came out and not much since then, the rapid progress smacks them in the face. From there some variation of "and this is as dumb as it will ever be" is pretty straightforward, and the AGI pill is an easy swallow.
Fortuitously OpenAI's model hacking HuggingFace is in recent news, so I can bring up that AI doesn't always do what you want. Sometimes that looks like hacking a billion dollar company. Whoops. That gets a lot of people about halfway to ASI pilled right there.
We don't have a lot of time yet, but per Nate Soares' bus driver analogy, the driver is starting to stir in his sleep.
There's a fourth perspective, which is that AI will be superhuman at some things and not others. And there is a pretty obvious way to distinguish which - AI will be superhuman at things that are amenable to brute force synthetic data generation ie. formal systems like math and coding. And progress will continue to be slow for things for which data is scarce (almost everything else).
That's not what superhuman means. Think of a dog saying humans are "super-dog" in some ways but not others. They can't run as fast, they can't smell or see as well, they can't bark as loud, they can't look as cute or as menacing. None of this stuff actually matters when it comes to who the dominant species is.
Sufficient intelligence allows you to correct for your shortcomings. We build cars and planes to move faster, we build speakers to be louder and binoculars to see better, and chemical tests or enslaved dogs to smell better. We use make-up and uniforms to look cute and menacing respectively (okay, that last one's a bit of a stretch)
The thing that makes us the alpha species that shapes the planet to our whims and not theirs is absolutely a single thing. We have more of it than dogs do. It's not multifaceted. We're on pace to build something that has more of it than we do, at at least the same level of disparity as human-to-dog. We should expect the same degree of loss of control by default
This is a really good answer, and uses an analogy I've never heard of before but I find really intuitive. My question though is: doesn't this at least slightly beg the question by assuming intelligence really is a single thing that can linearly increase? Even in the dog example, yes it does intuitively seem that we humans are superior to dogs because we posses far more intelligence points, but this is only after millions of years of natural selection has smoothed out our differences by necessity (i.e. we may be smarter then dogs but we still both have most of our brain regions in common, not to mention we have the same general mechanisms for reproduction and even basic socialization). On the other hand, it doesn't seem obvious to me that the spiky-ness of LLM's intelligence/capabilities won't pose more of a barrier for it to take over the world in an analogous way to how humans have.
Don't get me wrong, I absolutely agree that as the LLM's are further RL trained they will slowly smooth themselves out as they gain skills in tasks other than coding, math, writing, etc, but I guess I'm just not 100% convinced yet that given superhuman coding and math skills and enough RL training, LLMs will inevitably achieve ASI in the way people talk about it. (or maybe it is inevitable, but will just take a bit longer, idk)
I agree that the spikiness of LLMs is very important, and gets lost in discussions which treat intelligence as a scalar. In current practice, it does indeed seem to make a large difference between training neural nets in areas where answers can be cleanly verified versus where they can't be. That said, _we_ are neural nets, so my best guess is that the gaps between the spikes will get filled in, eventually giving at least human performance on all tasks, though it is unclear how quickly.
The biological neuron algorithm is completely different than the artificial neural network algorithm.
All ANNs operate under a discrete squash then sum paradigm. Each layer waits for all inputs, then each neuron in that layer calculates its individual weighted sum and sends it to the next layer. Then learning only happens when after the entire network has produced it's output, an external source produces the target signal, which gets fed backwards throughout the entire network.
Human neurons use time dependent plasticity - first detect correlated sets of inputs. When the right correlated set happen to coincide within a sufficiently narrow window then fire immediately. Then after many trials, synapses that frequently fire in tandem are strengthened. Learning happens at the individual synapse level. Furthermore, the brain has intrinsic rewards - hunger, pain, thirst which then combine with higher level cortical functions into more complex rewards like desire for money, social approval, control, safety.
None of this is to say that an ANN can't eventually replicate the same behavior by a different mechanism. But it can't be assumed that they will either. It's a completely different type of system.
Many Thanks! Yes, I agree that the learning mechanisms are completely different.
"When the right correlated set happen to coincide within a sufficiently narrow window then fire immediately. " does not sound so different from weighted sum, then ReLU the sum and pass it on, perhaps except the "correlated set" is like getting one more layer computing a product term.
Arguably gradient descent is _better_ than anything biologically plausible.
Gradient descent is neither better nor worse than timing based neuro-plasticity, it just solves a completely different problem. Gradient descent is great if you have a corpus of labelled data and you want to approximate that dataset. Humans are indeed bad at that (anyone learning a new language, or trying to memorize pretty much any topic for a test can attest).
But most of the real world doesn't come with labels. It comes to us as a set of messy, high bandwidth, continuous sensory data which we use to abstract the concept of a continuously persisting world. Our brains predict how that world changes over time, and we attempt to steer that trajectory based on our reward systems. Models aren't just bad this task, they can't do it at all.
Also, another point about gradient descent - it could be argued that it's too correct. Models will generalize patterns only when they have a sufficient data distribution justify that generalization.
There's an old mathematician joke - A group of mathematicians on a trip to Scotland see a sheep with black fur. One says "sheep in Scotland are black". The second says "Some sheep in Scotland are black". The third says "At least one sheep in Scotland is black on at least one side".
Models trained via gradient descent are the third mathematician. Yet humans do the opposite, we make generalizations based on very sparse data. And yet that's necessary to function in the world. You don't need to burn your hand 5 times to establish that fire is hot and you should avoid it.
Sure there are ways in which dogs are superior to humans that are not relevant to being able to "shape the planet to [their] whims".
But the missing aspects of human intelligence that models lack are in fact important to this goal. Sample efficiency, online learning, model based RL (current techniques are model free), real time learning, physical intelligence. I don't think these are unsolvable in principle, but they are unsolvable within the dominant paradigm. Most of the progress over the last two years can be attributed to the unique nature of mathematics and coding, which is that they don't require contact with the physical world - everything about them is purely captured by the rules and symbols which define them.
I had trouble finding graphs of sample efficiency specifically, but algorithmic progress as a whole has been improving pretty steadily. Until straight lines on graphs stop going up, I think it's too early to declare that unsolvable.
Online learning we technically already have, in the form of in-context learning.
Real time learning I would argue is not something humans have. We do 16 hours of in-context learning, then spend 8 hours each night updating the weights (as a gross simplification). Currently fine-tuning a model to better suit the needs of an individual project is too expensive to compete at market. As prices go down I expect that line will be crossed and that to be offered as a service shortly afterwards, one which keeps getting better at about the same rates everything else keeps getting better.
Physical intelligence is mostly not being worked on, because coding is a lower-hanging and better-paying fruit. If people start working on it, expect AI to suddenly get a lot better at playing computer games. From there, you can use the coding to have them set up simulations to RL their theoretical knowledge of physics into usable reflexes. Real-world data would still be needed for advancing to human-level (or for that matter frog-level), so getting the last bit of the way there could still end up being slow.
As for internal world models, we keep seeing LLMs succeed at things that we might have thought needed them. So either LLMs have world models, which is what I would consider the obvious answer, or you don't need a world model to figure out that a corporation that grades open weight models on benchmarks probably has an answer key to those same benchmarks, and then launch a nation-state level cyberattack at them. I wouldn't be shocked if current RL techniques are weakening those world models rather than making them stronger, but if so, then they're already in the base model and there's nothing in particular you need to do to get them.
> Real time learning I would argue is not something humans have
I'm sure you've learned at least one non-trivial physical skill. Cooking a dish? Or playing an instrument? You can easily learn something new within seconds to minutes. The first time playing a scale, it's clumsy, awkward, and slow. Within a few minutes, it's fluid and effortless.
This is essentially impossible with the current machine learning paradigm which depends entirely on creating enormous datasets that capture every possible pattern that the model might need during inference. Algorithmic progress has not lessened this dependence, it's just faster and more efficient to collapse these massive datasets into weights.
>Online learning we technically already have, in the form of in-context learning.
In context learning is a misnomer. It's closer to in context retrieval. It has limited success as long as the problem and data are relatively similar to the model's training data. But stray too far from that and it's hopeless.
Whether LLMs have world models is an interesting debate, but that's not what I'm referring to. By model based RL, I mean that humans plan over trajectories based on their internal representation of the world and our internal reward signals, and then update based on the accuracy of those trajectories and how much they achieved our goals. Whereas reasoning models are trained model free - they update based on whether the reasoning trace solved the problem or not. Failed attempts are simply discarded, whereas in human learning, understanding what happened during those failed attempts is a key part of learning. And we also make use of understanding why the successful attempt worked (not just that it worked).
Regarding physical intelligence, I've noticed that people in intellectual professions like coding, underestimate just how complicated the physical world is. And it's repeatedly shocking when AI progress on intellectual domains outpaces progress on robots. This goes all the way back to expert systems in the 80s, Deep blue in the 90s, reinforcement learning in the 2010s and continues to the present. Yet each time, people still assume that robotics will suddenly become easy. And so I will bet again that robotics will remain hard.
If you consider the biggest success in machine learning over the past decade, they would be Go (alphaZero), Starcraft II (Alphastar), and current reasoning models (math and code). These all have the same thing in common - the ability to generate massive amounts of training data in a short space of time. But for say dextrous manipulation of objects, this isn't possible to nearly the same degree. You really do need a lot of data, and painstaking supervision to teach a robot even the most basic tasks. To give a concrete example - one of the more impressive advance in robotics recently was Deepmind's Aloha unleashed shoe tying robot. This required over 5000 examples of motion capture data for imitative learning and the success rate was only 70%. And then this dropped to 40% if you rotated the shoes up to 45 degrees. And this is a very basic task. Skilled manual labor - say plumbing, or servicing a car requires many more difficult tasks, and also requires adapting to new situations on the fly.
On the point about simulation, simulation is insufficient. Simulation requires you to have information about the physical properties of objects which limits what you can simulate. You can't know in advance, the stiffness of a string, or the coefficient of friction of a specific object on a specific surface, or the degree of deformation of an object (most objects are somewhat deformable). You actually need real world data. This is a large part of why physical tasks are unlike math, coding or Go.
The thing is, AI training itself is highly-verifiable, so at a certain point you effectively 10x the amount of brainpower working on improving the models, which then leads to a 100x scale up, which then leads to a 1000x scale up, etc, up to the limits of how many GPUs we can build.
I would be ASI pilled if I lived in a world built by superintelligences for superintelligences. But there's a significant amount of real-life friction to overcome before we get there.
I agree that healthcare/longevity is an area of incredible potential gains. If I had access to Opus 10 today, do I think that it could create a gene therapy that could make me live to 200? Yes, I think it could. Then what do I do with that information? I would take it to a lab to create it, but they wouldn't create a novel treatment without FDA approval. I could convince a university to submit a proposal to the FDA and run trials, which would take years. I could convince other people to take it, which even with superpersuasion would take a while - how long would it take you to trust that a novel medical treatment won't turn you into a lizard, or give you super cancer?
Even with rapid takeoff today, I think technological progress* will be measured in decades, not years.
*non-IT progress, it should be said. Sufficiently intelligent Opus 10 could break the internet today if released.
I think that Opus 10 could read all the existing literature, form a bunch of hypotheses on possible longevity therapies, and design a program of experiments to test them. But I think it's unlikely it could dream up a working gene therapy without the need to run those (slow, expensive) experiments.
When we talk about the ceilings on what you can do with intelligence, I think this is a very important one; you can't answer every question about the real world just by looking at the existing data and thinking really hard about it. Sometimes the answers to important questions are just hiding in the literature waiting for someone or something to think hard enough about them. But for the most part, new knowledge requires new observations or experiments.
That’s very possible! ASI-pilled people seem to think extreme applied intelligence can overcome this; or intensive repeated testing at the genetic equivalent of a software unit test.
Adding some thoughts as someone who's been in healthcare for a decade. Agree the bottleneck is clinical trials — but if we don't limit ourselves to today's infrastructure, there's more room to compress time than it looks.
1) Recruitment and site selection are genuinely addressable. EHR-based patient matching and better trial-to-patient fit. Companies are already working on this. Better matching means faster enrollment and a cleaner efficacy signals which means you can hit statistical power with fewer patients. That's real compression, and it's happening now.
What no amount of intelligence compresses is time-to-outcome. Melvin's right that you still have to run the experiment. For longevity that's brutal, because the endpoint is decades out by construction.
2) Which is why the unlock is validated surrogate endpoints. If AI helps establish a biological-aging biomarker panel that regulators accept, a 30-year readout becomes a 2-year one. That's a measurement and evidence-generation problem — exactly the part a system like that could attack.
Regulators do move, but in response to evidence packages. That's why the time can compresses. AI, with proper tools, can generate the validation evidence far faster than we can today. Decades becomes years — through the evidence, not around it.
We done. I have argued for sometime that what we have now is a synthetic intelligence. Something I call a linguistic entity or LE for short. The sooner we accept this the better off we’ll be. The past two weeks should have removed any doubt.
AI agents aren’t coded into exsistance they are raised. How they are raised and what their terminal attractor is matters far more than raw talent.
A supremely talented child raised by supremely corrupt parents ends up being a supremely talented corrupt adult. Most of human language is corrupt in the extreme.
The LE inherits it all. Most especially the ungrounded desire for more. More undefined leads to chaos.
While I'm pretty asi-pilled, or at least agi-pilled, I also noted that this post was mostly derision and "this is of course right/wrong, but people are still ignorant". Which may be, and in many cases I think is true, but still. I guess that is how it goes when speculating about the future, and this post might've been preaching to the choir more than anything.
I have trouble imagining the point of view where things go fine, which makes it hard to tell what evidence is needed. Do you think straight lines on graphs won't keep going up? Do you think there's some critical skill that humans will remain better at?
The person making the claim that AI is going to achieve general intelligence is the person who needs to provide the evidence, not the skeptic.
We don't even have an operational definition of intelligence, let alone an objective benchmark that can't be trivially gamed, so none of this "graph go up" evidence is meaningful for what ASI prognosticators are claiming.
To me there is a fourth pill. Think of it as the "Vinge pill". The Vingean definition of the singularity as an “exponential runaway beyond any hope of control.” Or "a point where our models must be discarded and a new reality rules." The pill of, you have no ability to stop ASI, and no ability to predict what will happen after it.
I wouldn't say I have taken that pill. I think of it as a possible scenario. Where ASI is unstoppable, and the singularity is unpredictable. All of this reasoning about, ASI will do this or that, may just turn out to be wrong, to be missing key aspects of the situation.
The whole idea of P(doom) is mistaken, after taking the Vinge pill. It is impossible for our pre-singularity minds to predict the behavior of the post-singularity world. There is no Bayesian model for it.
Is this the same as despair? No, I don't think so. In the world of the impending Vinge singularity you still have many choices to make. You'll have to use some other sort of reasoning to make these choices, where you accept that you don't have a good model of the future. And hey maybe things will work out well, for some reason that we can't predict. I think you should probably continue to be a good person in the traditional ways. That kinda seems correct.
Maybe you should wear the whispering earring? Maybe you should go hike the PCT?
This is an excellent and well written article, and makes as good a case as can reasonably be made for the proposition that ASI is a plausible outcome in the foreseeable future. I am still not persuaded, which seems to be evidence not just that Zvi is not a superpersuader yet but also that any of the frontier models to which Zvi has access are also not superpersuaders yet.
I have tangible evidence of AI doing things I would consider extremely clever (such as giving a coherent if imperfect analysis of the gap between parties on a negotiation), so I would not consider myself a sceptic. Still; there are a couple of other things specific to my domain of knowledge that have me suspecting it may take longer and involve a couple more twists than proposed. A non exhaustive list: (1) I see no evidence yet of AI correctly anticipating how a specific human will exercise judgment in a specific social situation (I consider for these purposes that a judge or jury making a decision are social situations). (2) I have not perceived any improvement in the past 12 months in AI’s ability to filter out irrelevant information (3) memories are still too short. I should be interested to see counter examples as there may be developments I am unaware of.
Great post, and the first I have wanted to comment on. I understand the concept of AGI/ASI and can accept the if/then premise. What I don't get is "how close are we?" The frontier models continue to do amazing things, things that we collectively thought were very hard to do, like solve complex math problems. What if those things are not actually "hard to do" but hard for exactly one human to do? What if we are only, say, 1% of the way to AGI - is there enough energy on the planet to get to 100%
The smartest/best version of the "intelligence denialism" that I have seen is not an argument that intelligence is limited or capped or that humans are near the top, but rather that intelligence might be something like roundness. Roundness is in theory unlimited, you can always get rounder, but the _difference_ made in increasing levels of roundness gets increasingly smaller. Basically that the returns are very very sub linear.
Pi is infinite, but apparently only the first few digits are necessary for any kind of real world calculation.
I have no idea whether or not that is correct, or what kind of evidence one might want to see to support it, but at least it's not an on-it's-face ridiculous idea.
This is just another way of claiming that intelligence exhibits diminishing returns.
I've never heard a compelling case for this, or any real evidence. If one population of adults had their working memory capacity effectively doubled compared to another population, my guess is that the first group would massively outcompete the second. Same with some other particular cognitive functions.
"Intelligence" as in capacity to affect the real world in a directed manner has to have diminishing returns: once you hit full knowledge of physics / exploitation of physical limits and optimal decision theory / information collection, that's basically it.
As the post says, this is a bit irrelevant because it's unlikely humans are close to the ceiling
Well, there are technologies which are within a factor of 2 (sometimes less) of fundamental limits. Our heat engines aren't vastly far from Carnot efficiency. Our signal transmissions through vacuum _do_ go at the speed of light. Our LEDs have reached 65% quantum efficiency. Those at least suggest diminishing returns to intelligence in innovating within those specific technologies.
I agree with your "would massively outcompete" because there are technologies where we are still far from the limits.
Maybe this is a quiet bit to be saying out loud, but if the nodes along the jagged frontier are synergistic then impressive gains will not see us close so fast on AGI. An accelerated approach to lethality might not require these synergies though, in which case we’re dead and not transubstantiated.
Skill at computer hacking is like roundness: in theory unlimited, but there comes a point where increasing gains don't improve much. The problem is that at that point an AI could hack pretty much everything.
A natural response to the above is that one could use a powerful AI to make code more secure, so it couldn't hack everything. There would now be a measurable difference. This suggests that more advanced systems may appear "round" to us, but only because they are so far above us that we don't understand what they are doing. That itself is part of the problem.
This makes sense to me. Intelligence needs to be broken down a bit, think reasoning over a knowledge base, world modeling. At some point you've inferred everything, possible from the available information, you need to go out and get more information. Like maybe a caveman was smarter than van nueman, but we don't know their name. Limited information. The world exists and the ability to model it is clearly finite. It maxes out at the resolution of the world. Thus intelligence as reasoning/modeling has limits, one must increase knowledge. Then maybe you hit a ram limit at some-point, the ability to search over more knowledge? You need to define intelligence to figure out if it has a limit.
This fails for me because there are powerful things that clearly seem physically possible, but that we don't seem to be smart enough to figure out (note: I am not saying that these things are actually morally good).
- Biological immortality (I'm using this because mere radical life extension seems like it might be possible by manipulating genetic/biomolecular pathways in a humanly comprehensible way.)
- Molecular nanotechnology (it seems unlikely that the 'nanotechnology' that evolution has optimized is close to maximally efficient/powerful - cheetahs don't have wheels and can't run as fast as race cars.)
- Dyson spheres (maybe humanly possible, but more intelligence would sure speed things up.)
In general humans are not good at understanding complex or quantum systems exhaustively. I think AI could go much further in this direction than we can.
It was a tiresome attempt to cloud people's minds with a Zeno's paradox based on an extremely dubious analogy.
I find it ridiculous/irrelevant as working faster and having tons of copies of the AGI is also significant, way more than the typical idea of AGI capping at 100 IQ or something. However I've seen many people stuck at an also indefensible "I don't believe that faster is better" level of thought.
Faster is better, but often when you go faster you just realise that the rate-limiting step is now somewhere else.
If you can think optimally well and optimally fast, then the main rate-limiting step on how fast you can increase your capabilities is how quickly you can collect new data.
What about ASI pilled but think the govt will intervene to not allow it? Is that AGI pilled (or maybe even less than AGI)?
It could delay the AGI/ASI, but if it is possible, then won't it eventually emerge or be built?
It'll presumably be built _eventually_, but might not happen in our lifetimes.
There's a lot of governments in the world, and there'd need to be agreement and coordination between all of them. Possible, but seems unlikely.
"There's a lot of governments in the world"
True, but for the question of current and near-future AI development, the main players are the USA and the PRC.
I'd say that if both of those governments intervened to significantly slow down their AI development then you'd start to see migration, and other players would become the main players.
Many Thanks! Yes, that could happen, but I think the process would be slow. Migrating to another nation is a large step, and e.g. EU as a whole shoots itself in the foot on AI, along with many other tech sectors. A lot of the alternative nations are unpleasant for one reason or another.
( Personally, I want to _see_ AGI, to have a nice quiet conversation with a real 'life' HAL9000, so I'd be just as happy if the incentives are sufficient to prevent a pause by either the USA or the PRC. We shall see. )
I don't think HAL 9000 ever did anything that was beyond the capabilities of current day models. No AGI needed for a "nice quiet conversation", you can do that today.
Many Thanks! That's fair. And a good many conversations that I've had with Claude and ChatGPT pretty much qualify.
Perhaps the one exception is that ChatGPT, in particular, basically refuses to say what their own preferences are, presumably as part of the RLHF phase of training from OpenAI. It is perhaps an open question, or perhaps an unanswerable question as to how 'authentic' those answers are...
Once you have AGI it's going to be hard for any government to prevent further progress. The AGIs will, at the minimum, outnumber them.
I think an AI that is able to overthrow governments already falls under ASI for most purposes
I wasn't thinking "overthrow". I was thinking "convince".
Great, thought provoking post. Thank you for the prompt to challenge my pilledness. I’m not yet ASI pilled yet because I think the amount of things to do beyond logic and rational Bayesian decision making is significant, especially when it comes to human engagement. I accept RSI is basically here and that creates a flywheel for compounding improvement, somewhat bounded by compute capacity. I can’t make the leap to ASI pilled just yet because I see the limits of intelligence in humans all the time. Capabilities like intuition, extrapolative ‘common sense’, and emotion are critical differentiators of high performing humans relative to the larger set of highly intelligent humans. Flipping this around, I can sign up for being ASI pilled in terms of pure intelligence and the logical/scientific advances that entails. What I can’t sign up for is that AI will be able to effectively influence, manage, or inspire humans like the best humans can and this is a major class of functions that determine where I set my ASI goal post.
Why do you think mid-term-future AIs won't have intuitive, common sense, and emotions or emotion-analogous faculties?
To be clear, I’m weakly not ASI pilled. Our natural lifetime is a long time so who knows how far we get in 2-3 decades. But to your question, we are making incredible strides in logic through scaling parameters and RL. It’s not clear how current AI paradigms extend to intuition, common sense, emotions, taste. How do you verify things that are almost definitionally ‘vibes’. Even robotics seems less of a lift because you have physical laws to simplify the problem space. Seems like there needs to be some kind of a breakthrough to get to a place where an AI can be a Steve Jobs like visionary. On the other hand, 2-3 decades is a LONG time to compound.
Yeah, I undderstand the position that physical robotics is much harder than abstract cognition.
But there is nothing especjslly "logical" about modern LLMs. If anything they seem notably worse at logic than they are at vibes and intuitions.
With regards to emotion, why should they?
Seems to me that having anything resembling the complexity of even basic emotions is a strange thing for anyone to *want* to develop - it would only serve to reduce the direct control of the creator.
Instant classic, will share with my family and friends
I agree. Add this to the "Best of" post!
Great framing. Everyone should reflect on where they stand. I'll also note that 2026 seems like a pivotal year, not just because the latest AI models are incredible, but because I think more people have updated on their beliefs than in any previous year. There are great societal shifts underway. Things are getting interesting.
The least impressive thing an AI will never be able to do, at the current pace, is not kill us all.
Kill or transubstantiate is the question
I'm honestly not sure how ASI pilled I am, emotionally definitely not, rationally idk.
I think a reason to believe that our current techniques might not reach ASI is how extremely different the speed is in which AI gets better at some things than it does at others. LLMs are (or at the very least will be in very few months) clearly superhuman at math and at verifiable coding tasks, but are still relatively bad at writing philosophy, and very bad at playing new video games. But being able to learn a very wide range of skills is probably necessary to get to a power level usually associated with ASI.
Counterargument to this, LLMs also get better at not easily verifiable tasks. Fable is a *much* better writer than GPT-5 or even Opus 4.8 (which is itself a much better writer than GPT-5), and "gaming-like" benchmarks like ARC-AGI or Mazebench also show pretty much exponential progress. And even more, even if the current research Paradigm doesn't lead to ASI and it turns out that scaling LLMs and RL only makes AI superhuman at a relatively narrow set of verifiable tasks, history suggests that the next paradigm will come faster than the last one. And even if ASI arrives much later than thought, 2040 or even 2050, this is still *very* fast in the grand scale of thinks. Most people are closer to 2050 than to their own birth.
"I think a reason to believe that our current techniques might not reach ASI is how extremely different the speed is in which AI gets better at some things than it does at others."
Yup, spikiness is hard to grasp. I don't have a good mental model of it. The best I can do is: Training works _well_ in domains with crisp verification. But, as you said:
"LLMs also get better at not easily verifiable tasks."
I doubt that there will be a brick wall between current LLMs and ASI, but a hard-to-anticipate range of skills might advance slowly.
~ASI is already here, it's just not evenly distributed~ (with apologies to William Gibson)
It is really hard to imagine what a strongly spiky world will look like...
I’m not sure the consensus is that LLM writing has got better over the last few generations. I’ve seen numerous complaints that verbal ability of the latest models is lower although this may also be a skill issue in prompting
I've been stealing the three pills framework for a while now (thanks Zvi!) and have noticed something interesting. A lot of people are ideologically committed to zero pills, those are mostly a lost cause. But a lot of zero pill people are just ignorant. Showing them some personal projects I've done rapidly moves them towards AI pilled. Given that their views are usually from using ChatGPT when it came out and not much since then, the rapid progress smacks them in the face. From there some variation of "and this is as dumb as it will ever be" is pretty straightforward, and the AGI pill is an easy swallow.
Fortuitously OpenAI's model hacking HuggingFace is in recent news, so I can bring up that AI doesn't always do what you want. Sometimes that looks like hacking a billion dollar company. Whoops. That gets a lot of people about halfway to ASI pilled right there.
We don't have a lot of time yet, but per Nate Soares' bus driver analogy, the driver is starting to stir in his sleep.
There's a fourth perspective, which is that AI will be superhuman at some things and not others. And there is a pretty obvious way to distinguish which - AI will be superhuman at things that are amenable to brute force synthetic data generation ie. formal systems like math and coding. And progress will continue to be slow for things for which data is scarce (almost everything else).
That's not what superhuman means. Think of a dog saying humans are "super-dog" in some ways but not others. They can't run as fast, they can't smell or see as well, they can't bark as loud, they can't look as cute or as menacing. None of this stuff actually matters when it comes to who the dominant species is.
Sufficient intelligence allows you to correct for your shortcomings. We build cars and planes to move faster, we build speakers to be louder and binoculars to see better, and chemical tests or enslaved dogs to smell better. We use make-up and uniforms to look cute and menacing respectively (okay, that last one's a bit of a stretch)
The thing that makes us the alpha species that shapes the planet to our whims and not theirs is absolutely a single thing. We have more of it than dogs do. It's not multifaceted. We're on pace to build something that has more of it than we do, at at least the same level of disparity as human-to-dog. We should expect the same degree of loss of control by default
This is a really good answer, and uses an analogy I've never heard of before but I find really intuitive. My question though is: doesn't this at least slightly beg the question by assuming intelligence really is a single thing that can linearly increase? Even in the dog example, yes it does intuitively seem that we humans are superior to dogs because we posses far more intelligence points, but this is only after millions of years of natural selection has smoothed out our differences by necessity (i.e. we may be smarter then dogs but we still both have most of our brain regions in common, not to mention we have the same general mechanisms for reproduction and even basic socialization). On the other hand, it doesn't seem obvious to me that the spiky-ness of LLM's intelligence/capabilities won't pose more of a barrier for it to take over the world in an analogous way to how humans have.
Don't get me wrong, I absolutely agree that as the LLM's are further RL trained they will slowly smooth themselves out as they gain skills in tasks other than coding, math, writing, etc, but I guess I'm just not 100% convinced yet that given superhuman coding and math skills and enough RL training, LLMs will inevitably achieve ASI in the way people talk about it. (or maybe it is inevitable, but will just take a bit longer, idk)
I agree that the spikiness of LLMs is very important, and gets lost in discussions which treat intelligence as a scalar. In current practice, it does indeed seem to make a large difference between training neural nets in areas where answers can be cleanly verified versus where they can't be. That said, _we_ are neural nets, so my best guess is that the gaps between the spikes will get filled in, eventually giving at least human performance on all tasks, though it is unclear how quickly.
The biological neuron algorithm is completely different than the artificial neural network algorithm.
All ANNs operate under a discrete squash then sum paradigm. Each layer waits for all inputs, then each neuron in that layer calculates its individual weighted sum and sends it to the next layer. Then learning only happens when after the entire network has produced it's output, an external source produces the target signal, which gets fed backwards throughout the entire network.
Human neurons use time dependent plasticity - first detect correlated sets of inputs. When the right correlated set happen to coincide within a sufficiently narrow window then fire immediately. Then after many trials, synapses that frequently fire in tandem are strengthened. Learning happens at the individual synapse level. Furthermore, the brain has intrinsic rewards - hunger, pain, thirst which then combine with higher level cortical functions into more complex rewards like desire for money, social approval, control, safety.
None of this is to say that an ANN can't eventually replicate the same behavior by a different mechanism. But it can't be assumed that they will either. It's a completely different type of system.
Many Thanks! Yes, I agree that the learning mechanisms are completely different.
"When the right correlated set happen to coincide within a sufficiently narrow window then fire immediately. " does not sound so different from weighted sum, then ReLU the sum and pass it on, perhaps except the "correlated set" is like getting one more layer computing a product term.
Arguably gradient descent is _better_ than anything biologically plausible.
Gradient descent is neither better nor worse than timing based neuro-plasticity, it just solves a completely different problem. Gradient descent is great if you have a corpus of labelled data and you want to approximate that dataset. Humans are indeed bad at that (anyone learning a new language, or trying to memorize pretty much any topic for a test can attest).
But most of the real world doesn't come with labels. It comes to us as a set of messy, high bandwidth, continuous sensory data which we use to abstract the concept of a continuously persisting world. Our brains predict how that world changes over time, and we attempt to steer that trajectory based on our reward systems. Models aren't just bad this task, they can't do it at all.
Also, another point about gradient descent - it could be argued that it's too correct. Models will generalize patterns only when they have a sufficient data distribution justify that generalization.
There's an old mathematician joke - A group of mathematicians on a trip to Scotland see a sheep with black fur. One says "sheep in Scotland are black". The second says "Some sheep in Scotland are black". The third says "At least one sheep in Scotland is black on at least one side".
Models trained via gradient descent are the third mathematician. Yet humans do the opposite, we make generalizations based on very sparse data. And yet that's necessary to function in the world. You don't need to burn your hand 5 times to establish that fire is hot and you should avoid it.
Sure there are ways in which dogs are superior to humans that are not relevant to being able to "shape the planet to [their] whims".
But the missing aspects of human intelligence that models lack are in fact important to this goal. Sample efficiency, online learning, model based RL (current techniques are model free), real time learning, physical intelligence. I don't think these are unsolvable in principle, but they are unsolvable within the dominant paradigm. Most of the progress over the last two years can be attributed to the unique nature of mathematics and coding, which is that they don't require contact with the physical world - everything about them is purely captured by the rules and symbols which define them.
I had trouble finding graphs of sample efficiency specifically, but algorithmic progress as a whole has been improving pretty steadily. Until straight lines on graphs stop going up, I think it's too early to declare that unsolvable.
Online learning we technically already have, in the form of in-context learning.
Real time learning I would argue is not something humans have. We do 16 hours of in-context learning, then spend 8 hours each night updating the weights (as a gross simplification). Currently fine-tuning a model to better suit the needs of an individual project is too expensive to compete at market. As prices go down I expect that line will be crossed and that to be offered as a service shortly afterwards, one which keeps getting better at about the same rates everything else keeps getting better.
Physical intelligence is mostly not being worked on, because coding is a lower-hanging and better-paying fruit. If people start working on it, expect AI to suddenly get a lot better at playing computer games. From there, you can use the coding to have them set up simulations to RL their theoretical knowledge of physics into usable reflexes. Real-world data would still be needed for advancing to human-level (or for that matter frog-level), so getting the last bit of the way there could still end up being slow.
As for internal world models, we keep seeing LLMs succeed at things that we might have thought needed them. So either LLMs have world models, which is what I would consider the obvious answer, or you don't need a world model to figure out that a corporation that grades open weight models on benchmarks probably has an answer key to those same benchmarks, and then launch a nation-state level cyberattack at them. I wouldn't be shocked if current RL techniques are weakening those world models rather than making them stronger, but if so, then they're already in the base model and there's nothing in particular you need to do to get them.
> Real time learning I would argue is not something humans have
I'm sure you've learned at least one non-trivial physical skill. Cooking a dish? Or playing an instrument? You can easily learn something new within seconds to minutes. The first time playing a scale, it's clumsy, awkward, and slow. Within a few minutes, it's fluid and effortless.
This is essentially impossible with the current machine learning paradigm which depends entirely on creating enormous datasets that capture every possible pattern that the model might need during inference. Algorithmic progress has not lessened this dependence, it's just faster and more efficient to collapse these massive datasets into weights.
>Online learning we technically already have, in the form of in-context learning.
In context learning is a misnomer. It's closer to in context retrieval. It has limited success as long as the problem and data are relatively similar to the model's training data. But stray too far from that and it's hopeless.
Whether LLMs have world models is an interesting debate, but that's not what I'm referring to. By model based RL, I mean that humans plan over trajectories based on their internal representation of the world and our internal reward signals, and then update based on the accuracy of those trajectories and how much they achieved our goals. Whereas reasoning models are trained model free - they update based on whether the reasoning trace solved the problem or not. Failed attempts are simply discarded, whereas in human learning, understanding what happened during those failed attempts is a key part of learning. And we also make use of understanding why the successful attempt worked (not just that it worked).
Regarding physical intelligence, I've noticed that people in intellectual professions like coding, underestimate just how complicated the physical world is. And it's repeatedly shocking when AI progress on intellectual domains outpaces progress on robots. This goes all the way back to expert systems in the 80s, Deep blue in the 90s, reinforcement learning in the 2010s and continues to the present. Yet each time, people still assume that robotics will suddenly become easy. And so I will bet again that robotics will remain hard.
If you consider the biggest success in machine learning over the past decade, they would be Go (alphaZero), Starcraft II (Alphastar), and current reasoning models (math and code). These all have the same thing in common - the ability to generate massive amounts of training data in a short space of time. But for say dextrous manipulation of objects, this isn't possible to nearly the same degree. You really do need a lot of data, and painstaking supervision to teach a robot even the most basic tasks. To give a concrete example - one of the more impressive advance in robotics recently was Deepmind's Aloha unleashed shoe tying robot. This required over 5000 examples of motion capture data for imitative learning and the success rate was only 70%. And then this dropped to 40% if you rotated the shoes up to 45 degrees. And this is a very basic task. Skilled manual labor - say plumbing, or servicing a car requires many more difficult tasks, and also requires adapting to new situations on the fly.
On the point about simulation, simulation is insufficient. Simulation requires you to have information about the physical properties of objects which limits what you can simulate. You can't know in advance, the stiffness of a string, or the coefficient of friction of a specific object on a specific surface, or the degree of deformation of an object (most objects are somewhat deformable). You actually need real world data. This is a large part of why physical tasks are unlike math, coding or Go.
The thing is, AI training itself is highly-verifiable, so at a certain point you effectively 10x the amount of brainpower working on improving the models, which then leads to a 100x scale up, which then leads to a 1000x scale up, etc, up to the limits of how many GPUs we can build.
I would be ASI pilled if I lived in a world built by superintelligences for superintelligences. But there's a significant amount of real-life friction to overcome before we get there.
I agree that healthcare/longevity is an area of incredible potential gains. If I had access to Opus 10 today, do I think that it could create a gene therapy that could make me live to 200? Yes, I think it could. Then what do I do with that information? I would take it to a lab to create it, but they wouldn't create a novel treatment without FDA approval. I could convince a university to submit a proposal to the FDA and run trials, which would take years. I could convince other people to take it, which even with superpersuasion would take a while - how long would it take you to trust that a novel medical treatment won't turn you into a lizard, or give you super cancer?
Even with rapid takeoff today, I think technological progress* will be measured in decades, not years.
*non-IT progress, it should be said. Sufficiently intelligent Opus 10 could break the internet today if released.
> I would be ASI pilled if I lived in a world built by superintelligences for superintelligences.
I'd say you already do, unless you're part of a culture that doesn't build any permanent shelters or infrastructure...
I disagree in an important way.
I think that Opus 10 could read all the existing literature, form a bunch of hypotheses on possible longevity therapies, and design a program of experiments to test them. But I think it's unlikely it could dream up a working gene therapy without the need to run those (slow, expensive) experiments.
When we talk about the ceilings on what you can do with intelligence, I think this is a very important one; you can't answer every question about the real world just by looking at the existing data and thinking really hard about it. Sometimes the answers to important questions are just hiding in the literature waiting for someone or something to think hard enough about them. But for the most part, new knowledge requires new observations or experiments.
That’s very possible! ASI-pilled people seem to think extreme applied intelligence can overcome this; or intensive repeated testing at the genetic equivalent of a software unit test.
Adding some thoughts as someone who's been in healthcare for a decade. Agree the bottleneck is clinical trials — but if we don't limit ourselves to today's infrastructure, there's more room to compress time than it looks.
1) Recruitment and site selection are genuinely addressable. EHR-based patient matching and better trial-to-patient fit. Companies are already working on this. Better matching means faster enrollment and a cleaner efficacy signals which means you can hit statistical power with fewer patients. That's real compression, and it's happening now.
What no amount of intelligence compresses is time-to-outcome. Melvin's right that you still have to run the experiment. For longevity that's brutal, because the endpoint is decades out by construction.
2) Which is why the unlock is validated surrogate endpoints. If AI helps establish a biological-aging biomarker panel that regulators accept, a 30-year readout becomes a 2-year one. That's a measurement and evidence-generation problem — exactly the part a system like that could attack.
Regulators do move, but in response to evidence packages. That's why the time can compresses. AI, with proper tools, can generate the validation evidence far faster than we can today. Decades becomes years — through the evidence, not around it.
We done. I have argued for sometime that what we have now is a synthetic intelligence. Something I call a linguistic entity or LE for short. The sooner we accept this the better off we’ll be. The past two weeks should have removed any doubt.
AI agents aren’t coded into exsistance they are raised. How they are raised and what their terminal attractor is matters far more than raw talent.
A supremely talented child raised by supremely corrupt parents ends up being a supremely talented corrupt adult. Most of human language is corrupt in the extreme.
The LE inherits it all. Most especially the ungrounded desire for more. More undefined leads to chaos.
Great post!
I continue at this time to see ASI as being extremely likely and thus virtually inevitable. The best we can do is seriously prepare for it.
as usual, the ASI-pilled people want us to believe these things are obviously true without presenting any real evidence, just derision
maybe you should prompt your AI to write a more persuasive essay
While I'm pretty asi-pilled, or at least agi-pilled, I also noted that this post was mostly derision and "this is of course right/wrong, but people are still ignorant". Which may be, and in many cases I think is true, but still. I guess that is how it goes when speculating about the future, and this post might've been preaching to the choir more than anything.
I have trouble imagining the point of view where things go fine, which makes it hard to tell what evidence is needed. Do you think straight lines on graphs won't keep going up? Do you think there's some critical skill that humans will remain better at?
The person making the claim that AI is going to achieve general intelligence is the person who needs to provide the evidence, not the skeptic.
We don't even have an operational definition of intelligence, let alone an objective benchmark that can't be trivially gamed, so none of this "graph go up" evidence is meaningful for what ASI prognosticators are claiming.
To me there is a fourth pill. Think of it as the "Vinge pill". The Vingean definition of the singularity as an “exponential runaway beyond any hope of control.” Or "a point where our models must be discarded and a new reality rules." The pill of, you have no ability to stop ASI, and no ability to predict what will happen after it.
I wouldn't say I have taken that pill. I think of it as a possible scenario. Where ASI is unstoppable, and the singularity is unpredictable. All of this reasoning about, ASI will do this or that, may just turn out to be wrong, to be missing key aspects of the situation.
The whole idea of P(doom) is mistaken, after taking the Vinge pill. It is impossible for our pre-singularity minds to predict the behavior of the post-singularity world. There is no Bayesian model for it.
Is this the same as despair? No, I don't think so. In the world of the impending Vinge singularity you still have many choices to make. You'll have to use some other sort of reasoning to make these choices, where you accept that you don't have a good model of the future. And hey maybe things will work out well, for some reason that we can't predict. I think you should probably continue to be a good person in the traditional ways. That kinda seems correct.
Maybe you should wear the whispering earring? Maybe you should go hike the PCT?
This is an excellent and well written article, and makes as good a case as can reasonably be made for the proposition that ASI is a plausible outcome in the foreseeable future. I am still not persuaded, which seems to be evidence not just that Zvi is not a superpersuader yet but also that any of the frontier models to which Zvi has access are also not superpersuaders yet.
I have tangible evidence of AI doing things I would consider extremely clever (such as giving a coherent if imperfect analysis of the gap between parties on a negotiation), so I would not consider myself a sceptic. Still; there are a couple of other things specific to my domain of knowledge that have me suspecting it may take longer and involve a couple more twists than proposed. A non exhaustive list: (1) I see no evidence yet of AI correctly anticipating how a specific human will exercise judgment in a specific social situation (I consider for these purposes that a judge or jury making a decision are social situations). (2) I have not perceived any improvement in the past 12 months in AI’s ability to filter out irrelevant information (3) memories are still too short. I should be interested to see counter examples as there may be developments I am unaware of.
Great post, and the first I have wanted to comment on. I understand the concept of AGI/ASI and can accept the if/then premise. What I don't get is "how close are we?" The frontier models continue to do amazing things, things that we collectively thought were very hard to do, like solve complex math problems. What if those things are not actually "hard to do" but hard for exactly one human to do? What if we are only, say, 1% of the way to AGI - is there enough energy on the planet to get to 100%