Claude Code launched in February and is already approaching a $1 billion run rate. Clearly Karpathy's impressions about coding capabilities don't align with what developers actually find valuable.
I feel like the focus on "massive GDP growth" is a mistake. It's setting the bar too high. As far as I can tell, no technology has ever clearly shown up in GDP growth. GDP is an aggregate that includes all sorts of things and the main direct influence is financial things like monetary shocks rather than technological disruption.
I think you could have superintelligent AI take over the world and kill off the vast majority of humans, but *still* the event would not show up in GDP growth. Because the AIs might just want to continue managing the monetary supply in a similar way to the way we have been, GDP won't fully capture all the prices that go to zero or the downstream effects of purely AI-run militaries, etc.
So all of this talk about GDP growth is just going to give the skeptics more evidence when yet another year rolls around and we haven't seen a GDP statistic that resembles nothing in known history.
It almost seems like he wants the AI systems to choose (?) or converge on (?) something like metaphysics, very abstracted conclusions several levels derived from specific knowledge. He wasn't wonderfully clear about what he actually expected in the "cognitive core". There _are_ very abstracted conclusions like object persistence that are useful across extremely wide domains that might fit what he seemed to imply, but I couldn't tell if that was what he actually meant.
> I don’t buy that knowledge tends to get in the way. If it does, then Skill Issue.
I interpreted Karpathy's point here as referring to memorization getting in the way of learning. A good paper diving into this dynamic, if I remember it correctly, is "Omnigrok: Grokking Beyond Algorithmic Data" by Liu, Michaud, and Tegmark
I think that's right. Also, a lot of the things LLMs memorize are silly and probably get in the way (e.g. hash values, names, results of math computations, page numbers).
> He comes back to this idea that knowing things is a disadvantage. I don’t get it.
I think I get it.
He doesn’t means, "for an optimal reasoner, knowing things is a disadvantage". Or "for most real-world agents, knowing things is a disadvantage".
He’s saying that in the current paradigm of pretraining => RL, knowing more things from pretraining is a disadvantage, because it gets in the way of RL. RL is supposed to be the thing that "train the cognitive core", but it will always take the shortest path, and often the shortest path is "regurgitate the teacher password from training data", which is not very helpful for the goal of "train the cognitive core". If it didn't have the teacher password in its training data, it would have to learn to reason to find the answer.
It’s very similar to your observation "if you don't want to learn, LLMs are a formidable tool for that", but applied to LLMs themselves. Humans can avoid learning by just using LLMs. LLMs can avoid learning to reason by just using memories from training data.
> My prediction is that the cognitive core hypothesis is wrong
It seems clearly right to me ? It’s all the point about learning "mental movements" from LessWrong, like "imagine worlds where hypothesis is true, imagine worlds where hypothesis is false, compare". Reasoning is made of steps, the dance is not at all trivial, you have to learn it. This looks like something very generic and compact to me.
Look at all the mistakes LLMs make. "Unable to get out of a prior". This is not a lack of knowledge, this is a clear misstep in the dance.
This. I totally agree with Karpathy's cognitive core hypothesis even though his other beliefs seem incoherent. I'd say that 1 billion weights is easily enough to describe the most important parts of reasoning. Possibly 10 million weights could be enough, depending on how much you can change the architecture. Right now we're doing learn language => RL for reasoning, when really for ASI we should be doing RL for reasoning => learn language.
Interesting, I agree with Karpathy on many things, but I don't believe that 1 billion weights is enough for the cognitive core. We know human infants have over 100 trillion synapses.
How much of that is redundant? We can look at individuals born with less cerebral cortex (usually due to hydrocephalus). About half of cortical loss at birth is not always crippling so there is some redundancy*, but 90% of cortical loss is completely devastating. So you do need trillions of synapses for intelligence.
Of course there isn't a one to one correlation between synapses and artificial neural network weights, but I think that actually strengthens the point - synapses are probably more powerful/complicated than single weights so that should push us towards a higher bound on how many weights are needed.
This reminds me of savants who have extraordinary mental abilities but those abilities cause large deficits in ordinary daily functioning. Rain man is probably the most salient modern example. But this was even explored back in 1942 by Jean Luis Borges when he wrote the short story "Funes the Memorious".
Basically agreed. I would phrase "it will always take the shortest path, and often the shortest path is "regurgitate the teacher password from training data"" as "blurt out a shallow pattern match from training data".
As you said: "Reasoning is made of steps, the dance is not at all trivial, you have to learn it."
I wonder how much is fixable just by doing something like: "Always look through each of the steps towards an answer, find the most questionable one, and search for evidence for and against it." But I assume the major labs have already tried umpteen variants on it...
I wonder if you could use one model you pre trained fully to train ANOTHER model that learns by creating reasoning traces that lead to the answer, with a third "proctor" LLM inspecting the final solution and looking for cheating such as incomplete reasoning, reasoning biased by knowing the answer, conclusions that don't follow from premises, etc.
So you essentially create the smaller core model bootstrapping off the existing ones.
I don't know. AI isn't a pill that makes you intelligent and well-informed. Unless we're talking about Chinese room style online interactions, in which case posting a photoshopped picture of yourself looking fit would seem to be the closer physical analogy.
In this analogy (and actually in the real world as well) I think people will gravitate towards activities that are only possible / fun when you already have high capabilities. When farming (gym) is no longer necessary, humans shift back to hunting & gathering allegories (games) for recreation and social status.
So trying to simplify jobs, to add efficiency with AI is a big part of what I do for work. Here's what I've learned. I present it here because I think it's relevant for GDP growth. It will be hard and slow, but it will be more than 2%. 4% is optimistic though.
If you look at the bottle necks of many real-world physical jobs, AI can support workers quite well, and add 20%-40% capacity for jobs in the physical world. Essentially many jobs, like fixing a car, or diagnosing a health condition, require a ton of paper work to ensure you get paid. It'll be much longer before robots are doing these jobs. So Toyota won't pay for a new transmission unless the repair tech does all the things to prove you need it, otherwise shops will just always rip the thing out and charge the warrantee. The way AI can fix this is, the person doing the difficult physical examination work will need to wear a camera and have it record what they do, then it can automate the paperwork, parts ordering, and tech can speak in any language. This means the tech gets about 12 minutes an hour back. Same with doctors. It will cost about 10k per employee to live stream video for AI to summarize and process. So worth it for any job that generates more than $90k in value. Additionally, the AI can help ensure things are done correctly, actually fixed, suggest things to try, so a less experience tech can do a more complex repair. In practice they fix an additional car per day per tech. With many fewer service advisors. Medical billing works very similarly. A camera can do all the insurance stuff, order the meds. No more data entry, fewer medical billing people.
You need a ton of data centers to do video 24/7 for all these jobs. But that is the future of AI in the economy. That is what it looks like, at least until we get reliable robots.
"I’m not convinced. As Dario points out, in theory you can put everything in the context window. You can do a lot better on memory and imitating continual learning than that with effort, and we’ve done remarkably little on such fronts."
This is where I agree with Andrei and not with Dario. You can't just put everything in context for two reasons.
The first is quite practical - proto-AGI needs to be able to digest video or at least sequential images (perhaps a screenshot every second though preferably better than that). Even with a context length of 100 million tokens, one frame per second screenshots runs you out of context length within minutes. And high quality video can run you out of context length in seconds.
The second is more subtle. In context length learning compresses knowledge but so far it has not shown to result in deeper understanding. It's not enough to compress knowledge, models need to be able to absorb deeper abstractions and apply those to new tasks. Right now, in context learning is the equivalent of an amnesiac who has to consult a book before making each decision. That's fine for some tasks (those that are primarily data retrieval) but it's inadequate for many if not most tasks.
To give a concrete example - I think it's fair to say a proto-AGI should be able to play a video game. Imagine a very simple video game sequence, avoiding a goomba in the original Mario. The first time you run into it and die. The second time you remember you died the first time so you think about how to avoid it - you use the jump button. By the third time, you don't even have to think about it - you see goomba, you automatically jump over it. But in the real game, there are dozens of little things you learn that are like this - and you no longer have to consult your memory of what happened before to solve new problems - you've abstracted the core properties that let you solve new challenges. Simply adding context length will never achieve this sort of learning.
Re: "And you’re going to say having access to ‘any economically valuable (digital) task at human performance or better’ only is +2% GDP growth? Really?"
I don't think that this is the most likely scenario, but let me try to steelman it with 3 possibilities:
1) The boring one. Yes, we get full AGI. Yes, it really can do any economically valuable human digital task. BUT - we got it to work by throwing so much test-time compute at it that it winds up having incremental costs LARGER than human wages.
2) There is much, much, much more crucial implicit knowledge in human workplaces than we realized. Yes, you can train an AI to come in to a workplace with all the knowledge of a bright intern. And we've also solved incremental learning, so they learn from experience as fast as a bright intern. But memoires and retrospectives or not, the crucial details on most workplace experiences AREN'T available except for encountering them "in the wild", and a career doles them out slowly, at paces that the AI can't speed up. It still takes 30 years to get someone/someAI with 30 years experience. And that experience is needed in order to properly fill senior roles.
3) Andrej's "march of nines". AIs still need to crank up reliability in the face of corner cases, and the corner cases are discovered, not predicted. (somewhat overlapping with (2))
That's true for augmenting human organizations, but perhaps less so for ones built bottom-up around AI agents? The industrial revolution wasn't about drop-in replacements for artesans, but a complete reimagining of the production ecosystem.
Many Thanks! I'm a bit uncertain which scenario you are replying to? Perhaps to (2)?
I agree that
"The industrial revolution wasn't about drop-in replacements for artesans, but a complete reimagining of the production ecosystem."
but this is somewhat more of a "normal technology" scenario. If the work flows _must_ be extensively reworked to take advantage of AI, then the process is likely to take decades. Yes, it will eventually transform the world, but on a time scale that needs to rebuild the production ecosystem from the ground up.
From the point of view of individual enterprises, this is a very risky path, with many blind alleys. It would happen. After all, incorporation of many normal technologies also happened, but they generally took decades _after_ the technology worked.
( I should say that my _actual_ expectation is that AGI, if we indeed get it, will be more rapidly transforming than this. I expect that the "drop-in replacement" approach will allow much faster diffusion of AGI technology than of a typical normal technology. I expect that implicit knowledge in workplaces will be a hindrance, but not as major a one as I portray in steelman (2). I _do_ agree that "drop-in" won't be the most effective way to use these AIs, but I expect that there will be a tail of further innovations in changing the production ecosystem after the initial "drop-in" shock. )
Actually I think a way we might consolidate these viewpoints is if new AI-first service companies crop up that serve as drop-in replacements for business functions for traditional companies.
I think many if not most organizations are going to be very hard pressed to develop such AI expertise in-house that allows them to move very fast, but they could leverage their position in the legacy ecosystem to secure funding that enables them to create these AI capabilities in parallel.
On the other hand, AI first companies could work at a considerable higher frequency than what human organizations are capable of, and creating vertical integration (given relevant field expertise) might be rapidly achievable, and thus a competing ecosystem could crop up relatively overnight.
If a single motivated, smart person without any experience can build such a business in months with current tech, clean slate projects have wild potential.
Many Thanks! That's a good point. This would be similar to how startups sometimes displace a dominant legacy company, but enhanced by the speedup possible from AI.
Hey, amigo… you don't happen to be ‘jeffreysoreff9588’ on YouTube, do you? If so, I feel like it's quite a coincidence that I just saw a comment from you on a random video, there! (If not, I guess it's still something of a coincidence!—not as interesting, though…)
‘Fun’ sure isn’t the way I’d describe going to the gym. More like an hour of misery, tempered by the fact that I know I’ll feel good about it afterward, due to a combination of endorphins from exercise and the knowledge that I made myself more healthy. I guess some people find it more fun than I do.
In any case, there’s another analogy to be made. Even though we have machines to move things, physical strength is still quite useful in day-to-day life because machines aren’t always immediately accessible. Perhaps ASI and robotics will fix that problem… But likewise, once we have ASI, as long as humans are still around, intelligence and knowledge will still be useful because only one’s own intelligence has immediate access to the thoughts, feelings, and desires in one’s brain. Well, unless the AI can read your mind. But in that case the obvious next step is brain augmentation and then the problem ceases to exist.
When you get fit, the feeling changes, from "miserable hurt shaky nauseous exhaustion" to "tired and stretched" and the endorphin after-high increases slightly.
Took about a year for me to get to that state. Hang in there!
Also depends on what you do at the gym. I've never found cardio remotely enjoyable, even after getting very good at it. Whereas resistance training e.g. weightlifting, IME rapidly goes from feeling uncomfortable to feeling OK, good even. Even doing something like heavy squats or deadlifts I find incomparably easier than cardio, because it's always just seconds or a minute of slightly-uncomfortable effort. But some people hate lifting and love cardio, so I imagine there's a genetic component too.
That will only advance as AI improves. A true AGI-level AI could very obviously do most accounting tasks on its own.
Except it is not the case.
LLMs need to be 100% reliable, not 99% reliable in order to substitute for jobs where attention to details is everything. And it is currently not there, and it is not making much progress on that front.
That's why the METR study is flawed. Relying on 50% completion is not nearly the same as said task being replaced by an Agent. It would be much more impressive if instead of doing longer task at 50% success rate, we were able to improve the success rate for a given task, even a 1 or 10min one.
There is no such thing as 100% reliability (no, humans aren't either, even when are paid for paying attention specifically, as I see constantly). LLM reliability has improved significantly over just last year, and already currently surpasses humans over some reference classes (mostly those we don't value much, though). The problem is that the rate of catastrophic failure is still significantly higher than the humans'.
When you introduce a team, you introduce checking after other humans, thus efficiency losses (yes, we are more used to human failure patterns than to LLM ones, so there is less inefficiency, but the losses are still real), and 100% reliability still doesn't exist.
Conversely, when tou're willing to tolerate efficiency losses, you can get an LLM to check outputs of another LLM in an adversarial setup (it just introduces a ton of friction as opposed to just dropping a prompt into an app). You can also do it with intermediate products of said LLM in a sufficiently complex pipeline. The result will be more reliable (possibly sufficiently reliable for the purpose)/easier to check on. Also, validating the results is often easier than generating them.
The point is: yes, LLMs are not good enough for drop-in replacement in most spheres yet. But you can get them to usability with some effort in many cases, and real humans tend to be worse than we imagine (source: worked as a QA for a while and interacted with QA in different capacity).
I work in the space of AI personalization and point 13 here (reproduced below) captures why AI feels terrible at lots of things where we feel it should't be.
This is top of mind for a lot of product teams right now and I am confident we will see remarkable progress in the next 18 months. Writing --> actions --> forecasting --> behaviour
13. “I also feel like it’s annoying to have to type out what I want in English because it’s too much typing. If I just navigate to the part of the code that I want, and I go where I know the code has to appear and I start typing out the first few letters, autocomplete gets it and just gives you the code. This is a very high information bandwidth to specify what you want.”
As a writer this resonates so, so much. There are many tasks where in theory the LLM could do it for me, but by the time I figure out how to get the LLM to do it for me, I might as well have gone and done it myself.
Whereas the autocomplete in gmail is actually good enough that it’s worth my brain scanning it to see if it’s what I wanted to type (or on occasion, a better version).
Hey Zvi, do you think you could comment on the recent Astral Codex Ten Substack regarding political spending by AI moguls? This has me very concerned.
https://www.astralcodexten.com/p/tech-pacs-are-closing-in-on-the-almonds?r=10ib&utm_medium=ios&triedRedirect=true
Claude Code launched in February and is already approaching a $1 billion run rate. Clearly Karpathy's impressions about coding capabilities don't align with what developers actually find valuable.
Not necessarily. He gives two examples of where he found it useful. Those limited use cases may be worth billions.
I feel like the focus on "massive GDP growth" is a mistake. It's setting the bar too high. As far as I can tell, no technology has ever clearly shown up in GDP growth. GDP is an aggregate that includes all sorts of things and the main direct influence is financial things like monetary shocks rather than technological disruption.
I think you could have superintelligent AI take over the world and kill off the vast majority of humans, but *still* the event would not show up in GDP growth. Because the AIs might just want to continue managing the monetary supply in a similar way to the way we have been, GDP won't fully capture all the prices that go to zero or the downstream effects of purely AI-run militaries, etc.
So all of this talk about GDP growth is just going to give the skeptics more evidence when yet another year rolls around and we haven't seen a GDP statistic that resembles nothing in known history.
One weirdness about Andrej's "cognitive core":
It almost seems like he wants the AI systems to choose (?) or converge on (?) something like metaphysics, very abstracted conclusions several levels derived from specific knowledge. He wasn't wonderfully clear about what he actually expected in the "cognitive core". There _are_ very abstracted conclusions like object persistence that are useful across extremely wide domains that might fit what he seemed to imply, but I couldn't tell if that was what he actually meant.
> I don’t buy that knowledge tends to get in the way. If it does, then Skill Issue.
I interpreted Karpathy's point here as referring to memorization getting in the way of learning. A good paper diving into this dynamic, if I remember it correctly, is "Omnigrok: Grokking Beyond Algorithmic Data" by Liu, Michaud, and Tegmark
I think that's right. Also, a lot of the things LLMs memorize are silly and probably get in the way (e.g. hash values, names, results of math computations, page numbers).
I know that Tentacle Porn exists, and I have a vague idea of what it involves. I feel like knowing more would get in the way.
Podcast episode for this post about a podcast:
https://open.substack.com/pub/dwatvpodcast/p/on-dwarkesh-patels-podcast-with-andrej
> He comes back to this idea that knowing things is a disadvantage. I don’t get it.
I think I get it.
He doesn’t means, "for an optimal reasoner, knowing things is a disadvantage". Or "for most real-world agents, knowing things is a disadvantage".
He’s saying that in the current paradigm of pretraining => RL, knowing more things from pretraining is a disadvantage, because it gets in the way of RL. RL is supposed to be the thing that "train the cognitive core", but it will always take the shortest path, and often the shortest path is "regurgitate the teacher password from training data", which is not very helpful for the goal of "train the cognitive core". If it didn't have the teacher password in its training data, it would have to learn to reason to find the answer.
It’s very similar to your observation "if you don't want to learn, LLMs are a formidable tool for that", but applied to LLMs themselves. Humans can avoid learning by just using LLMs. LLMs can avoid learning to reason by just using memories from training data.
> My prediction is that the cognitive core hypothesis is wrong
It seems clearly right to me ? It’s all the point about learning "mental movements" from LessWrong, like "imagine worlds where hypothesis is true, imagine worlds where hypothesis is false, compare". Reasoning is made of steps, the dance is not at all trivial, you have to learn it. This looks like something very generic and compact to me.
Look at all the mistakes LLMs make. "Unable to get out of a prior". This is not a lack of knowledge, this is a clear misstep in the dance.
This. I totally agree with Karpathy's cognitive core hypothesis even though his other beliefs seem incoherent. I'd say that 1 billion weights is easily enough to describe the most important parts of reasoning. Possibly 10 million weights could be enough, depending on how much you can change the architecture. Right now we're doing learn language => RL for reasoning, when really for ASI we should be doing RL for reasoning => learn language.
Interesting, I agree with Karpathy on many things, but I don't believe that 1 billion weights is enough for the cognitive core. We know human infants have over 100 trillion synapses.
How much of that is redundant? We can look at individuals born with less cerebral cortex (usually due to hydrocephalus). About half of cortical loss at birth is not always crippling so there is some redundancy*, but 90% of cortical loss is completely devastating. So you do need trillions of synapses for intelligence.
Of course there isn't a one to one correlation between synapses and artificial neural network weights, but I think that actually strengthens the point - synapses are probably more powerful/complicated than single weights so that should push us towards a higher bound on how many weights are needed.
Great points.
This reminds me of savants who have extraordinary mental abilities but those abilities cause large deficits in ordinary daily functioning. Rain man is probably the most salient modern example. But this was even explored back in 1942 by Jean Luis Borges when he wrote the short story "Funes the Memorious".
Basically agreed. I would phrase "it will always take the shortest path, and often the shortest path is "regurgitate the teacher password from training data"" as "blurt out a shallow pattern match from training data".
As you said: "Reasoning is made of steps, the dance is not at all trivial, you have to learn it."
I wonder how much is fixable just by doing something like: "Always look through each of the steps towards an answer, find the most questionable one, and search for evidence for and against it." But I assume the major labs have already tried umpteen variants on it...
I wonder if you could use one model you pre trained fully to train ANOTHER model that learns by creating reasoning traces that lead to the answer, with a third "proctor" LLM inspecting the final solution and looking for cheating such as incomplete reasoning, reasoning biased by knowing the answer, conclusions that don't follow from premises, etc.
So you essentially create the smaller core model bootstrapping off the existing ones.
> Why do they go to the gym? Because it’s fun, it’s healthy, and you look hot when you have a six-pack.
Does gym survive a magic pill that gives a six-pack and full health with no downsides ? I’m not sure, and it’s the better comparison with AGI.
I don't know. AI isn't a pill that makes you intelligent and well-informed. Unless we're talking about Chinese room style online interactions, in which case posting a photoshopped picture of yourself looking fit would seem to be the closer physical analogy.
In this analogy (and actually in the real world as well) I think people will gravitate towards activities that are only possible / fun when you already have high capabilities. When farming (gym) is no longer necessary, humans shift back to hunting & gathering allegories (games) for recreation and social status.
So trying to simplify jobs, to add efficiency with AI is a big part of what I do for work. Here's what I've learned. I present it here because I think it's relevant for GDP growth. It will be hard and slow, but it will be more than 2%. 4% is optimistic though.
If you look at the bottle necks of many real-world physical jobs, AI can support workers quite well, and add 20%-40% capacity for jobs in the physical world. Essentially many jobs, like fixing a car, or diagnosing a health condition, require a ton of paper work to ensure you get paid. It'll be much longer before robots are doing these jobs. So Toyota won't pay for a new transmission unless the repair tech does all the things to prove you need it, otherwise shops will just always rip the thing out and charge the warrantee. The way AI can fix this is, the person doing the difficult physical examination work will need to wear a camera and have it record what they do, then it can automate the paperwork, parts ordering, and tech can speak in any language. This means the tech gets about 12 minutes an hour back. Same with doctors. It will cost about 10k per employee to live stream video for AI to summarize and process. So worth it for any job that generates more than $90k in value. Additionally, the AI can help ensure things are done correctly, actually fixed, suggest things to try, so a less experience tech can do a more complex repair. In practice they fix an additional car per day per tech. With many fewer service advisors. Medical billing works very similarly. A camera can do all the insurance stuff, order the meds. No more data entry, fewer medical billing people.
You need a ton of data centers to do video 24/7 for all these jobs. But that is the future of AI in the economy. That is what it looks like, at least until we get reliable robots.
"I’m not convinced. As Dario points out, in theory you can put everything in the context window. You can do a lot better on memory and imitating continual learning than that with effort, and we’ve done remarkably little on such fronts."
This is where I agree with Andrei and not with Dario. You can't just put everything in context for two reasons.
The first is quite practical - proto-AGI needs to be able to digest video or at least sequential images (perhaps a screenshot every second though preferably better than that). Even with a context length of 100 million tokens, one frame per second screenshots runs you out of context length within minutes. And high quality video can run you out of context length in seconds.
The second is more subtle. In context length learning compresses knowledge but so far it has not shown to result in deeper understanding. It's not enough to compress knowledge, models need to be able to absorb deeper abstractions and apply those to new tasks. Right now, in context learning is the equivalent of an amnesiac who has to consult a book before making each decision. That's fine for some tasks (those that are primarily data retrieval) but it's inadequate for many if not most tasks.
To give a concrete example - I think it's fair to say a proto-AGI should be able to play a video game. Imagine a very simple video game sequence, avoiding a goomba in the original Mario. The first time you run into it and die. The second time you remember you died the first time so you think about how to avoid it - you use the jump button. By the third time, you don't even have to think about it - you see goomba, you automatically jump over it. But in the real game, there are dozens of little things you learn that are like this - and you no longer have to consult your memory of what happened before to solve new problems - you've abstracted the core properties that let you solve new challenges. Simply adding context length will never achieve this sort of learning.
Re: "And you’re going to say having access to ‘any economically valuable (digital) task at human performance or better’ only is +2% GDP growth? Really?"
I don't think that this is the most likely scenario, but let me try to steelman it with 3 possibilities:
1) The boring one. Yes, we get full AGI. Yes, it really can do any economically valuable human digital task. BUT - we got it to work by throwing so much test-time compute at it that it winds up having incremental costs LARGER than human wages.
2) There is much, much, much more crucial implicit knowledge in human workplaces than we realized. Yes, you can train an AI to come in to a workplace with all the knowledge of a bright intern. And we've also solved incremental learning, so they learn from experience as fast as a bright intern. But memoires and retrospectives or not, the crucial details on most workplace experiences AREN'T available except for encountering them "in the wild", and a career doles them out slowly, at paces that the AI can't speed up. It still takes 30 years to get someone/someAI with 30 years experience. And that experience is needed in order to properly fill senior roles.
3) Andrej's "march of nines". AIs still need to crank up reliability in the face of corner cases, and the corner cases are discovered, not predicted. (somewhat overlapping with (2))
That's true for augmenting human organizations, but perhaps less so for ones built bottom-up around AI agents? The industrial revolution wasn't about drop-in replacements for artesans, but a complete reimagining of the production ecosystem.
Many Thanks! I'm a bit uncertain which scenario you are replying to? Perhaps to (2)?
I agree that
"The industrial revolution wasn't about drop-in replacements for artesans, but a complete reimagining of the production ecosystem."
but this is somewhat more of a "normal technology" scenario. If the work flows _must_ be extensively reworked to take advantage of AI, then the process is likely to take decades. Yes, it will eventually transform the world, but on a time scale that needs to rebuild the production ecosystem from the ground up.
From the point of view of individual enterprises, this is a very risky path, with many blind alleys. It would happen. After all, incorporation of many normal technologies also happened, but they generally took decades _after_ the technology worked.
( I should say that my _actual_ expectation is that AGI, if we indeed get it, will be more rapidly transforming than this. I expect that the "drop-in replacement" approach will allow much faster diffusion of AGI technology than of a typical normal technology. I expect that implicit knowledge in workplaces will be a hindrance, but not as major a one as I portray in steelman (2). I _do_ agree that "drop-in" won't be the most effective way to use these AIs, but I expect that there will be a tail of further innovations in changing the production ecosystem after the initial "drop-in" shock. )
Actually I think a way we might consolidate these viewpoints is if new AI-first service companies crop up that serve as drop-in replacements for business functions for traditional companies.
I think many if not most organizations are going to be very hard pressed to develop such AI expertise in-house that allows them to move very fast, but they could leverage their position in the legacy ecosystem to secure funding that enables them to create these AI capabilities in parallel.
On the other hand, AI first companies could work at a considerable higher frequency than what human organizations are capable of, and creating vertical integration (given relevant field expertise) might be rapidly achievable, and thus a competing ecosystem could crop up relatively overnight.
Case in point, this small business a 17-y.o. Danish man built on his own: https://www.reddit.com/r/automation/s/ImHxHmpA4C
If a single motivated, smart person without any experience can build such a business in months with current tech, clean slate projects have wild potential.
Many Thanks! That's a good point. This would be similar to how startups sometimes displace a dominant legacy company, but enhanced by the speedup possible from AI.
Hey, amigo… you don't happen to be ‘jeffreysoreff9588’ on YouTube, do you? If so, I feel like it's quite a coincidence that I just saw a comment from you on a random video, there! (If not, I guess it's still something of a coincidence!—not as interesting, though…)
Many Thanks! Yes, that is my userid on YouTube.
‘Fun’ sure isn’t the way I’d describe going to the gym. More like an hour of misery, tempered by the fact that I know I’ll feel good about it afterward, due to a combination of endorphins from exercise and the knowledge that I made myself more healthy. I guess some people find it more fun than I do.
In any case, there’s another analogy to be made. Even though we have machines to move things, physical strength is still quite useful in day-to-day life because machines aren’t always immediately accessible. Perhaps ASI and robotics will fix that problem… But likewise, once we have ASI, as long as humans are still around, intelligence and knowledge will still be useful because only one’s own intelligence has immediate access to the thoughts, feelings, and desires in one’s brain. Well, unless the AI can read your mind. But in that case the obvious next step is brain augmentation and then the problem ceases to exist.
When you get fit, the feeling changes, from "miserable hurt shaky nauseous exhaustion" to "tired and stretched" and the endorphin after-high increases slightly.
Took about a year for me to get to that state. Hang in there!
Also depends on what you do at the gym. I've never found cardio remotely enjoyable, even after getting very good at it. Whereas resistance training e.g. weightlifting, IME rapidly goes from feeling uncomfortable to feeling OK, good even. Even doing something like heavy squats or deadlifts I find incomparably easier than cardio, because it's always just seconds or a minute of slightly-uncomfortable effort. But some people hate lifting and love cardio, so I imagine there's a genetic component too.
That will only advance as AI improves. A true AGI-level AI could very obviously do most accounting tasks on its own.
Except it is not the case.
LLMs need to be 100% reliable, not 99% reliable in order to substitute for jobs where attention to details is everything. And it is currently not there, and it is not making much progress on that front.
That's why the METR study is flawed. Relying on 50% completion is not nearly the same as said task being replaced by an Agent. It would be much more impressive if instead of doing longer task at 50% success rate, we were able to improve the success rate for a given task, even a 1 or 10min one.
There is no such thing as 100% reliability (no, humans aren't either, even when are paid for paying attention specifically, as I see constantly). LLM reliability has improved significantly over just last year, and already currently surpasses humans over some reference classes (mostly those we don't value much, though). The problem is that the rate of catastrophic failure is still significantly higher than the humans'.
There is 99% reliability, maybe not at the individual level but at team level at least. That is what accountants are paid for.
The problem of having an LLM doing even 1% of the job is that you have to check everything it has done, so you're not saving any time in reality.
There is very little difference between an LLM doing 0% of a task or achieving only 80% of reliability for said task.
I don't know how people can't comprehend this.
When you introduce a team, you introduce checking after other humans, thus efficiency losses (yes, we are more used to human failure patterns than to LLM ones, so there is less inefficiency, but the losses are still real), and 100% reliability still doesn't exist.
Conversely, when tou're willing to tolerate efficiency losses, you can get an LLM to check outputs of another LLM in an adversarial setup (it just introduces a ton of friction as opposed to just dropping a prompt into an app). You can also do it with intermediate products of said LLM in a sufficiently complex pipeline. The result will be more reliable (possibly sufficiently reliable for the purpose)/easier to check on. Also, validating the results is often easier than generating them.
The point is: yes, LLMs are not good enough for drop-in replacement in most spheres yet. But you can get them to usability with some effort in many cases, and real humans tend to be worse than we imagine (source: worked as a QA for a while and interacted with QA in different capacity).
I'm sympathetic to your general point, but I'd take 99% reliable over the last four accountants I've had who were all significantly worse!
Well, my point is that agents are nowhere near 99%, closer to 50%; so as is they're useless and NO GAIN of productivity AT ALL.
Zvi, could you explain for new readers what you mean by Skill Issue?
Is it one skill per economically valuable task, for all tasks in the economy, the way we humans have done it over centuries and millennia?
Or is it One Skill To Rule Them All, and in the darkness bind them?
If the latter, what would that look like?
I work in the space of AI personalization and point 13 here (reproduced below) captures why AI feels terrible at lots of things where we feel it should't be.
This article by Pete captures the current problem excellently. https://koomen.dev/essays/horseless-carriages/
This is top of mind for a lot of product teams right now and I am confident we will see remarkable progress in the next 18 months. Writing --> actions --> forecasting --> behaviour
13. “I also feel like it’s annoying to have to type out what I want in English because it’s too much typing. If I just navigate to the part of the code that I want, and I go where I know the code has to appear and I start typing out the first few letters, autocomplete gets it and just gives you the code. This is a very high information bandwidth to specify what you want.”
As a writer this resonates so, so much. There are many tasks where in theory the LLM could do it for me, but by the time I figure out how to get the LLM to do it for me, I might as well have gone and done it myself.
Whereas the autocomplete in gmail is actually good enough that it’s worth my brain scanning it to see if it’s what I wanted to type (or on occasion, a better version).
"I propose a podcast which is nothing but Dwarkesh Patel watching Star Trek for the first time and reacting."
Yes, please.