>Dwarkesh fires back that there are a lot of skills that are instance-specific and require on-the-job or continual learning, which he’s been emphasizing a lot for a while. I continue to not see a contradiction, or why it would be that hard to store and make available that knowledge as needed even if it’s hard for the LLM to permanently learn it.
In a sense, continual learning and storing compressed knowledge could be the same thing. Indeed, the weights of a neural network are a compressed representation of it's training data. If there were a general way for AI agents to summarize the knowledge they've gained and keep them in context (while cleaning up unneeded information) I think that would count as continual learning, even if the weights aren't changing.
However, that's still a very difficult problem. This is particularly an issue with multimodal information. A lossless representation of a single second of HD video or a series of screenshots would already fill up the context length of most models. Storing visual information requires compressing and abstracting relevant information and no one has demonstrated a robust way to do this.
Current multi-model models will tokenize an image or video first without having any sense of the task that it's trying to accomplish. Then they do reasoning over the tokens they've extracted. But imagine trying to do any task like this - you see an image and you write down as much information as you can think of about it. Then your visual memory of that image gets wiped and you have to work with your text description. It's crippling. If instead you knew what you were looking for, your search pattern would be much more concise and useful.
Man, I'd love to see what Dwarkesh would ask you, how you'd answer, and how he'd respond; that would be a great podcast; can we make that happen, please?
I agree it's confusing that Ilya implies that the age of research ever paused. I'm pretty sure that there has been more non-LLM AI research this year than any other (and that the same was true at the end of 2024). Deepmind and many other labs are constantly churning out new architectures*. And occasionally one surfaces to public awareness as a possible replacement for the transformer. But so far none of them have performed well at scale.
Maybe Ilya and his team have such good research taste that they can see what all of the other world class researchers missed. We'll see.
Relatedly, it could be argued that Anthropic has benefited from mostly focusing on scale, training methods and data quality rather than research into alternative architectures. They've spent much fewer resources than OpenAI or Deepmind, with a smaller team yet their models are on par.
*Titans, nested learning, large diffusion models for example.
I intuitively agree with your point that human-efficient learning in AI would constitute ASI. But I wonder what exactly that would be.
Consider myself as an example—I'm not even a particularly efficient human learner, with decent but far from exceptional calibration and world-modeling abilities. Yet even so, I outperform current AIs in terms of (certain kinds of) world models and achieve better calibration on various specific frontiers (but this is quite possibly a skill issue not an actual gap).
Imagine I could absorb and process information at even a fraction of an AI's rate. I'd immediately vault into the top echelon of, eg, lawyers globally, despite not having practiced for over a decade. I still might not reach the absolute elite tier, though, because mastering courtroom argument and client persuasion at the highest level might require emotional adaptations I couldn't learn.
Which is the question: how transformative would even "mediocre" human learning efficiency combined with current AI processing capacity be?
Otherwise put: is the current architecture and hardware already "enough" such that achieving human-level learning efficiency immediately triggers takeoff?
Or would human-level learning merely allow AIs to finally reach the asymptote of human expert performance across domains, without necessarily breaking through it (which of course would still be transformative due to reproducibility and speed, but very differently so)?
My gut reaction to both the current absence of incremental learning and human's currently superior sample efficiency is that there is a good chance that we need another significant innovation. My suspicion is something like a third kind of layer in the neural networks, somewhat like the introduction of attention layers.
Remember that perceptron layers, _by themselves_, are "complete" in the sense that they can approximate any smooth function. In that sense the attention layers are "redundant". Yet they helped a _lot_, and, AFAIK, are present in all SOTA models.
I forgot who mentioned this, but one of the commenters here mentioned that the human cortex has a motif of six approximately repeating layers. This partially fuels my suspicion that we are missing one (or more). I asked Claude Opus about this, and the most promising (promising-looking to me) current work was Kolmogorov-Arnold Networks, which make the spline function carry additional information, on top of what the now-classic perceptron/attention combination carries. Though these aren't, strictly speaking, an additional separate type of layer.
Claude said that these KANs _had_ shown improved sample efficiencies, which is one of the areas where we _know_ we need improvement to reach AGI. As Ilya Sutskever said, a human can learn to drive in 10 hours. And we do it with neural networks. So it should be possible to do sample efficient learning with a neural net (of _some_ sort), not needing e.g. augmentation with other software tools. I also see sample efficiency and incremental learning as closely linked, since incremental learning tends to dole out new information a little at a time, _not_ gigabytes of repeated and varied examples, albeit incremental learning needs to also solve the additional problem of catastrophic forgetting.
Claude also cited three other techniques, albeit I'm skeptical of one of them and unfamiliar with two of the others:
1) memory-augmented architectures, RAG and related techniques. I'm skeptical here, since it seems the learning isn't well integrated with the pre-training knowledge. It seems like giving an amnesiac a diary and a library.
2) state-space models (I'm too unfamiliar with them to really comment)
3) Fast weights and modern Hopfield networks. I'm mostly too unfamiliar to comment. I _do_ see the analogy in having multiple time scales for memories to the human need for sleep to consolidate learning into long term memory.
I think I agree with you. I don't fully buy that keeping everything in context is sufficient - real learning involves fully internalizing intuitions, rules of thumb, nuances, small optimizations. Adding to context does feel like the equivalent of an amnesiac with a notepad.
I think Anthropic researchers would respond by stating that sample efficient learning could be gradual rather than all or nothing. I remember one of them stating that sample efficiency has it's own scaling laws - as models increased in size from 2022 to present, the number of samples you need to get them to learn something new has decreased. And the reason is that smarter models learn more from each sample. Learning happens when models either get the right answer, or assign a higher probability to the correct answer. Having a baseline smarter model means that learning happens more often.
Depending on how robust this scaling law is, there might not be need for a single discrete breakthrough.
Many Thanks! Yes, that sounds reasonable. There might just ("just"?) need to be a bunch of small incremental improvements in sample efficiency and learning. To go meta with this: For improvements in sample efficiency (or in other capabilities of the models!), there might be a distribution of jumps from improvements, with some innovations yielding small improvements and others yielding large ones. Perhaps there is a power law with some exponent...
>I remember one of them stating that sample efficiency has it's own scaling laws - as models increased in size from 2022 to present, the number of samples you need to get them to learn something new has decreased. And the reason is that smarter models learn more from each sample. Learning happens when models either get the right answer, or assign a higher probability to the correct answer.
I'm glad to hear that. It does sound hopeful. I do think that the gap between human and LLM sample efficiencies is large enough that there is "space" for a substantial innovation, so to speak, though I'd be happy to see improvement of LLMs to human baselines in sample efficiency and incremental learning by any means, incremental or eureka moment.
Re alignment, I recently released a musical satire, You Get the Moon, about a real estate deal whereby the ASIs and their human agents get the Moon and we get the Earth. On all the streaming services
Podcast episode for this post:
https://open.substack.com/pub/dwatvpodcast/p/on-dwarkesh-patels-second-interview?utm_campaign=post-expanded-share&utm_medium=web
Thank you! Just wanted to say I really appreciate these for days when I don't have time to sit down and actually read the post
Thank you for saying so! I'm really glad I can help make an audio version a reality.
very well summarized. TY.
How many Hilbert dimensions to move the goal posts beyond your ability to catch them?
More than you got buddy.
>Dwarkesh fires back that there are a lot of skills that are instance-specific and require on-the-job or continual learning, which he’s been emphasizing a lot for a while. I continue to not see a contradiction, or why it would be that hard to store and make available that knowledge as needed even if it’s hard for the LLM to permanently learn it.
In a sense, continual learning and storing compressed knowledge could be the same thing. Indeed, the weights of a neural network are a compressed representation of it's training data. If there were a general way for AI agents to summarize the knowledge they've gained and keep them in context (while cleaning up unneeded information) I think that would count as continual learning, even if the weights aren't changing.
However, that's still a very difficult problem. This is particularly an issue with multimodal information. A lossless representation of a single second of HD video or a series of screenshots would already fill up the context length of most models. Storing visual information requires compressing and abstracting relevant information and no one has demonstrated a robust way to do this.
Current multi-model models will tokenize an image or video first without having any sense of the task that it's trying to accomplish. Then they do reasoning over the tokens they've extracted. But imagine trying to do any task like this - you see an image and you write down as much information as you can think of about it. Then your visual memory of that image gets wiped and you have to work with your text description. It's crippling. If instead you knew what you were looking for, your search pattern would be much more concise and useful.
Man, I'd love to see what Dwarkesh would ask you, how you'd answer, and how he'd respond; that would be a great podcast; can we make that happen, please?
Love the podcast transcript to Zvi take SFT data, as usual
I agree it's confusing that Ilya implies that the age of research ever paused. I'm pretty sure that there has been more non-LLM AI research this year than any other (and that the same was true at the end of 2024). Deepmind and many other labs are constantly churning out new architectures*. And occasionally one surfaces to public awareness as a possible replacement for the transformer. But so far none of them have performed well at scale.
Maybe Ilya and his team have such good research taste that they can see what all of the other world class researchers missed. We'll see.
Relatedly, it could be argued that Anthropic has benefited from mostly focusing on scale, training methods and data quality rather than research into alternative architectures. They've spent much fewer resources than OpenAI or Deepmind, with a smaller team yet their models are on par.
*Titans, nested learning, large diffusion models for example.
You need to be on the Dwarkesh Podcast, zvi
I intuitively agree with your point that human-efficient learning in AI would constitute ASI. But I wonder what exactly that would be.
Consider myself as an example—I'm not even a particularly efficient human learner, with decent but far from exceptional calibration and world-modeling abilities. Yet even so, I outperform current AIs in terms of (certain kinds of) world models and achieve better calibration on various specific frontiers (but this is quite possibly a skill issue not an actual gap).
Imagine I could absorb and process information at even a fraction of an AI's rate. I'd immediately vault into the top echelon of, eg, lawyers globally, despite not having practiced for over a decade. I still might not reach the absolute elite tier, though, because mastering courtroom argument and client persuasion at the highest level might require emotional adaptations I couldn't learn.
Which is the question: how transformative would even "mediocre" human learning efficiency combined with current AI processing capacity be?
Otherwise put: is the current architecture and hardware already "enough" such that achieving human-level learning efficiency immediately triggers takeoff?
Or would human-level learning merely allow AIs to finally reach the asymptote of human expert performance across domains, without necessarily breaking through it (which of course would still be transformative due to reproducibility and speed, but very differently so)?
My gut reaction to both the current absence of incremental learning and human's currently superior sample efficiency is that there is a good chance that we need another significant innovation. My suspicion is something like a third kind of layer in the neural networks, somewhat like the introduction of attention layers.
Remember that perceptron layers, _by themselves_, are "complete" in the sense that they can approximate any smooth function. In that sense the attention layers are "redundant". Yet they helped a _lot_, and, AFAIK, are present in all SOTA models.
I forgot who mentioned this, but one of the commenters here mentioned that the human cortex has a motif of six approximately repeating layers. This partially fuels my suspicion that we are missing one (or more). I asked Claude Opus about this, and the most promising (promising-looking to me) current work was Kolmogorov-Arnold Networks, which make the spline function carry additional information, on top of what the now-classic perceptron/attention combination carries. Though these aren't, strictly speaking, an additional separate type of layer.
Claude said that these KANs _had_ shown improved sample efficiencies, which is one of the areas where we _know_ we need improvement to reach AGI. As Ilya Sutskever said, a human can learn to drive in 10 hours. And we do it with neural networks. So it should be possible to do sample efficient learning with a neural net (of _some_ sort), not needing e.g. augmentation with other software tools. I also see sample efficiency and incremental learning as closely linked, since incremental learning tends to dole out new information a little at a time, _not_ gigabytes of repeated and varied examples, albeit incremental learning needs to also solve the additional problem of catastrophic forgetting.
Claude also cited three other techniques, albeit I'm skeptical of one of them and unfamiliar with two of the others:
1) memory-augmented architectures, RAG and related techniques. I'm skeptical here, since it seems the learning isn't well integrated with the pre-training knowledge. It seems like giving an amnesiac a diary and a library.
2) state-space models (I'm too unfamiliar with them to really comment)
3) Fast weights and modern Hopfield networks. I'm mostly too unfamiliar to comment. I _do_ see the analogy in having multiple time scales for memories to the human need for sleep to consolidate learning into long term memory.
I think I agree with you. I don't fully buy that keeping everything in context is sufficient - real learning involves fully internalizing intuitions, rules of thumb, nuances, small optimizations. Adding to context does feel like the equivalent of an amnesiac with a notepad.
I think Anthropic researchers would respond by stating that sample efficient learning could be gradual rather than all or nothing. I remember one of them stating that sample efficiency has it's own scaling laws - as models increased in size from 2022 to present, the number of samples you need to get them to learn something new has decreased. And the reason is that smarter models learn more from each sample. Learning happens when models either get the right answer, or assign a higher probability to the correct answer. Having a baseline smarter model means that learning happens more often.
Depending on how robust this scaling law is, there might not be need for a single discrete breakthrough.
Many Thanks! Yes, that sounds reasonable. There might just ("just"?) need to be a bunch of small incremental improvements in sample efficiency and learning. To go meta with this: For improvements in sample efficiency (or in other capabilities of the models!), there might be a distribution of jumps from improvements, with some innovations yielding small improvements and others yielding large ones. Perhaps there is a power law with some exponent...
>I remember one of them stating that sample efficiency has it's own scaling laws - as models increased in size from 2022 to present, the number of samples you need to get them to learn something new has decreased. And the reason is that smarter models learn more from each sample. Learning happens when models either get the right answer, or assign a higher probability to the correct answer.
I'm glad to hear that. It does sound hopeful. I do think that the gap between human and LLM sample efficiencies is large enough that there is "space" for a substantial innovation, so to speak, though I'd be happy to see improvement of LLMs to human baselines in sample efficiency and incremental learning by any means, incremental or eureka moment.
Re alignment, I recently released a musical satire, You Get the Moon, about a real estate deal whereby the ASIs and their human agents get the Moon and we get the Earth. On all the streaming services