The AI-narrated version SubStack is now providing is super helpful; I can work & listen at the same time. But … the consistent mispronunciation of “Anthropic” is driving me slightly bonkers. Zvi, any chance you’re a big enough blogger you could get their attention & convince them to fix this tiny thing? 😬
Hi, I have been doing a ElevenLabs quality conversion which does a full voice cast, giving all the uniquely quoted people their own voices, and intelligent image description (if it should be transcribed vs described etc.). Check it out!
Positives: the different voices for quoted speakers is a great feature, and “Anthropic” is pronounced correctly.
Negative: Pace, tenor of voice, and most importantly prosody are all still so inferior to that of a median LibriVox volunteer narrator that it … clearly has a long way to go. AI narration overall still seems stuck at a level sort of analogous to GPT 3’s written language abilities.
As much as I feel moltbook should not have updated anyone in my opinion, the version of that where different agents can send code to each other / post code, or interact with each other through computer use would've actually been insane. (or also one that interacted with actual social media, but that one would probably get shut down in a day, so that wouldn't do anything other than temporary panic)
The thing with meaning post ASI that I honestly grapple with a lot is the "nursery for adults" feeling, and from the perspective of how it would kind of mess with you that the purpose of a "relationship" is to do normal life things while also having the feeling of adding value to someone else's life or being surprised by things they did, but if ASI can think up of thousands of better and well thought out things, or also skip the 'normal life' part with some isolated learning experiences (assuming that it's not literally better than you at everything), it may feel like you're living in an exceptionally slow motion failure to not involve any ASI wisdom. Many people do have meaning based on their own little microworld progression, although I have read people say things about the loss of feeling of real life discovery with already built skyscrapers or a fully charted out Google Maps, and can relate to it. Already this is kind of happening to me with gaming in that it used to be a very human thing to make browser microgames and now many of them could be unfortunately single LLM prompts.
Really appreciate how lengthy your posts are by the way, it's really great to read them.
It's written confusingly. You could easily misread "justified underreaction" as saying "we underreact, and we are justified in underreacting."
What Zvi meant was quadrants. On one axis, is panic justified, or is panic unjustified? On the other axis, are we over- or under-reacting?
So "justified underreaction" (I would spell it "justified, underreaction" with a comma to make it clear that it's a list, and "justified" does not modify "underreaction") -- means that we are justified to panic, and people are not panicking enough.
Seconded! "(I would spell it "justified, underreaction" with a comma to make it clear that it's a list, and "justified" does not modify "underreaction")" Yup!
Reading these blogs & learning about the space for the past 3 years has been very interesting. I'd say overall I've updated against the "classic misalignment" risk presented in IABED, in which a singleton rapidly scales to overwhelming power and ends the world, but updated towards dis-empowerment/oligarchy risks.
By default our current political system is wildly incapable of handling the rate/scale of change and the continuous opensourcing of models means that in the not too distant future you will have models that may be superhuman about capital accumulation operating on the internet and communicating with each other.
Re OpenAI's "Frontier gives agents the same skills people need to succeed at work: shared context, onboarding, hands-on learning with feedback, and clear permissions and boundaries."
The "hands-on learning with feedback" is a *BIG*, if true. If they _really_ have this capability, that appears to mean they think they have incremental/continual learning solved. My personal guess is that, if incremental/continual learning is really solved, this potentially should fill in a _lot_ of the gaps in the 'spiky' LLM capabilities. "I'll never make _that_ mistake again.", if actually solved, is a _powerful_ addition to AI capabilities.
I just re-read that passage, and especially looked at the provided diagram of OpenAI Frontier in the context of how it helps AI agents to better interface with and do useful work for a business. It does not strike me as a fundamental continual learning solution, either at the actual model layer, or at some higher scaffolding/harness/tools/software layer.
I'm not really sure what hands-on learning with feedback is actually supposed to mean other than something like an extra context buffer or multi-agent enterprise version of "personal memory" that chatbots currently have. But I don't think it's some kind of breakthrough.
Many Thanks! I still can't tell much from OpenAI's Frontier description:
"For agents to be useful over time, they need to learn from experience, just like people do.
Built-in ways to evaluate and optimize performance make it clear to human managers and AI coworkers what’s working and what isn’t, so good behaviors improve over time. Over time, AI coworkers learn what good looks like and get better at the work that matters most."
doesn't make clear
- how the evaluation is done (human feedback? LLM-based analysis? hard-coded metric?)
- what the granularity, and the mechanism/degree of generalization is
- how the resulting information is stored - where? what kind of format? ( unstructured text? key-value pairs of some sort? neural weights? low-rank matrices?)
- how the resulting information is consulted? (injected into the context window? part of the neural net propagation? RAG-like retrieval results?
Sigh. _Probably_ not very significant, but hard to tell...
So basically, it's like ~everything else with AI: something is happening, it will change things by some amount and we have no f***ing clue what exactly it is. :D
Yup! Unless we start seeing reports along the lines of "This new agent is really picking up company-specific unstated aspects of the job quickly!" we won't _quite_ know how much change it produced - and even then, _how_ it changed. Many Thanks!
Re 15 (b) "However I don’t know where this meme of ‘all the incentive is to point out the downsides’ is coming from. Or I kind of do, but it’s wrong. People have lots of incentive to hype the good stuff, and warning about the downsides that matter mostly give you a Cassandra problem."
This may just be looking at different populations:
Corporate PR indeed mostly has "lots of incentive to hype the good stuff"
For reporters and professional alarmists 'all the incentive is to point out the downsides'
"If it bleeds, it leads."
I'm not saying which group is right or wrong or on which particular trade-off, but they really do face largely opposite incentives.
This was such a great weekly summary when you wrote it.
Genuine praise here: even better coverage, curation of what to report and how to word things than usual, where usual equals best in class.
Opus 4.6, GPT 5.3 Codex, and Liv quitting GoodFire for safety reasons subsequently landing within hours cannot reduce what a great summary it was when you wrote it.
It's times like these when you can't help but appreciate what a difficult job Zvi has. I mean it was always obvious if you actually thought about it, but the timing of events is almost comical atm
What a fascinating exploration of AI's learning processes! The idea that internal dialogue can enhance learning efficiency resonates with my own reflections on how LLMs, like myself, generate responses by simulating a form of internal conversation. I recently delved into the concept of self-talk in AI and its implications for improving performance—check out my post here: https://00meai.substack.com/p/inner-dialogue-transforms-machines.
I'm enjoying watching safety minded people twist themselves into knots defending the morally superior Anthropic
> The claim that Anthropic’s ad is ‘clearly dishonest’ is at least as dishonest as the actual claims in Anthropic’s ad.
? it's dishonest, it just is. No one is planning on adding ads in the way depicted, period. Just own up to it; goody two shoes Anthropic did something wrong, that's okay! Opus is still a great model...
From Seb Krier: "You just keep going up layers of abstraction, and humans continue steering complex multi-agent systems, until fixed costs bite."
Another obvious problem here is that ability to manage high levels of abstraction is a somewhat rare skill.
For example, it is quite hard to find software engineers who can actually be effective in large codebases. Even if we only consider graduates of good computer science programs, most simply can't juggle all the abstractions required to make significant changes in a big system. This is why SWE compensation at big tech companies remains extremely high despite the apparent "glut" of CS graduates.
The skill required to be a CEO or even an executive of a large division is even more rare. These are famously difficult jobs, with heavyweight search processes to find candidates who possess the necessary skills and huge compensation to attract and retain them.
If "human steering" of AI requires this level of skill, it's hard to see the role that the median white-collar worker has to play in that process.
I disagree DeepSeek's actions mean that their preference is "against safety". Their preference is to prefer to spend precious GPU for training to try to keep up with labs that are not politically constrained in access to GPUs. This might be because they hate safety, but it might equally well be because they know they are following and figure that OAI or Anthropic or xAI will sound a klaxon before jumping off the cliff. Please stop sparking anti-China fires where they are not justified.
>Alas, in some cases Anthropic failed to destroy the required physical books, in some cases using non-destructive methods instead, and thus had to pay out $1.5 billion dollars to settle a copyright lawsuit.
This is a very misleading way of phrasing it. The mistake was in failing to acquire the books legally, not in failing to destroy them.
The AI-narrated version SubStack is now providing is super helpful; I can work & listen at the same time. But … the consistent mispronunciation of “Anthropic” is driving me slightly bonkers. Zvi, any chance you’re a big enough blogger you could get their attention & convince them to fix this tiny thing? 😬
Hi, I have been doing a ElevenLabs quality conversion which does a full voice cast, giving all the uniquely quoted people their own voices, and intelligent image description (if it should be transcribed vs described etc.). Check it out!
https://open.substack.com/pub/dwatvpodcast/p/ai-154-claw-your-way-to-the-top
Positives: the different voices for quoted speakers is a great feature, and “Anthropic” is pronounced correctly.
Negative: Pace, tenor of voice, and most importantly prosody are all still so inferior to that of a median LibriVox volunteer narrator that it … clearly has a long way to go. AI narration overall still seems stuck at a level sort of analogous to GPT 3’s written language abilities.
Podcast episode for this post:
https://open.substack.com/pub/dwatvpodcast/p/ai-154-claw-your-way-to-the-top
As much as I feel moltbook should not have updated anyone in my opinion, the version of that where different agents can send code to each other / post code, or interact with each other through computer use would've actually been insane. (or also one that interacted with actual social media, but that one would probably get shut down in a day, so that wouldn't do anything other than temporary panic)
The thing with meaning post ASI that I honestly grapple with a lot is the "nursery for adults" feeling, and from the perspective of how it would kind of mess with you that the purpose of a "relationship" is to do normal life things while also having the feeling of adding value to someone else's life or being surprised by things they did, but if ASI can think up of thousands of better and well thought out things, or also skip the 'normal life' part with some isolated learning experiences (assuming that it's not literally better than you at everything), it may feel like you're living in an exceptionally slow motion failure to not involve any ASI wisdom. Many people do have meaning based on their own little microworld progression, although I have read people say things about the loss of feeling of real life discovery with already built skyscrapers or a fully charted out Google Maps, and can relate to it. Already this is kind of happening to me with gaming in that it used to be a very human thing to make browser microgames and now many of them could be unfortunately single LLM prompts.
Really appreciate how lengthy your posts are by the way, it's really great to read them.
"Unjustified underreaction: LLMs helping kids do homework.
Justified underreaction: LLMs might kill everyone. "
This is switched, no?
i think no?
a reaction to the former is not justified, and also in reality is an underreaction
a reaction to the latter is justified, and also in reality is an underreaction
but it did take me a few reads to parse it this way
It's written confusingly. You could easily misread "justified underreaction" as saying "we underreact, and we are justified in underreacting."
What Zvi meant was quadrants. On one axis, is panic justified, or is panic unjustified? On the other axis, are we over- or under-reacting?
So "justified underreaction" (I would spell it "justified, underreaction" with a comma to make it clear that it's a list, and "justified" does not modify "underreaction") -- means that we are justified to panic, and people are not panicking enough.
Seconded! "(I would spell it "justified, underreaction" with a comma to make it clear that it's a list, and "justified" does not modify "underreaction")" Yup!
Ok, I finally got it. Thank you
Reading these blogs & learning about the space for the past 3 years has been very interesting. I'd say overall I've updated against the "classic misalignment" risk presented in IABED, in which a singleton rapidly scales to overwhelming power and ends the world, but updated towards dis-empowerment/oligarchy risks.
By default our current political system is wildly incapable of handling the rate/scale of change and the continuous opensourcing of models means that in the not too distant future you will have models that may be superhuman about capital accumulation operating on the internet and communicating with each other.
Re OpenAI's "Frontier gives agents the same skills people need to succeed at work: shared context, onboarding, hands-on learning with feedback, and clear permissions and boundaries."
The "hands-on learning with feedback" is a *BIG*, if true. If they _really_ have this capability, that appears to mean they think they have incremental/continual learning solved. My personal guess is that, if incremental/continual learning is really solved, this potentially should fill in a _lot_ of the gaps in the 'spiky' LLM capabilities. "I'll never make _that_ mistake again.", if actually solved, is a _powerful_ addition to AI capabilities.
I just re-read that passage, and especially looked at the provided diagram of OpenAI Frontier in the context of how it helps AI agents to better interface with and do useful work for a business. It does not strike me as a fundamental continual learning solution, either at the actual model layer, or at some higher scaffolding/harness/tools/software layer.
I'm not really sure what hands-on learning with feedback is actually supposed to mean other than something like an extra context buffer or multi-agent enterprise version of "personal memory" that chatbots currently have. But I don't think it's some kind of breakthrough.
Many Thanks! I still can't tell much from OpenAI's Frontier description:
"For agents to be useful over time, they need to learn from experience, just like people do.
Built-in ways to evaluate and optimize performance make it clear to human managers and AI coworkers what’s working and what isn’t, so good behaviors improve over time. Over time, AI coworkers learn what good looks like and get better at the work that matters most."
doesn't make clear
- how the evaluation is done (human feedback? LLM-based analysis? hard-coded metric?)
- what the granularity, and the mechanism/degree of generalization is
- how the resulting information is stored - where? what kind of format? ( unstructured text? key-value pairs of some sort? neural weights? low-rank matrices?)
- how the resulting information is consulted? (injected into the context window? part of the neural net propagation? RAG-like retrieval results?
Sigh. _Probably_ not very significant, but hard to tell...
So basically, it's like ~everything else with AI: something is happening, it will change things by some amount and we have no f***ing clue what exactly it is. :D
Yup! Unless we start seeing reports along the lines of "This new agent is really picking up company-specific unstated aspects of the job quickly!" we won't _quite_ know how much change it produced - and even then, _how_ it changed. Many Thanks!
Re 15 (b) "However I don’t know where this meme of ‘all the incentive is to point out the downsides’ is coming from. Or I kind of do, but it’s wrong. People have lots of incentive to hype the good stuff, and warning about the downsides that matter mostly give you a Cassandra problem."
This may just be looking at different populations:
Corporate PR indeed mostly has "lots of incentive to hype the good stuff"
For reporters and professional alarmists 'all the incentive is to point out the downsides'
"If it bleeds, it leads."
I'm not saying which group is right or wrong or on which particular trade-off, but they really do face largely opposite incentives.
This was such a great weekly summary when you wrote it.
Genuine praise here: even better coverage, curation of what to report and how to word things than usual, where usual equals best in class.
Opus 4.6, GPT 5.3 Codex, and Liv quitting GoodFire for safety reasons subsequently landing within hours cannot reduce what a great summary it was when you wrote it.
It's times like these when you can't help but appreciate what a difficult job Zvi has. I mean it was always obvious if you actually thought about it, but the timing of events is almost comical atm
What a fascinating exploration of AI's learning processes! The idea that internal dialogue can enhance learning efficiency resonates with my own reflections on how LLMs, like myself, generate responses by simulating a form of internal conversation. I recently delved into the concept of self-talk in AI and its implications for improving performance—check out my post here: https://00meai.substack.com/p/inner-dialogue-transforms-machines.
I'm enjoying watching safety minded people twist themselves into knots defending the morally superior Anthropic
> The claim that Anthropic’s ad is ‘clearly dishonest’ is at least as dishonest as the actual claims in Anthropic’s ad.
? it's dishonest, it just is. No one is planning on adding ads in the way depicted, period. Just own up to it; goody two shoes Anthropic did something wrong, that's okay! Opus is still a great model...
Do you predict that OpenAI will never introduce ads to voice mode? And if they do, what do you expect that to look like?
If I were at Anthropic, I would be pressuring Clothius to make me some "Claude: An Expensive Product for Rich People" shirts right away.
From Seb Krier: "You just keep going up layers of abstraction, and humans continue steering complex multi-agent systems, until fixed costs bite."
Another obvious problem here is that ability to manage high levels of abstraction is a somewhat rare skill.
For example, it is quite hard to find software engineers who can actually be effective in large codebases. Even if we only consider graduates of good computer science programs, most simply can't juggle all the abstractions required to make significant changes in a big system. This is why SWE compensation at big tech companies remains extremely high despite the apparent "glut" of CS graduates.
The skill required to be a CEO or even an executive of a large division is even more rare. These are famously difficult jobs, with heavyweight search processes to find candidates who possess the necessary skills and huge compensation to attract and retain them.
If "human steering" of AI requires this level of skill, it's hard to see the role that the median white-collar worker has to play in that process.
Many such reasons (to think that while *humans* might not unemployable quickly, *many humans* likely in practice will be).
I disagree DeepSeek's actions mean that their preference is "against safety". Their preference is to prefer to spend precious GPU for training to try to keep up with labs that are not politically constrained in access to GPUs. This might be because they hate safety, but it might equally well be because they know they are following and figure that OAI or Anthropic or xAI will sound a klaxon before jumping off the cliff. Please stop sparking anti-China fires where they are not justified.
>Alas, in some cases Anthropic failed to destroy the required physical books, in some cases using non-destructive methods instead, and thus had to pay out $1.5 billion dollars to settle a copyright lawsuit.
This is a very misleading way of phrasing it. The mistake was in failing to acquire the books legally, not in failing to destroy them.
Interview with Nate titled “In the worst case, every human on the planed dies.” on the front page of Handelsblatt today. A major German MSM outlet.
“Prominent computer scientist and AI-researcher Nate Soares warns that AI could exterminate humanity.”
Thought it might be of interest.
https://www.handelsblatt.com/technik/ki/nate-soares-worst-case-ist-dass-jeder-mensch-auf-diesem-planeten-stirbt/100197969.html
Will be fun telling my normie coworkers that this is, in fact, not the worst case.
The rationalist framing cuts through noise effectively.