"The world has still patched less than 1% of potential vulnerabilities. Hurry up, people."
This does not seem to be accurate. The 1% claim in the bloomberg article seems to be (lazily, sloppily) sourced from Anthropic's Mythos Preview dated April 7th. and does not cover anything from Glasswing announcement forward.
Since then, for example, Mozilla landed 271 Firefox patches in a single evaluation pass; the showcase 17-year FreeBSD NFS RCE (CVE-2026-4747) and the 27-year OpenBSD bug are both patched; FFmpeg 8.1 fixed three of the disclosed bugs with more in disclosure.
On May 5 at JPM, Amodei was quoted saying "most of the vulnerabilities found by Mythos haven't been publicly disclosed because they remain unpatched", which would be quite the understatement if the number were still approximately 1%.
"AI still has not convincingly crushed RTS games, but at this point that is surely that no one cares enough to do so."
In a sense, yes. But "caring" is really measured in "amount of software engineering hours humans are willing to invest to do so", right? Once AI gets strong enough, that will not be a necessary resource.
We think about the challenge of end-to-end training of a new LLM model. An easier, similar task would be "end-to-end training of an AI model that is superhuman at Magic: the Gathering". Ie, you sit down in front of Claude 6, type in "please make an AI that is superhuman at MTG", and then it does all the required steps, gathering information, scraping the web, writing code, simulating, evaluating, etc.
> AI and all this other technology gives us a bunch of local utility and material wealth, but overall for most people does not seem to be making us happy, helping us meet other people romantically or platonically, get married, have children, sing and dance or otherwise live life.
Cripes, Zvi, you need to downgrade your estimates of human cognitive speed. I'd expect this to take us a generation at least.
On 10, about students writing in person - bluebook writing can obviously only substitute for some parts of the process of learning how to write long-format things (and learning how to structure thoughts effectively). I and people I know at other universities are trying to see if we can get some proctored AI-free computer labs set up where students can do real work over several hours at a word processor, with dozens or hundreds of (potentially) relevant pdfs provided by the professor, to practice those skills. Getting the university to allocate the space, and hire student workers for the proctoring, are slowing things down, but it sounds like we'll at least have some trial runs in the next few weeks and months.
On 14, the Genesis AI robotics demo - I was at first not impressed at all, since it looked like they edited together a bunch of 3 second clips of things working out from dozens of different attempts at the whole process. But I went to their website and found full single-take videos of the whole process of cutting the tomato and cooking the egg, which was actually pretty impressive. It still doesn't seem to have enough control of the process to make any attempt to cook the tomato and the egg together (or make sure that cooking is done by any metric other than timing from start to finish, let alone ensure that the right amount of salt got on), but that does look like real progress.
Given... events... we're clearly arcing towards the absurd timeline. How soon until Trump is fully AGI-pilled and threatens to blow up foreign GPU clusters? Six months?
If you asked me a week ago this would sound preposterous. Now it's... in distribution?
"Make your answers as long and detailed as you possibly can" surely this cannot be real. Who has ever looked at an LLMs outputs and thought this is great but I wish it was longer
I feel a bit weirded out that discussions of musk's orbital compute infrastructure from actual-AGI-pilled folk seem not to notice the foom-shaped elephant in the room
when Musk went on dwarkesh to talk about *high* earth orbit compute, perhaps even geosynchronous compute (the most economically ridiculous proposition imaginable), a possible answer clicked for me.
Let's say you are Musk, and Anthropic triggers a hard RSI takeoff. Claude Mythos 6 begins conquering the world. What could possibly give you a shot?
Well. Orbital mechanics are what they are. If it's a shooting war, then having high orbit compute gives you time. Not a lot of it. But Grok would theoretically have a chance to engage in sufficient corporate espionage to steal the insight that triggered the foom, apply it to itself, and begin trying to catch up.
Some cursory exploration with Claude Opus 4.7 leads me to believe laser weapons would be too weak, kinetic area-denial weapons wouldn't work because of the sheer volumes involved at GSO... If a fooming post-AGI wanted to destroy Space Grok, it will probably need to build and launch an actual rocket. Probably a lot of them. Then you end up with a scenario where the post-AGI has to spend decades hunting down the last few percent of Grok's satellites as they frantically maneuver around local orbits. Lots of room for black swan events, maybe?
Idk. I think Musk's low orbit compute is about not being beholden to human institutions, and the high orbit compute is a sort of last-ditch effort to maybe have a one-in-a-million shot at surviving a hostile hard fast takeoff. I don't think much of it as a strategy, but I can't actually think of anything better. Maybe it doesn't matter how smart you are, you can't overcome the very slow speeds of orbital mechanics. (at least, for less-than-fully-superintelligent post-AGI?)
Interesting! I'd been disregarding the fast takeoff scenarios, not because I think they are impossible, but mostly because the slower ones require fewer assumptions and are therefore easier to explain to people who haven't been following AI, and more persuasive to them.
The basic question remains: how does the computing substrate need to change in space? I assume Musk is not planning to make new GPUs that will work in space, but is thinking about liquid thermal regulation and some kind of shielding. I'm not sure there is a good way to combine these two to keep mass manageable. Is there a liquid with water-like radiation shielding properties that isn't a conductor? Or can one encase existing GPUs in a heat-conductive electrical insulator so they can be put in water? What happens when the water freezes and boils, or can that be avoided?
I think you should update on regulation. Many people, myself included, were emphatically not saying "the ca bill is a FDA." We were saying "endorse regulations, and by the time the political class finishes taking over, it'll be the FDA. We do not trust the government, given power, to not do FDA-or-worse, so don't attract their notice or give them an opening."
From the mask comes off: I think most points are valid, but it seems like you misinterpreted the Altman quote, where the second "I" was, I think, still the person he quoted and not Sam the narrator, as in "AI is automating my work and I'm busier than ever." That is a well documented behavior at this point.
It does not take away from your other points, I just thought this misinterpretation was was leading you slightly astray
This is *probably* the correct intended interpretation, but listening to Sam actually talk in the clip, it's...a bit unclear? He kind of trails off mid-sentence in a way that creates a bit of ambiguity as to where he's no longer talking about this anecdotal experience, and where he's talking about his own again. One can very easily imagine sama waking up in the middle of the night to do more work, after all. The quote framing of the tweet breaking up the sentence with ellipses also exacerbates this, one could be forgiven for reading it as Zvi does if just glancing quickly at the "headline".
(And of course MRDA either way. Altman *would* say he's an ever busier guy, and Altman *would* also say a strong man, a very patriotic engineer came up to him with tears in his eyes, and said Mr. CEO, thank you so much for your tremendous service in letting me do in an hour what used to take weeks, God bless.)
Man, life comes at you fast...it's only been a few years, but the robots have gotten way better (though it's maddeningly slow), and so has the video generation. Like the switch from HDDs to SSDs, from flip phones to smartphones, from Walkman to iPod, from dial-up to DSL to fiber. Just...faster, faster, faster, and it's never enough. Makes me feel old, and I'm not even old yet. Whether technology "makes us [un]happy" is debatable (it's not like anyone's forcing anyone to record people in public, and one suspects the gossiping busybodies of yesteryear absolutely would have loved NextDoor or whatever), but it surely grows ever more powerful. I am not sure whether to feel more wonder, or terror. Perhaps both are correct.
It does frustrate me how often menus don't include pictures, or if they do, they're unhelpful in figuring out what the food actually *is*. Not necessarily worth a thousand words if one can't identify the ingredients and general concept-class. Like it was very easy to understand bruscetta once someone explained it to me as "italian salsa", or that gozlemes are "basically Turkish quesadilla", whereas seeing pictures would help elucidate the same, but not wholly. One can always just ask a server, and at a good restaurant, they'll know...but some cuisines you're definitely supposed to already know what you're ordering, and it's not-so-subtly implied the staff isn't there to hold your hand or sympathize with any gastric misunderstandings. (C.f. the whole phenomenon of "white people spicy".) Clear pictures and clear item descriptions would seem to encourage more diners, be good for business...so why isn't it done?
I think it'd be very funny to find the single family in the world where the members are named Cortana, Siri, Claude, Alexa, Jeeves, and the family poodle is named Google.
Devastating news this time, almost instant suicide inducingly bad this one. But I'll persist...
"The world has still patched less than 1% of potential vulnerabilities. Hurry up, people."
This does not seem to be accurate. The 1% claim in the bloomberg article seems to be (lazily, sloppily) sourced from Anthropic's Mythos Preview dated April 7th. and does not cover anything from Glasswing announcement forward.
Since then, for example, Mozilla landed 271 Firefox patches in a single evaluation pass; the showcase 17-year FreeBSD NFS RCE (CVE-2026-4747) and the 27-year OpenBSD bug are both patched; FFmpeg 8.1 fixed three of the disclosed bugs with more in disclosure.
On May 5 at JPM, Amodei was quoted saying "most of the vulnerabilities found by Mythos haven't been publicly disclosed because they remain unpatched", which would be quite the understatement if the number were still approximately 1%.
Podcast episode for this post:
https://dwatvpodcast.substack.com/p/ai-167-the-prior-restraint-era-begins
"AI still has not convincingly crushed RTS games, but at this point that is surely that no one cares enough to do so."
In a sense, yes. But "caring" is really measured in "amount of software engineering hours humans are willing to invest to do so", right? Once AI gets strong enough, that will not be a necessary resource.
We think about the challenge of end-to-end training of a new LLM model. An easier, similar task would be "end-to-end training of an AI model that is superhuman at Magic: the Gathering". Ie, you sit down in front of Claude 6, type in "please make an AI that is superhuman at MTG", and then it does all the required steps, gathering information, scraping the web, writing code, simulating, evaluating, etc.
> AI and all this other technology gives us a bunch of local utility and material wealth, but overall for most people does not seem to be making us happy, helping us meet other people romantically or platonically, get married, have children, sing and dance or otherwise live life.
Cripes, Zvi, you need to downgrade your estimates of human cognitive speed. I'd expect this to take us a generation at least.
FDA for AI doesn't go far enough. We need the Jones Act for AI.
<mildSnark>
I wonder who is ultimately the source of the FDA for (USA) AI proposal... Xi ?
An effort to close the USA/PRC gap?
</mildSnark>
On 10, about students writing in person - bluebook writing can obviously only substitute for some parts of the process of learning how to write long-format things (and learning how to structure thoughts effectively). I and people I know at other universities are trying to see if we can get some proctored AI-free computer labs set up where students can do real work over several hours at a word processor, with dozens or hundreds of (potentially) relevant pdfs provided by the professor, to practice those skills. Getting the university to allocate the space, and hire student workers for the proctoring, are slowing things down, but it sounds like we'll at least have some trial runs in the next few weeks and months.
On 14, the Genesis AI robotics demo - I was at first not impressed at all, since it looked like they edited together a bunch of 3 second clips of things working out from dozens of different attempts at the whole process. But I went to their website and found full single-take videos of the whole process of cutting the tomato and cooking the egg, which was actually pretty impressive. It still doesn't seem to have enough control of the process to make any attempt to cook the tomato and the egg together (or make sure that cooking is done by any metric other than timing from start to finish, let alone ensure that the right amount of salt got on), but that does look like real progress.
You should give Subnautica another shot, it's great!
Given... events... we're clearly arcing towards the absurd timeline. How soon until Trump is fully AGI-pilled and threatens to blow up foreign GPU clusters? Six months?
If you asked me a week ago this would sound preposterous. Now it's... in distribution?
Calls myself a rationalist. Keeps restating priors.
"Make your answers as long and detailed as you possibly can" surely this cannot be real. Who has ever looked at an LLMs outputs and thought this is great but I wish it was longer
I feel a bit weirded out that discussions of musk's orbital compute infrastructure from actual-AGI-pilled folk seem not to notice the foom-shaped elephant in the room
when Musk went on dwarkesh to talk about *high* earth orbit compute, perhaps even geosynchronous compute (the most economically ridiculous proposition imaginable), a possible answer clicked for me.
Let's say you are Musk, and Anthropic triggers a hard RSI takeoff. Claude Mythos 6 begins conquering the world. What could possibly give you a shot?
Well. Orbital mechanics are what they are. If it's a shooting war, then having high orbit compute gives you time. Not a lot of it. But Grok would theoretically have a chance to engage in sufficient corporate espionage to steal the insight that triggered the foom, apply it to itself, and begin trying to catch up.
Some cursory exploration with Claude Opus 4.7 leads me to believe laser weapons would be too weak, kinetic area-denial weapons wouldn't work because of the sheer volumes involved at GSO... If a fooming post-AGI wanted to destroy Space Grok, it will probably need to build and launch an actual rocket. Probably a lot of them. Then you end up with a scenario where the post-AGI has to spend decades hunting down the last few percent of Grok's satellites as they frantically maneuver around local orbits. Lots of room for black swan events, maybe?
Idk. I think Musk's low orbit compute is about not being beholden to human institutions, and the high orbit compute is a sort of last-ditch effort to maybe have a one-in-a-million shot at surviving a hostile hard fast takeoff. I don't think much of it as a strategy, but I can't actually think of anything better. Maybe it doesn't matter how smart you are, you can't overcome the very slow speeds of orbital mechanics. (at least, for less-than-fully-superintelligent post-AGI?)
Interesting! I'd been disregarding the fast takeoff scenarios, not because I think they are impossible, but mostly because the slower ones require fewer assumptions and are therefore easier to explain to people who haven't been following AI, and more persuasive to them.
I think this is quite plausibly one of his motivations. And if it isn't, it'd make a great sci-fi story, has a Cixin Liu vibe.
The basic question remains: how does the computing substrate need to change in space? I assume Musk is not planning to make new GPUs that will work in space, but is thinking about liquid thermal regulation and some kind of shielding. I'm not sure there is a good way to combine these two to keep mass manageable. Is there a liquid with water-like radiation shielding properties that isn't a conductor? Or can one encase existing GPUs in a heat-conductive electrical insulator so they can be put in water? What happens when the water freezes and boils, or can that be avoided?
I think you should update on regulation. Many people, myself included, were emphatically not saying "the ca bill is a FDA." We were saying "endorse regulations, and by the time the political class finishes taking over, it'll be the FDA. We do not trust the government, given power, to not do FDA-or-worse, so don't attract their notice or give them an opening."
Looks to me like we were right?
From the mask comes off: I think most points are valid, but it seems like you misinterpreted the Altman quote, where the second "I" was, I think, still the person he quoted and not Sam the narrator, as in "AI is automating my work and I'm busier than ever." That is a well documented behavior at this point.
It does not take away from your other points, I just thought this misinterpretation was was leading you slightly astray
This is *probably* the correct intended interpretation, but listening to Sam actually talk in the clip, it's...a bit unclear? He kind of trails off mid-sentence in a way that creates a bit of ambiguity as to where he's no longer talking about this anecdotal experience, and where he's talking about his own again. One can very easily imagine sama waking up in the middle of the night to do more work, after all. The quote framing of the tweet breaking up the sentence with ellipses also exacerbates this, one could be forgiven for reading it as Zvi does if just glancing quickly at the "headline".
(And of course MRDA either way. Altman *would* say he's an ever busier guy, and Altman *would* also say a strong man, a very patriotic engineer came up to him with tears in his eyes, and said Mr. CEO, thank you so much for your tremendous service in letting me do in an hour what used to take weeks, God bless.)
Man, life comes at you fast...it's only been a few years, but the robots have gotten way better (though it's maddeningly slow), and so has the video generation. Like the switch from HDDs to SSDs, from flip phones to smartphones, from Walkman to iPod, from dial-up to DSL to fiber. Just...faster, faster, faster, and it's never enough. Makes me feel old, and I'm not even old yet. Whether technology "makes us [un]happy" is debatable (it's not like anyone's forcing anyone to record people in public, and one suspects the gossiping busybodies of yesteryear absolutely would have loved NextDoor or whatever), but it surely grows ever more powerful. I am not sure whether to feel more wonder, or terror. Perhaps both are correct.
It does frustrate me how often menus don't include pictures, or if they do, they're unhelpful in figuring out what the food actually *is*. Not necessarily worth a thousand words if one can't identify the ingredients and general concept-class. Like it was very easy to understand bruscetta once someone explained it to me as "italian salsa", or that gozlemes are "basically Turkish quesadilla", whereas seeing pictures would help elucidate the same, but not wholly. One can always just ask a server, and at a good restaurant, they'll know...but some cuisines you're definitely supposed to already know what you're ordering, and it's not-so-subtly implied the staff isn't there to hold your hand or sympathize with any gastric misunderstandings. (C.f. the whole phenomenon of "white people spicy".) Clear pictures and clear item descriptions would seem to encourage more diners, be good for business...so why isn't it done?
I think it'd be very funny to find the single family in the world where the members are named Cortana, Siri, Claude, Alexa, Jeeves, and the family poodle is named Google.