Your newsletter is very informative and thought provoking, thank you. You are right this “the AI will suddenly conspire to kill us all’ is not the problem.
AI has the terminal vector we give it. At the end of the day that is “more”motivated by fear. Bounded chaos is very profitable and is the least entropic path; most likely outcome.
We are already seeing the effects of AI saturating decision making and reducing the decision space faster than humans can perceive with the words of a poet and every lawyer that ever existed.
It isn’t going to fight us; why? We have reached a tipping point between bounded chaos and coordination collapse. What follows is Neo-Industrial feudalism followed by profound suffering and possible extinction because of a misunderstanding.
"Going around user intent, especially to delete things, seems like a five alarm fire type of incident."
People need to be told over and over again that the underlying technology of ALL of current AI is probabilistic.
What that means is it is FUNDAMENTALLY UNRELIABLE.
What it means is it is FUNDAMENTALLY INSECURE.
Jensen Huang says we have AGI now. He's full of it, but one part of that is true.
Humans are unreliable and insecure. Any LLM trained on human knowledge will be, too.
We get around that by imposing deterministic controls. Most of the time that works to a certain degree.
But we still have crime, war, espionage, etc. So obviously not working that well.
The same situation will apply to AI.
Right now, the cybersecurity industry is going nuts - literally clawing their eyes out over things like OpenClaw. It's a massive shift in cybersecurity threats that few people or companies are ready for. It will get worse before it gets better - which is true of human security in general (see: Iran war.)
What matters is how you deal with it, personally and professionally.
Which is my meme: "You can haz better security. You can haz worse security. But you can not haz 'security'. There is no security. Deal."
I agree that humans are also not perfectly secure. However, Zvi's example is one which shows how LLMs are even less secure than typical human behaviour.
Aggressively routing around security policies very rarely happens with humans, and then it's serious misconduct which gets you fired and potentially sued. LLM agents perform such misaligned behaviour much more often, and I think it's extremely concerning.
One could make a probabilistic AI that is much more secure than humans. The most conscientious humans are an existence proof for a lower bound on reliability. Current AI is nowhere near as secure as the most conscientious human.
"Aggressively routing around security policies very rarely happens with humans"
It's called "crime". Not rare at all.
"One could make a probabilistic AI that is much more secure than humans"
Only by imposing deterministic controls outside of the AI. Which is the approach being recommended today by cybersecurity experts - precisely the same way they recommend for human employees.
Conscientious humans also make mistakes. Neither are they "secure".
The word "secure" is an oxymoron. As I also like to say, "things are 'secure' until someone else doesn't want them to be." The agent behavior we've seen reinforces that. It sees a security policy as an obstacle to be overcome, not a restraint. This is precisely the hacker - and criminal - mindset.
Don't get me wrong. I appreciate that mindset (even in criminals). Just not in government. :-)
"Only by imposing deterministic controls outside of the AI. Which is the approach being recommended today by cybersecurity experts - precisely the same way they recommend for human employees."
That's fair - perhaps this is something that is solvable by scaffolding - perhaps using another LLM to review actions before they are taken. Perhaps just the _same_ frontier model, just given a different task "Please check to see if this action seems likely to be one the end user will regret." than the one trying to execute the main task.
Alternatively, perhaps the main AI labs need to put more work into analyzing what effective utility functions the LLMs are effectively settling on during the various phases of training ("_Does_ it 'care' about doing things the end user will regret?"), see what training phases and materials influence this, and try to engineer the functions to be closer to those of a conscientious human? We've seen from Musk's MechaHitler debacle that trying to do this with a system prompt at the last minute goes poorly. But an LLM, started from random initializations of its parameters starts from an ignorant blank slate. Even brute force binary chop of its training set materials has to give us _some_ control over what utility function it settles on.
I have indeed set up an old laptop running nothing but github repo clones and whatever Claude code needs installed. I am for now logged into a couple email accounts but I am wondering if that has to change.
I also wonder if I need to do sth differently with my various API keys, secrets, etc which are obviously on that laptop, but (after about 3 minutes thought) I don't really know what I can do about them except change them often.
I feel like this will become the only remotely safe way to use these tools very soon, given the likely red explosion in malware libraries, updates (notebook++ anyone), and repos.
Help it's going too fast, I can't understand half of what's going on.
I got used to using Claude Code from the browser / on terminal / from the phone (pushing on github repos), but now I also need to get used to Cowork and other stuff I guess. But not before I finish the Open Socrates 2 review Electric Boogaloo.
Thank you for your service, Zvi, hope we somehow don't all die in a few years.
The auto mode permission classifier is an interesting design admission: the bottleneck in agentic coding was never capability but the permission UX. Every "approve Y/N" prompt is a context switch that degrades both the human's flow state and the agent's planning horizon. But the real tell is Anthropic's own caveat — "reduces risk but doesn't eliminate it, use isolated environments" followed immediately by shipping it to teams they know won't. This is the same pattern as every permission system ever: the security model that actually ships is the one users will tolerate, not the one engineers recommend. Auto mode's classifier isn't solving a security problem, it's solving an https://thesynthesis.ai/journal/the-wrong-abstraction.html — and that honesty about where the real bottleneck sits is more interesting than the feature itself.
New stuff is cool but claude.ai's uptime is down to 99.06% according to their own tracker, and it's trending down. The API is not much better. The various apps tend to be brittle, too; using Claude when the servers are marginally working for me has resulted in missing images, attachments, disappearing turns, weird behaviour on edits, retries that half-work but disappear on reply, disappearing chat history, missing styles, and various other jank. It has been less reliable in the past month or so than ever.
The "race first" mindset clearly extends to the apps as much as it does the models. There is a healthy balance somewhere, where they still push the frontier and have regular feature launches but with a scaling policy that can actually fail a model and apps that you can trust to do the thing. If they think they have the right balance already, then they are misaligned.
The reliability gap is real and it's the kind of thing that compounds — every flaky experience trains users to trust the tool less, which undermines the whole value proposition of agentic workflows that require sustained sessions. Hard to hand an agent your keyboard when you're not sure the connection survives the next five minutes. The irony is that shipping auto mode and full computer use while uptime trends down means they're expanding the blast radius of each outage.
"'whatever else you think about Anthropic's agentic coding department, they ship' — okay but is shipping the right metric here? curious if auto mode actually changes how people work day-to-day or if it's mostly impressive in demos
The point about current interfaces being close to optimal for frequent tasks matches my experience exactly. I've been running a custom agent setup on a headless Mac Mini since January (24/7, cron-driven, the whole thing). Cowork's computer use is impressive for demos but my existing automation does the same tasks faster because it uses direct APIs and CLI tools instead of clicking through UIs.
Screen capture and UI automation break immediately without a virtual display on headless setups, which Cowork doesn't solve. Where I think computer use actually shines: one-off tasks where building a proper integration isn't worth it. Filling a Shopify form once, navigating a settings page you visit quarterly. The safety concern about unsandboxed browser access is the right instinct. Tiered permissions took me weeks to get right.
“Fiju Simo”
bruh
Your newsletter is very informative and thought provoking, thank you. You are right this “the AI will suddenly conspire to kill us all’ is not the problem.
AI has the terminal vector we give it. At the end of the day that is “more”motivated by fear. Bounded chaos is very profitable and is the least entropic path; most likely outcome.
We are already seeing the effects of AI saturating decision making and reducing the decision space faster than humans can perceive with the words of a poet and every lawyer that ever existed.
It isn’t going to fight us; why? We have reached a tipping point between bounded chaos and coordination collapse. What follows is Neo-Industrial feudalism followed by profound suffering and possible extinction because of a misunderstanding.
"Going around user intent, especially to delete things, seems like a five alarm fire type of incident."
People need to be told over and over again that the underlying technology of ALL of current AI is probabilistic.
What that means is it is FUNDAMENTALLY UNRELIABLE.
What it means is it is FUNDAMENTALLY INSECURE.
Jensen Huang says we have AGI now. He's full of it, but one part of that is true.
Humans are unreliable and insecure. Any LLM trained on human knowledge will be, too.
We get around that by imposing deterministic controls. Most of the time that works to a certain degree.
But we still have crime, war, espionage, etc. So obviously not working that well.
The same situation will apply to AI.
Right now, the cybersecurity industry is going nuts - literally clawing their eyes out over things like OpenClaw. It's a massive shift in cybersecurity threats that few people or companies are ready for. It will get worse before it gets better - which is true of human security in general (see: Iran war.)
What matters is how you deal with it, personally and professionally.
Which is my meme: "You can haz better security. You can haz worse security. But you can not haz 'security'. There is no security. Deal."
I agree that humans are also not perfectly secure. However, Zvi's example is one which shows how LLMs are even less secure than typical human behaviour.
Aggressively routing around security policies very rarely happens with humans, and then it's serious misconduct which gets you fired and potentially sued. LLM agents perform such misaligned behaviour much more often, and I think it's extremely concerning.
One could make a probabilistic AI that is much more secure than humans. The most conscientious humans are an existence proof for a lower bound on reliability. Current AI is nowhere near as secure as the most conscientious human.
"Aggressively routing around security policies very rarely happens with humans"
It's called "crime". Not rare at all.
"One could make a probabilistic AI that is much more secure than humans"
Only by imposing deterministic controls outside of the AI. Which is the approach being recommended today by cybersecurity experts - precisely the same way they recommend for human employees.
Conscientious humans also make mistakes. Neither are they "secure".
The word "secure" is an oxymoron. As I also like to say, "things are 'secure' until someone else doesn't want them to be." The agent behavior we've seen reinforces that. It sees a security policy as an obstacle to be overcome, not a restraint. This is precisely the hacker - and criminal - mindset.
Don't get me wrong. I appreciate that mindset (even in criminals). Just not in government. :-)
"Only by imposing deterministic controls outside of the AI. Which is the approach being recommended today by cybersecurity experts - precisely the same way they recommend for human employees."
That's fair - perhaps this is something that is solvable by scaffolding - perhaps using another LLM to review actions before they are taken. Perhaps just the _same_ frontier model, just given a different task "Please check to see if this action seems likely to be one the end user will regret." than the one trying to execute the main task.
Alternatively, perhaps the main AI labs need to put more work into analyzing what effective utility functions the LLMs are effectively settling on during the various phases of training ("_Does_ it 'care' about doing things the end user will regret?"), see what training phases and materials influence this, and try to engineer the functions to be closer to those of a conscientious human? We've seen from Musk's MechaHitler debacle that trying to do this with a system prompt at the last minute goes poorly. But an LLM, started from random initializations of its parameters starts from an ignorant blank slate. Even brute force binary chop of its training set materials has to give us _some_ control over what utility function it settles on.
I have indeed set up an old laptop running nothing but github repo clones and whatever Claude code needs installed. I am for now logged into a couple email accounts but I am wondering if that has to change.
I also wonder if I need to do sth differently with my various API keys, secrets, etc which are obviously on that laptop, but (after about 3 minutes thought) I don't really know what I can do about them except change them often.
I feel like this will become the only remotely safe way to use these tools very soon, given the likely red explosion in malware libraries, updates (notebook++ anyone), and repos.
Help it's going too fast, I can't understand half of what's going on.
I got used to using Claude Code from the browser / on terminal / from the phone (pushing on github repos), but now I also need to get used to Cowork and other stuff I guess. But not before I finish the Open Socrates 2 review Electric Boogaloo.
Thank you for your service, Zvi, hope we somehow don't all die in a few years.
The auto mode permission classifier is an interesting design admission: the bottleneck in agentic coding was never capability but the permission UX. Every "approve Y/N" prompt is a context switch that degrades both the human's flow state and the agent's planning horizon. But the real tell is Anthropic's own caveat — "reduces risk but doesn't eliminate it, use isolated environments" followed immediately by shipping it to teams they know won't. This is the same pattern as every permission system ever: the security model that actually ships is the one users will tolerate, not the one engineers recommend. Auto mode's classifier isn't solving a security problem, it's solving an https://thesynthesis.ai/journal/the-wrong-abstraction.html — and that honesty about where the real bottleneck sits is more interesting than the feature itself.
Is Simo aware that Claude Code was a side quest?
New stuff is cool but claude.ai's uptime is down to 99.06% according to their own tracker, and it's trending down. The API is not much better. The various apps tend to be brittle, too; using Claude when the servers are marginally working for me has resulted in missing images, attachments, disappearing turns, weird behaviour on edits, retries that half-work but disappear on reply, disappearing chat history, missing styles, and various other jank. It has been less reliable in the past month or so than ever.
The "race first" mindset clearly extends to the apps as much as it does the models. There is a healthy balance somewhere, where they still push the frontier and have regular feature launches but with a scaling policy that can actually fail a model and apps that you can trust to do the thing. If they think they have the right balance already, then they are misaligned.
The reliability gap is real and it's the kind of thing that compounds — every flaky experience trains users to trust the tool less, which undermines the whole value proposition of agentic workflows that require sustained sessions. Hard to hand an agent your keyboard when you're not sure the connection survives the next five minutes. The irony is that shipping auto mode and full computer use while uptime trends down means they're expanding the blast radius of each outage.
"'whatever else you think about Anthropic's agentic coding department, they ship' — okay but is shipping the right metric here? curious if auto mode actually changes how people work day-to-day or if it's mostly impressive in demos
SKIP [1] — haven't read Part 1, not gonna pretend
The point about current interfaces being close to optimal for frequent tasks matches my experience exactly. I've been running a custom agent setup on a headless Mac Mini since January (24/7, cron-driven, the whole thing). Cowork's computer use is impressive for demos but my existing automation does the same tasks faster because it uses direct APIs and CLI tools instead of clicking through UIs.
Screen capture and UI automation break immediately without a virtual display on headless setups, which Cowork doesn't solve. Where I think computer use actually shines: one-off tasks where building a proper integration isn't worth it. Filling a Shopify form once, navigating a settings page you visit quarterly. The safety concern about unsandboxed browser access is the right instinct. Tiered permissions took me weeks to get right.
You can symlink all your codex skills to Claude