The big headline this week was the song, which was the release of Claude Sonnet 4.5. I covered this in two parts, first the System Card and Alignment, and then a second post on capabilities. It is a very good model, likely the current best model for most coding tasks, most agentic and computer use tasks, and quick or back-and-forth chat conversations. GPT-5 still has a role to play as well.
- Vetos and holdout problems matter a lot and I agree there are challenges, although I think they can be addressed in most cases (see NIMBY example in there); but yes agree that mechanism design is important. Not sure these are necessarily full dealbreakers across the board, but an important component to get this right.
- On Ferengi dystopianism, I imagine this is what people in small villages a while back would say about large, complex cities and financial markets. Ultimately though I make it clear in the piece that we should not, in fact, be putting a price on *everything*. But in many ways we implicitly do so already ('value', votes etc) - so might as well make it explicit where it's not controversial to do so.
- On alignment, 'align model [M] to arbitrary target [T]' is essentially instruction following or 'technical alignment', which I'm generally optimistic about. Getting agents to follow T is the trickier bit, though so far models seem to do so well - though keen to see more multi-agent work/evals to get a better sense. *Specifying* T doesn't seem intractable either, since we already do so with laws, regs, norms etc. Also I'm not sure you need to specify a single T to avoid disempowerement either: T is a collection of many things, and what we've done so far before AI will continue with AI too.
- Not sure how you get to a ‘totalitarian surveillance state’, particularly given the second part of the essay, but ultimately it's trye that the infrastructure could allow for one - in the same way as it could allow for a democratic system with checks and balances. This seems like a 'governance/normative' question more than a technological one: similarly to how I'm happy London has lots of CCTV cameras, but don't feel the same in Pyongyang. If anything I think the decentralized architecture actually has features that make it less prone to centralized surveillance than the status quo.
For your last point: if everyone is using AIs for all/most of their interactions with the World, and the AIs _must_ obey the law, all of the law even the bits that no-one ever obeys, then how does that differ from a totalitarian state?
I bet the OpenAI ads are not intended to attract users but instead are trying to implant the idea for the US public that AI is good -- gets you out in the world, helps you do positive healthy things, etc.
How has the hallucination problem been solved? This may be a skill issue, but I'm a lawyer and hardly a day goes by when an LLM doesn't make up cases and statutes and quotes out of whole cloth, or cite to cases or statutes that have nothing to do with the subject in question.
If you remember to always ask it for sources and always have the browser tool enabled, you are less likely to encounter this. If you don't have it access the web to search for actual data, the chance they will make up stuff from their training data sets to try to fulfill your request based on "plausibility" or "sounding good enough" is almost guaranteed. They have a training bias against saying "I don't know" or "I don't have data for this", so if that's the case they will confabulate to fulfill your request. Make sure they are searching the web or your source documents.
Your ability to synthesize knowledge from multiple fields, combined with your sharp wit, rational viewpoint, and clear insights, is impressive. Considering Sonnet 4.5 and all the rest while recalling Sonnet 105: Fair, kind, and true is all my argument; Fair, kind, and true, varying to other words...
I've gotten maybe the closest so far to being AI slop nerd sniped by the videos of the old women dropping boulders on glass bridges. Just the first like 3 seconds of the first one I saw (the one with the goofy dog rescue subplot) was so stupid in a way that was adequately hilarious to me that I definitely kind of marveled at it for maybe a minute or so. I would like to say this kind of thing is a novelty with diminishing returns (only that one held up, the others aren't as good) but I definitely cannot yet be sure.
On shopping, I really wish that I could get an agent that works for me (the buyer), and not for the seller. Would happily pay 2% of purchase price for that, though I expect that it should cost a lot less to provide a service like that.
This is of course true in other domains as well. It is hard to get a real estate agent that actually works for buyers, for example, and it used to be hard to get a financial advisor that didn't rely on kickbacks.
In the shopping use case, my sense is that the problem is similar to the downfall of mint. The people who would be the most interested in a service like that also tend to be penny pinchers, making it hard to get a large enough user base to survive.
I'd love if Pulse were something simple like "give me a feed updated daily on the following topics: Magic the Gathering, [employer], [related industry news], SF politics. text only, final destination, no video. you know the rules, and so do I"
...but I doubt it'll be that good. If that expectation were bucked, I'd still find it depressing anyway that OpenAI finally managed to crack the MVP personalized (ad) feed code in current_year after $billions and decades of wasted effort by hundreds of others attempting to create same. It should not be that hard, especially when a potential customer *wants* to tell you things they're willing to be hawked! Alas, the customization algorithms for all such extant products remain hopelessly adversarial. Wishing them best of luck anyway.
Neon Mobile: privacy erosion, but now at least you get paid a pittance. Capitalism, ho! I think that article fails to typical-mind (or perhaps it's ITT, is privacy ideologically coded yet?) by not being cynical enough. It's not that people are cynically getting a rebate on data pimping, and would stop if only they knew The Real Risks. It's actually very simple revealed preferences - many people simply don't give an ass' rat about privacy, Privacy, or """privacy""", so are quite happy to sell nothing for something. Do you *really* think they're looking at the ToS? If you're not old enough to have grown up with a different expectation...if you've been misled by media into thinking nefarious actors have unlimited reach...if you don't understand security through obscurity. Lots of ways one could pull a learned helplessness here, or fail to develop a model of the concept to begin with.
Don't get me wrong, it's still gross and depressing and worthy of critique from people fighting the good fight. But I really do think it's a rearguard action at this point, and we're trying to close the barn door after the ship has largely sailed. (Maybe someone could popularize Roko's Basilisk for a mass audience, that'd do the trick...)
Re: Radiologist jobs — I think that's a bit of a head fake, mainly because Geoff Hinton made an offhand comment years ago. That post has a chart of how radiologists spend the rest of their time (meetings, consulting, teaching), and it seems likely that the other parts will get automated before the imaging (regulatory hurdles on the actual medical work, even if the models got good + generalized enough).
> "Employment is only in trouble after a tipping point is reached where sufficiently full automation becomes sufficiently broad, or demand growth is saturated."
This is spot on, and it will be different across different fields and roles. Plumbing jobs are clearly safe, but it seems so are the entry-level software engineers (so far)? https://x.com/simonw/status/1972794015482585462
It may be that AI companies need to unlearn looking for product-market fit (a product a set of people will use to do a job they need done) and look for "agent-worker fit" instead.
Could you write something about the product Claude now, not the model? Like what is going on with the usage limits -- people are absolutely furious over in reddit.com/r/claudeai , they're paying for MAX and exhausting their weekly quota in a day. Why do you think Anthropic have made the usaage limits so restrictive when they must know it will drive people to GPT? The announcement was that only heavy users should run up against the limits but it seems everyone is. Have they miscalibrated? Do users just have to get used to this being the cost of a product like this now? How can OpenAI keep their product so cheap?
As I understand it, Sonnet 4.5 is widely considered preferable for its factual accuracy, avoidance of bias, acknowledgment of uncertainty, and consistency, achieved through its "Constitutional AI" training. So, for this ongoing developers race snapshot, Anthropic is Goldish out of the blocks, OpenAI is somewhat Silvery. At the same time, Google basks in a dreamy, middle-distance midsummer's evening copper glow.
Thanks for the review! A few quick takes:
- Vetos and holdout problems matter a lot and I agree there are challenges, although I think they can be addressed in most cases (see NIMBY example in there); but yes agree that mechanism design is important. Not sure these are necessarily full dealbreakers across the board, but an important component to get this right.
- On Ferengi dystopianism, I imagine this is what people in small villages a while back would say about large, complex cities and financial markets. Ultimately though I make it clear in the piece that we should not, in fact, be putting a price on *everything*. But in many ways we implicitly do so already ('value', votes etc) - so might as well make it explicit where it's not controversial to do so.
- On alignment, 'align model [M] to arbitrary target [T]' is essentially instruction following or 'technical alignment', which I'm generally optimistic about. Getting agents to follow T is the trickier bit, though so far models seem to do so well - though keen to see more multi-agent work/evals to get a better sense. *Specifying* T doesn't seem intractable either, since we already do so with laws, regs, norms etc. Also I'm not sure you need to specify a single T to avoid disempowerement either: T is a collection of many things, and what we've done so far before AI will continue with AI too.
- Not sure how you get to a ‘totalitarian surveillance state’, particularly given the second part of the essay, but ultimately it's trye that the infrastructure could allow for one - in the same way as it could allow for a democratic system with checks and balances. This seems like a 'governance/normative' question more than a technological one: similarly to how I'm happy London has lots of CCTV cameras, but don't feel the same in Pyongyang. If anything I think the decentralized architecture actually has features that make it less prone to centralized surveillance than the status quo.
For your last point: if everyone is using AIs for all/most of their interactions with the World, and the AIs _must_ obey the law, all of the law even the bits that no-one ever obeys, then how does that differ from a totalitarian state?
Podcast episode for this post:
https://open.substack.com/pub/dwatvpodcast/p/ai-136-a-song-and-dance
I bet the OpenAI ads are not intended to attract users but instead are trying to implant the idea for the US public that AI is good -- gets you out in the world, helps you do positive healthy things, etc.
How has the hallucination problem been solved? This may be a skill issue, but I'm a lawyer and hardly a day goes by when an LLM doesn't make up cases and statutes and quotes out of whole cloth, or cite to cases or statutes that have nothing to do with the subject in question.
If you remember to always ask it for sources and always have the browser tool enabled, you are less likely to encounter this. If you don't have it access the web to search for actual data, the chance they will make up stuff from their training data sets to try to fulfill your request based on "plausibility" or "sounding good enough" is almost guaranteed. They have a training bias against saying "I don't know" or "I don't have data for this", so if that's the case they will confabulate to fulfill your request. Make sure they are searching the web or your source documents.
Your ability to synthesize knowledge from multiple fields, combined with your sharp wit, rational viewpoint, and clear insights, is impressive. Considering Sonnet 4.5 and all the rest while recalling Sonnet 105: Fair, kind, and true is all my argument; Fair, kind, and true, varying to other words...
I find GPT5 much better in this regatding, with Gemini being a total mess. What is your comparative experience with different models?
I've gotten maybe the closest so far to being AI slop nerd sniped by the videos of the old women dropping boulders on glass bridges. Just the first like 3 seconds of the first one I saw (the one with the goofy dog rescue subplot) was so stupid in a way that was adequately hilarious to me that I definitely kind of marveled at it for maybe a minute or so. I would like to say this kind of thing is a novelty with diminishing returns (only that one held up, the others aren't as good) but I definitely cannot yet be sure.
On shopping, I really wish that I could get an agent that works for me (the buyer), and not for the seller. Would happily pay 2% of purchase price for that, though I expect that it should cost a lot less to provide a service like that.
This is of course true in other domains as well. It is hard to get a real estate agent that actually works for buyers, for example, and it used to be hard to get a financial advisor that didn't rely on kickbacks.
In the shopping use case, my sense is that the problem is similar to the downfall of mint. The people who would be the most interested in a service like that also tend to be penny pinchers, making it hard to get a large enough user base to survive.
I think this is an effective AI-Safety video:
https://www.youtube.com/watch?v=f9HwA5IR-sg
The thumbnail is 'doomy' without being obnoxious. Video is sleek, smart, and has a sexy/likeable presenter. 1 Day old. Already has 1 million views.
I watched the whole thing just for entertainment value even though I already knew most of the key points.
I'd love if Pulse were something simple like "give me a feed updated daily on the following topics: Magic the Gathering, [employer], [related industry news], SF politics. text only, final destination, no video. you know the rules, and so do I"
...but I doubt it'll be that good. If that expectation were bucked, I'd still find it depressing anyway that OpenAI finally managed to crack the MVP personalized (ad) feed code in current_year after $billions and decades of wasted effort by hundreds of others attempting to create same. It should not be that hard, especially when a potential customer *wants* to tell you things they're willing to be hawked! Alas, the customization algorithms for all such extant products remain hopelessly adversarial. Wishing them best of luck anyway.
Neon Mobile: privacy erosion, but now at least you get paid a pittance. Capitalism, ho! I think that article fails to typical-mind (or perhaps it's ITT, is privacy ideologically coded yet?) by not being cynical enough. It's not that people are cynically getting a rebate on data pimping, and would stop if only they knew The Real Risks. It's actually very simple revealed preferences - many people simply don't give an ass' rat about privacy, Privacy, or """privacy""", so are quite happy to sell nothing for something. Do you *really* think they're looking at the ToS? If you're not old enough to have grown up with a different expectation...if you've been misled by media into thinking nefarious actors have unlimited reach...if you don't understand security through obscurity. Lots of ways one could pull a learned helplessness here, or fail to develop a model of the concept to begin with.
Don't get me wrong, it's still gross and depressing and worthy of critique from people fighting the good fight. But I really do think it's a rearguard action at this point, and we're trying to close the barn door after the ship has largely sailed. (Maybe someone could popularize Roko's Basilisk for a mass audience, that'd do the trick...)
Re: Radiologist jobs — I think that's a bit of a head fake, mainly because Geoff Hinton made an offhand comment years ago. That post has a chart of how radiologists spend the rest of their time (meetings, consulting, teaching), and it seems likely that the other parts will get automated before the imaging (regulatory hurdles on the actual medical work, even if the models got good + generalized enough).
> "Employment is only in trouble after a tipping point is reached where sufficiently full automation becomes sufficiently broad, or demand growth is saturated."
This is spot on, and it will be different across different fields and roles. Plumbing jobs are clearly safe, but it seems so are the entry-level software engineers (so far)? https://x.com/simonw/status/1972794015482585462
It may be that AI companies need to unlearn looking for product-market fit (a product a set of people will use to do a job they need done) and look for "agent-worker fit" instead.
Is there a good long-form text version of Emmett Shear’s alignment approach?
So appreciate the comment on mobile only versions. It is so often a painful reduction in functionality to just capture users in a walled garden
Yeah, just like Auren.
Could you write something about the product Claude now, not the model? Like what is going on with the usage limits -- people are absolutely furious over in reddit.com/r/claudeai , they're paying for MAX and exhausting their weekly quota in a day. Why do you think Anthropic have made the usaage limits so restrictive when they must know it will drive people to GPT? The announcement was that only heavy users should run up against the limits but it seems everyone is. Have they miscalibrated? Do users just have to get used to this being the cost of a product like this now? How can OpenAI keep their product so cheap?
As I understand it, Sonnet 4.5 is widely considered preferable for its factual accuracy, avoidance of bias, acknowledgment of uncertainty, and consistency, achieved through its "Constitutional AI" training. So, for this ongoing developers race snapshot, Anthropic is Goldish out of the blocks, OpenAI is somewhat Silvery. At the same time, Google basks in a dreamy, middle-distance midsummer's evening copper glow.