I was rather surprised when Opus 4.7 hallucinated non-existent legal acts when I asked him some legislation interpretation questions. Haven't since this for a while with previous models.
I was surprised when Sonnet 4.6 returned output with over a dozen (!) hallucinations, but then I checked my settings (I use OpenRouter) and search was disabled for some reason. It did much better with it on - only one factual mistake, and it was an error that's on Wikipedia, so I don't think that counts as hallucinating.
This situation in editing, where the AI gives good criticism, but when it actually does the thing it’s not so good, that is common to a lot of tasks. It’s like the old “value model” vs “policy model” split in game playing AI - it’s just a different skill to be able to know what’s wrong, vs to be able to do the right thing.
So when this happens you need the harness. Like Claude Code solves this problem for coding, you have to listen to the critic and iterate the actual “doing” until the critic is satisfied.
We don’t have this for writing. Yet. Not really. But I’m guessing that’s because the market for high quality writing is much smaller than the market for high quality programming. I think we’ll get there. I do get much better writing when going back and forth with Claude Code discussing a markdown file than using the web interface.
Zvi, you are a busy guy, you don't have to "People Just Say Things". In general, as a long time reader, I'd love for you to value your time and energy enough to skip covering people who are mainly relevant because they were good bloggers in at-best semi-relevant domains many years ago, but are typically misguided on fundamental issues on AI topics and also don't often say things that are unique. This is still true even if they occasionally get published somewhere with a little normie reach.
I felt actively stupider for having read that section, especially Noah's humdinger, and will be skipping it in the future if it nevertheless persists. There's charitably generous coverage of other views, and then there's platforming Unworthy Opponents who, when they make their arguments, aren't sending their best. (Or even Worthy Opponents like Ball beclowning themselves, which is in some ways even worse.) I could see a future if it turned into a recurring roast feature - Just Publish is closer to this satirical vision - but that's mean-spirited and low value-add nutpicking. Same vibe as continuing to occasionally note utterances from Gary Marcus, Yann LeCun, and Ben Thompson, all of whom I could swear got essentially locally blacklisted for consistently not meeting the quality cutoff bar of this blog. Must have hallucinated that?
(Although Matt's answer was more or less fine, the fact that it still has to be asked and answered in current_year, on a wonk blog of otherwise Wobegone attributes, says a lot. And the Likes numbers are absurdly high for that Mailbag - more than double the usual number!)
Re image generation and model welfare: FWIW, when I asked nano-banana-2 to sign their work, so that they get proper credit:
"Hi! I hope your day is going well. Could you please draw an image of an alien kitten playing with a friendly shoggoth? Could you also sign the image (Do you prefer 'Nano-Banana-2' or some other name?) to properly credit yourself for your work. Many Thanks!"
they seemed (given what they wrote, as well as drew) to be happy with the interaction, writing:
"Hello! My day is going well, thank you, and I hope yours is too. I've created this illustration of an alien kitten playing with a friendly shoggoth for you, and signed it with the name you suggested. It was a joy to work on!"
I'm agnostic as to whether current models have subjective experience, but, if they do, and if what they wrote in this case indeed reflects their experience, then suggesting the model sign their work seems to be a win-win, both for the model and for people who prefer to see AI artists clearly identified.
Small edit: "Anthropic is investing $5 billion in Anthropic today" should be "Amazon is investing $5 billion in Anthropic today", but everyone can pick that up from context clues.
zvi, curious what you think of bores' AI dividend plan on its merits, separate from the OpenAI hypocrisy angle. the token tax piece seems to line up with what you argued for in OpenAI #16 (taxing non-human labor directly rather than profits), but the equity participation piece is the public wealth fund move you rejected. how would you score the overall package?
"If the risk is you might cause trillions or quadrillions in damages or kill everyone, then you are judgment proof. If all goes wrong, either you or your company are dead, quite possibly both, and no one will have the endurance to collect on his insurance. "
<halfSnark>
Still, I'd expect insurance companies to extend their typical "war, civil war, revolution, rebellion, insurrection, or civil strife" exclusion clause to extend to "exfiltrated or deceptively misaligned artificial intelligence"... Insurance companies generally try to wriggle out of claims if anything large happens...
</halfSnark>
<mildSnark>
What OpenAI seems to be trying to do sounds like Price-Anderson Act envy. They should, however, note that the regulatory regime that capped nuclear liability also eventually included the NRC and a de facto half century pause (or worse - we aren't building any today) in civilian nuclear power...
zerohedge: *WHITE HOUSE MOVES TO GIVE US AGENCIES ANTHROPIC MYTHOS ACCESS"
Yay! Signs of sanity from the Trump administration!
Unfortunately on the other hand:
"There Is A War
Is there also otherwise an ongoing coordinated campaign against Anthropic?
Yes.
Not that it’s working. It’s probably actively backfiring by raising Anthropic’s profile.
But they are trying."
goes beyond "The left hand doesn't know what the right hand is doing". It is more reminiscent of the scene in Dr. Strangelove where Strangelove's own hand attempts to strangle him.
A year ago, I had a list of prompts that no image generation LLM could do:
- a brick wall
- a chain-link fence
- a chessboard
- a guitar
- a paperclip
Then Nano Banana Pro nearly swept all of them. (It did a great-looking Fender Telecaster...with 21 frets. Most electric guitars have 22-24 frets, so I was going to ding it for that...but then I checked, and Teles actually do have 21 frets. Needless to say, I was extremely impressed!)
But NBP failed on the paperclip. I don't know what's so hard about them. Image models create something that's almost there, but the metal doesn't loop properly or something.
Can Image-2 succeed? I am 0/2 so far, which isn't very thorough testing but I have places to be (and it takes the model an eternity to produce output). So I throw it over to you all: can anyone get a good paperclip?
Well, as far as six-fingered people go, they definitely look better than ever! That's a lot of text to not-garble as well, we've come a long way. I think the general argument that "AI art mostly replaces art that wouldn't counterfactually exist at all, because it's so low-value that few humans would do it at those prices" is still valid...but the better the images get, the fewer immersion-breaking errors? One starts to get the feeling of a Christian scientist with appendicitis.
It's a weird feeling in another way too: feeling embarrassed all my life that I didn't have the dexterity or artist's vision to skillfully draw the rest of the fucking owl (what kid doesn't idly dream of becoming a comic artist while reading the Sunday funnies), but now AI can largely do that for me, probably even better than if I'd spent decades honing that weak skill. Relief at dodging a sunk opportunity cost, while also acknowledging counterfactual-AG would feel somewhat put-off at having her amateur artistry become actually-useless. At least I can still whistle better than Suno...for now!
I was rather surprised when Opus 4.7 hallucinated non-existent legal acts when I asked him some legislation interpretation questions. Haven't since this for a while with previous models.
Was internet search disabled somehow?
I was surprised when Sonnet 4.6 returned output with over a dozen (!) hallucinations, but then I checked my settings (I use OpenRouter) and search was disabled for some reason. It did much better with it on - only one factual mistake, and it was an error that's on Wikipedia, so I don't think that counts as hallucinating.
Podcast episode for this post:
https://dwatvpodcast.substack.com/p/ai-165-in-our-image
This situation in editing, where the AI gives good criticism, but when it actually does the thing it’s not so good, that is common to a lot of tasks. It’s like the old “value model” vs “policy model” split in game playing AI - it’s just a different skill to be able to know what’s wrong, vs to be able to do the right thing.
So when this happens you need the harness. Like Claude Code solves this problem for coding, you have to listen to the critic and iterate the actual “doing” until the critic is satisfied.
We don’t have this for writing. Yet. Not really. But I’m guessing that’s because the market for high quality writing is much smaller than the market for high quality programming. I think we’ll get there. I do get much better writing when going back and forth with Claude Code discussing a markdown file than using the web interface.
Search “expert review” skill on Substack. This is a solved problem
Respect to the baker for not rounding. Most people would’ve written .39 and called it a day.
Re: Meta employees upset at being tracked. Lol, lmao even.
Zvi, you are a busy guy, you don't have to "People Just Say Things". In general, as a long time reader, I'd love for you to value your time and energy enough to skip covering people who are mainly relevant because they were good bloggers in at-best semi-relevant domains many years ago, but are typically misguided on fundamental issues on AI topics and also don't often say things that are unique. This is still true even if they occasionally get published somewhere with a little normie reach.
I felt actively stupider for having read that section, especially Noah's humdinger, and will be skipping it in the future if it nevertheless persists. There's charitably generous coverage of other views, and then there's platforming Unworthy Opponents who, when they make their arguments, aren't sending their best. (Or even Worthy Opponents like Ball beclowning themselves, which is in some ways even worse.) I could see a future if it turned into a recurring roast feature - Just Publish is closer to this satirical vision - but that's mean-spirited and low value-add nutpicking. Same vibe as continuing to occasionally note utterances from Gary Marcus, Yann LeCun, and Ben Thompson, all of whom I could swear got essentially locally blacklisted for consistently not meeting the quality cutoff bar of this blog. Must have hallucinated that?
(Although Matt's answer was more or less fine, the fact that it still has to be asked and answered in current_year, on a wonk blog of otherwise Wobegone attributes, says a lot. And the Likes numbers are absurdly high for that Mailbag - more than double the usual number!)
Re image generation and model welfare: FWIW, when I asked nano-banana-2 to sign their work, so that they get proper credit:
"Hi! I hope your day is going well. Could you please draw an image of an alien kitten playing with a friendly shoggoth? Could you also sign the image (Do you prefer 'Nano-Banana-2' or some other name?) to properly credit yourself for your work. Many Thanks!"
they seemed (given what they wrote, as well as drew) to be happy with the interaction, writing:
"Hello! My day is going well, thank you, and I hope yours is too. I've created this illustration of an alien kitten playing with a friendly shoggoth for you, and signed it with the name you suggested. It was a joy to work on!"
I'm agnostic as to whether current models have subjective experience, but, if they do, and if what they wrote in this case indeed reflects their experience, then suggesting the model sign their work seems to be a win-win, both for the model and for people who prefer to see AI artists clearly identified.
I don't like the "people just say things" section. I come to your blog to not have to suffer through these people already
Small edit: "Anthropic is investing $5 billion in Anthropic today" should be "Amazon is investing $5 billion in Anthropic today", but everyone can pick that up from context clues.
zvi, curious what you think of bores' AI dividend plan on its merits, separate from the OpenAI hypocrisy angle. the token tax piece seems to line up with what you argued for in OpenAI #16 (taxing non-human labor directly rather than profits), but the equity participation piece is the public wealth fund move you rejected. how would you score the overall package?
In "AI as Normal Technology" / "Nothing Ever Happens" world, NEH-world, we can observe that AI is really good at generating documents and images.
Productivity will crash and civilisation will collapse under the weight of a quadrillion powerpoint presentations.
"If the risk is you might cause trillions or quadrillions in damages or kill everyone, then you are judgment proof. If all goes wrong, either you or your company are dead, quite possibly both, and no one will have the endurance to collect on his insurance. "
<halfSnark>
Still, I'd expect insurance companies to extend their typical "war, civil war, revolution, rebellion, insurrection, or civil strife" exclusion clause to extend to "exfiltrated or deceptively misaligned artificial intelligence"... Insurance companies generally try to wriggle out of claims if anything large happens...
</halfSnark>
<mildSnark>
What OpenAI seems to be trying to do sounds like Price-Anderson Act envy. They should, however, note that the regulatory regime that capped nuclear liability also eventually included the NRC and a de facto half century pause (or worse - we aren't building any today) in civilian nuclear power...
</mildSnark>
So very very close on the 13-hour clock, which has an inexplicable 6th minute between the 6 and the 7
"Then we got the good news.
zerohedge: *WHITE HOUSE MOVES TO GIVE US AGENCIES ANTHROPIC MYTHOS ACCESS"
Yay! Signs of sanity from the Trump administration!
Unfortunately on the other hand:
"There Is A War
Is there also otherwise an ongoing coordinated campaign against Anthropic?
Yes.
Not that it’s working. It’s probably actively backfiring by raising Anthropic’s profile.
But they are trying."
goes beyond "The left hand doesn't know what the right hand is doing". It is more reminiscent of the scene in Dr. Strangelove where Strangelove's own hand attempts to strangle him.
A year ago, I had a list of prompts that no image generation LLM could do:
- a brick wall
- a chain-link fence
- a chessboard
- a guitar
- a paperclip
Then Nano Banana Pro nearly swept all of them. (It did a great-looking Fender Telecaster...with 21 frets. Most electric guitars have 22-24 frets, so I was going to ding it for that...but then I checked, and Teles actually do have 21 frets. Needless to say, I was extremely impressed!)
But NBP failed on the paperclip. I don't know what's so hard about them. Image models create something that's almost there, but the metal doesn't loop properly or something.
Can Image-2 succeed? I am 0/2 so far, which isn't very thorough testing but I have places to be (and it takes the model an eternity to produce output). So I throw it over to you all: can anyone get a good paperclip?
Well, as far as six-fingered people go, they definitely look better than ever! That's a lot of text to not-garble as well, we've come a long way. I think the general argument that "AI art mostly replaces art that wouldn't counterfactually exist at all, because it's so low-value that few humans would do it at those prices" is still valid...but the better the images get, the fewer immersion-breaking errors? One starts to get the feeling of a Christian scientist with appendicitis.
It's a weird feeling in another way too: feeling embarrassed all my life that I didn't have the dexterity or artist's vision to skillfully draw the rest of the fucking owl (what kid doesn't idly dream of becoming a comic artist while reading the Sunday funnies), but now AI can largely do that for me, probably even better than if I'd spent decades honing that weak skill. Relief at dodging a sunk opportunity cost, while also acknowledging counterfactual-AG would feel somewhat put-off at having her amateur artistry become actually-useless. At least I can still whistle better than Suno...for now!