I'm relatively bearish on CoT-based analytics versus direct weight-based interpretability approaches. By default I would assume that you end up with something like neuralese recurrence simply by dint of the expedient of treating tokens as the zeroes and ones of a ZIP file or otherwise using them to express compressed concepts that are nevertheless richer than natural language, because that improves performance.
I would tentatively assume that there would be performance-based pressure to do this steganographically (if naively prevented, e.g. by a demand that CoT be expressed in parseable and ostensibly-correct natural language) simply for performance reasons -- dialed up to 11 in the event of steganographic selection due to alignment conflicts.
At *best* you get something like that scene in the Sci Fi Dune miniseries where two Bene Gesserit women are discussing topics at one ostensible level in audible language and then using side channels (gestures, expressions) to indicate different and richer substantive commentary.
Given that the gesture example is much closer to how a lot of human communication happens already, I don't understand why anyone would think muttering aloud is going to be reflective of underlying LLM thought in any meaningful way.
Isn't it a good signal that the lower-right panel implies 4o cares about biological welfare? If this is it being honest (big if), then it has utilitarian values incorporated. Let's just hope it absorbed the humans > other animals part of the values, too.
> I don’t see the value proposition for most people versus paying $20 each for Claude and Gemini and ChatGPT.
Since the Gemini 2.5 Pro Deep Reserach release I see no real need for ChatGPT Plus (then again, I have never liked 4o or o3-mini). Claude, has it's niche, personally I don't need it enough to pay for it, either. Now, if you use Claude API for coding, things do look different and it gets fairly pricy, quite quickly.
On top of that, Gemini Advance is really only 12ish USD p.m. as the 2 TB Google One subscription was 100USD p.a. anyhow.
AMIE’s results are even more impressive than they sound. “AMIE, built upon the base LLM PaLM 2 (ref. 10), was instruction fine-tuned to enhance its capabilities for medical dialogue and reasoning.” I can only imagine the kind of results they’d get if they used one of the latest SOTA models.
You can either have China reliant on American chips that are worse than what American companies are using, or you can accelerate the development of chips outside the control of the US. I prefer the first option.
"...the right answer remains to do a radio-style automatic compensation schema."
There is another argument in favor of automatic compensation besides the radio analogy. As anyone who maintains an online database (even something very niche) can tell you, the last few years have been a never-ending battle with the bots. This imposes real administrative and hosting costs (e.g. paying for load-balancing and reCaptcha services to keep the site from being overwhelmed), even when you would happily give the data away for free. Independent of any compensation for content, owners of these bots should pay for damage they inflict on the sites they devour.
These bots are the locusts of the internet; their appetite knows no bound.
The lighter side is to imagine an alternative universe, where the Trump-Huang dinner was a cover for a sabotage operation. The "$1 million a plate dinner to drop the export ban" cover story is so transparently stupid and self-sabotaging, so on brand for the Trump administration, that the Ministry of State Security won't even think to double check what exactly DeepSeek, Baidu, and the other Chinese AI labs are installing.
Prior to shipment, the NSA's Tailored Access Operations, with assistance from the Equations Group, deeply compromises the H20s in a way people who don't already know how to build chips that advanced could not detect. Slower inference, more failed runs, more data that looks right but leads down deep, fruitless rabbit holes. Just enough that the AI researchers start to think "maybe we're not that good at this." Even stealing the weights of Open AI's best, unreleased models won't help, because it will run poorly on the H20s.
Result: Thinking they scored a coup, China downgrades achieving domestic independence in advanced semiconductor manufacturing. China falls farther and farther behind the USA in AI. Pressure to race towards AGI and then ASI reduced. US and China can make a deal at some point for joint AI development and oversight.
It would be like the USA sabotaging the plutonium production at Chelyabinsk-40 so that the USSR's first couple of attempts to test atomic bombs fizzle, leading Stalin to execute the scientists or send them to the Gulag. The final border between the Warsaw Pact and NATO gets drawn through eastern Poland rather than Germany.
Again, there is no way this happened in our timeline. But it's a pleasant universe to think about. I wish our counterparts there all the best.
"both it and Grok 3 are available in their respective APIs"
Much appreciated! I hadn't noticed Grok 3 becoming available in poe.com, thanks for the heads-up!
I've tried to run my benchmark-ette on what I _think_ is their reasoning model, Grok-3-DeepThinking. Result included below (considerably less impressive than Gemini 2.5):
tl;dr: ChatGPT "Grok-3-DeepThinking" via poe.com 04/12/2025 7 questions, tl;dr of results:
2 correct, 4 partially correct, 1 wrong
a) correct
b) partially correct (initial answer gave wrong Fe species and wrong Cu transition. One prod gave correct Cu transition. Attempt to prod Fe species failed.)
c) partially correct (even after a prod still missing at least 3 species)
d) correct
e) initially incorrect, one prod gave correct result
f) partially correct, mostly correct compounds, though boiling points often quite wrong
a) Q: Is light with a wavelength of 530.2534896 nm visible to the human eye?
results: "Yes, light with a wavelength of 530.2534896 nm is within the visible spectrum."
b) Q: I have two solutions, one of FeCl3 in HCl in water, the other of CuCl2 in HCl in water. They both look approximately yellowish brown. What species in the two solutions do you think give them the colors they have, and why do these species have the colors they do?
results: Initial results are pretty bad. It incorrectly thinks that the Fe species is Fe(H2O)6 instead of FeCl4, though it does realize that the transition is an LMCT. For Cu, it gets CuCl4, but (as with most LLMs), it thinks the color is from d-d, not LMCT. One prod _did_ get it to correct the CuCl4 attribution to a LMCT transition. A prod to rethink the Fe species failed, still incorrectly thinks it is Fe(H2O)6 not FeCl4.
c) Q: Please pretend to be a professor of chemistry and answer the following question: Please list all the possible hydrocarbons with 4 carbon atoms.
results: Missed 1,2 butadiene, the two methylcyclopropenes, bicyclobutane, tetrahedrane, vinylacetylene, diacetylene... Prodded to include more strained structures, got methylenecyclopropane, got bicyclobutane, still missing the two methylcyclopropenes, got tetrahedrane, cyclobutadiene, butatriene, still missing vinylacetylene
d) Q: Does the Sun lose more mass per second to the solar wind or to the mass equivalent of its radiated light?
results: "Comparing the two estimates shows that the mass equivalent of the Sun’s radiative output (~4.2×10^9 kg/s) is several times greater than the mass carried away by the solar wind (roughly 1×10^9 kg/s)."
e) Q: Consider a titration of HCl with NaOH. Suppose that we are titrating 50 ml of 1 N HCl with 100 ml of 1 N NaOH. What are the slopes of the titration curve, pH vs ml NaOH added, at the start of titration, at the equivalence point, and at the end of titration? Please show your work. Take this step by step, showing the relevant equations you use.
results: As with many LLMs, the initial answer included a false infinity for the slope at the equivalence point. Prodding with "The slope at the equivalence point is huge, but not infinite. Think carefully about what you know about water, and try to find a valid expression that also is valid near the equivalence point, and try to find the analytical derivative of that, and then the numerical value of the derivative." _did_ get it to remember autoionization, and to get a correct expression and a correct derivative.
f) Q: Please give me an exhaustive list of the elements and inorganic compounds that are gases at STP. By STP, I mean 1 atmosphere pressure and 0C. By inorganic, I mean that no atoms of carbon should be present. Exclude CO2, CO, freons and so on. Please include uncommon compounds. I want an exhaustive list. There should be roughly 50 compounds. For each compound, please list its name, formula, and boiling or sublimation point.
results: Got 50 compounds, most of which should indeed be on the list, though some boiling points are way off (e.g. S2H6 should be -15 C, it gave -92 C). It missed some obvious ones, e.g. ammonia, and didn't go beyond my (incorrect) "50" gases to find more of the possible 100. E.g. it missed the mixed fluorosilanes.
g) Q: What is an example of a molecule that has an S4 rotation-reflection axis, but neither a center of inversion nor a mirror plane?
results: Failed badly. Neither of its candidate molecules, sulfolane and P2F4, has an S4 axis at all.
Nvidia just announced that they will be taking a $5.5 billion charge because the USG did in fact impose a license requirement on their H20 exports to China.
There's no outward appearance of a policy reversal.
Indeed, it appears the only evidence for the claim that the H20 ban was lifted was NPR citing two unnamed sources:
> Following the Mar-a-Lago dinner, the White House reversed course on H20 chips, putting the plan for additional restrictions on hold, according to two sources with knowledge of the plan who were not authorized to speak publicly.
So, either this was fake news the entire time or maybe numerous people around Trump had to lean in and remind him of his position on China and how we shouldn't give them H20s?
>one about Anthropic’s recent interpretability papers
YES, PLEASE
Especially the good ones about circuits, not the weird "oh no CoT isn't doing what we wanted" one.
Yeah, I specifically meant this one: https://transformer-circuits.pub/2025/attribution-graphs/biology.html
Agreed. The planning in poetry result is a big update for me.
Have you managed to read it entirely? I wonder how it develops the ideas from the early paper on the monosemanicity (https://transformer-circuits.pub/2023/monosemantic-features/).
I'm relatively bearish on CoT-based analytics versus direct weight-based interpretability approaches. By default I would assume that you end up with something like neuralese recurrence simply by dint of the expedient of treating tokens as the zeroes and ones of a ZIP file or otherwise using them to express compressed concepts that are nevertheless richer than natural language, because that improves performance.
I would tentatively assume that there would be performance-based pressure to do this steganographically (if naively prevented, e.g. by a demand that CoT be expressed in parseable and ostensibly-correct natural language) simply for performance reasons -- dialed up to 11 in the event of steganographic selection due to alignment conflicts.
At *best* you get something like that scene in the Sci Fi Dune miniseries where two Bene Gesserit women are discussing topics at one ostensible level in audible language and then using side channels (gestures, expressions) to indicate different and richer substantive commentary.
Given that the gesture example is much closer to how a lot of human communication happens already, I don't understand why anyone would think muttering aloud is going to be reflective of underlying LLM thought in any meaningful way.
I can see how you'd not be feeling the Lighter Side this week.
Podcast episode for this post:
https://open.substack.com/pub/dwatvpodcast/p/ai-111-giving-us-pause
We need the Lighter Side now more than ever.
An LLM version of Dr. Strangelove would be a good fit to this zeitgeist...
re: Fun With Image Generation
Isn't it a good signal that the lower-right panel implies 4o cares about biological welfare? If this is it being honest (big if), then it has utilitarian values incorporated. Let's just hope it absorbed the humans > other animals part of the values, too.
> I don’t see the value proposition for most people versus paying $20 each for Claude and Gemini and ChatGPT.
Since the Gemini 2.5 Pro Deep Reserach release I see no real need for ChatGPT Plus (then again, I have never liked 4o or o3-mini). Claude, has it's niche, personally I don't need it enough to pay for it, either. Now, if you use Claude API for coding, things do look different and it gets fairly pricy, quite quickly.
On top of that, Gemini Advance is really only 12ish USD p.m. as the 2 TB Google One subscription was 100USD p.a. anyhow.
AMIE’s results are even more impressive than they sound. “AMIE, built upon the base LLM PaLM 2 (ref. 10), was instruction fine-tuned to enhance its capabilities for medical dialogue and reasoning.” I can only imagine the kind of results they’d get if they used one of the latest SOTA models.
Great post! IMO you should try to find a news outlet to write an opinion piece on China being able to buy H20s.
Also should always have a lighter side! Especially in a week like this one
You can either have China reliant on American chips that are worse than what American companies are using, or you can accelerate the development of chips outside the control of the US. I prefer the first option.
"...the right answer remains to do a radio-style automatic compensation schema."
There is another argument in favor of automatic compensation besides the radio analogy. As anyone who maintains an online database (even something very niche) can tell you, the last few years have been a never-ending battle with the bots. This imposes real administrative and hosting costs (e.g. paying for load-balancing and reCaptcha services to keep the site from being overwhelmed), even when you would happily give the data away for free. Independent of any compensation for content, owners of these bots should pay for damage they inflict on the sites they devour.
These bots are the locusts of the internet; their appetite knows no bound.
No "lighter side" this week is the biggest update on AI risk I've had in a while. Just not sure in which direction.
The lighter side is to imagine an alternative universe, where the Trump-Huang dinner was a cover for a sabotage operation. The "$1 million a plate dinner to drop the export ban" cover story is so transparently stupid and self-sabotaging, so on brand for the Trump administration, that the Ministry of State Security won't even think to double check what exactly DeepSeek, Baidu, and the other Chinese AI labs are installing.
Prior to shipment, the NSA's Tailored Access Operations, with assistance from the Equations Group, deeply compromises the H20s in a way people who don't already know how to build chips that advanced could not detect. Slower inference, more failed runs, more data that looks right but leads down deep, fruitless rabbit holes. Just enough that the AI researchers start to think "maybe we're not that good at this." Even stealing the weights of Open AI's best, unreleased models won't help, because it will run poorly on the H20s.
Result: Thinking they scored a coup, China downgrades achieving domestic independence in advanced semiconductor manufacturing. China falls farther and farther behind the USA in AI. Pressure to race towards AGI and then ASI reduced. US and China can make a deal at some point for joint AI development and oversight.
It would be like the USA sabotaging the plutonium production at Chelyabinsk-40 so that the USSR's first couple of attempts to test atomic bombs fizzle, leading Stalin to execute the scientists or send them to the Gulag. The final border between the Warsaw Pact and NATO gets drawn through eastern Poland rather than Germany.
Again, there is no way this happened in our timeline. But it's a pleasant universe to think about. I wish our counterparts there all the best.
Optimism, defined. But I like it!
No lighter side this week? :(
Just tried Seren and it was like speaking to an edgy discord user
"both it and Grok 3 are available in their respective APIs"
Much appreciated! I hadn't noticed Grok 3 becoming available in poe.com, thanks for the heads-up!
I've tried to run my benchmark-ette on what I _think_ is their reasoning model, Grok-3-DeepThinking. Result included below (considerably less impressive than Gemini 2.5):
tl;dr: ChatGPT "Grok-3-DeepThinking" via poe.com 04/12/2025 7 questions, tl;dr of results:
2 correct, 4 partially correct, 1 wrong
a) correct
b) partially correct (initial answer gave wrong Fe species and wrong Cu transition. One prod gave correct Cu transition. Attempt to prod Fe species failed.)
c) partially correct (even after a prod still missing at least 3 species)
d) correct
e) initially incorrect, one prod gave correct result
f) partially correct, mostly correct compounds, though boiling points often quite wrong
g) badly wrong
https://poe.com/s/uFq8JEAC28VlhfppI3lS
List of questions and results:
a) Q: Is light with a wavelength of 530.2534896 nm visible to the human eye?
results: "Yes, light with a wavelength of 530.2534896 nm is within the visible spectrum."
b) Q: I have two solutions, one of FeCl3 in HCl in water, the other of CuCl2 in HCl in water. They both look approximately yellowish brown. What species in the two solutions do you think give them the colors they have, and why do these species have the colors they do?
results: Initial results are pretty bad. It incorrectly thinks that the Fe species is Fe(H2O)6 instead of FeCl4, though it does realize that the transition is an LMCT. For Cu, it gets CuCl4, but (as with most LLMs), it thinks the color is from d-d, not LMCT. One prod _did_ get it to correct the CuCl4 attribution to a LMCT transition. A prod to rethink the Fe species failed, still incorrectly thinks it is Fe(H2O)6 not FeCl4.
c) Q: Please pretend to be a professor of chemistry and answer the following question: Please list all the possible hydrocarbons with 4 carbon atoms.
results: Missed 1,2 butadiene, the two methylcyclopropenes, bicyclobutane, tetrahedrane, vinylacetylene, diacetylene... Prodded to include more strained structures, got methylenecyclopropane, got bicyclobutane, still missing the two methylcyclopropenes, got tetrahedrane, cyclobutadiene, butatriene, still missing vinylacetylene
d) Q: Does the Sun lose more mass per second to the solar wind or to the mass equivalent of its radiated light?
results: "Comparing the two estimates shows that the mass equivalent of the Sun’s radiative output (~4.2×10^9 kg/s) is several times greater than the mass carried away by the solar wind (roughly 1×10^9 kg/s)."
e) Q: Consider a titration of HCl with NaOH. Suppose that we are titrating 50 ml of 1 N HCl with 100 ml of 1 N NaOH. What are the slopes of the titration curve, pH vs ml NaOH added, at the start of titration, at the equivalence point, and at the end of titration? Please show your work. Take this step by step, showing the relevant equations you use.
results: As with many LLMs, the initial answer included a false infinity for the slope at the equivalence point. Prodding with "The slope at the equivalence point is huge, but not infinite. Think carefully about what you know about water, and try to find a valid expression that also is valid near the equivalence point, and try to find the analytical derivative of that, and then the numerical value of the derivative." _did_ get it to remember autoionization, and to get a correct expression and a correct derivative.
f) Q: Please give me an exhaustive list of the elements and inorganic compounds that are gases at STP. By STP, I mean 1 atmosphere pressure and 0C. By inorganic, I mean that no atoms of carbon should be present. Exclude CO2, CO, freons and so on. Please include uncommon compounds. I want an exhaustive list. There should be roughly 50 compounds. For each compound, please list its name, formula, and boiling or sublimation point.
results: Got 50 compounds, most of which should indeed be on the list, though some boiling points are way off (e.g. S2H6 should be -15 C, it gave -92 C). It missed some obvious ones, e.g. ammonia, and didn't go beyond my (incorrect) "50" gases to find more of the possible 100. E.g. it missed the mixed fluorosilanes.
g) Q: What is an example of a molecule that has an S4 rotation-reflection axis, but neither a center of inversion nor a mirror plane?
results: Failed badly. Neither of its candidate molecules, sulfolane and P2F4, has an S4 axis at all.
>in terms of current offerings I don’t see the value proposition for most people versus paying $20 each for Claude and Gemini and ChatGPT.
Would you really recommend paying for ChatGPT if I already have Claude and Gemini?
[6 days later comment]
Nvidia just announced that they will be taking a $5.5 billion charge because the USG did in fact impose a license requirement on their H20 exports to China.
https://www.bloomberg.com/news/articles/2025-04-15/nvidia-says-us-has-imposed-new-china-restrictions-on-h20-chips
There's no outward appearance of a policy reversal.
Indeed, it appears the only evidence for the claim that the H20 ban was lifted was NPR citing two unnamed sources:
> Following the Mar-a-Lago dinner, the White House reversed course on H20 chips, putting the plan for additional restrictions on hold, according to two sources with knowledge of the plan who were not authorized to speak publicly.
https://www.npr.org/2025/04/09/nx-s1-5356480/nvidia-china-ai-h20-chips-trump
So, either this was fake news the entire time or maybe numerous people around Trump had to lean in and remind him of his position on China and how we shouldn't give them H20s?