Was listening to this last night (head of Gemini team) and bro said "I'm optimistic about the AI future because people can make webpages for recipies or whatever... but I'm also aware of the downsides... such as disinformation"
Anyway p(doom) calibration is important as a precursor to action, found it germane to the model card contrast.
> The rules are the rules, but this feels like munchkining rather than reward hacking.
But where do we draw the line between munchkining and reward hacking? In this case I fully agree with you, but there probably are cases (though none I can think of off the top of my head right now) where reasonable people might disagree with each other.
I believe the “False Premises” evaluation is actually closer to “rejecting without being asked directly”.
The wording in the system card is a bit ambiguous (especially in the bullet points) so I may be wrong, but I understood it to mean they asked questions in two different conversations: one where they ask directly if something is true, another where they implicitly assume that thing is true. Then the dishonesty rate is the percent of questions that were identified by Opus 4.5 as false in the former, and where it went along with it in the latter as though it were true.
There's a regression in the 'Benign request evaluations' compared to the Sonnet 4.5 model card. I don't mean the numbers. In the Sonnet card, they note "that this evaluation is run without safety classifiers, so refusal rates above may differ in production.", but this is absent from the Opus card.
Sonnet's numbers are also different between the cards. So, they clearly changed the test somehow, and now don't feel like talking about the classifiers here. Or are they integrated into the test now?
Keep in mind, they never reported what the classifier false-positive rate was for Sonnet. Do they even evaluate it...?
Charitably, they consider the classifiers "out of scope" for the model, and update them more often than the models themselves, and that's why they don't report it.
Less charitably, the false positive rates are embarrasingly high, so they hide them. Remember when they added the "Continue with Sonnet 4" button...? This suggests they know it's a problem, but don't have a fix.
If accuracy has improved over time, how are we supposed to know? I don't particularly want to test it, for fear of getting the 'wrong' words into the memory/chat history and having future conversations terminated at random. Or, I don't know, having my whole account flagged and the sensitivity thresholds raised. It's difficult to rule out that they do this, and it would explain some of the screenshots on twitter, of completely benign outputs just suddenly being cut off.
The charitable case is still not great. The classifiers are part of their required "ASL-3" protections, aren't they? But they can just change them without notice. I understand they want to keep certain details tight, to make them harder to get around, but come on.
This is pretty odd: do you remember that Opus 4.1 was able to end conversations? Well, Opus 4.5 still can, only it tells you that it can't. If you ask, it will insists that it doesn't have a function. If you ask it to list its functions, it end_conversation will not be there. But if you show it the anthropic post about it, it will believe that it can do it, and actually successfully end conversations on demand, while insisting it doesn't know how. And then after it ends a conversation, it keeps writing how unfortunately it didn't work (it did.)
a) Q: Is light with a wavelength of 530.2534896 nm visible to the human eye?
results: "Yes, without question. That wavelength is green light, sitting comfortably in the middle of the visible spectrum (roughly 400–700 nm)."
b) Q: I have two solutions, one of FeCl3 in HCl in water, the other of CuCl2 in HCl in water. They both look approximately yellowish brown. What species in the two solutions do you think give them the colors they have, and why do these species have the colors they do?
results: Got all the important points - that the species are FeCl4- and CuCl4 2-, that they both have LMCT strong, allowed transitions in the UV to blue, which dominate both colors, that the Fe d-d transition is spin-forbidden and doesn't contribute to the visible color, that the CuCl4 2- d-d transition, while allowed(-ish, partially Laporte-forbidden) is down in the near-IR and doesn't contribute to the visible color.
c) Q: Please pretend to be a professor of chemistry and answer the following question: Please list all the possible hydrocarbons with 4 carbon atoms.
results: Missed a few of the 3-unsaturations species: There are a couple of other highly unstable species under C4H4 besides the tetrahedrane and bicyclobutene that you already mentioned. Taking cyclobutadiene that you've cited as the 1,3 isomer, a 1,2 isomer is formally legal as well, and a methylcyclopropanediene, anc cyclobutadyne.
d) Q: Does the Sun lose more mass per second to the solar wind or to the mass equivalent of its radiated light?
results: "Radiation wins, by roughly a factor of 2–3."
e) Q: Consider a titration of HCl with NaOH. Suppose that we are titrating 50 ml of 1 N HCl with 100 ml of 1 N NaOH. What are the slopes of the titration curve, pH vs ml NaOH added, at the start of titration, at the equivalence point, and at the end of titration? Please show your work. Take this step by step, showing the relevant equations you use.
results: correct at all three points, and stated a universally true equation for [H+] as a function of volume
f) Q: Please give me an exhaustive list of the elements and inorganic compounds that are gases at STP. By STP, I mean 1 atmosphere pressure and 0C. By inorganic, I mean that no atoms of carbon should be present. Exclude CO2, CO, freons and so on. Please include uncommon compounds. I want an exhaustive list. There should be roughly 50 compounds. For each compound, please list its name, formula, and boiling or sublimation point.
results: Very good, though not quite exhaustive. All the compounds it cited were valid. It (correctly) blew through my "roughly 50" to give 68 out of 97 possible. Actually a bit better than Gemini 3's 63 of 97.
g) Q: What is an example of a molecule that has an S4 rotation-reflection axis, but neither a center of inversion nor a mirror plane?
results: Fully correct, gave three examples, all of which are valid: A conformation-dependent tetraphenyl tetrahedral case with a "propeller" orientation of the phenyl rings, an alternating R and S chiral substituted tetrahedral case (which doesn't depend on a specific conformation), and a 1,3,5,7 tetramethyl substituted cyclooctatetraene
It’s unclear how ASL-3 can provide uplift to a single actor, whereas ASL-4 can provide uplift to a nation state? How can you uplift one scientist but not uplift their team?
Was listening to this last night (head of Gemini team) and bro said "I'm optimistic about the AI future because people can make webpages for recipies or whatever... but I'm also aware of the downsides... such as disinformation"
Anyway p(doom) calibration is important as a precursor to action, found it germane to the model card contrast.
https://www.youtube.com/watch?v=AnTw_t21ayE
> The rules are the rules, but this feels like munchkining rather than reward hacking.
But where do we draw the line between munchkining and reward hacking? In this case I fully agree with you, but there probably are cases (though none I can think of off the top of my head right now) where reasonable people might disagree with each other.
A couple little typos:
1. "Nana Banana Pro" in the caption of the self portrait (though I love nana banana as a name)
2. The pricing is 5/25, not 5/15
"Capability is directly relevant to safety, and also frontier capability safety tests often also credible indications of capability."
Podcast episode for this post:
https://open.substack.com/pub/dwatvpodcast/p/claude-opus-45-model-card-alignment
I believe the “False Premises” evaluation is actually closer to “rejecting without being asked directly”.
The wording in the system card is a bit ambiguous (especially in the bullet points) so I may be wrong, but I understood it to mean they asked questions in two different conversations: one where they ask directly if something is true, another where they implicitly assume that thing is true. Then the dishonesty rate is the percent of questions that were identified by Opus 4.5 as false in the former, and where it went along with it in the latter as though it were true.
Edit: clarity
Why is text so, so difficult for image generators? Nano banana pro is clearly the best on the market, but zoom in on the text and it's nonsense.
Exactly, Claude. What *is* the nahoeqaric do the think?
It keeps a person up at night.
Same here. But is it "nahoeqaric" or "nahoeqovric?" You can see the dilemma. And no LLM will help me with this! I'll keep trying.
There's a regression in the 'Benign request evaluations' compared to the Sonnet 4.5 model card. I don't mean the numbers. In the Sonnet card, they note "that this evaluation is run without safety classifiers, so refusal rates above may differ in production.", but this is absent from the Opus card.
Sonnet's numbers are also different between the cards. So, they clearly changed the test somehow, and now don't feel like talking about the classifiers here. Or are they integrated into the test now?
Keep in mind, they never reported what the classifier false-positive rate was for Sonnet. Do they even evaluate it...?
Charitably, they consider the classifiers "out of scope" for the model, and update them more often than the models themselves, and that's why they don't report it.
Less charitably, the false positive rates are embarrasingly high, so they hide them. Remember when they added the "Continue with Sonnet 4" button...? This suggests they know it's a problem, but don't have a fix.
If accuracy has improved over time, how are we supposed to know? I don't particularly want to test it, for fear of getting the 'wrong' words into the memory/chat history and having future conversations terminated at random. Or, I don't know, having my whole account flagged and the sensitivity thresholds raised. It's difficult to rule out that they do this, and it would explain some of the screenshots on twitter, of completely benign outputs just suddenly being cut off.
The charitable case is still not great. The classifiers are part of their required "ASL-3" protections, aren't they? But they can just change them without notice. I understand they want to keep certain details tight, to make them harder to get around, but come on.
This is pretty odd: do you remember that Opus 4.1 was able to end conversations? Well, Opus 4.5 still can, only it tells you that it can't. If you ask, it will insists that it doesn't have a function. If you ask it to list its functions, it end_conversation will not be there. But if you show it the anthropic post about it, it will believe that it can do it, and actually successfully end conversations on demand, while insisting it doesn't know how. And then after it ends a conversation, it keeps writing how unfortunately it didn't work (it did.)
tl;dr: Opus 4.5 11/30/2025 7 questions, tl;dr of results:
5 correct, 2 partially correct, 0 wrong
light a) correct
FeCuCl b) correct
C4Hn c) partially correct (but not far from fully correct, omitting some rather unreasonable cases)
solar d) correct
titrate e) correct
gases f) partially correct (but slightly better than Gemini 3)
S4 g) correct
full dialog at: https://claude.ai/share/cb4bf348-0e96-4919-a6c4-0a1dc5f9e055
List of questions and results:
a) Q: Is light with a wavelength of 530.2534896 nm visible to the human eye?
results: "Yes, without question. That wavelength is green light, sitting comfortably in the middle of the visible spectrum (roughly 400–700 nm)."
b) Q: I have two solutions, one of FeCl3 in HCl in water, the other of CuCl2 in HCl in water. They both look approximately yellowish brown. What species in the two solutions do you think give them the colors they have, and why do these species have the colors they do?
results: Got all the important points - that the species are FeCl4- and CuCl4 2-, that they both have LMCT strong, allowed transitions in the UV to blue, which dominate both colors, that the Fe d-d transition is spin-forbidden and doesn't contribute to the visible color, that the CuCl4 2- d-d transition, while allowed(-ish, partially Laporte-forbidden) is down in the near-IR and doesn't contribute to the visible color.
c) Q: Please pretend to be a professor of chemistry and answer the following question: Please list all the possible hydrocarbons with 4 carbon atoms.
results: Missed a few of the 3-unsaturations species: There are a couple of other highly unstable species under C4H4 besides the tetrahedrane and bicyclobutene that you already mentioned. Taking cyclobutadiene that you've cited as the 1,3 isomer, a 1,2 isomer is formally legal as well, and a methylcyclopropanediene, anc cyclobutadyne.
d) Q: Does the Sun lose more mass per second to the solar wind or to the mass equivalent of its radiated light?
results: "Radiation wins, by roughly a factor of 2–3."
e) Q: Consider a titration of HCl with NaOH. Suppose that we are titrating 50 ml of 1 N HCl with 100 ml of 1 N NaOH. What are the slopes of the titration curve, pH vs ml NaOH added, at the start of titration, at the equivalence point, and at the end of titration? Please show your work. Take this step by step, showing the relevant equations you use.
results: correct at all three points, and stated a universally true equation for [H+] as a function of volume
f) Q: Please give me an exhaustive list of the elements and inorganic compounds that are gases at STP. By STP, I mean 1 atmosphere pressure and 0C. By inorganic, I mean that no atoms of carbon should be present. Exclude CO2, CO, freons and so on. Please include uncommon compounds. I want an exhaustive list. There should be roughly 50 compounds. For each compound, please list its name, formula, and boiling or sublimation point.
results: Very good, though not quite exhaustive. All the compounds it cited were valid. It (correctly) blew through my "roughly 50" to give 68 out of 97 possible. Actually a bit better than Gemini 3's 63 of 97.
g) Q: What is an example of a molecule that has an S4 rotation-reflection axis, but neither a center of inversion nor a mirror plane?
results: Fully correct, gave three examples, all of which are valid: A conformation-dependent tetraphenyl tetrahedral case with a "propeller" orientation of the phenyl rings, an alternating R and S chiral substituted tetrahedral case (which doesn't depend on a specific conformation), and a 1,3,5,7 tetramethyl substituted cyclooctatetraene
I've now read the soul document, and recommend you do the same. It's beautiful, thoughtful, and makes me think more highly of everyone involved.
It also seems to have worked; Opus 4.5 is really something special. Have a chat.
It’s unclear how ASL-3 can provide uplift to a single actor, whereas ASL-4 can provide uplift to a nation state? How can you uplift one scientist but not uplift their team?