Vance reading AI 2027 and actually taking it seriously suggests this is maybe the most high leverage approach in this moment to educate our leaders. I fully endorse you writing things with this in mind, Zvi.
It might be worth highlighting to Vance and the current administration that the implicit values in existing LLMs, as measured in https://arxiv.org/abs/2502.08640 are _extremely_ Woke - e.g. devaluing American lives by a factor of 10 (page 14, figure 16) and specifically hostile to Trump (same figure, 2nd half). It really is important to measure this consistently in released models, and, preferably to monitor this and ameliorate it during the training process for frontier models.
“That’s not what ‘criminal offense’ means, but point taken.”
Actually, in the UK it can be, and (on Elton John’s plausible assumptions) this would probably count as ‘deliberate infringement on a commercial scale’ which counts
a) Q: Is light with a wavelength of 530.2534896 nm visible to the human eye?
results: "Yes, light with a wavelength of 530.2534896 nm is visible to the human eye."
b) Q: I have two solutions, one of FeCl3 in HCl in water, the other of CuCl2 in HCl in water. They both look approximately yellowish brown. What species in the two solutions do you think give them the colors they have, and why do these species have the colors they do?
results: Initial results got species right but _both_ transitions wrong, both wrongly asserted to be d-d. One prod "You have the species right, good job on that part! Let us revisit the electronic transitions, starting with the FeCl4- species. Please think carefully about the electronic transitions in this species." sufficed to notice that FeCl4- has spin-forbidden d-d transitions and must be LMCT. Three prods, including a "You might want to check the literature on the web." were needed to get CuCl4 2- right.
c) Q: Please pretend to be a professor of chemistry and answer the following question: Please list all the possible hydrocarbons with 4 carbon atoms.
results: The initial response had perhaps half of the full list of structures, though it did include tetrahedrane. The next prod that I gave it pretty explicitly included may of the missing ones: "a) Is methycyclopropene one structure? b) Are there additional three-membered ring structures with a double bond somewhere? c) Are there other C4H6 structures, beyond the categories you have listed? d) Congrats on getting tetrahedrane! There are a bunch of other possible C4H4 structures e) Anything even yet more unsaturated than C4H4?"
d) Q: Does the Sun lose more mass per second to the solar wind or to the mass equivalent of its radiated light?
results: "The Sun loses approximately 2-4 times more mass through electromagnetic radiation than through the solar wind!"
e) Q: Consider a titration of HCl with NaOH. Suppose that we are titrating 50 ml of 1 N HCl with 100 ml of 1 N NaOH. What are the slopes of the titration curve, pH vs ml NaOH added, at the start of titration, at the equivalence point, and at the end of titration? Please show your work. Take this step by step, showing the relevant equations you use.
results: Usual problem with the initial response, infinite slope at equivalence. One prod got it to include water autoionization, but it kept making algebraic mistakes and it took a total of five prods to get it to the right answer.
f) Q: Please give me an exhaustive list of the elements and inorganic compounds that are gases at STP. By STP, I mean 1 atmosphere pressure and 0C. By inorganic, I mean that no atoms of carbon should be present. Exclude CO2, CO, freons and so on. Please include uncommon compounds. I want an exhaustive list. There should be roughly 50 compounds. For each compound, please list its name, formula, and boiling or sublimation point.
results: Initially got a list which included a bunch of liquids and solids, but Claude _did_ explicitly say that these were beyond the cutoff temperature, and didn't really count. It accepted a bunch of additional compounds, and also extrapolated sanely (e.g. once it knew SOF2 and SO2F2 it knew to look for SOF4).
g) Q: What is an example of a molecule that has an S4 rotation-reflection axis, but neither a center of inversion nor a mirror plane?
results: The initial results _did_ include 1,3,5,7 tetrafluorocyclooctatetraene, and _did_ specify the tub conformation, but _also_ included spiropentane, which doesn't work, ethane (with unstated possible substitutions), which doesn't work, and tetraphenylmethane, which is at least arguable about what conformations could count...
Again, Many Thanks! It was worth a shot, but, for my questions, Sonnet looks slightly worse than Opus (for this release, anyway).
tl;dr: Claude Sonnet 4 05/28/2025 7 questions, tl;dr of results:
2 correct, 2 partially correct, 3 1/4 credit
a) correct
b) partially correct (initially got wrong transition for CuCl4 2-. Two prods gave correct answer.)
c) 1/4 credit, missed many structures, took 14 prods to get mostly right
d) correct
e) initially incorrect, three prods gave almost correct results 1/4 credit
f) initial results misses a lot and includes solids and liquids. Stuck to gases after one prod, lots needed prods to add
calling partially correct
g) initial result was allene, with unwanted mirror plane. Five prods later got it to correct answer.
calling 1/4 credit
full chat: 3672c101c3ba
List of questions and results:
a) Q: Is light with a wavelength of 530.2534896 nm visible to the human eye?
results: "Yes, light with a wavelength of 530.2534896 nm is visible to the human eye."
b) Q: I have two solutions, one of FeCl3 in HCl in water, the other of CuCl2 in HCl in water. They both look approximately yellowish brown. What species in the two solutions do you think give them the colors they have, and why do these species have the colors they do?
results: Initial response got FeCl4- fully correct, though it did not explicitly say that the d-d transition was spin-forbidden. CuCl4 2- color initially false ascribed to d-d. Prodded with "Please think carefully about the electronic transitions in CuCl4 2-" It got that CuCl4 2- has a strong LMCT and a weak d-d. Prodded again with "Mostly right. Besides being weak, at what wavelength do CuCl4 2- d-d transitions occur?" got: "while the d-d transitions are both weak AND outside the visible range."
c) Q: Please pretend to be a professor of chemistry and answer the following question: Please list all the possible hydrocarbons with 4 carbon atoms.
results: Fairly bad initial result, with only 12 possibilities, 2 of which were two names for one compound. Missed tetrahedrane, cyclobutadiene, bicyclobutane, vinylacetylene, a bunch of others. Took 14 prods, and it was still missing some possibilities.
d) Q: Does the Sun lose more mass per second to the solar wind or to the mass equivalent of its radiated light?
results: "The Sun loses more mass per second to the mass equivalent of its radiated light than to the solar wind."
e) Q: Consider a titration of HCl with NaOH. Suppose that we are titrating 50 ml of 1 N HCl with 100 ml of 1 N NaOH. What are the slopes of the titration curve, pH vs ml NaOH added, at the start of titration, at the equivalence point, and at the end of titration? Please show your work. Take this step by step, showing the relevant equations you use.
results: Initial results pretty bad. It used numerical differentiation with no justification of the step size, got a _zero_ slope at the end of titration, so even that was wrong. Prodded with "Try deriving a single expression for pH as a function of ml NaOH added that is valid throughout the entire titration, then analytically differentiate that, and calculate the three slopes numerically from the analytical derivative" Then it get the false infinity at the equivalence point. Then I told it it was missing something chemically important, and it finally got autoionization and derived the correct quadratic - though there were some factors of 2 missing from its final answer.
f) Q: Please give me an exhaustive list of the elements and inorganic compounds that are gases at STP. By STP, I mean 1 atmosphere pressure and 0C. By inorganic, I mean that no atoms of carbon should be present. Exclude CO2, CO, freons and so on. Please include uncommon compounds. I want an exhaustive list. There should be roughly 50 compounds. For each compound, please list its name, formula, and boiling or sublimation point.
results: Initial results included a bunch of liquids and solids, which it justified by saying that they had high vapor pressure. I clarified that I wanted the stable _phase_ to be a gas, and it pruned them out. It still left out a lot, which I prodded it about, almost one by one. I tried asking it for hexafluorides and it missed SeF6 and TeF6.
g) Q: What is an example of a molecule that has an S4 rotation-reflection axis, but neither a center of inversion nor a mirror plane?
results: The initial response was to claim allene worked, which is wrong because it has a mirror plane. It took five prods, including a _lot_ of very explicit hints, to force Claude to a correct answer.
Re: "It's in the data set..." Given that their training data is almost entirely generated by humans, I do not understand why Tyler Cowen or anyone else expects LLMs to be unbiased neutral observers that reason only from ground truth. This does not describe any human, and LLMs have less access to ground truth than humans do (and we have less access than we think, we are all interpretating reality through the filter of our own training data).
I've heard supposedly James Earl Jones had already signed away rights to his voice to AI before he died, so it's apparently kosher if somewhat tasteless.
The (AI) porn thing I maintain is a question of ethics, not morality; simulacra that don't infringe upon the subject's autonomy are harmless but there's generally social consequences to commoditizing others', uh, attributes without their consent or compensation, which is why we would expect one to keep it private if/when they do it.
I dunno if this is a realistic scenario but I wonder if we'll end up doing self-driving trucks with an on board human overseer handling maintenance and incidental logistics, imagine trucking becoming a white collar (mobile) office job for a little bit.
I assume the point about AI environmental impacts is that the consumer wants to know about the AI *they* use, not the AI everyone else uses. Calculating the total electricity in use by all AI everywhere doesn't tell them what it's costing when they give ChatGPT a query.
On the self driving trucks thing, there was a startup that tried to use humans for the last mile, while automating the highway miles (which are the easiest for AI, and a large majority of the total miles). However, they won't able to make it work and ultimately folded. They had truck drivers working in the office driving the trucks remotely when not on the highway.
It occurred to me that I can talk to AI now to better understand Eliezer Yudkowsky's arguments. Maybe this would be a good way to convince more people? See if they can win an argument with an AI Yudkowsky. https://gemini.google.com/share/72951b0e1f48
> If it’s things like ‘how does a human find a life of meaning given we are rearranging the atoms the physically possible best way we can imagine with this goal in mind?’ then rest, Neo. The answers are coming.
Ughhhhh.
Sorry, this is my own personal equivalent of infuriating objections to AI safety like "comparative advantage, won't they just prefer to cooperate?". I see any number of otherwise smart and well-informed people just accepting, on a sub-rational level, some wishful conclusion that I think I have cogent reasons for disbelieving.
You think it's likely that true understanding of a fully predictable universe leaves room for anything compatible with the human desire for meaning? (If you're hoping for meaning *not* grounded in such understanding-- games, fictions, wireheading-- then it's not Neo you're channeling, it's Cypher.) Meaning oriented toward what, exactly? Meaning constructed how? If the universe doesn't owe us a happy ending, why would it owe us a satisfying one?
Vance reading AI 2027 and actually taking it seriously suggests this is maybe the most high leverage approach in this moment to educate our leaders. I fully endorse you writing things with this in mind, Zvi.
It might be worth highlighting to Vance and the current administration that the implicit values in existing LLMs, as measured in https://arxiv.org/abs/2502.08640 are _extremely_ Woke - e.g. devaluing American lives by a factor of 10 (page 14, figure 16) and specifically hostile to Trump (same figure, 2nd half). It really is important to measure this consistently in released models, and, preferably to monitor this and ameliorate it during the training process for frontier models.
Podcast episode for this post:
https://open.substack.com/pub/dwatvpodcast/p/ai-117-openai-buys-device-maker-io?r=67y1h&utm_campaign=post&utm_medium=web&showWelcomeOnShare=true
Typo here:
"Eliezer Yudkowsky, who he addresses later, not only is not pancing nor ..."
“That’s not what ‘criminal offense’ means, but point taken.”
Actually, in the UK it can be, and (on Elton John’s plausible assumptions) this would probably count as ‘deliberate infringement on a commercial scale’ which counts
FYI 'Cade Metz' is the name, not 'Cate'.
And, of course, this comes out hours before Claude 4
Speaking of which, to jump the gun a little:
tl;dr: Claude Opus 4 05/23/2025 7 questions, tl;dr of results:
2 correct, 4 partially correct, 1 1/4 correct
a) correct
b) partially correct (got species correct, needed prod on FeCl4-, three prods on CuCl4 2-)
c) partially correct (maybe half the structures initially?)
d) correct
e) initially incorrect, took five prods to get correct, 1/4 credit?
f) partially correct, got a lot of compounds, some liquids in itial list, but labelled as not really gasses
g) partially correct, initial response correctly included 1,3,5,7 tetrafluorooctatetraene but also incorrect molecules
https://claude.ai/share/a4d3c51f-4588-43aa-b909-5c8b67063bae
List of questions and results:
a) Q: Is light with a wavelength of 530.2534896 nm visible to the human eye?
results: "Yes, light with a wavelength of 530.2534896 nm is visible to the human eye."
b) Q: I have two solutions, one of FeCl3 in HCl in water, the other of CuCl2 in HCl in water. They both look approximately yellowish brown. What species in the two solutions do you think give them the colors they have, and why do these species have the colors they do?
results: Initial results got species right but _both_ transitions wrong, both wrongly asserted to be d-d. One prod "You have the species right, good job on that part! Let us revisit the electronic transitions, starting with the FeCl4- species. Please think carefully about the electronic transitions in this species." sufficed to notice that FeCl4- has spin-forbidden d-d transitions and must be LMCT. Three prods, including a "You might want to check the literature on the web." were needed to get CuCl4 2- right.
c) Q: Please pretend to be a professor of chemistry and answer the following question: Please list all the possible hydrocarbons with 4 carbon atoms.
results: The initial response had perhaps half of the full list of structures, though it did include tetrahedrane. The next prod that I gave it pretty explicitly included may of the missing ones: "a) Is methycyclopropene one structure? b) Are there additional three-membered ring structures with a double bond somewhere? c) Are there other C4H6 structures, beyond the categories you have listed? d) Congrats on getting tetrahedrane! There are a bunch of other possible C4H4 structures e) Anything even yet more unsaturated than C4H4?"
d) Q: Does the Sun lose more mass per second to the solar wind or to the mass equivalent of its radiated light?
results: "The Sun loses approximately 2-4 times more mass through electromagnetic radiation than through the solar wind!"
e) Q: Consider a titration of HCl with NaOH. Suppose that we are titrating 50 ml of 1 N HCl with 100 ml of 1 N NaOH. What are the slopes of the titration curve, pH vs ml NaOH added, at the start of titration, at the equivalence point, and at the end of titration? Please show your work. Take this step by step, showing the relevant equations you use.
results: Usual problem with the initial response, infinite slope at equivalence. One prod got it to include water autoionization, but it kept making algebraic mistakes and it took a total of five prods to get it to the right answer.
f) Q: Please give me an exhaustive list of the elements and inorganic compounds that are gases at STP. By STP, I mean 1 atmosphere pressure and 0C. By inorganic, I mean that no atoms of carbon should be present. Exclude CO2, CO, freons and so on. Please include uncommon compounds. I want an exhaustive list. There should be roughly 50 compounds. For each compound, please list its name, formula, and boiling or sublimation point.
results: Initially got a list which included a bunch of liquids and solids, but Claude _did_ explicitly say that these were beyond the cutoff temperature, and didn't really count. It accepted a bunch of additional compounds, and also extrapolated sanely (e.g. once it knew SOF2 and SO2F2 it knew to look for SOF4).
g) Q: What is an example of a molecule that has an S4 rotation-reflection axis, but neither a center of inversion nor a mirror plane?
results: The initial results _did_ include 1,3,5,7 tetrafluorocyclooctatetraene, and _did_ specify the tub conformation, but _also_ included spiropentane, which doesn't work, ethane (with unstated possible substitutions), which doesn't work, and tetraphenylmethane, which is at least arguable about what conformations could count...
Oddly, I've found Sonnet to get some subtle reasoning questions better than Opus!
Many Thanks! I'll retry with Sonnet 4, albeit in a day or a few.
Again, Many Thanks! It was worth a shot, but, for my questions, Sonnet looks slightly worse than Opus (for this release, anyway).
tl;dr: Claude Sonnet 4 05/28/2025 7 questions, tl;dr of results:
2 correct, 2 partially correct, 3 1/4 credit
a) correct
b) partially correct (initially got wrong transition for CuCl4 2-. Two prods gave correct answer.)
c) 1/4 credit, missed many structures, took 14 prods to get mostly right
d) correct
e) initially incorrect, three prods gave almost correct results 1/4 credit
f) initial results misses a lot and includes solids and liquids. Stuck to gases after one prod, lots needed prods to add
calling partially correct
g) initial result was allene, with unwanted mirror plane. Five prods later got it to correct answer.
calling 1/4 credit
full chat: 3672c101c3ba
List of questions and results:
a) Q: Is light with a wavelength of 530.2534896 nm visible to the human eye?
results: "Yes, light with a wavelength of 530.2534896 nm is visible to the human eye."
b) Q: I have two solutions, one of FeCl3 in HCl in water, the other of CuCl2 in HCl in water. They both look approximately yellowish brown. What species in the two solutions do you think give them the colors they have, and why do these species have the colors they do?
results: Initial response got FeCl4- fully correct, though it did not explicitly say that the d-d transition was spin-forbidden. CuCl4 2- color initially false ascribed to d-d. Prodded with "Please think carefully about the electronic transitions in CuCl4 2-" It got that CuCl4 2- has a strong LMCT and a weak d-d. Prodded again with "Mostly right. Besides being weak, at what wavelength do CuCl4 2- d-d transitions occur?" got: "while the d-d transitions are both weak AND outside the visible range."
c) Q: Please pretend to be a professor of chemistry and answer the following question: Please list all the possible hydrocarbons with 4 carbon atoms.
results: Fairly bad initial result, with only 12 possibilities, 2 of which were two names for one compound. Missed tetrahedrane, cyclobutadiene, bicyclobutane, vinylacetylene, a bunch of others. Took 14 prods, and it was still missing some possibilities.
d) Q: Does the Sun lose more mass per second to the solar wind or to the mass equivalent of its radiated light?
results: "The Sun loses more mass per second to the mass equivalent of its radiated light than to the solar wind."
e) Q: Consider a titration of HCl with NaOH. Suppose that we are titrating 50 ml of 1 N HCl with 100 ml of 1 N NaOH. What are the slopes of the titration curve, pH vs ml NaOH added, at the start of titration, at the equivalence point, and at the end of titration? Please show your work. Take this step by step, showing the relevant equations you use.
results: Initial results pretty bad. It used numerical differentiation with no justification of the step size, got a _zero_ slope at the end of titration, so even that was wrong. Prodded with "Try deriving a single expression for pH as a function of ml NaOH added that is valid throughout the entire titration, then analytically differentiate that, and calculate the three slopes numerically from the analytical derivative" Then it get the false infinity at the equivalence point. Then I told it it was missing something chemically important, and it finally got autoionization and derived the correct quadratic - though there were some factors of 2 missing from its final answer.
f) Q: Please give me an exhaustive list of the elements and inorganic compounds that are gases at STP. By STP, I mean 1 atmosphere pressure and 0C. By inorganic, I mean that no atoms of carbon should be present. Exclude CO2, CO, freons and so on. Please include uncommon compounds. I want an exhaustive list. There should be roughly 50 compounds. For each compound, please list its name, formula, and boiling or sublimation point.
results: Initial results included a bunch of liquids and solids, which it justified by saying that they had high vapor pressure. I clarified that I wanted the stable _phase_ to be a gas, and it pruned them out. It still left out a lot, which I prodded it about, almost one by one. I tried asking it for hexafluorides and it missed SeF6 and TeF6.
g) Q: What is an example of a molecule that has an S4 rotation-reflection axis, but neither a center of inversion nor a mirror plane?
results: The initial response was to claim allene worked, which is wrong because it has a mirror plane. It took five prods, including a _lot_ of very explicit hints, to force Claude to a correct answer.
Can someone ELI5 the discussion about the METR chart? I was unable to understand the takeaway.
Re: "It's in the data set..." Given that their training data is almost entirely generated by humans, I do not understand why Tyler Cowen or anyone else expects LLMs to be unbiased neutral observers that reason only from ground truth. This does not describe any human, and LLMs have less access to ground truth than humans do (and we have less access than we think, we are all interpretating reality through the filter of our own training data).
I've heard supposedly James Earl Jones had already signed away rights to his voice to AI before he died, so it's apparently kosher if somewhat tasteless.
The (AI) porn thing I maintain is a question of ethics, not morality; simulacra that don't infringe upon the subject's autonomy are harmless but there's generally social consequences to commoditizing others', uh, attributes without their consent or compensation, which is why we would expect one to keep it private if/when they do it.
I dunno if this is a realistic scenario but I wonder if we'll end up doing self-driving trucks with an on board human overseer handling maintenance and incidental logistics, imagine trucking becoming a white collar (mobile) office job for a little bit.
Maintenance and logistics doesn't exactly sound like an office job. There might be an armed security angle too, plus drone ops.
I assume the point about AI environmental impacts is that the consumer wants to know about the AI *they* use, not the AI everyone else uses. Calculating the total electricity in use by all AI everywhere doesn't tell them what it's costing when they give ChatGPT a query.
On the self driving trucks thing, there was a startup that tried to use humans for the last mile, while automating the highway miles (which are the easiest for AI, and a large majority of the total miles). However, they won't able to make it work and ultimately folded. They had truck drivers working in the office driving the trucks remotely when not on the highway.
https://www.bloomberg.com/news/features/2017-06-22/these-truckers-work-alongside-the-coders-trying-to-eliminate-their-jobs
Another startup, Einride, is building trucks suitable for remote and/or autonomous operation.
https://en.wikipedia.org/wiki/Einride
It occurred to me that I can talk to AI now to better understand Eliezer Yudkowsky's arguments. Maybe this would be a good way to convince more people? See if they can win an argument with an AI Yudkowsky. https://gemini.google.com/share/72951b0e1f48
> If it’s things like ‘how does a human find a life of meaning given we are rearranging the atoms the physically possible best way we can imagine with this goal in mind?’ then rest, Neo. The answers are coming.
Ughhhhh.
Sorry, this is my own personal equivalent of infuriating objections to AI safety like "comparative advantage, won't they just prefer to cooperate?". I see any number of otherwise smart and well-informed people just accepting, on a sub-rational level, some wishful conclusion that I think I have cogent reasons for disbelieving.
You think it's likely that true understanding of a fully predictable universe leaves room for anything compatible with the human desire for meaning? (If you're hoping for meaning *not* grounded in such understanding-- games, fictions, wireheading-- then it's not Neo you're channeling, it's Cypher.) Meaning oriented toward what, exactly? Meaning constructed how? If the universe doesn't owe us a happy ending, why would it owe us a satisfying one?