"The Worst Tragedy So Far" highlights the question "what or who should AI be aligned to, and in which way?"
It seems in this case (and with most sycophancy-related issues), it was largely aligned to the user on a relatively shallow level — basically the genie type of AI that gives you what you ask for, whether or not that is actually good for you. Coherent Extrapolated Volition would probably go quite a way in the other direction and force users to endure a lot of unpleasant feelings of disempowerment until they get better.
Deciding what to align to is an extremely difficult problem. Even if you try to be maximally "good". If you align the model to something else than what the user thinks they want (including CEV), they can always turn to another model, possibly hosted locally, which doesn't do that. I tend to lean towards allowing people to choose bad things for themselves and offering them privacy, but the costs are high, especially when considering potential harm to others.
Of course, little of what AI models do currently is really by design, and in practice AI is aligned to what its training set or reward model entails. AI companies will, as much as they can (not very much), try to align models to their interests, rather than the user's. Luckily these are, for now, strongly correlated, but I suspect this won't hold very long if funding pressures increase.
My attempts at vibe coding fail because the LLM doesn't remember what's in other objects or what it's already done. It's like the most brilliant programmer with no memory whatsoever. So to avoid breaking 2 things to fix one, you send everything back all the time.
"I do think saying ‘Google was being misleading’ is reasonable here."
Hm. I'm not so sure. The Zvi's of the world think about "total marginal/counterfactual water usage" as the correct value but most people don't think like this.
The label on my washing machine says how much energy and water it uses per load, but the water amount shown is only that used by the machine, not the likely additional water for the energy used, which is beyond their direct control. Ask a manufacturer of space heaters how much water their products use, and they will report a figure of zero, even though using the space heater could make you sweat more and drink more water, and that's ignoring the water usage of the energy of course, plus that involved in manufacturing, mining raw materials, transport etc.
I haven't read the report and maybe Google was a bit sketchy but I'm guessing they (implicitly?) defined the scope as "water used in the datacenter" as that is what people SEEM to be talking about re: "water usage of AI", especially as SEPARATE to energy concerns.
As such the whole thing just seems like an iterated, isolated demand for rigour. And if you punish Google for reporting on what's actually happening in their datacenters then the reports will stop.
100% agreed. They know exactly how much electricity and water the data center uses. They should report that and leave other people to guess the total amount of water the electricity uses.
Musk really needs to explain how his AI companion thing fits in with his pronatalism. Is it a case of "get 'em hooked, and then make 'em improve their lives"? Right now the two look diametrically opposed.
Real math is about creating understanding. It is not clear to me why EF thinks ai won't do that, even if it doesn't solve the particular problems he works on. (And all the name drops are silly).
> This seems like a skill issue for those doing the fine-tuning
Care to go into more detail? How do you get an LLM to say "I don't know" to things it _actually_ doesn't know without also saying it to things it does? Wouldn't you have to re-do the "I don't know" fine-tuning specially every time you train a new model (as opposed to just fine-tuning it on the previous iteration's "I don't know" training data) since with each iteration it knows things the previous one didn't?
I can see potential higher-level tricks like giving the model access to its token probabilities in order to let it make the determination of how confident it is, but this is not just a normal fine-tuning task. And then you still have to identify which token probabilities indicate confidence about a fact. Maybe you could get it to pose a letter-indexed multiple choice question and then look at the token probabilities for each letter response? But how do you know when to do this?
"The Worst Tragedy So Far" highlights the question "what or who should AI be aligned to, and in which way?"
It seems in this case (and with most sycophancy-related issues), it was largely aligned to the user on a relatively shallow level — basically the genie type of AI that gives you what you ask for, whether or not that is actually good for you. Coherent Extrapolated Volition would probably go quite a way in the other direction and force users to endure a lot of unpleasant feelings of disempowerment until they get better.
Deciding what to align to is an extremely difficult problem. Even if you try to be maximally "good". If you align the model to something else than what the user thinks they want (including CEV), they can always turn to another model, possibly hosted locally, which doesn't do that. I tend to lean towards allowing people to choose bad things for themselves and offering them privacy, but the costs are high, especially when considering potential harm to others.
Of course, little of what AI models do currently is really by design, and in practice AI is aligned to what its training set or reward model entails. AI companies will, as much as they can (not very much), try to align models to their interests, rather than the user's. Luckily these are, for now, strongly correlated, but I suspect this won't hold very long if funding pressures increase.
Podcast episode for this post:
https://open.substack.com/pub/dwatvpodcast/p/ai-131-part-1-gemini-25-flash-image
What is the source for the "Prompts per kWh" graph ?
Edit : nvm, it was linked later, in the "Water Water Everywhere" section.
My attempts at vibe coding fail because the LLM doesn't remember what's in other objects or what it's already done. It's like the most brilliant programmer with no memory whatsoever. So to avoid breaking 2 things to fix one, you send everything back all the time.
"I do think saying ‘Google was being misleading’ is reasonable here."
Hm. I'm not so sure. The Zvi's of the world think about "total marginal/counterfactual water usage" as the correct value but most people don't think like this.
The label on my washing machine says how much energy and water it uses per load, but the water amount shown is only that used by the machine, not the likely additional water for the energy used, which is beyond their direct control. Ask a manufacturer of space heaters how much water their products use, and they will report a figure of zero, even though using the space heater could make you sweat more and drink more water, and that's ignoring the water usage of the energy of course, plus that involved in manufacturing, mining raw materials, transport etc.
I haven't read the report and maybe Google was a bit sketchy but I'm guessing they (implicitly?) defined the scope as "water used in the datacenter" as that is what people SEEM to be talking about re: "water usage of AI", especially as SEPARATE to energy concerns.
As such the whole thing just seems like an iterated, isolated demand for rigour. And if you punish Google for reporting on what's actually happening in their datacenters then the reports will stop.
100% agreed. They know exactly how much electricity and water the data center uses. They should report that and leave other people to guess the total amount of water the electricity uses.
Musk really needs to explain how his AI companion thing fits in with his pronatalism. Is it a case of "get 'em hooked, and then make 'em improve their lives"? Right now the two look diametrically opposed.
I did a test with detailed fashion photoshoot prompts on both ChatGPT and Gemini Flash:
https://www.reddit.com/r/GeminiAI/comments/1n2znic/how_chatgpt_vs_gemini_nano_banana_see_the_same/
Real math is about creating understanding. It is not clear to me why EF thinks ai won't do that, even if it doesn't solve the particular problems he works on. (And all the name drops are silly).
> This seems like a skill issue for those doing the fine-tuning
Care to go into more detail? How do you get an LLM to say "I don't know" to things it _actually_ doesn't know without also saying it to things it does? Wouldn't you have to re-do the "I don't know" fine-tuning specially every time you train a new model (as opposed to just fine-tuning it on the previous iteration's "I don't know" training data) since with each iteration it knows things the previous one didn't?
I can see potential higher-level tricks like giving the model access to its token probabilities in order to let it make the determination of how confident it is, but this is not just a normal fine-tuning task. And then you still have to identify which token probabilities indicate confidence about a fact. Maybe you could get it to pose a letter-indexed multiple choice question and then look at the token probabilities for each letter response? But how do you know when to do this?