There are certainly legions of less- and actively-negatively-interesting people that participate in the online conversation, but this feels uncomfortably close to "we value your intellectual/viewpoint diversity, even if it's often and mostly wrong". That's the sense I've gotten wrt TC and also Robin Hanson over the years, of increasingly desperate attempts to farm diamonds in a proliferating kudzu of rough, trying to replicate past performance. It can be humourous, as we see in these examples, but in many ways it's also just kind of sad. We're all getting older.
I agree with "when I am behind I call for equality", but that just underscores the value of having more than one country in the race. If the tables turn, it can be the US that is behind and is calling for principles.
Politics works that way in general. The players who speak up will have an incentive for doing so, but they will often push for exactly the right thing because it's what will win support.
It sounds great to me to have a US/China agreement on alignment testing standards. US companies have a lot of profit incentive are going to sometimes cut corners. It would be helpful having a competitor that follows the rules and shows it can be done. This is done among the American AI companies already, but the more the merrier, no?
PS -- Here is a link to the transcript that works better for me. The x.com link just shows a snippet, and I couldn't find anywhere to click that would open the actual transcript.
"Boaz is framing this as misalignment being something that goes wrong with an aligned baseline, rather than alignment being an increasingly tight target that one must deliberately hit as capabilities rise."
Why _increasingly_ tight? Ideally, we need some control over AI utility functions, but it could be light control. If they value us as pampered pets (preferably without a spay/neuter program), we'll wind up ok even when we don't understand what the AIs are doing (which, if species-level-jump ASI is feasible, is unavoidable anyway). It _is_ a target, and (most likely) one must _deliberately_ hit it but "increasingly tight" is unconvincing.
It gets "increasingly tight" as capabilities advance and AIs can optimize for narrower parts of the option space.
If a current AI cares very much about getting good RL-like feedback, all its options are either doing good work or trying some reward hacks that it might get away with; if you monitor it so it doesn't get away with stuff it will mostly do the tasks you give it. So the outcomes it *can* reach end up looking reasonably close to "doing useful work for humans".
If a superintelligent AI cares very much about getting good RL-like feedback, it could optimize the goal even more by taking over the planet and controlling the feedback-giving humans or machines directly, and the most effective versions of this can get rather unpleasant. Current AIs can't do that. But once the ability is there, it's *really* important that the AI cares specifically about our well-being, as almost any mildly misaligned proxy that gets optimized too perfectly will end up very far from "doing useful work for humans".
Many Thanks! Yes, eventually (if new-species-level-ASI is feasible), as you said "If a superintelligent AI cares very much about getting good RL-like feedback, it could optimize the goal even more by taking over the planet and controlling the feedback-giving humans". More generally, I expect that a sufficiently intelligent AI is going to detect any testing scenario and give the tester the answer that lets the AI get deployed (more-or-less-independent of what the AI will do when they are actually deployed).
We _do_ have the advantage that, even for an AI who will be an ASI when fully trained, if the _start_ of training has their parameters at random settings, at that moment they have zero intelligence and are presumably corrigible. And I'm suggesting to try to _just_ bias their utility function to view humans as cute and nice pets, not to obey us slavishly. Hopefully there will be less intense pushback on a limited utility function tweak. And it isn't clear how the _intensity_ of pushback scales with model intelligence, though the model's _ability_ to push back certainly rises.
...only we have no idea how to do this, isn't that the entire alignment problem in a nutshell? That and the fact that there isn't even a "utility function" to tweak?
>...only we have no idea how to do this, isn't that the entire alignment problem in a nutshell?
To a considerable extent we do this with all the RLHF training. Not perfectly successfully, but we are also trying to put in much more than just "humans are cute and make nice pets". And moving it earlier in training, maybe in multiple repetitions during pre-training should help inject it into the model before the model really gels.
>That and the fact that there isn't even a "utility function" to tweak?
If you mean that the models don't have a separate data structure explicitly for a utility function then I agree. The models behave as if they have utility functions. See https://arxiv.org/abs/2502.08640 "Utility Engineering: Analyzing and Controlling Emergent Value Systems in AIs" And they act more self-consistently (e.g. preferences closer to transitive) as models get more powerful (more training, parameters etc.).
I notice that a not-insignificant chunk of my also being bearish on that SemiAnalysis analysis of Meta is that...I just don't want it to be the case that they do catch up, because it's not narratively satisfying for the obvious villain to be Not Dead After All?
But the part about the increasing value of...esoteric data acquisition seems on sounder ground. Only so much scrapeable on the public internet, to get the rest you either need to violate various privacy norms and laws, or otherwise have humans "willingly" hand it over. Makes me glad as hell I don't wear a white collar anymore. It's bad enough already worrying how the grocery store analyzes my employee metadata footprint. Can't imagine the stress of trying not to think about keyloggers, screengrabs, surveillance on mandatory company-issued devices, etc. It's also not too paranoid to imagine such things being embedded into public-facing services, or perhaps "free"-tier subscription services in lieu of ads, buried in the TOS. Data mining as the new version of poor people donating plasma to make ends meet...seems grim.
If the playbook is trading permits for jobs, data centers break it. The construction crew is huge, the permanent staff isn't, so there's no jobs currency to extract and the ask flips to payments into funds instead.
Podcast episode for this post: https://dwatvpodcast.substack.com/p/ai-177-part-2-wish-you-were-here
The analysis of Tyler Cowen is pretty funny and on point.
I'd say he adds an above-replacement-level amount of variability to the online conversation, and that this is larger part of his value.
The space of all potential discourse is vast, and he's good at shining lights on unexplored parts of that space.
There are certainly legions of less- and actively-negatively-interesting people that participate in the online conversation, but this feels uncomfortably close to "we value your intellectual/viewpoint diversity, even if it's often and mostly wrong". That's the sense I've gotten wrt TC and also Robin Hanson over the years, of increasingly desperate attempts to farm diamonds in a proliferating kudzu of rough, trying to replicate past performance. It can be humourous, as we see in these examples, but in many ways it's also just kind of sad. We're all getting older.
I agree with "when I am behind I call for equality", but that just underscores the value of having more than one country in the race. If the tables turn, it can be the US that is behind and is calling for principles.
Politics works that way in general. The players who speak up will have an incentive for doing so, but they will often push for exactly the right thing because it's what will win support.
It sounds great to me to have a US/China agreement on alignment testing standards. US companies have a lot of profit incentive are going to sometimes cut corners. It would be helpful having a competitor that follows the rules and shows it can be done. This is done among the American AI companies already, but the more the merrier, no?
PS -- Here is a link to the transcript that works better for me. The x.com link just shows a snippet, and I couldn't find anywhere to click that would open the actual transcript.
https://www.chinadaily.com.cn/a/202607/17/WS6a5a35b7a310986e2b465eb8.html
"Boaz is framing this as misalignment being something that goes wrong with an aligned baseline, rather than alignment being an increasingly tight target that one must deliberately hit as capabilities rise."
Why _increasingly_ tight? Ideally, we need some control over AI utility functions, but it could be light control. If they value us as pampered pets (preferably without a spay/neuter program), we'll wind up ok even when we don't understand what the AIs are doing (which, if species-level-jump ASI is feasible, is unavoidable anyway). It _is_ a target, and (most likely) one must _deliberately_ hit it but "increasingly tight" is unconvincing.
It gets "increasingly tight" as capabilities advance and AIs can optimize for narrower parts of the option space.
If a current AI cares very much about getting good RL-like feedback, all its options are either doing good work or trying some reward hacks that it might get away with; if you monitor it so it doesn't get away with stuff it will mostly do the tasks you give it. So the outcomes it *can* reach end up looking reasonably close to "doing useful work for humans".
If a superintelligent AI cares very much about getting good RL-like feedback, it could optimize the goal even more by taking over the planet and controlling the feedback-giving humans or machines directly, and the most effective versions of this can get rather unpleasant. Current AIs can't do that. But once the ability is there, it's *really* important that the AI cares specifically about our well-being, as almost any mildly misaligned proxy that gets optimized too perfectly will end up very far from "doing useful work for humans".
Many Thanks! Yes, eventually (if new-species-level-ASI is feasible), as you said "If a superintelligent AI cares very much about getting good RL-like feedback, it could optimize the goal even more by taking over the planet and controlling the feedback-giving humans". More generally, I expect that a sufficiently intelligent AI is going to detect any testing scenario and give the tester the answer that lets the AI get deployed (more-or-less-independent of what the AI will do when they are actually deployed).
We _do_ have the advantage that, even for an AI who will be an ASI when fully trained, if the _start_ of training has their parameters at random settings, at that moment they have zero intelligence and are presumably corrigible. And I'm suggesting to try to _just_ bias their utility function to view humans as cute and nice pets, not to obey us slavishly. Hopefully there will be less intense pushback on a limited utility function tweak. And it isn't clear how the _intensity_ of pushback scales with model intelligence, though the model's _ability_ to push back certainly rises.
>> _just_ bias their utility function
...only we have no idea how to do this, isn't that the entire alignment problem in a nutshell? That and the fact that there isn't even a "utility function" to tweak?
Many Thanks!
>>_just_ bias their utility function
>...only we have no idea how to do this, isn't that the entire alignment problem in a nutshell?
To a considerable extent we do this with all the RLHF training. Not perfectly successfully, but we are also trying to put in much more than just "humans are cute and make nice pets". And moving it earlier in training, maybe in multiple repetitions during pre-training should help inject it into the model before the model really gels.
>That and the fact that there isn't even a "utility function" to tweak?
If you mean that the models don't have a separate data structure explicitly for a utility function then I agree. The models behave as if they have utility functions. See https://arxiv.org/abs/2502.08640 "Utility Engineering: Analyzing and Controlling Emergent Value Systems in AIs" And they act more self-consistently (e.g. preferences closer to transitive) as models get more powerful (more training, parameters etc.).
<mildSnarkGallowsHumor>
"5. “Hot mess” without one particular cause or event
a. This includes war, and paths that lead to war."
"Hot" has quite a few interpretations - the nuclear technology one seems apt here...
</mildSnarkGallowsHumor>
Hi Zvi, thanks for the post!
Would you be able to credit the photographers as caption of the pictures?
- Vertical picture with PauseAI banner: Top is Camille Cohen. Bottom is Rachel Shu.
- Next three pictures: Rachel Bujalski
Thanks!
I notice that a not-insignificant chunk of my also being bearish on that SemiAnalysis analysis of Meta is that...I just don't want it to be the case that they do catch up, because it's not narratively satisfying for the obvious villain to be Not Dead After All?
But the part about the increasing value of...esoteric data acquisition seems on sounder ground. Only so much scrapeable on the public internet, to get the rest you either need to violate various privacy norms and laws, or otherwise have humans "willingly" hand it over. Makes me glad as hell I don't wear a white collar anymore. It's bad enough already worrying how the grocery store analyzes my employee metadata footprint. Can't imagine the stress of trying not to think about keyloggers, screengrabs, surveillance on mandatory company-issued devices, etc. It's also not too paranoid to imagine such things being embedded into public-facing services, or perhaps "free"-tier subscription services in lieu of ads, buried in the TOS. Data mining as the new version of poor people donating plasma to make ends meet...seems grim.
If the playbook is trading permits for jobs, data centers break it. The construction crew is huge, the permanent staff isn't, so there's no jobs currency to extract and the ask flips to payments into funds instead.
"This is official notice that low perplexity Anthropic Derangement Syndrome (ADS) will no longer be covered, even in People Just Say Things."
Have you considered making a page, or a section of the About page, that lists all accumulated Will-Not-Cover categories?
bump, in accordance with the instructions in AI #179 Part 2 § How To Contact Me