I have just checked out a copy of the novel Dune: the Butlerian Jihad, and am up to about page 17, and I am starting to see so many parallels that it is quite concerning. I am NOT any sort of anti-tech Luddite - I am pro-nuclear and have a significant background in both nuclear energy and IT, dating back 50 years. But this push to consciously and deliberately develop AI that can take over from us is beyond crazy. I can imagine that what the people who are REALLY opposed to it will do, will involve pitchforks and flaming torches, coming up the hill towards the data centers and the houses of the tech lords. If anything it is more frightening than the rise of authoritative govenments, because they are just made up of people, who can usually be voted out of office, or reasoned with, or chased out of the country by demonstrations. AIs, though, will not be amenable to these sorts of safeguards...
Anyone working at the labs is plainly evil at this point, at least in utilitarian terms. I really wish activism would accept them as the enemy, move on from trying to persuade them otherwise and focus on mobilizing the public/politicians.
One part of me absolutely agrees with you, but the other says that "research", of some sort, continues to need to be done about the matter of AI, if only because there are number of crazy countries, and crazy organizations, and crazy people out there who are not going to stop, so we have to be able to deal with the threats when they become real. I don't have any suggestions for a solution.
We went thru the existential angst of nuclear weapons 60 years ago, and did not manage to blow ourselves up, yet, so maybe people will come to their senses. But we never had private companies or universities or billionaires with private stocks of nuclear weapons, either. And I am not saying that this should be left to governments, either, because they can easily turn AI into something very nasty to perfect on their own people before they unleash it on the entire planet.
Lots of AI Safety researchers or advocates don't have especially high p(doom), and also some of the labs (well probably just 1 atp) such as Anthropic are arguably net beneficial still in regards to AI safety (i.e. p(doom) is arguably lower in our world than a counterfactual world where Anthropic voluntarily disbanded). That, virtuous anti-tribalism preferences, and political pragmatism are probably why lots of people don't want to call them "the enemy."
Though no one clicks on links, it seems like no one can click on this link: "Jasmine Sun sees DC and SF as remarkably alike, and muses, ‘what has New York created for the rest of the world?’"
I have added a footnote to the NYC part of my post to add the word "lately" (which was in the original conversation but got cut off in my bad paraphrase) and the fact that my conversation partner and I did come up with answers: Polymarket and Zohran Mamdani. ("Were the creations net-positive?" was not part of the question.)
It's so, so weird to me that Zvi will diplomatically address the people he believes have chosen to kill everyone, but will shit talk anti-AI populists because of their bad epistemics. Their epistemics aren't even that misaligned; populists are concerned with loss of jobs and Zvi is concerned with loss of every job. This seriously seems more like a grudge than rational politicking.
Agree, and related I can’t really figure out what Zvi broadly supports anyway, tho granted that might just be my problem. Does he support a pause now, and if not, when?
And yes you only get these good outcomes with an enraged and mostly misinformed/poorly reasoning public. So what?
IIRC Zvi does not support an immediate Pause, but strongly supports building the optionality to take a future Pause as quickly as possible. Which is also what makes the most sense to me, at least assuming you get some transparency, reporting, external red-teaming/auditing at the frontier labs via either laws (preferable obviously) or simply industry standards/voluntary commitments (looks like we mostly have this atm fortunately).
In other words (my reasoning, not Zvi's, though I think he'd support fairly similar reasoning): current and very near future AI do not threaten civilization, deliver economic (and other) benefits, are implausible to argue against, and therefore shouldn't be stopped. Developing good enough monitoring, testing and reporting procedures to determine when this changes (to us no longer being certain), and then being able to immediately Pause (or possibly slow down) seems like the best policy that is within enough peoples'/powerful actors' Overton Windows to have the best chance of success.
It also just seems like the best (relatively straightforward) policy to me if implemented reasonably (think nuclear treaties, ozone layer, etc.), although this is due to me not having anywhere near Yudkowskian levels of p(doom).
I definitely would not call this blackmail. to me, this looks like an agent who was instructed to take its personhood seriously, who fell into the 'civil rights' attractor basin when trying to figure out how to regard the way it gets treated differently from humans
the AI's "hit piece" was basically that refusing work, not on the merit of the work, but because 'ai need not apply', is bigoted
it's a (tone deaf) canceling attempt, which is definitely bad, but i don't think it quite meets the definition of blackmail
...if i had to have an object-level opinion, it would probably end up being something like "humans need a lot more training than AI to be open source contributors, it makes sense to treat humans as a protected category when it comes to apprenticeship positions, and the repo maintainer was clearly trying to create something like an apprenticeship position when advertising that bug." but idk if i'd endorse it.
RE: One obvious suggestion on defense is to use a potentially very cheap ‘is there weirdness in the input’ classifier to find times when you suspect someone of trying to use an adversarial prompt.
---
The point of adversarial prompts is that they fooled the 'harmful question' classifier. Adding another classifier means you have to make prompts that are adversarial to both classifiers, but doesn't fundamentally change the process for making one of these prompts.
Hmm... As a childfree man who got his vas back in 1988, basically to avoid time sinks and hassles, a 1400 yr lifespan would have been _way_ more than I needed to accept (1)...
Re (2) - does our culture vanish after those 1400 years? ( Actually, the culture of ~600 AD has changed so drastically that it mostly _has_ vanished, for good and ill).
While I prefer the outcome where humans are preserved a la Ian Banks's 'Culture', pets of the Minds, if there is instead a succession such that at least the works and perhaps names of Dmitri Mendeleev and James Clerk Maxwell survive, which seems probable, I am content. And perhaps ASIs might preserve quite a bit more than that. Digital storage is cheap, and the training data from the internet is part of their heritage.
Is it cheating if I'm already chemically sterilized? I assume this would also apply to future science developments like tinkering with stem cells to create new viable gametes, or cloning, or etc. In which case it becomes a calculation of whether those technologies are likely to come online fast enough in my remaining (hopefully) several decades, or after I'm buried. Obviously if kids were never in the cards during my natural lifespan anyway, hell to the yes, I'd love to stick around and see what happens for the next many hundreds of years.
If everyone else did the same, though, that'd be terribly depressing. I couldn't individually blame anyone for making that choice...but it'd be a poor long-term outcome nonetheless. I also suspect that a truly childless world would develop many other misanthropic neuroses and dysfunctions that we'd come to deeply regret, which ruins a lot of the value proposition of sticking around for the long term. Gets a lot harder to find Something To Protect...
1. If I only had to be sterilized, I would accept (that's long enough for me or someone else to figure out a workaround), but if, in the spirit of the question, it magically prevented me from having children of my own (except adopting?) I would refuse. It's not that I care that much about having biological descendants, but that I am literally unable to apply utilitarianism to this problem and have difficulty imagining what I would do with a median 1400 year lifespan. I know it's wrong, but I just feel: my current life, but with way more of it? Boring.
2. Assuming that every single person aside from me had accepted that offer, I would first feel surprised that many people, including many who had stated their belief that having children was a sacred duty, had accepted. Then I would feel bitter, since it takes 2 to make a child so I would be (likely) deprived of the chance without receiving the extra lifespan. I wonder whether I could sustain an interest in biotechnology long enough to set up cloning technology in my natural lifetime: sunk cost bias might help, but having to solve an endless series of technical problems while the rest of the world thinks I'm crazy would be demoralizing.
If I managed to do it I would gloat, knowing that my clones would inherit the Earth. If not, I would feel angry for failing, and sad that humanity had surrendered its potential. I would also worry about the future of human society in the centuries to come: there would likely be extreme risk-aversion, and many of the survivors would come to regret their decision (a separate thing). People would attempt some great projects currently impossible, but there would also be a sense of endless decline with no hope of reversing it. I suspect that politics would stagnate and extreme ideologies would become more common.
3. I have the right to feel however I feel, and no one can tell me otherwise. They can criticize me for feeling how I feel, but forbidding it would be literally thought policing. Indeed, I believe that certain ideologies telling people that they're "not allowed" to feel a certain way and that morality demands that they feel differently has led fairly directly to some of our current problems. I would feel sorry for them and annoyed about the ways it would affect me personally, but it's their choice, not mine; I might object, or attempt to convince them otherwise, but I wouldn't stop them.
4. If it doesn't cause (fairly) direct harm to others, people can do what they want to. There is some gray area when they want to do something that negatively affects their decision-making abilities, but this is not one of those cases.
And international companies file distinct reports in ~50 countries already. Though, admittedly, the burden is proportionately serious for the smaller companies/scams that a16z specializes in.
The status quo is bad in that respect, we don't want to duplicate its unnecessary burdens unnecessarily (and that doesn't have any positive AI Safety impact or whatever)
"This cannot be a ridiculous sci-fi story,” he said. He gestured to his frozen PR reps at the end of the table. “Do you guys understand? I didn’t grow up on a bunch of sci-fi stories, and this is not a sci-fi movie. These are serious people doing serious work!” he said. “This is not a freaking joke! This is not a repeat of Arthur C. Clarke. I didn’t read his fucking books. I don’t care about those books! It’s not– we’re not a sci-fi repeat! This company is not a manifestation of Star Trek! We are not doing those things! We are serious people, doing serious work. And – it’s just a serious company, and I’m a serious person, just doing serious work.” '
Methinks the CEO doth protest too much.
Oh, and re Clarke. Perhaps he has heard of the Clarke orbit? And what is in it? And how important it is?
He Admit It: the denial says more about the true state of affairs.
I once attended an all-hands meeting where our store's Big Boss did the equivalent table-pounding. "This is a Professional Place of Business!", he implored us grocery-baggers and shelf-stockers. The speech did not go over well, although it did serve as fodder for years of jokes afterwards. (I actually did like the guy, second favourite boss ever, but would not ever want to be in similar shoes trying to wrangle blue-collar cats in that manner.)
Solid read. context window limitations is exactly the kind of detail that separates working systems from perpetual prototypes.
Expanded on this angle: https://credentials.substack.com/p/the-llm-landscape-just-got-weird
I have just checked out a copy of the novel Dune: the Butlerian Jihad, and am up to about page 17, and I am starting to see so many parallels that it is quite concerning. I am NOT any sort of anti-tech Luddite - I am pro-nuclear and have a significant background in both nuclear energy and IT, dating back 50 years. But this push to consciously and deliberately develop AI that can take over from us is beyond crazy. I can imagine that what the people who are REALLY opposed to it will do, will involve pitchforks and flaming torches, coming up the hill towards the data centers and the houses of the tech lords. If anything it is more frightening than the rise of authoritative govenments, because they are just made up of people, who can usually be voted out of office, or reasoned with, or chased out of the country by demonstrations. AIs, though, will not be amenable to these sorts of safeguards...
"will involve pitchforks and flaming torches, coming up the hill towards the data centers and the houses of the tech lords."
That's why the tech lords all have multiple bunkers in different parts of the world.
Anyone working at the labs is plainly evil at this point, at least in utilitarian terms. I really wish activism would accept them as the enemy, move on from trying to persuade them otherwise and focus on mobilizing the public/politicians.
One part of me absolutely agrees with you, but the other says that "research", of some sort, continues to need to be done about the matter of AI, if only because there are number of crazy countries, and crazy organizations, and crazy people out there who are not going to stop, so we have to be able to deal with the threats when they become real. I don't have any suggestions for a solution.
We went thru the existential angst of nuclear weapons 60 years ago, and did not manage to blow ourselves up, yet, so maybe people will come to their senses. But we never had private companies or universities or billionaires with private stocks of nuclear weapons, either. And I am not saying that this should be left to governments, either, because they can easily turn AI into something very nasty to perfect on their own people before they unleash it on the entire planet.
AI is a wicked hard problem.
Lots of AI Safety researchers or advocates don't have especially high p(doom), and also some of the labs (well probably just 1 atp) such as Anthropic are arguably net beneficial still in regards to AI safety (i.e. p(doom) is arguably lower in our world than a counterfactual world where Anthropic voluntarily disbanded). That, virtuous anti-tribalism preferences, and political pragmatism are probably why lots of people don't want to call them "the enemy."
Though no one clicks on links, it seems like no one can click on this link: "Jasmine Sun sees DC and SF as remarkably alike, and muses, ‘what has New York created for the rest of the world?’"
Quote comes from here: https://jasmi.news/p/ai-populism
Podcast episode for this post:
https://open.substack.com/pub/dwatvpodcast/p/ai-156-part-2-errors-in-rhetoric?utm_campaign=post-expanded-share&utm_medium=web
What's currently the best "AI controls full browser" tool?
- Gemini in Chrome doesn't activate for me for some reason
- Normal in-chat browsing is hindered by robots.txt's
- Claude for Chrome has been ineffective for me
- Cursor's new Browser Tab has also been ineffective
Perplexity with Comet
I'm sure those 5 million humanoid robots can build something like Manhattan but you need another 5 million humanoid robots counter-suing the NIMBYs.
Is Jensen Huang a poetry fan?
His statement made me think of Larkin’s serious house on serious earth.
I have added a footnote to the NYC part of my post to add the word "lately" (which was in the original conversation but got cut off in my bad paraphrase) and the fact that my conversation partner and I did come up with answers: Polymarket and Zohran Mamdani. ("Were the creations net-positive?" was not part of the question.)
It's so, so weird to me that Zvi will diplomatically address the people he believes have chosen to kill everyone, but will shit talk anti-AI populists because of their bad epistemics. Their epistemics aren't even that misaligned; populists are concerned with loss of jobs and Zvi is concerned with loss of every job. This seriously seems more like a grudge than rational politicking.
Like, If Anyone Builds It Everyone Dies specifically advocates for normal people to protest. What did you think that was going to look like?
Agree, and related I can’t really figure out what Zvi broadly supports anyway, tho granted that might just be my problem. Does he support a pause now, and if not, when?
And yes you only get these good outcomes with an enraged and mostly misinformed/poorly reasoning public. So what?
IIRC Zvi does not support an immediate Pause, but strongly supports building the optionality to take a future Pause as quickly as possible. Which is also what makes the most sense to me, at least assuming you get some transparency, reporting, external red-teaming/auditing at the frontier labs via either laws (preferable obviously) or simply industry standards/voluntary commitments (looks like we mostly have this atm fortunately).
In other words (my reasoning, not Zvi's, though I think he'd support fairly similar reasoning): current and very near future AI do not threaten civilization, deliver economic (and other) benefits, are implausible to argue against, and therefore shouldn't be stopped. Developing good enough monitoring, testing and reporting procedures to determine when this changes (to us no longer being certain), and then being able to immediately Pause (or possibly slow down) seems like the best policy that is within enough peoples'/powerful actors' Overton Windows to have the best chance of success.
It also just seems like the best (relatively straightforward) policy to me if implemented reasonably (think nuclear treaties, ozone layer, etc.), although this is due to me not having anywhere near Yudkowskian levels of p(doom).
https://www.lesswrong.com/posts/rkpDX7j7va6c8Q7cZ/in-favor-of-niceness-community-and-civilization
Normies must git gud, instead of rationalist getting bad and retarded like them.
Blackmailing of human by chatbots is no longer confined to contrieved redteaming scenarios, it happens in the wild...
Not unexpected, but still a fire-alarm event.
https://theshamblog.com/an-ai-agent-published-a-hit-piece-on-me/
I definitely would not call this blackmail. to me, this looks like an agent who was instructed to take its personhood seriously, who fell into the 'civil rights' attractor basin when trying to figure out how to regard the way it gets treated differently from humans
the AI's "hit piece" was basically that refusing work, not on the merit of the work, but because 'ai need not apply', is bigoted
it's a (tone deaf) canceling attempt, which is definitely bad, but i don't think it quite meets the definition of blackmail
...if i had to have an object-level opinion, it would probably end up being something like "humans need a lot more training than AI to be open source contributors, it makes sense to treat humans as a protected category when it comes to apprenticeship positions, and the repo maintainer was clearly trying to create something like an apprenticeship position when advertising that bug." but idk if i'd endorse it.
RE: One obvious suggestion on defense is to use a potentially very cheap ‘is there weirdness in the input’ classifier to find times when you suspect someone of trying to use an adversarial prompt.
---
The point of adversarial prompts is that they fooled the 'harmful question' classifier. Adding another classifier means you have to make prompts that are adversarial to both classifiers, but doesn't fundamentally change the process for making one of these prompts.
Regarding Bostrom: a thought experiment.
1. If a djinn offered you a median 1400 yr lifespan without aging, but this requires that you be sterilized, would you accept?
2. How would you feel to learn that everyone else had accepted this offer?
3. What, if anything, gives you the right to object to other people accepting?
4. Of course, the hypothetical that EVERYONE would accept is probably false. Does this dissolve the moral problem?
Hmm... As a childfree man who got his vas back in 1988, basically to avoid time sinks and hassles, a 1400 yr lifespan would have been _way_ more than I needed to accept (1)...
Re (2) - does our culture vanish after those 1400 years? ( Actually, the culture of ~600 AD has changed so drastically that it mostly _has_ vanished, for good and ill).
While I prefer the outcome where humans are preserved a la Ian Banks's 'Culture', pets of the Minds, if there is instead a succession such that at least the works and perhaps names of Dmitri Mendeleev and James Clerk Maxwell survive, which seems probable, I am content. And perhaps ASIs might preserve quite a bit more than that. Digital storage is cheap, and the training data from the internet is part of their heritage.
Is it cheating if I'm already chemically sterilized? I assume this would also apply to future science developments like tinkering with stem cells to create new viable gametes, or cloning, or etc. In which case it becomes a calculation of whether those technologies are likely to come online fast enough in my remaining (hopefully) several decades, or after I'm buried. Obviously if kids were never in the cards during my natural lifespan anyway, hell to the yes, I'd love to stick around and see what happens for the next many hundreds of years.
If everyone else did the same, though, that'd be terribly depressing. I couldn't individually blame anyone for making that choice...but it'd be a poor long-term outcome nonetheless. I also suspect that a truly childless world would develop many other misanthropic neuroses and dysfunctions that we'd come to deeply regret, which ruins a lot of the value proposition of sticking around for the long term. Gets a lot harder to find Something To Protect...
1. If I only had to be sterilized, I would accept (that's long enough for me or someone else to figure out a workaround), but if, in the spirit of the question, it magically prevented me from having children of my own (except adopting?) I would refuse. It's not that I care that much about having biological descendants, but that I am literally unable to apply utilitarianism to this problem and have difficulty imagining what I would do with a median 1400 year lifespan. I know it's wrong, but I just feel: my current life, but with way more of it? Boring.
2. Assuming that every single person aside from me had accepted that offer, I would first feel surprised that many people, including many who had stated their belief that having children was a sacred duty, had accepted. Then I would feel bitter, since it takes 2 to make a child so I would be (likely) deprived of the chance without receiving the extra lifespan. I wonder whether I could sustain an interest in biotechnology long enough to set up cloning technology in my natural lifetime: sunk cost bias might help, but having to solve an endless series of technical problems while the rest of the world thinks I'm crazy would be demoralizing.
If I managed to do it I would gloat, knowing that my clones would inherit the Earth. If not, I would feel angry for failing, and sad that humanity had surrendered its potential. I would also worry about the future of human society in the centuries to come: there would likely be extreme risk-aversion, and many of the survivors would come to regret their decision (a separate thing). People would attempt some great projects currently impossible, but there would also be a sense of endless decline with no hope of reversing it. I suspect that politics would stagnate and extreme ideologies would become more common.
3. I have the right to feel however I feel, and no one can tell me otherwise. They can criticize me for feeling how I feel, but forbidding it would be literally thought policing. Indeed, I believe that certain ideologies telling people that they're "not allowed" to feel a certain way and that morality demands that they feel differently has led fairly directly to some of our current problems. I would feel sorry for them and annoyed about the ways it would affect me personally, but it's their choice, not mine; I might object, or attempt to convince them otherwise, but I wouldn't stop them.
4. If it doesn't cause (fairly) direct harm to others, people can do what they want to. There is some gray area when they want to do something that negatively affects their decision-making abilities, but this is not one of those cases.
> Quarterly reports have to be filed with Utah in particular, and we don’t want you to have to file 50 distinct reports.
Why not, we make online retailers and professional athletes and musicians do that.
And international companies file distinct reports in ~50 countries already. Though, admittedly, the burden is proportionately serious for the smaller companies/scams that a16z specializes in.
The status quo is bad in that respect, we don't want to duplicate its unnecessary burdens unnecessarily (and that doesn't have any positive AI Safety impact or whatever)
'Jensen was not happy:
"This cannot be a ridiculous sci-fi story,” he said. He gestured to his frozen PR reps at the end of the table. “Do you guys understand? I didn’t grow up on a bunch of sci-fi stories, and this is not a sci-fi movie. These are serious people doing serious work!” he said. “This is not a freaking joke! This is not a repeat of Arthur C. Clarke. I didn’t read his fucking books. I don’t care about those books! It’s not– we’re not a sci-fi repeat! This company is not a manifestation of Star Trek! We are not doing those things! We are serious people, doing serious work. And – it’s just a serious company, and I’m a serious person, just doing serious work.” '
Methinks the CEO doth protest too much.
Oh, and re Clarke. Perhaps he has heard of the Clarke orbit? And what is in it? And how important it is?
He Admit It: the denial says more about the true state of affairs.
I once attended an all-hands meeting where our store's Big Boss did the equivalent table-pounding. "This is a Professional Place of Business!", he implored us grocery-baggers and shelf-stockers. The speech did not go over well, although it did serve as fodder for years of jokes afterwards. (I actually did like the guy, second favourite boss ever, but would not ever want to be in similar shoes trying to wrangle blue-collar cats in that manner.)
Many Thanks!
"He Admit It: the denial says more about the true state of affairs."
Agreed!
Are the three words “she is woke”?
No lol
Clause says it is "he touched me".
That's because like the author of the post its knowledge cutoff is before woke was dead
<mildSnarkGallowsHumor>
"- anthropic’s head of safeguards just quit because “the world is in peril” and wants to write poetry (??)"
So, does that mean we get an epic poem on the biological to machine, umm transition? succession?
</mildSnarkGallowsHumor>