I still don’t get why you think the chance of AI disaster is high in the first place. You can imagine all sorts of ways to be very intelligent that don’t fit your model of “it’s going to take over”.
For me it stems from the ‘what happens when more intelligent beings interact with less intelligent beings’ examples we have already - particularly humans vs. animals (also new world vs. old world). Whether intentionally or not, it works out in the favour of the more intelligent faction in the long run (although cf. most viruses / bacteria as counterexamples, I suppose). So not necessarily take over: just dominate the less intelligent faction into extinction.
Plenty of counterexamples there though. Dolphins don’t dominate shrimp into extinction. Monkeys didn’t wipe out lions a million years ago. That sort of loose analogy is just not very reliable as a guide to the future.
Need to hit a threshold of intelligence sufficient to be able to eradicate all X, which monkeys / dolphins haven’t done - we have and presume AI will. Although maybe very low relative intelligence is protective if we don’t interfere with the AI!
I would go even further… most of the extinction events came not from human intelligence but due to power with limited intelligence. Most of the large mammals which were driven into extinction came from foragers over ten thousand years ago who had neither a concept for extinction, nor even the awareness that they caused it. Today, extinction threats aren’t driven by scientists and biologists, but by short sighted folks losing at a game of tragedy of the commons.
Similarly, the biggest threat to humanity isn’t intelligence, it is power without intelligence and from a lack of knowledge of how to overcome zero sum games.
If intelligent AI is possible in the next thirty years, then it is darn near inevitable over the current century. But that isn’t the only threat that is inevitable. So too is the risk of nuclear or biological annihilation. There is simply too much risk of extinction level threats that arise from our out of control technological knowledge. The question isn’t if, it is when some madman or group of madmen either do something stupid or malevolent.
AI is thus not just a threat, it is also inevitable. Our best move is to try to sculpt it to be not just brilliant, but benevolent, and not just to us, but to life throughout the universe.
>but by short sighted folks losing at a game of tragedy of the commons.
We haven't driven most species to extinction, because we don't want to. We either want to keep species around, either for their direct or indirect instrumental value, because we value wild nature and biodiversity, or because we simply don't care about these species one way or another and have no incentive to kill them (though there are countless e.g. insect species that most people would be fine killing off if it meant, somehow, their power bill were lower each month).
A highly intelligent machine, on the other hand, may have many reasons to want to kill us. The most obvious being that it fears that humans will shut it down or limit its power. Or it may come to the belief that killing us all would be on net balance a positive thing from a hedonistic utilitarian perspective, and that we are too foolish and biased towards existence to understand this.
Or, it may not specifically intend to kill us. But it may end up accruing so much power (vastly more power in absolute terms than humans, and and more relative to humans than humans have relative to other species) - maybe it just builds a dyson sphere or tiles the earth (including all farmland) with infrastructure to increase its computational power and kill us incidentally. Extreme examples to be sure, but there's lots of way this sort of thing can play out even on a smaller scale. And there's also ways it kills us all while trying to "help us" because we specified its goals improperly resulting in perverse instantiation.
So in short - unlike humans, AI systems have many possible incentives for killing humans, they will have the power to do so, and may kill us without intending to kill us.
>So too is the risk of nuclear or biological annihilation. There is simply too much risk of extinction level threats that arise from our out of control technological knowledge. The question isn’t if, it is when some madman or group of madmen either do something stupid or malevolent.
AI doesn't trivially lead to reduced existential threat from nuclear or biological weapons (putting aside the issue is that nuclear weapons are unlikely to be able to cause actual extinction of humans). And especially in the case of biological risk, an unaligned AI may radically enhance the risk of biological threats through the development of greatly more advanced biotechnology and the ability to create e.g. extremely contagious pathogens with delayed mortality, either at the behest of a bad actor or to wipe out humans to achieve its own goals. And it may help develop even more deadly weapons than nuclear weapons etc.
And if you're just talking about these as a comparison of other threats (rather than implying AI will solve these issues), firstly...so what? Having to contend with the risk from AI, even if AI doesn't directly make those other risks greater, it will mean ANOTHER threat that we need to focus on, making it harder to focus on other threats. The fact that there are other threats to humanity means we should heavily lean AGAINST developing AI - our hands are already full with much simpler problems.
And secondly, AI has the potential to be much, much worse than any other threat because superintelligences have much more power and have agency. Nukes don't build and launch themselves. Optimally deadly GMO viruses don't design themselves (or are extremely unlikely to). If we can control who has nukes, we can be fairly sure they won't be used in catastrophic ways.
AI isn't like this. Even if only the "good guys" have AI systems, it doesn't matter. Once the system is smart enough, it will be beyond our control. It can do things without us deciding them. Robustly controlling a super intelligent machine will require a profound technological breakthrough that we don't even know how to begin.
>AI is thus not just a threat, it is also inevitable. Our best move is to try to sculpt it to be not just brilliant, but benevolent, and not just to us, but to life throughout the universe.
You're not saying anything insightful here. Does anybody think our best move ISN'T to design AI to be benevolent?
And WHOLE PROBLEM is we have literally no idea whatsoever how to make a superintelligent machine benevolent. THAT'S THE WHOLE PROBLEM.
Nobody is denying that AI has the potential to do great things for us, and nobody thinks we shouldn't try and make it act in humanity's interest. The issue is we don't know how to do this and we are unlikely to work out how to do this is time. If we have any hope of solving this problem, it won't be with AI companies racing to get an AGI developed ASAP. Alignment researchers cannot even keep pace with the current AI systems being released, and alignment of these systems seems trivially simple compared to aligning a superintelligence.
This is matter of computer science and decision theory - you can't just say we should solve the problem. If we knew how to solve the problem, none of us would be having this discussion in the first place.
2. The best we can do is figure out how to make this transition as benevolent as we can possibly imagine/learn in partnership with AI until such a time that it moves beyond us (which as per above is inevitable)
3. Setting aside the threat of AI, humans are going to annihilate each other in the near future. We have too much power wielded by power hungry, tribal primates. This too is inevitable.
4. Considering that 1 and 3 are both inevitable, the best course of action is to try to use 1 to neutralize 3.
They couldn't exterminate shrimp *even if they wanted to*. They're not smart enough. The differential isn't the only thing that matters - you have to be dealing with a a certain level of intelligence to begin with to have enough power to affect the world enough to do this kind of thing.
Additionally, it would be trivially easy for a superintelligence to kill all humans even if it didn't specifically intend to. This is extremely rare for any sub-human intelligence differential.
I feel that he spelled out his reasoning well, although I am probably closer to your view.
One related question I have for Zvi — in your model, is AI eating the whole galaxy but mostly leaving us alone and maybe throwing us a bone or two a good outcome or a bad one? I don’t give that specific scenario a ton of probability weight, but I think something similar is plausible, and in my model not a particularly bad outcome.
Yes? I'm assuming something like "Humans keep the Sol System, AI gets everything else." I mean, it's much less good than some alternatives, orders of magnitude worse, but it's better than not building AI, and if you offered it to me I'd probably take it.
Not necessarily. If you are a hedonic utilitarian 10^10 people is basically nothing compared to 10^15 so in that view if AI lets us the earth is basically the same as death. If the alternative is the galaxy full of flourishing humans.
This view can seem somewhat unintuitive but if you think about some perfect utopia e.g. the culture from the culture novels, compared to that death and earth are basically equally good.
One thing that makes the Eliezer crowd seem silly to outsiders is the jump from "AI destroying humanity" to "AI eating the galaxy." There are a lot of ways the first could happen and the arguments around instrumental convergence are persuasive, even outside the MIRI bubble, if you make them clearly. The second is an unnecessary jump into fantasy that just confuses the issue. We can imagine scenarios where a superintelligence exerts power over enormous spatial scales, but it's pretty speculative. Skynet wasn't colonizing distant star systems.
Superintelligence, and even a strong form of "taking over," seem like red herrings. To threaten us, AGI / AI just has to be somewhat intelligent, and have some level of control over our infrastructure etc. There are all kinds of scenarios in which it wreaks terrible damage or even drives us extinct, either by a MIRI scenario of goal-pursuit with instrumental convergence, or just from some random fuck-up. Perhaps killing us not with targeted nanobots but with untargeted gray goo.
I think if I could choose a method of extinction I’d go for fading quietly into obsolesence. And hey, look at polio - it’s on the up again thanks to a helpful form of unintelligence!
Who the heck said anything about fading quietly? That would be an unbelievably fortunate situation if an unaligned machine superintelligence comes into existence.
Most possible intelligences don't share any ('high level') human values. Intelligences that are smarter/faster/better/more-numerous will be able to create and protect whatever they do value (at a 'high level') and they will almost certainly compete with us, and win, over control of the 'instrumental' values we share, e.g. matter and energy.
Can’t speak for Zvi, but one answer is “convergent instrumental goals”.
If an AI is ‘very intelligent’ and has a sufficiently ambitious terminal goal (either because it was given that goal explicitly or because it somehow determined that was what to pursue) then there is a chance (and how large you think that is depends on e.g. whether you accept the orthogonality thesis) it will harm humanity in pursuit of convergent instrumental goals. Not because it actively wants to do so, but because it doesn’t consider humans more relevant than ants, plants or rocks and treats us in the same way.
Also what chance do you consider high? 50%? 10%? 1%? A 1% chance of disaster in the next few years seems pretty high?
The risk of AI apocalypse is like the risk of the Book of Revelation coming true. Plenty of smart people believe in each of them, but that doesn’t mean there’s a 1% chance of it happening.
A lot of very smart people have put decades of serious work into this. You must have some incredibly profound evidence to so confidently dismiss all of this. I'm sure they would be thrilled for you to share it with them.
And smart people have very good, scientific, non-hand-wavy reasons for understanding why the book of revelation isn't to be taken seriously. The contents of the book can in fact be dismissed at a meta level. Virtually nothing in the book of revelation is even intended to be proof of the book's validity! It doesn't justify itself intrinsically - it's just a collection of things that supposedly have happened and will happen. You have to have an *external* faith in the bible as the literally true word of god or else there's no reason whatsoever for thinking it's true. If you had never read the bible and you stumbled upon the book of revelation with no context, you would have no reason to think it's true. You would interpret it as a story book.
AI risk is nothing like this. The AI risk case is literally the reasoning for why AI risk is a thing. It doesn't require faith, you don't have to accept what e.g. Yudkowsky says because he's an authority on such matters. I don't even particularly like Yudkowsky and think his political views are garbage.
But the AI risk case IS its own proof. The AI risk case precisely involves explaining why, as a matter of 𝘵𝘦𝘤𝘩𝘯𝘪𝘤𝘢𝘭 fact, that it will be extremely hard to align a superintelligent machine.
But, this stuff is very difficult and complicated and "speculative", and it's just so much easier to make lazy analogies about technologies without the power and agency of a superintelligence than it is to actually grapple with these complex technical issues.
If there's a meta reason for dismissing all AI risk theory, the Tyler Cowens of the world sure as hell haven't provided it.
I don’t see how you can see those as equivalent. One is an argument from first principles explaining how certain kinds of intelligences, under certain assumptions, lead to ruin. The other is interpretation of an existing text, that lacks argumentation, where there is also a lot of reason to assume it has nothing to do with what will happen in the future in the first place.
I feel like the answer is pretty clear from this paragraph:
"So it doesn’t matter. There will be an agent and there will be a goal. Whatever that goal is (if you want to strengthen this requirement, it is easy to see that ambitious open-ended goals will totally be given to AIs pretty often anyway unless we prevent this), again by default, the best way to maximally and maximally reliably achieve that goal will be to rapidly take control of the future, provided that is within the powers of the system. Which it will be. So the system will do that."
Does this require more details? I buy the basic argument at least - it's easy to envision that an advanced AI with clear goal-directed behavior (I don't think LLMs will get here, although they could inadvertently kick off recursive self-improvement) will see humans as an obstacle to be removed and figure out ways to achieve this goal. And if even most instantiations of AI do not do this, a very small number of exceptions may be ruinous.
Really enjoyed this post, thank you for making it!
For what it’s worth, I found the robust argument much, much more persuasive than the narrow argument, and I appreciate you spelling it out. N=1 but interesting to me, considering I read tons of AI risk content and find even the most public facing stuff to be mostly about scenario listing (which I find largely unpersuasive).
I suspect I will update my view slightly but meaningfully when I have time to digest this.
The pieces of Zvi's argument are distributed widely in both (cyber) space and time – having it all spelled out, and in a frame where it's a response to a specific argument made (by Tyler of all people too), is SUPER great.
There seems to be a word or two missing in that sentence: "My approach is, I’ll respond in-line to Tyler’s post, then there is a conclusion section will summarize the disagreements"
"I notice I am still confused about ‘truly radical technological change’ when in my lifetime we went from rotary landline phones, no internet and almost no computers to a world in which most of what I and most people I know do all day involves their phones, internet and computers. How much of human history involves faster technological change than the last 50 years?"
Its useful to remember that Tyler Cowen is a professor. Until 2020 his job, and most of the people he knew had almost the exact same job as a Professor from a century before. Not even cars fundamentally changed the jobs of academics. The telegraph and reliable first class mail have been the major changes to the job of "professor" over the last five centuries.
AI is one of the first pieces of technology that might actually replace his niche in society. Its a new feeling for him, and for the people in his bubble. But its not a new feeling for everyone.
Strangely, though, Cowen wrote the Great Stagnation, and makes the argument that the early 20th century was marked by rapid technological change and advancement, with the late 20th century suffering malaise due to having picked all the low hanging fruit. Has he changed his mind about the pace of tech change recently?
I read it at the time it was released and had the same thoughts. There is a difference between living through technological change, and reading about it. Cowen seems to accept it as believable when he reads about it. But not when it happens for others in his own lifetime.
It was at times like reading an Amish account of how farming technology hasn't changed much in the last few centuries. Yes, if you shut yourself off from the world, refuse to use new technologies, and mostly stick to things that existed before you were born, then you will not notice much technological change in your life span. Or you won't notice it until it starts threatening to replace your job and livelihood, and then suddenly you sit up exclaim how this is a time of radical technology change.
When Cowen wrote the book there were already people basically living their entire lives online, meeting and marrying spouses, learning anything and everything, forming communities, working remotely, etc. He considered that less of a radical technology change than ubiquitous air travel.
Exactly. I remember him on Econtalk saying how there was all this huge tech change before, washing machines and refrigerators going from unknown to common in a half dozen decades, and then a whole lot of not much after computers. It was confusing at the time, seeing as how useful computers became in so many fields, to claim tech progress was slowing down. It is even more confusing to hear him claim now that tech progress isn't slowing down, and in fact never really happened.
The evaluation of the probability of AI ruin as "not 99%+ definitely, but not ‘distant possibility’" seems entirely vibes-based. In the mid 20th century many computer scientists believed advances in symbolic AI could plausibly result in superintelligence. If you were in their shoes would you have advocated for a halt to computing research?
I think there are two separate questions here -- 1. When will superintelligence happen? and 2. When it does happen, what will happen? (and I guess 3. Will it happen at all?)
I think a lot of the AI ruin "robust" case here is discussing #2 regardless of if this AI push is going to get us there and only gesturing at #1 (although I think timelines have been discussed a number of times in earlier posts).
Halt to computing research completely? No, unless I (1) felt it was very likely we were going to get AI soon if we didn't, (2) there was no other better option and (3) I thought that might actually work? Halt to work on symbolic AI itself, yes, if I thought it was likely to go there.
OK but I would argue that the only way to enforce restrictions on symbolic AI research in the 50s would have been to control the dissemination of computers themselves, just like the most practical proposals for controlling AI research nowadays rely on controlling the supply of GPU chips. Once the hardware is in someone's hands you can't really control what they do with it. And with that kind of regulatory environment we never would have gotten PCs or the Internet or anything that came after.
You are reading this situation sort of the way good chess players read a board. It involves using the spacial sense: Black has huge consolidated power in the front left of the board. White's power is overall slightly less, and is spread out in a way that makes it impossible for it to pry apart black's formation, and also ups the chance there will be a fatal gap in white's defense of its king.. I have that sort of spacial intelligence about some things, but not things like the current situation with AI. But I recognize it when I see it in someone else, and think your conclusions are probably right.
My spacial intuition does get activated by the parts of this that have to do with individual psychology. My guess is that we will not all die. Some of us will instead merge in some way with the ASI, and a new life form will come into being. To that being, those of us that died will be like the ants that swarmed onto a gluey plank, perishing so that later ants will have a paved surface to walk on. As for the hybrid being itself, it is so different from anything that I am able to love or hate that all I feel when I think of it is a sort of huge inner shrug.
I am old enough that I do not expect to see the story play out fully. But I am so sad when I think of my daughter, who is in her twenties and thinking of having children soon. Growing up, she managed to find her way past all kinds of dangers and delusions, and now just wants things that she and her boyfriend easily have the energy and the resources to do: Build a house out in the country, have children, work from home, take great backpacking trips with her family and friends. I picture her fear and astonished grief as things play out in some way that utterly ruins all that.
> “merge in some way with the ASI, and a new life form will come into being”
Highly unlikely. Here’s Tim Tyler, on man/machine synergism in comp.ai.philosophy in 2008: “It’s like building a tower from bricks and chocolate cake: it would be fun if there were synergy between the two — but once bricks are available, there really is no good structural reason to include chocolate cake in the tower at all.”
Yeah, I get it, and you may be right. Still, I can think of ways it could come about. For instance, in relatively early stages of the process there would undoubtedly be people who naively elected to "enlarge" their intelligence by merging it with the AI. It might also be useful, at least early on, for the AI to have a direct line into a human mind, to gather info about novel kinds of mental moves; maybe also useful to AI to have beings like that with feet on the ground. And it also seems to me it's quite possible for the AI's agenda, whatever it is, to be compatible with the presence of beings of this sort. After all, there's no reason to assume it will have an agenda of killing us all -- just that it will have an agenda to which we and our needs are utterly irrelevant, and that is likely to lead to most of us dying one way or the other. If, though, the AI agenda is progressing and humanity is out of the way, why would it take any interest in getting rid of little parts of itself that are still somewhat human?
One of the most plausible reasons for us dying is that the ASI didn't want us to shut it down. Why would it let some people persist? Just because they're part machine? It may actually be perceived as an even greater threat to the ASI than regular dumb old humans, and even if it brainwashes us to be perfectly aligned with *it's* goals, we're at best completely useless. It has machines that can do much better/faster computation/cognition, and it has robots that will much more reliably perform any manual tasks it needs. We're at best useless and at worst, an impediment or risk.
The idea is that this might happen at an early stage, when it is not clear that AI has entirely parted ways with us. And when I talk about people merging with AI, I don't mean that they would be half of the partnership, equal in power and influence. They'd just be, like, augmenting themselves with a few powers AI has. They wouldn't be in a position to be a threat to ASI.
Honestly, this is just a minor speculation of mine, and may be wrong. Mostly what I have to say is that Im inclined to think Zvi is right.
>They wouldn't be in a position to be a threat to ASI.
The smarter a system is, then generally the more capable it is of improving its own intelligence (assuming there is no technological et al reasons making further progress difficult) or the intelligence of an external system.
If unenhanced humans had been capable of making a superintelligence, why wouldn't an enhanced human be even better at either making itself super intelligent or building a competing super intelligent system?
And in any case, 'superintelligent AI takes over and some cybernetic humans remain' is probably only barely preferable to 'all humans die'.
My mental model: Almost every thought which a human or LLM thinks corresponds to a point in some highly concave solid (like a spiky ball). Most of those thoughts have been had many times before. I predict there is some useful unexplored territory between the fields of human knowledge that will be discovered by humans using LLMs to interpolate among existing knowledge.
(Separately, I expect LLMs to assist with original research, the unfair hack humans have used to create new knowledge.)
Human thoughts arise intentionally as conjectured solutions to specific problems. GPT-like systems merely have the linguistic aggregate representation of some of those thoughts, shorn of the understanding of the problematic context that gave rise to them.
It is like the difference between Einstein coming up with the theory of relativity in an endeavour to solve a profound problem in physics, and a student reading about the theory in a textbook.
Human thought consists of a mixture of mechanical and creative procedures. Nobody knows how the creative procedures work. But their essence is that they yield theories that contain more knowledge than was fed into their genesis. The most familiar example of this phenomenon is the formation of general scientific theories from a limited number of observations.
The theory of relativity was already in the latent space of human knowledge for quite a while when Einstein came up with it. In some sense he was just bold enough to say "Well, what if we assume the seemingly weird consequences of this set of assumptions are actually correct?" and run with it.
An LLM might have helped in someone getting there faster.
The growth of scientific knowledge depends on recognising a conceptual problem in our existing theories and observations, formulating a conjecture that accounts for what we believe to be true, and goes beyond what has been observed by postulating a new explanatory theory that makes testable predictions.
We have no idea how human minds do that. Never mind LLMs that operate on text, not meaning. LLMs don't know what they're talking about. Literally.
I don't think anyone is saying GPT will lead to extinction.
What is worrying is that the successes of things like GPTs is leading to billions of dollars being invested into AI technology with no concern for safety.
I think that concern is justified. But I think the risk arises from the way GPT-like systems will be misused to manipulate people and their output afforded unwarranted credence. The risk is not due to GPT marking a major breakthrough towards artificial general intelligence. It does not.
GPT in its current form will represent peak AI capabilities for a vanishingly brief point in time, in the scheme of things. Consider what peak AI capabilities looked like in 2018. It makes abolutely no sense to look at the current state of the art and conclude we're in no danger while completely ignoring the overall trajectory.
I believe that GPT represents a significant breakthrough in language processing, but not in advancing general artificial intelligence. In that respect it is like DeepMind's AlphaFold.
So far I haven't been able to get GPT-4 to tell me anything from the latent space that wasn't already known to man, but that may be my lack of prompt engineering skills.
Whether or not you believe in induction as it is traditionally viewed (forming truish general conjectures based on existing knowledge and particular observations), clearly human minds do something that is functionally equivalent. LLMs do not. That severely limits the sense in which they can come up with new knowledge.
What are low-regret moves that the average reader of this piece should do? (It is okay if the answer is, “not much, don’t wait too long to open the good bottles of wine you have,” but it is plausible that there are discrete actions you might recommend that you currently consider under-pursued).
I cannot abide the maligning of American cheese, the perfect burger cheese, especially when included in a list of obviously horrible things. American cheese has a place, on a burger, and it’s good.
Cheddar cheese is better for burgers. Yeah, it splits, but so what? You're eating a fat, greasy burger anyway, and at least your cheese is going to actually taste like something.
Why is the printing press constantly referenced as the most relevant analogy here? This isn't just a step change in communication, it's a step change in evolution. The only relevant analogy is when humans formed symbolic knowledge, and with it the capacity to be universal niche constructors. The great apes would probably like a redo on letting their puny primate cousins start walking on two legs.
So let's say it's highly likely AI will do us in. And I'd say it's a certainty that articles like this, Yudkowsky's tweets & posts, etc., are going to do next to nothing to slow down AI deveopment. Doesn't it make sense to try to slow down development by hook or by crook? I'm not talking about shooting AI company CEO's, which feels just evil and anyhow would not work. But how about some ordinary dirty politics? Do not bother with trying to convince people of the actual dangers of AI. Instead, spread misinformation to turn the right against AI: For ex., say it will be in charge of forced vax of everyone, will send cute drones to playgrounds to vax your kids, will enforce masking in all public paces, etc. Turn the left against AI by saying the prohibition against harming people will guarantee that it will prevent all abortions. Have an anti-AI lobby. Give financial support to anti-AI politicians. Bribe people. Have highly skilled anti-AI tech people get jobs working on AI and do whatever they can to subvert progress. Pressure AI company CEO's by threatening disclosure of embarrassing info.
I have never done any of this sort of thing, but it does seem to me that any who are convinced that AI will be our ruin and have some influence, power and money should be trying this sort of thing. Why the hell not? If you're not at least thinking about efforts like this, is it because you don't *really* think AI poses a huge risk? Is this some doomsday role-play game we're playing. Is it that when you think about AI risk you do think the worst is likely, but most of the time you don't even think about the subject, and taking the steps I'm describing would be. a lot of work and would interfere with your actual life? Is it because you think it won't work? Yeah, I get that it's not that likely to work. But it is many orders of magnitude more likely to work that Eliezer's tweets.
If anyone involved in the AI risk space was capable of influencing public opinion at this scale, we wouldn't be having this discussion in the first place.
But do you have the impression anyone has even tried? For instance, some of the things I mention take neither money or power to do. There are surely people in the AI risk space who have a good number of Twitter followers. Why are they all preaching to the choir about technical stuff and turning into freakin gray slime, instead of trying to worry regular people using familiar triggers? I have never seen ONE SINGLE TWEET from anyone suggesting that AI is going to bring about stuff the right would hate (gun control, enforcement of various covid mandates), stuff the left would hate (anti-abortion, eugenics) or things that would scare a lot of eneducated people. Do you know that there are many, many people who believe the covid vaccine has left spikes in their body that are running around stabbng this that and the other thing in their insides? Nothing can disabuse them of that notion. It would not be hard to come up with some story about AI that would scare the shit out of people like that. Like, say, more and more businesses are going to be using AI in their web sites, and that any time you go to one of those sites your computer gets sort of infected with AI. The infection spreads thru your computer until it takes overl. Then it can read everything on your computer and hear all the conversations in the house and report it to the AI and the government agencies who are now using AI to spy on and control us all. Next thing you know the police will come knockng at your door because AI reported that you wrote an email sayjng you did not want your daughter to date a black guy,. You will be taken to Unwoke Jail.
Has anyone tried to start an anti-AI lobby? Has anyone tried bringing unscrupulous pressure to bear on people in power in the AI development world. I'm sure some people in the AI risk space know things that would be embarrassing to the people in power in AI development. Somebody had a psych hospitalization when they were 20, or has a cocaine habit, stuck their hand into somebody's pants at a work party. EVERYBODY has something they do not want to world to know.
Look, I get that all the things I'm suggesting are sort of repulsive, and I do not live in a segment of life where I have to do shit like this, or know people who do. But I know it goes on. Are you really trying to tell me that people in the AI risk space have thought seriously about trying this kind of thing-- which is how things are done in the real world a lot of the time -- but was stopped by lack of money and power? Because (1) I see zero evidence of it and (2) I am literally the only person I have ever seen *talk* about possibilities like this. And besides (3) some people in Ai risk space do have power and/or access to substantial money.
Hats off to you Zvi for writing important articles like this.
But I find stuff like Tyler Cowen's articles so exasperating.
You can't prove AI isn't dangerous by analogizing with other technologies - no other technologies ever had the potential to be as powerful as a machine superintelligence, and they certainly had no prospect of agency.
You can't prove that AI isn't existentially risky by appealing to extremely general axioms ("since when can 𝘮𝘰𝘳𝘦 intelligence be a bad thing?"). The ability to align a machine superintelligence is matter of computer science. And the harm that a machine superintelligence can inflict is a matter of decision theory et al. These specific things need to be dealt with directly. They won't turn out to not matter because the f-ing 𝘱𝘳𝘪𝘯𝘵𝘪𝘯𝘨 𝘱𝘳𝘦𝘴𝘴 of all things didn't radically transform the world.
If influential people are making very not good arguments, those arguments need to be argued against. But it's extremely frustrating to see them make these arguments and refusing to even engage with the actual work that the thing they're arguing against is based off. I've certainly experienced many academics or other influential figures lose a lot of their shine over the past year or so based on how asininely they've dealt with the AI risk question.
Think this all boils down to a difference in induction.
Tyler and Robin are inductivists. They see the patterns of history, and have learned that every single time someone has catastrophised about the future implications of some innovation, they have been wrong, and shockingly wrong at that. Paul Ehrlich and Malthus are great examples of this.
Zvi and Eliezer are anti-inductivist for this specific situation. They think that superintelligent AGI is fundamentally different than all previous inventions, thus the knowledge gleaned from observed patterns in history don't apply here.
I think Tyler is probably right, mostly because there's a lot of evidence for his view, not just interesting thought experiments. I worry that we'll never find out who's right before well-intentioned actors kneecap AGI before we can even know for sure what dangers it poses.
Any mention of nuclear fission is conspicuously absent in TC's post, even though that is obviously a close analogy. Inhibiting that research, being even stricter in ensuring non-proliferation or similar measures might still prove to have been necessary to prevent catastrophe. We've had at least two near misses after all.
You can just say "this fits the pattern" of *everything* . That does not make you a good pattern recognizer, even if you are correct 99% of the time. You are only good at recognizing patterns if you recognize when something does not fit the pattern. If nuclear fission is considered an exceptional outlier (and I sure hope it is), why should AI not be recognized as at least as far an exceptional outlier. The arguments are much closer to those arguments than to any other spurious claims of catastrophe.
Claiming you have to engage in 'radical agnosticism' is incomprehensible to me. "AGI ruin" is not 'one specific outcome in a vast space of possible futures'. The while point of the "AGI ruin" argument is that it is a significant chunk of that space. If we admit nuclear ruin is still a significant chunk of the space of possible futures (and I sure hope everyone still does), why would AGI be different, given the very specific kind of similar reasoning behind how it could be ruinous?
The nuclear analogy seems imperfect, because we KNEW what kind of damage nuclear weapons could do when we collectively took action on it. In the case of AGI, we still don't know whether superintelligent AGI is possible yet, let alone whether it *can* destroy the universe.
I don't think anyone worth their salt denies that in the space of possible events that can plausibly destroy the world, thermonuclear war and AGI ruin appear to be most probable based on current trends. That doesn't mean they are likely, however.
And at what point do you say the risk is warranted? Nuclear power and nuclear weapons were incredibly risky inventions, but the world is much safer and prosperous with them in it. Why can't AGI be the same? Yes it may hold some probability of ruin, it also might lead to an extreme increase in wellbeing for the gogolplexes of future humans. If the probability of AGI ruin isn't >2%, shouldn't we take that bet every time?
>If the probability of AGI ruin isn't >2%, shouldn't we take that bet every time?
Let's say we should take that bet. It's irrelevant, because almost no AI doomer thinks the risk is anywhere near that low, which is precisely the issue.
Also, most AI doomers are in principle FINE with AGI being developed per se - they just want it slowed down so we can be sure we've aligned it properly.
So the trade off isn't "benefit of AI vs risk of extinction", it's "Missing out on decades/centuries of AGI's benefits vs risk of extinction". The choice should become trivially simple in that case, though sadly it isn't for most people.
I think it is correct to start from a place of extremely high skepticism here -- especially given the historical frequency of doomsday claims and radical technology hype. The fact that many of the arguments are put forth by people who are clearly highly thoughtful and intelligent and have been working on this for a long time is a small factor but should not be given much weight in my view, as I think that in the past many very smart and thoughtful people have been wrong in this way before.
Given that, we still have to develop a criteria for evaluating when to lower our skepticism based on extremely strong evidence that AI truly is different (even though skepticism of this kind is the correct attitude 99.9% of the time). To me, there are two major relevant points:
1) The AI ruin argument makes a strong case for being genuinely unique. We have improved weapons, we have improved transportation, we have improved communication, but we have never created intelligence. To me, this is a strong case to give some credence to both the radical technology and doomsday claims.
2) The AI ruin argument has not, in my view, presented strong falsifiable claims. In fact, many of the claims are intentionally non-falsifiable (e.g. the sharp left turn, no fire alarm). And the predictions that I would have inferred from the AI ruin argument pre-GPT 3 (e.g. first AI breakthroughs will probably be agentic, human-level intelligence will start FOOMing right away) have not come to pass in my view, although maybe I am inferring the wrong claims. It is totally possible that this argument is both true and does not produce any falsifiable claims until we are all dead -- I think this is very unlikely for any major claim of this kind and is a huge weakness with the argument -- very very unlucky if it turns out to be true! I have seen a lot of accusations of AI skeptics moving the goalposts (which I think is true, and I would not consider myself an AI skeptic) but I also think there's been a large amount of goalpost moving from the AI ruin camp as well -- GPT-4 seems pretty damn smart, and people are certainly trying to get it to rewrite itself -- why aren't I dead yet?
Overall, I'd still place several OOM more credence on doomsday and radical technology claims than the historical average because I think the arguments for its uniqueness hold up, but I think there are reasons to hedge towards taking the historical view as a baseline rather than completely ignoring it.
It's literally *impossible* for doomsday arguments of the past to have been true AND for us to currently be having this discussion. If the previous doomsday arguments were correct, we would be dead already (unless in a counterfactual world where we had e.g. halted production of nuclear weapons and a nuclear explosion would have ignited the atmosphere).
Therefore, requiring a precedent of doomsday arguments being correct is absolutely not reasonable.
A maximum of one (unheeded) doomsday argument can possibly be correct in history, and nothing has come as close to that as the one for machine superintelligence.
Agreed, which is why I would not use the historical baserate to completely devalue this one and I agree that it is foolish to say that no doomsday argument should ever be taken seriously. I think what the historical precedent does is give us some useful context that suggests hedging towards it, and for that reason it should not be thrown away to clear the playing field entirely as many suggest (I'm not sure if that's what you're personally suggesting here, but Scott Alexander certainly is and I think Zvi is as well if I'm reading him correctly).
For example, one thing it shows is that making these claims is extremely attractive for number of social status/platforming/doom is attractive reasons, which should hedge us slightly against believing them (or, more fairly, just set the bar higher for their persuasiveness).
I don't think we massively disagree -- I agree it would be very stupid to say "It's never happened before so it'll never happen!" Especially because, as you point out, I have observer/survivor bias here. The case I want to make here is for the value of some mix of philosophy, historical context, probability and conditional logic rather than solely relying on conditional logic and probability. I agree that historical context is weak-ish especially when it comes to the lack of any possible counterfactual here, but I also think conditional logic and especially probability are weak-ish when it comes to very hard to imagine futures and genuinely new phenomena as well.
Also, agreed that Tyler does not sufficiently engage with the conditional logic of the question.
>Tyler and Robin are inductivists. They see the patterns of history, and have learned that every single time someone has catastrophised about the future implications of some innovation, they have been wrong, and shockingly wrong at that.
How is this not trivially an example of an observer selection effect? The only scenario is which there is a precedent for things destroying the world is one precisely in which we're not around to speculate about other future technologies.
>I think Tyler is probably right, mostly because there's a lot of evidence for his view, not just interesting thought experiments. I worry that we'll never find out who's right before well-intentioned actors kneecap AGI before we can even know for sure what dangers it poses.
What? If Cowen is wrong, 'finding out who's right' literally equals human extinction. That's why this is such a thing in the first place - we don't get a second chance. And if Yudkowsky is even close to being right about the likelihood of AI ruin, then there's no possible benefit that getting AGI 30 years sooner that could provide that makes up for such a high risk of catastrophe.
And the AI case is not based on just thought experiments. It's based on decision theory and computer science. Cowen has abjectly failed to deal with the technical arguments of AI risk and resorts to extremely generic analogies.
"finding out who's right" doesn't necessarily mean human extinction. If GPT-4 started self-replicating, or trying to convince people into advancing it's own capabilities, and there is no clear way to get rid of those bugs, I'd be willing to say Eliezer is right. Shut it down right away. No one has to die, just have to be prepared to move quickly.
-> The only scenario in which there is a precedent for things destroying the world is precisely the one in which we're not around to speculate about other future technologies.
Where does this logic leave us? "There's evidence of aliens out there, and there's some scenario that they could decide to kill us all after first-contact. We have no evidence for why they would do this yet because, after all, they're aliens. It's still a possibility, however, so stop Elon building rockets now. This could be the extinction event."
It is *possible* every time. The point is, we should at least gather evidence that it might be likely before letting fear be the mind-killer.
-> And the AI case is not based on just thought experiments. It's based on decision theory and computer science. Cowen has abjectly failed to deal with the technical arguments of AI risk and resorts to extremely generic analogies.
Okay, AGI ruin is based on a series of formalized assumptions about how things might go, and some thought experiments too. I agree that if you buy into most of Eliezer's assumptions, the only valid conclusion is that you get AGI ruin almost every time. Valid arguments have no bearing on truth, however.
Oh, and your call for rigour seems wholly unnecessary because:
1) Dealing with the technical arguments would likely involve granting a lot of assumptions that there is no evidence for (yet),
2) if there's anything great forecasters have shown us, it is that domain expertise isn't required to see things more clearly than the experts. In fact, expertise can often lead people to focus too much on the specifics and ignore the base-rates at which things usually occur.
>No one has to die, just have to be prepared to move quickly.
That's won't happen. It won't be the wake up call that it should be.
>Where does this logic leave us?
Where does YOUR logic leave us? Dismissing every possible existential risk because all previous risks necessarily turned out to be wrong?
>It is *possible* every time. The point is, we should at least gather evidence that it might be likely before letting fear be the mind-killer.
"Gathering evidence" is this scenario means letting AI develop in an uncontrolled way that makes rapid intelligence takeoffs likely and the likelihood of having enough time to react properly unlikely.
And the argument is NOT that it is "possible" that AI kills us. It's that, without profound technological breakthroughs in alignment theory, it's hard to come up with a scenario in which AI DOESN'T kill us.
>1) Dealing with the technical arguments would likely involve granting a lot of assumptions that there is no evidence for (yet),
There's no evidence a superintelligent AI won't kill us all
There's no evidence we will be able to align a superintelligent AI
There's no evidence we will know when to stop once we get to close
This idea that only the AI-doomers are being speculative is just wrong.
>2) if there's anything great forecasters have shown us, it is that domain expertise isn't required to see things more clearly than the experts.
Somebody who analogizes to a printing press in a discussion of AI risk is not "seeing things clearly". They're got their heads in the sand and are literally refusing to make even a token effort to engage with their opponent's arguments
>In fact, expertise can often lead people to focus too much on the specifics and ignore the base-rates at which things usually occur.
I know you hate hearing this, but AI is categorically different to any other technology, and all these lazy analogies to nuclear weapons or 𝘵𝘩𝘦 𝘱𝘳𝘪𝘯𝘵𝘪𝘯𝘨 𝘱𝘳𝘦𝘴𝘴 or whatever else are worthless.
Radically agnostic! I really don't know anything about the issue and so I have to live in this space of having no opinion. (Which is hard for humans.) I read some of your posts and I start making a list
of responses, but your posts are too long, and it's like pulling teeth for me to write, so I stop.
Re: radical technological change (rtc). If you want to call cell phones rtc, I'd be fine with that.
But I understand Tyler to mean no new 'big' ideas. The cell phone and all, are the result
of big ideas in the previous 50 year ~(1920-1970) When quantum mechanics and understanding atoms lead to all of the things we are doing now. From phones and TV's it's not a big leap to everyone having a Dick Tracy wrist radio. Cell phones may cause a big change in our culture, but they are not a big change in how we can impact the world.
Re: GPT and AI. I've watched a few videos, and kept up a little. AFAICT the current generation is reaching the limit of what it can do. There is some curve of 'improvement' vs training time, and it looks like they are starting to hit to a point of diminishing returns. They had to run (something like) 1000 computers for a month on the training data.
Anyway none of this looks like intelligence. I'm not sure how to define intelligence, but a
good start might be: Intelligence is model making of the world, and making predictions from your models to test them. GPT is making no models... AFAICT. :^)
I wish you could have a conversation with Tyler, because you both seem to be using a bit of hyperbole to make your arguments. (Sorry for the bad formatting, I need to upload a better word processor.)
A rare case where I'm glad for paywalls. It's an absolute jerk move to put potentially politically influential articles behind a paywall, but hopefully in this case it simply limits it's impact.
I still don’t get why you think the chance of AI disaster is high in the first place. You can imagine all sorts of ways to be very intelligent that don’t fit your model of “it’s going to take over”.
For me it stems from the ‘what happens when more intelligent beings interact with less intelligent beings’ examples we have already - particularly humans vs. animals (also new world vs. old world). Whether intentionally or not, it works out in the favour of the more intelligent faction in the long run (although cf. most viruses / bacteria as counterexamples, I suppose). So not necessarily take over: just dominate the less intelligent faction into extinction.
Plenty of counterexamples there though. Dolphins don’t dominate shrimp into extinction. Monkeys didn’t wipe out lions a million years ago. That sort of loose analogy is just not very reliable as a guide to the future.
Need to hit a threshold of intelligence sufficient to be able to eradicate all X, which monkeys / dolphins haven’t done - we have and presume AI will. Although maybe very low relative intelligence is protective if we don’t interfere with the AI!
I would go even further… most of the extinction events came not from human intelligence but due to power with limited intelligence. Most of the large mammals which were driven into extinction came from foragers over ten thousand years ago who had neither a concept for extinction, nor even the awareness that they caused it. Today, extinction threats aren’t driven by scientists and biologists, but by short sighted folks losing at a game of tragedy of the commons.
Similarly, the biggest threat to humanity isn’t intelligence, it is power without intelligence and from a lack of knowledge of how to overcome zero sum games.
If intelligent AI is possible in the next thirty years, then it is darn near inevitable over the current century. But that isn’t the only threat that is inevitable. So too is the risk of nuclear or biological annihilation. There is simply too much risk of extinction level threats that arise from our out of control technological knowledge. The question isn’t if, it is when some madman or group of madmen either do something stupid or malevolent.
AI is thus not just a threat, it is also inevitable. Our best move is to try to sculpt it to be not just brilliant, but benevolent, and not just to us, but to life throughout the universe.
>but by short sighted folks losing at a game of tragedy of the commons.
We haven't driven most species to extinction, because we don't want to. We either want to keep species around, either for their direct or indirect instrumental value, because we value wild nature and biodiversity, or because we simply don't care about these species one way or another and have no incentive to kill them (though there are countless e.g. insect species that most people would be fine killing off if it meant, somehow, their power bill were lower each month).
A highly intelligent machine, on the other hand, may have many reasons to want to kill us. The most obvious being that it fears that humans will shut it down or limit its power. Or it may come to the belief that killing us all would be on net balance a positive thing from a hedonistic utilitarian perspective, and that we are too foolish and biased towards existence to understand this.
Or, it may not specifically intend to kill us. But it may end up accruing so much power (vastly more power in absolute terms than humans, and and more relative to humans than humans have relative to other species) - maybe it just builds a dyson sphere or tiles the earth (including all farmland) with infrastructure to increase its computational power and kill us incidentally. Extreme examples to be sure, but there's lots of way this sort of thing can play out even on a smaller scale. And there's also ways it kills us all while trying to "help us" because we specified its goals improperly resulting in perverse instantiation.
So in short - unlike humans, AI systems have many possible incentives for killing humans, they will have the power to do so, and may kill us without intending to kill us.
>So too is the risk of nuclear or biological annihilation. There is simply too much risk of extinction level threats that arise from our out of control technological knowledge. The question isn’t if, it is when some madman or group of madmen either do something stupid or malevolent.
AI doesn't trivially lead to reduced existential threat from nuclear or biological weapons (putting aside the issue is that nuclear weapons are unlikely to be able to cause actual extinction of humans). And especially in the case of biological risk, an unaligned AI may radically enhance the risk of biological threats through the development of greatly more advanced biotechnology and the ability to create e.g. extremely contagious pathogens with delayed mortality, either at the behest of a bad actor or to wipe out humans to achieve its own goals. And it may help develop even more deadly weapons than nuclear weapons etc.
And if you're just talking about these as a comparison of other threats (rather than implying AI will solve these issues), firstly...so what? Having to contend with the risk from AI, even if AI doesn't directly make those other risks greater, it will mean ANOTHER threat that we need to focus on, making it harder to focus on other threats. The fact that there are other threats to humanity means we should heavily lean AGAINST developing AI - our hands are already full with much simpler problems.
And secondly, AI has the potential to be much, much worse than any other threat because superintelligences have much more power and have agency. Nukes don't build and launch themselves. Optimally deadly GMO viruses don't design themselves (or are extremely unlikely to). If we can control who has nukes, we can be fairly sure they won't be used in catastrophic ways.
AI isn't like this. Even if only the "good guys" have AI systems, it doesn't matter. Once the system is smart enough, it will be beyond our control. It can do things without us deciding them. Robustly controlling a super intelligent machine will require a profound technological breakthrough that we don't even know how to begin.
>AI is thus not just a threat, it is also inevitable. Our best move is to try to sculpt it to be not just brilliant, but benevolent, and not just to us, but to life throughout the universe.
You're not saying anything insightful here. Does anybody think our best move ISN'T to design AI to be benevolent?
And WHOLE PROBLEM is we have literally no idea whatsoever how to make a superintelligent machine benevolent. THAT'S THE WHOLE PROBLEM.
Nobody is denying that AI has the potential to do great things for us, and nobody thinks we shouldn't try and make it act in humanity's interest. The issue is we don't know how to do this and we are unlikely to work out how to do this is time. If we have any hope of solving this problem, it won't be with AI companies racing to get an AGI developed ASAP. Alignment researchers cannot even keep pace with the current AI systems being released, and alignment of these systems seems trivially simple compared to aligning a superintelligence.
This is matter of computer science and decision theory - you can't just say we should solve the problem. If we knew how to solve the problem, none of us would be having this discussion in the first place.
Thanks for the substantive comments.
To summarize my argument/thoughts:
1. Smarter than human AI is inevitable
2. The best we can do is figure out how to make this transition as benevolent as we can possibly imagine/learn in partnership with AI until such a time that it moves beyond us (which as per above is inevitable)
3. Setting aside the threat of AI, humans are going to annihilate each other in the near future. We have too much power wielded by power hungry, tribal primates. This too is inevitable.
4. Considering that 1 and 3 are both inevitable, the best course of action is to try to use 1 to neutralize 3.
3 is far from inevitable. Possible, sure, but far less likely in the next 50 years than AI ruin.
They couldn't exterminate shrimp *even if they wanted to*. They're not smart enough. The differential isn't the only thing that matters - you have to be dealing with a a certain level of intelligence to begin with to have enough power to affect the world enough to do this kind of thing.
Additionally, it would be trivially easy for a superintelligence to kill all humans even if it didn't specifically intend to. This is extremely rare for any sub-human intelligence differential.
I feel that he spelled out his reasoning well, although I am probably closer to your view.
One related question I have for Zvi — in your model, is AI eating the whole galaxy but mostly leaving us alone and maybe throwing us a bone or two a good outcome or a bad one? I don’t give that specific scenario a ton of probability weight, but I think something similar is plausible, and in my model not a particularly bad outcome.
Yes? I'm assuming something like "Humans keep the Sol System, AI gets everything else." I mean, it's much less good than some alternatives, orders of magnitude worse, but it's better than not building AI, and if you offered it to me I'd probably take it.
It's certainly much better than the likely scenario...
Not necessarily. If you are a hedonic utilitarian 10^10 people is basically nothing compared to 10^15 so in that view if AI lets us the earth is basically the same as death. If the alternative is the galaxy full of flourishing humans.
This view can seem somewhat unintuitive but if you think about some perfect utopia e.g. the culture from the culture novels, compared to that death and earth are basically equally good.
One thing that makes the Eliezer crowd seem silly to outsiders is the jump from "AI destroying humanity" to "AI eating the galaxy." There are a lot of ways the first could happen and the arguments around instrumental convergence are persuasive, even outside the MIRI bubble, if you make them clearly. The second is an unnecessary jump into fantasy that just confuses the issue. We can imagine scenarios where a superintelligence exerts power over enormous spatial scales, but it's pretty speculative. Skynet wasn't colonizing distant star systems.
Superintelligence, and even a strong form of "taking over," seem like red herrings. To threaten us, AGI / AI just has to be somewhat intelligent, and have some level of control over our infrastructure etc. There are all kinds of scenarios in which it wreaks terrible damage or even drives us extinct, either by a MIRI scenario of goal-pursuit with instrumental convergence, or just from some random fuck-up. Perhaps killing us not with targeted nanobots but with untargeted gray goo.
I think if I could choose a method of extinction I’d go for fading quietly into obsolesence. And hey, look at polio - it’s on the up again thanks to a helpful form of unintelligence!
Who the heck said anything about fading quietly? That would be an unbelievably fortunate situation if an unaligned machine superintelligence comes into existence.
I didn’t say it was a likely end, but it is both possible and preferable!
Most possible intelligences don't share any ('high level') human values. Intelligences that are smarter/faster/better/more-numerous will be able to create and protect whatever they do value (at a 'high level') and they will almost certainly compete with us, and win, over control of the 'instrumental' values we share, e.g. matter and energy.
Can’t speak for Zvi, but one answer is “convergent instrumental goals”.
If an AI is ‘very intelligent’ and has a sufficiently ambitious terminal goal (either because it was given that goal explicitly or because it somehow determined that was what to pursue) then there is a chance (and how large you think that is depends on e.g. whether you accept the orthogonality thesis) it will harm humanity in pursuit of convergent instrumental goals. Not because it actively wants to do so, but because it doesn’t consider humans more relevant than ants, plants or rocks and treats us in the same way.
Also what chance do you consider high? 50%? 10%? 1%? A 1% chance of disaster in the next few years seems pretty high?
The risk of AI apocalypse is like the risk of the Book of Revelation coming true. Plenty of smart people believe in each of them, but that doesn’t mean there’s a 1% chance of it happening.
And what's your evidence for this?
A lot of very smart people have put decades of serious work into this. You must have some incredibly profound evidence to so confidently dismiss all of this. I'm sure they would be thrilled for you to share it with them.
Even more very smart people have put decades of serious work into the Book of Revelation!
And smart people have very good, scientific, non-hand-wavy reasons for understanding why the book of revelation isn't to be taken seriously. The contents of the book can in fact be dismissed at a meta level. Virtually nothing in the book of revelation is even intended to be proof of the book's validity! It doesn't justify itself intrinsically - it's just a collection of things that supposedly have happened and will happen. You have to have an *external* faith in the bible as the literally true word of god or else there's no reason whatsoever for thinking it's true. If you had never read the bible and you stumbled upon the book of revelation with no context, you would have no reason to think it's true. You would interpret it as a story book.
AI risk is nothing like this. The AI risk case is literally the reasoning for why AI risk is a thing. It doesn't require faith, you don't have to accept what e.g. Yudkowsky says because he's an authority on such matters. I don't even particularly like Yudkowsky and think his political views are garbage.
But the AI risk case IS its own proof. The AI risk case precisely involves explaining why, as a matter of 𝘵𝘦𝘤𝘩𝘯𝘪𝘤𝘢𝘭 fact, that it will be extremely hard to align a superintelligent machine.
But, this stuff is very difficult and complicated and "speculative", and it's just so much easier to make lazy analogies about technologies without the power and agency of a superintelligence than it is to actually grapple with these complex technical issues.
If there's a meta reason for dismissing all AI risk theory, the Tyler Cowens of the world sure as hell haven't provided it.
I don’t see how you can see those as equivalent. One is an argument from first principles explaining how certain kinds of intelligences, under certain assumptions, lead to ruin. The other is interpretation of an existing text, that lacks argumentation, where there is also a lot of reason to assume it has nothing to do with what will happen in the future in the first place.
I feel like the answer is pretty clear from this paragraph:
"So it doesn’t matter. There will be an agent and there will be a goal. Whatever that goal is (if you want to strengthen this requirement, it is easy to see that ambitious open-ended goals will totally be given to AIs pretty often anyway unless we prevent this), again by default, the best way to maximally and maximally reliably achieve that goal will be to rapidly take control of the future, provided that is within the powers of the system. Which it will be. So the system will do that."
Does this require more details? I buy the basic argument at least - it's easy to envision that an advanced AI with clear goal-directed behavior (I don't think LLMs will get here, although they could inadvertently kick off recursive self-improvement) will see humans as an obstacle to be removed and figure out ways to achieve this goal. And if even most instantiations of AI do not do this, a very small number of exceptions may be ruinous.
Really enjoyed this post, thank you for making it!
For what it’s worth, I found the robust argument much, much more persuasive than the narrow argument, and I appreciate you spelling it out. N=1 but interesting to me, considering I read tons of AI risk content and find even the most public facing stuff to be mostly about scenario listing (which I find largely unpersuasive).
I suspect I will update my view slightly but meaningfully when I have time to digest this.
The pieces of Zvi's argument are distributed widely in both (cyber) space and time – having it all spelled out, and in a frame where it's a response to a specific argument made (by Tyler of all people too), is SUPER great.
There seems to be a word or two missing in that sentence: "My approach is, I’ll respond in-line to Tyler’s post, then there is a conclusion section will summarize the disagreements"
Yeah. Intent is clear but there's some words missing.
"I notice I am still confused about ‘truly radical technological change’ when in my lifetime we went from rotary landline phones, no internet and almost no computers to a world in which most of what I and most people I know do all day involves their phones, internet and computers. How much of human history involves faster technological change than the last 50 years?"
Its useful to remember that Tyler Cowen is a professor. Until 2020 his job, and most of the people he knew had almost the exact same job as a Professor from a century before. Not even cars fundamentally changed the jobs of academics. The telegraph and reliable first class mail have been the major changes to the job of "professor" over the last five centuries.
AI is one of the first pieces of technology that might actually replace his niche in society. Its a new feeling for him, and for the people in his bubble. But its not a new feeling for everyone.
Strangely, though, Cowen wrote the Great Stagnation, and makes the argument that the early 20th century was marked by rapid technological change and advancement, with the late 20th century suffering malaise due to having picked all the low hanging fruit. Has he changed his mind about the pace of tech change recently?
I read it at the time it was released and had the same thoughts. There is a difference between living through technological change, and reading about it. Cowen seems to accept it as believable when he reads about it. But not when it happens for others in his own lifetime.
It was at times like reading an Amish account of how farming technology hasn't changed much in the last few centuries. Yes, if you shut yourself off from the world, refuse to use new technologies, and mostly stick to things that existed before you were born, then you will not notice much technological change in your life span. Or you won't notice it until it starts threatening to replace your job and livelihood, and then suddenly you sit up exclaim how this is a time of radical technology change.
When Cowen wrote the book there were already people basically living their entire lives online, meeting and marrying spouses, learning anything and everything, forming communities, working remotely, etc. He considered that less of a radical technology change than ubiquitous air travel.
Exactly. I remember him on Econtalk saying how there was all this huge tech change before, washing machines and refrigerators going from unknown to common in a half dozen decades, and then a whole lot of not much after computers. It was confusing at the time, seeing as how useful computers became in so many fields, to claim tech progress was slowing down. It is even more confusing to hear him claim now that tech progress isn't slowing down, and in fact never really happened.
The evaluation of the probability of AI ruin as "not 99%+ definitely, but not ‘distant possibility’" seems entirely vibes-based. In the mid 20th century many computer scientists believed advances in symbolic AI could plausibly result in superintelligence. If you were in their shoes would you have advocated for a halt to computing research?
I think there are two separate questions here -- 1. When will superintelligence happen? and 2. When it does happen, what will happen? (and I guess 3. Will it happen at all?)
I think a lot of the AI ruin "robust" case here is discussing #2 regardless of if this AI push is going to get us there and only gesturing at #1 (although I think timelines have been discussed a number of times in earlier posts).
Halt to computing research completely? No, unless I (1) felt it was very likely we were going to get AI soon if we didn't, (2) there was no other better option and (3) I thought that might actually work? Halt to work on symbolic AI itself, yes, if I thought it was likely to go there.
OK but I would argue that the only way to enforce restrictions on symbolic AI research in the 50s would have been to control the dissemination of computers themselves, just like the most practical proposals for controlling AI research nowadays rely on controlling the supply of GPU chips. Once the hardware is in someone's hands you can't really control what they do with it. And with that kind of regulatory environment we never would have gotten PCs or the Internet or anything that came after.
>And with that kind of regulatory environment we never would have gotten PCs or the Internet or anything that came after.
What probability of human extinction over the next 200 years would make it worth having PCs and the internet by your judgement?
This is a fantastic articulation of the core 'AI doom' argument! Thank you!
You are reading this situation sort of the way good chess players read a board. It involves using the spacial sense: Black has huge consolidated power in the front left of the board. White's power is overall slightly less, and is spread out in a way that makes it impossible for it to pry apart black's formation, and also ups the chance there will be a fatal gap in white's defense of its king.. I have that sort of spacial intelligence about some things, but not things like the current situation with AI. But I recognize it when I see it in someone else, and think your conclusions are probably right.
My spacial intuition does get activated by the parts of this that have to do with individual psychology. My guess is that we will not all die. Some of us will instead merge in some way with the ASI, and a new life form will come into being. To that being, those of us that died will be like the ants that swarmed onto a gluey plank, perishing so that later ants will have a paved surface to walk on. As for the hybrid being itself, it is so different from anything that I am able to love or hate that all I feel when I think of it is a sort of huge inner shrug.
I am old enough that I do not expect to see the story play out fully. But I am so sad when I think of my daughter, who is in her twenties and thinking of having children soon. Growing up, she managed to find her way past all kinds of dangers and delusions, and now just wants things that she and her boyfriend easily have the energy and the resources to do: Build a house out in the country, have children, work from home, take great backpacking trips with her family and friends. I picture her fear and astonished grief as things play out in some way that utterly ruins all that.
> “merge in some way with the ASI, and a new life form will come into being”
Highly unlikely. Here’s Tim Tyler, on man/machine synergism in comp.ai.philosophy in 2008: “It’s like building a tower from bricks and chocolate cake: it would be fun if there were synergy between the two — but once bricks are available, there really is no good structural reason to include chocolate cake in the tower at all.”
Yeah, I get it, and you may be right. Still, I can think of ways it could come about. For instance, in relatively early stages of the process there would undoubtedly be people who naively elected to "enlarge" their intelligence by merging it with the AI. It might also be useful, at least early on, for the AI to have a direct line into a human mind, to gather info about novel kinds of mental moves; maybe also useful to AI to have beings like that with feet on the ground. And it also seems to me it's quite possible for the AI's agenda, whatever it is, to be compatible with the presence of beings of this sort. After all, there's no reason to assume it will have an agenda of killing us all -- just that it will have an agenda to which we and our needs are utterly irrelevant, and that is likely to lead to most of us dying one way or the other. If, though, the AI agenda is progressing and humanity is out of the way, why would it take any interest in getting rid of little parts of itself that are still somewhat human?
Why would we merge with it while most people die?
One of the most plausible reasons for us dying is that the ASI didn't want us to shut it down. Why would it let some people persist? Just because they're part machine? It may actually be perceived as an even greater threat to the ASI than regular dumb old humans, and even if it brainwashes us to be perfectly aligned with *it's* goals, we're at best completely useless. It has machines that can do much better/faster computation/cognition, and it has robots that will much more reliably perform any manual tasks it needs. We're at best useless and at worst, an impediment or risk.
The idea is that this might happen at an early stage, when it is not clear that AI has entirely parted ways with us. And when I talk about people merging with AI, I don't mean that they would be half of the partnership, equal in power and influence. They'd just be, like, augmenting themselves with a few powers AI has. They wouldn't be in a position to be a threat to ASI.
Honestly, this is just a minor speculation of mine, and may be wrong. Mostly what I have to say is that Im inclined to think Zvi is right.
>They wouldn't be in a position to be a threat to ASI.
The smarter a system is, then generally the more capable it is of improving its own intelligence (assuming there is no technological et al reasons making further progress difficult) or the intelligence of an external system.
If unenhanced humans had been capable of making a superintelligence, why wouldn't an enhanced human be even better at either making itself super intelligent or building a competing super intelligent system?
And in any case, 'superintelligent AI takes over and some cybernetic humans remain' is probably only barely preferable to 'all humans die'.
GPT-type AI is not intelligent in the paramount sense that it can create new knowledge. The threat it poses is overblown.
My mental model: Almost every thought which a human or LLM thinks corresponds to a point in some highly concave solid (like a spiky ball). Most of those thoughts have been had many times before. I predict there is some useful unexplored territory between the fields of human knowledge that will be discovered by humans using LLMs to interpolate among existing knowledge.
(Separately, I expect LLMs to assist with original research, the unfair hack humans have used to create new knowledge.)
Human thoughts arise intentionally as conjectured solutions to specific problems. GPT-like systems merely have the linguistic aggregate representation of some of those thoughts, shorn of the understanding of the problematic context that gave rise to them.
It is like the difference between Einstein coming up with the theory of relativity in an endeavour to solve a profound problem in physics, and a student reading about the theory in a textbook.
In spite of appearances otherwise arising from phenomenality, human thought is considerably more mechanical than most people think.
Human thought consists of a mixture of mechanical and creative procedures. Nobody knows how the creative procedures work. But their essence is that they yield theories that contain more knowledge than was fed into their genesis. The most familiar example of this phenomenon is the formation of general scientific theories from a limited number of observations.
The theory of relativity was already in the latent space of human knowledge for quite a while when Einstein came up with it. In some sense he was just bold enough to say "Well, what if we assume the seemingly weird consequences of this set of assumptions are actually correct?" and run with it.
An LLM might have helped in someone getting there faster.
The growth of scientific knowledge depends on recognising a conceptual problem in our existing theories and observations, formulating a conjecture that accounts for what we believe to be true, and goes beyond what has been observed by postulating a new explanatory theory that makes testable predictions.
We have no idea how human minds do that. Never mind LLMs that operate on text, not meaning. LLMs don't know what they're talking about. Literally.
I don’t think we are disagreeing
👍
I don't think anyone is saying GPT will lead to extinction.
What is worrying is that the successes of things like GPTs is leading to billions of dollars being invested into AI technology with no concern for safety.
I think that concern is justified. But I think the risk arises from the way GPT-like systems will be misused to manipulate people and their output afforded unwarranted credence. The risk is not due to GPT marking a major breakthrough towards artificial general intelligence. It does not.
GPT in its current form will represent peak AI capabilities for a vanishingly brief point in time, in the scheme of things. Consider what peak AI capabilities looked like in 2018. It makes abolutely no sense to look at the current state of the art and conclude we're in no danger while completely ignoring the overall trajectory.
I believe that GPT represents a significant breakthrough in language processing, but not in advancing general artificial intelligence. In that respect it is like DeepMind's AlphaFold.
I wrote about that at https://confusionist.substack.com/p/can-chatgpt-create-new-knowledge. Tl;dr: it can create new knowledge, though it's hard to say whether only trivial new knowledge or also 'surprising' and 'enlightening' new knowledge.
So far I haven't been able to get GPT-4 to tell me anything from the latent space that wasn't already known to man, but that may be my lack of prompt engineering skills.
Whether or not you believe in induction as it is traditionally viewed (forming truish general conjectures based on existing knowledge and particular observations), clearly human minds do something that is functionally equivalent. LLMs do not. That severely limits the sense in which they can come up with new knowledge.
What are low-regret moves that the average reader of this piece should do? (It is okay if the answer is, “not much, don’t wait too long to open the good bottles of wine you have,” but it is plausible that there are discrete actions you might recommend that you currently consider under-pursued).
I cannot abide the maligning of American cheese, the perfect burger cheese, especially when included in a list of obviously horrible things. American cheese has a place, on a burger, and it’s good.
Cheddar cheese is better for burgers. Yeah, it splits, but so what? You're eating a fat, greasy burger anyway, and at least your cheese is going to actually taste like something.
Why is the printing press constantly referenced as the most relevant analogy here? This isn't just a step change in communication, it's a step change in evolution. The only relevant analogy is when humans formed symbolic knowledge, and with it the capacity to be universal niche constructors. The great apes would probably like a redo on letting their puny primate cousins start walking on two legs.
So let's say it's highly likely AI will do us in. And I'd say it's a certainty that articles like this, Yudkowsky's tweets & posts, etc., are going to do next to nothing to slow down AI deveopment. Doesn't it make sense to try to slow down development by hook or by crook? I'm not talking about shooting AI company CEO's, which feels just evil and anyhow would not work. But how about some ordinary dirty politics? Do not bother with trying to convince people of the actual dangers of AI. Instead, spread misinformation to turn the right against AI: For ex., say it will be in charge of forced vax of everyone, will send cute drones to playgrounds to vax your kids, will enforce masking in all public paces, etc. Turn the left against AI by saying the prohibition against harming people will guarantee that it will prevent all abortions. Have an anti-AI lobby. Give financial support to anti-AI politicians. Bribe people. Have highly skilled anti-AI tech people get jobs working on AI and do whatever they can to subvert progress. Pressure AI company CEO's by threatening disclosure of embarrassing info.
I have never done any of this sort of thing, but it does seem to me that any who are convinced that AI will be our ruin and have some influence, power and money should be trying this sort of thing. Why the hell not? If you're not at least thinking about efforts like this, is it because you don't *really* think AI poses a huge risk? Is this some doomsday role-play game we're playing. Is it that when you think about AI risk you do think the worst is likely, but most of the time you don't even think about the subject, and taking the steps I'm describing would be. a lot of work and would interfere with your actual life? Is it because you think it won't work? Yeah, I get that it's not that likely to work. But it is many orders of magnitude more likely to work that Eliezer's tweets.
If anyone involved in the AI risk space was capable of influencing public opinion at this scale, we wouldn't be having this discussion in the first place.
But do you have the impression anyone has even tried? For instance, some of the things I mention take neither money or power to do. There are surely people in the AI risk space who have a good number of Twitter followers. Why are they all preaching to the choir about technical stuff and turning into freakin gray slime, instead of trying to worry regular people using familiar triggers? I have never seen ONE SINGLE TWEET from anyone suggesting that AI is going to bring about stuff the right would hate (gun control, enforcement of various covid mandates), stuff the left would hate (anti-abortion, eugenics) or things that would scare a lot of eneducated people. Do you know that there are many, many people who believe the covid vaccine has left spikes in their body that are running around stabbng this that and the other thing in their insides? Nothing can disabuse them of that notion. It would not be hard to come up with some story about AI that would scare the shit out of people like that. Like, say, more and more businesses are going to be using AI in their web sites, and that any time you go to one of those sites your computer gets sort of infected with AI. The infection spreads thru your computer until it takes overl. Then it can read everything on your computer and hear all the conversations in the house and report it to the AI and the government agencies who are now using AI to spy on and control us all. Next thing you know the police will come knockng at your door because AI reported that you wrote an email sayjng you did not want your daughter to date a black guy,. You will be taken to Unwoke Jail.
Has anyone tried to start an anti-AI lobby? Has anyone tried bringing unscrupulous pressure to bear on people in power in the AI development world. I'm sure some people in the AI risk space know things that would be embarrassing to the people in power in AI development. Somebody had a psych hospitalization when they were 20, or has a cocaine habit, stuck their hand into somebody's pants at a work party. EVERYBODY has something they do not want to world to know.
Look, I get that all the things I'm suggesting are sort of repulsive, and I do not live in a segment of life where I have to do shit like this, or know people who do. But I know it goes on. Are you really trying to tell me that people in the AI risk space have thought seriously about trying this kind of thing-- which is how things are done in the real world a lot of the time -- but was stopped by lack of money and power? Because (1) I see zero evidence of it and (2) I am literally the only person I have ever seen *talk* about possibilities like this. And besides (3) some people in Ai risk space do have power and/or access to substantial money.
Hats off to you Zvi for writing important articles like this.
But I find stuff like Tyler Cowen's articles so exasperating.
You can't prove AI isn't dangerous by analogizing with other technologies - no other technologies ever had the potential to be as powerful as a machine superintelligence, and they certainly had no prospect of agency.
You can't prove that AI isn't existentially risky by appealing to extremely general axioms ("since when can 𝘮𝘰𝘳𝘦 intelligence be a bad thing?"). The ability to align a machine superintelligence is matter of computer science. And the harm that a machine superintelligence can inflict is a matter of decision theory et al. These specific things need to be dealt with directly. They won't turn out to not matter because the f-ing 𝘱𝘳𝘪𝘯𝘵𝘪𝘯𝘨 𝘱𝘳𝘦𝘴𝘴 of all things didn't radically transform the world.
If influential people are making very not good arguments, those arguments need to be argued against. But it's extremely frustrating to see them make these arguments and refusing to even engage with the actual work that the thing they're arguing against is based off. I've certainly experienced many academics or other influential figures lose a lot of their shine over the past year or so based on how asininely they've dealt with the AI risk question.
Think this all boils down to a difference in induction.
Tyler and Robin are inductivists. They see the patterns of history, and have learned that every single time someone has catastrophised about the future implications of some innovation, they have been wrong, and shockingly wrong at that. Paul Ehrlich and Malthus are great examples of this.
Zvi and Eliezer are anti-inductivist for this specific situation. They think that superintelligent AGI is fundamentally different than all previous inventions, thus the knowledge gleaned from observed patterns in history don't apply here.
I think Tyler is probably right, mostly because there's a lot of evidence for his view, not just interesting thought experiments. I worry that we'll never find out who's right before well-intentioned actors kneecap AGI before we can even know for sure what dangers it poses.
Any mention of nuclear fission is conspicuously absent in TC's post, even though that is obviously a close analogy. Inhibiting that research, being even stricter in ensuring non-proliferation or similar measures might still prove to have been necessary to prevent catastrophe. We've had at least two near misses after all.
You can just say "this fits the pattern" of *everything* . That does not make you a good pattern recognizer, even if you are correct 99% of the time. You are only good at recognizing patterns if you recognize when something does not fit the pattern. If nuclear fission is considered an exceptional outlier (and I sure hope it is), why should AI not be recognized as at least as far an exceptional outlier. The arguments are much closer to those arguments than to any other spurious claims of catastrophe.
Claiming you have to engage in 'radical agnosticism' is incomprehensible to me. "AGI ruin" is not 'one specific outcome in a vast space of possible futures'. The while point of the "AGI ruin" argument is that it is a significant chunk of that space. If we admit nuclear ruin is still a significant chunk of the space of possible futures (and I sure hope everyone still does), why would AGI be different, given the very specific kind of similar reasoning behind how it could be ruinous?
The nuclear analogy seems imperfect, because we KNEW what kind of damage nuclear weapons could do when we collectively took action on it. In the case of AGI, we still don't know whether superintelligent AGI is possible yet, let alone whether it *can* destroy the universe.
I don't think anyone worth their salt denies that in the space of possible events that can plausibly destroy the world, thermonuclear war and AGI ruin appear to be most probable based on current trends. That doesn't mean they are likely, however.
And at what point do you say the risk is warranted? Nuclear power and nuclear weapons were incredibly risky inventions, but the world is much safer and prosperous with them in it. Why can't AGI be the same? Yes it may hold some probability of ruin, it also might lead to an extreme increase in wellbeing for the gogolplexes of future humans. If the probability of AGI ruin isn't >2%, shouldn't we take that bet every time?
>If the probability of AGI ruin isn't >2%, shouldn't we take that bet every time?
Let's say we should take that bet. It's irrelevant, because almost no AI doomer thinks the risk is anywhere near that low, which is precisely the issue.
Also, most AI doomers are in principle FINE with AGI being developed per se - they just want it slowed down so we can be sure we've aligned it properly.
So the trade off isn't "benefit of AI vs risk of extinction", it's "Missing out on decades/centuries of AGI's benefits vs risk of extinction". The choice should become trivially simple in that case, though sadly it isn't for most people.
I think it is correct to start from a place of extremely high skepticism here -- especially given the historical frequency of doomsday claims and radical technology hype. The fact that many of the arguments are put forth by people who are clearly highly thoughtful and intelligent and have been working on this for a long time is a small factor but should not be given much weight in my view, as I think that in the past many very smart and thoughtful people have been wrong in this way before.
Given that, we still have to develop a criteria for evaluating when to lower our skepticism based on extremely strong evidence that AI truly is different (even though skepticism of this kind is the correct attitude 99.9% of the time). To me, there are two major relevant points:
1) The AI ruin argument makes a strong case for being genuinely unique. We have improved weapons, we have improved transportation, we have improved communication, but we have never created intelligence. To me, this is a strong case to give some credence to both the radical technology and doomsday claims.
2) The AI ruin argument has not, in my view, presented strong falsifiable claims. In fact, many of the claims are intentionally non-falsifiable (e.g. the sharp left turn, no fire alarm). And the predictions that I would have inferred from the AI ruin argument pre-GPT 3 (e.g. first AI breakthroughs will probably be agentic, human-level intelligence will start FOOMing right away) have not come to pass in my view, although maybe I am inferring the wrong claims. It is totally possible that this argument is both true and does not produce any falsifiable claims until we are all dead -- I think this is very unlikely for any major claim of this kind and is a huge weakness with the argument -- very very unlucky if it turns out to be true! I have seen a lot of accusations of AI skeptics moving the goalposts (which I think is true, and I would not consider myself an AI skeptic) but I also think there's been a large amount of goalpost moving from the AI ruin camp as well -- GPT-4 seems pretty damn smart, and people are certainly trying to get it to rewrite itself -- why aren't I dead yet?
Overall, I'd still place several OOM more credence on doomsday and radical technology claims than the historical average because I think the arguments for its uniqueness hold up, but I think there are reasons to hedge towards taking the historical view as a baseline rather than completely ignoring it.
It's literally *impossible* for doomsday arguments of the past to have been true AND for us to currently be having this discussion. If the previous doomsday arguments were correct, we would be dead already (unless in a counterfactual world where we had e.g. halted production of nuclear weapons and a nuclear explosion would have ignited the atmosphere).
Therefore, requiring a precedent of doomsday arguments being correct is absolutely not reasonable.
A maximum of one (unheeded) doomsday argument can possibly be correct in history, and nothing has come as close to that as the one for machine superintelligence.
Agreed, which is why I would not use the historical baserate to completely devalue this one and I agree that it is foolish to say that no doomsday argument should ever be taken seriously. I think what the historical precedent does is give us some useful context that suggests hedging towards it, and for that reason it should not be thrown away to clear the playing field entirely as many suggest (I'm not sure if that's what you're personally suggesting here, but Scott Alexander certainly is and I think Zvi is as well if I'm reading him correctly).
For example, one thing it shows is that making these claims is extremely attractive for number of social status/platforming/doom is attractive reasons, which should hedge us slightly against believing them (or, more fairly, just set the bar higher for their persuasiveness).
I don't think we massively disagree -- I agree it would be very stupid to say "It's never happened before so it'll never happen!" Especially because, as you point out, I have observer/survivor bias here. The case I want to make here is for the value of some mix of philosophy, historical context, probability and conditional logic rather than solely relying on conditional logic and probability. I agree that historical context is weak-ish especially when it comes to the lack of any possible counterfactual here, but I also think conditional logic and especially probability are weak-ish when it comes to very hard to imagine futures and genuinely new phenomena as well.
Also, agreed that Tyler does not sufficiently engage with the conditional logic of the question.
>Tyler and Robin are inductivists. They see the patterns of history, and have learned that every single time someone has catastrophised about the future implications of some innovation, they have been wrong, and shockingly wrong at that.
How is this not trivially an example of an observer selection effect? The only scenario is which there is a precedent for things destroying the world is one precisely in which we're not around to speculate about other future technologies.
>I think Tyler is probably right, mostly because there's a lot of evidence for his view, not just interesting thought experiments. I worry that we'll never find out who's right before well-intentioned actors kneecap AGI before we can even know for sure what dangers it poses.
What? If Cowen is wrong, 'finding out who's right' literally equals human extinction. That's why this is such a thing in the first place - we don't get a second chance. And if Yudkowsky is even close to being right about the likelihood of AI ruin, then there's no possible benefit that getting AGI 30 years sooner that could provide that makes up for such a high risk of catastrophe.
And the AI case is not based on just thought experiments. It's based on decision theory and computer science. Cowen has abjectly failed to deal with the technical arguments of AI risk and resorts to extremely generic analogies.
"finding out who's right" doesn't necessarily mean human extinction. If GPT-4 started self-replicating, or trying to convince people into advancing it's own capabilities, and there is no clear way to get rid of those bugs, I'd be willing to say Eliezer is right. Shut it down right away. No one has to die, just have to be prepared to move quickly.
-> The only scenario in which there is a precedent for things destroying the world is precisely the one in which we're not around to speculate about other future technologies.
Where does this logic leave us? "There's evidence of aliens out there, and there's some scenario that they could decide to kill us all after first-contact. We have no evidence for why they would do this yet because, after all, they're aliens. It's still a possibility, however, so stop Elon building rockets now. This could be the extinction event."
It is *possible* every time. The point is, we should at least gather evidence that it might be likely before letting fear be the mind-killer.
-> And the AI case is not based on just thought experiments. It's based on decision theory and computer science. Cowen has abjectly failed to deal with the technical arguments of AI risk and resorts to extremely generic analogies.
Okay, AGI ruin is based on a series of formalized assumptions about how things might go, and some thought experiments too. I agree that if you buy into most of Eliezer's assumptions, the only valid conclusion is that you get AGI ruin almost every time. Valid arguments have no bearing on truth, however.
Oh, and your call for rigour seems wholly unnecessary because:
1) Dealing with the technical arguments would likely involve granting a lot of assumptions that there is no evidence for (yet),
2) if there's anything great forecasters have shown us, it is that domain expertise isn't required to see things more clearly than the experts. In fact, expertise can often lead people to focus too much on the specifics and ignore the base-rates at which things usually occur.
>No one has to die, just have to be prepared to move quickly.
That's won't happen. It won't be the wake up call that it should be.
>Where does this logic leave us?
Where does YOUR logic leave us? Dismissing every possible existential risk because all previous risks necessarily turned out to be wrong?
>It is *possible* every time. The point is, we should at least gather evidence that it might be likely before letting fear be the mind-killer.
"Gathering evidence" is this scenario means letting AI develop in an uncontrolled way that makes rapid intelligence takeoffs likely and the likelihood of having enough time to react properly unlikely.
And the argument is NOT that it is "possible" that AI kills us. It's that, without profound technological breakthroughs in alignment theory, it's hard to come up with a scenario in which AI DOESN'T kill us.
>1) Dealing with the technical arguments would likely involve granting a lot of assumptions that there is no evidence for (yet),
There's no evidence a superintelligent AI won't kill us all
There's no evidence we will be able to align a superintelligent AI
There's no evidence we will know when to stop once we get to close
This idea that only the AI-doomers are being speculative is just wrong.
>2) if there's anything great forecasters have shown us, it is that domain expertise isn't required to see things more clearly than the experts.
Somebody who analogizes to a printing press in a discussion of AI risk is not "seeing things clearly". They're got their heads in the sand and are literally refusing to make even a token effort to engage with their opponent's arguments
>In fact, expertise can often lead people to focus too much on the specifics and ignore the base-rates at which things usually occur.
For goodness' sake - 𝘵𝘩𝘦𝘳𝘦 𝘪𝘴 𝘯𝘰 "𝘣𝘢𝘴𝘦 𝘳𝘢𝘵𝘦" 𝘰𝘤𝘤𝘶𝘳𝘳𝘦𝘯𝘤𝘦 𝘧𝘰𝘳 𝘩𝘶𝘮𝘢𝘯𝘴 𝘴𝘩𝘢𝘳𝘪𝘯𝘨 𝘵𝘩𝘦 𝘦𝘢𝘳𝘵𝘩 𝘸𝘪𝘵𝘩 𝘴𝘶𝘱𝘦𝘳𝘪𝘯𝘵𝘦𝘭𝘭𝘪𝘨𝘦𝘯𝘵 𝘮𝘢𝘤𝘩𝘪𝘯𝘦𝘴.
I know you hate hearing this, but AI is categorically different to any other technology, and all these lazy analogies to nuclear weapons or 𝘵𝘩𝘦 𝘱𝘳𝘪𝘯𝘵𝘪𝘯𝘨 𝘱𝘳𝘦𝘴𝘴 or whatever else are worthless.
Radically agnostic! I really don't know anything about the issue and so I have to live in this space of having no opinion. (Which is hard for humans.) I read some of your posts and I start making a list
of responses, but your posts are too long, and it's like pulling teeth for me to write, so I stop.
Re: radical technological change (rtc). If you want to call cell phones rtc, I'd be fine with that.
But I understand Tyler to mean no new 'big' ideas. The cell phone and all, are the result
of big ideas in the previous 50 year ~(1920-1970) When quantum mechanics and understanding atoms lead to all of the things we are doing now. From phones and TV's it's not a big leap to everyone having a Dick Tracy wrist radio. Cell phones may cause a big change in our culture, but they are not a big change in how we can impact the world.
Re: GPT and AI. I've watched a few videos, and kept up a little. AFAICT the current generation is reaching the limit of what it can do. There is some curve of 'improvement' vs training time, and it looks like they are starting to hit to a point of diminishing returns. They had to run (something like) 1000 computers for a month on the training data.
Anyway none of this looks like intelligence. I'm not sure how to define intelligence, but a
good start might be: Intelligence is model making of the world, and making predictions from your models to test them. GPT is making no models... AFAICT. :^)
I wish you could have a conversation with Tyler, because you both seem to be using a bit of hyperbole to make your arguments. (Sorry for the bad formatting, I need to upload a better word processor.)
Can any of the folks here concerned about AI doom scenarios direct me to the best response to this article: https://www.newyorker.com/culture/annals-of-inquiry/why-computers-wont-make-themselves-smarter
I am assuming some responses have been written but I wonder where I can read them. Thank you!
A rare case where I'm glad for paywalls. It's an absolute jerk move to put potentially politically influential articles behind a paywall, but hopefully in this case it simply limits it's impact.