We are living through the ultimate underdog story. If we win, it'll be bigger than Greece 2004, bigger than Liverpool 2005, bigger even than Leicester 2016. If we win, Peter Drury will obliterate Shakespeare in the art of the spoken English word. Humanity's "remontada"? We will be there.
I don’t think the Comet King is a good role model. His mantra, “Somebody has to and nobody else will”, appealed for me for a while but then I realized how bitter and resentful it is. If your mission is to protect the world and those you love, you can do better than having disdain for everyone else.
Much better would be “I’m honored to have a vital role” or “I will protect those who cannot protect themselves” (that second one is from Brandon Sanderson).
I’m still workshopping alternatives to the Comet King’s slogan to find something as catchy, and would love to hear ideas from others.
Going hard on selflessly helping others didn't work out so great for Emiya, at least in most possible worlds, but I consider that story an inspiration anyway. "The thought of wanting to help can't be wrong." I similarly like the Comet King's mantra because it's an imperative: one *must* do this duty, it's not supererogatory. Bitter, yes - would that others would step up, oftentimes it only takes one, and not even a hero at that. But that's the key point to my mind, that anyone can be that somebody. You don't need a magic sword or an angelic dad or Providence, RI to walk the Lawful path. Just the conviction to unflinchingly face the myriad opportunities each day to Do The Right Thing rather than the easy thing, rather than reflexively looking away. (Part of why making the right thing easy is a crucial part of good system design!) This doesn't even mean getting them all right, just seeing the branches at all versus passively going along towards the default Bad End. If it were easy, somebody else would have, and nobody would have to. And I'd rather be somebody than nobody. One must imagine Jalaketu happy, at the end of all things.
The day that Dario renames the lab to Anthropics is when I'll know to pack it up, pack it in. Although by the same principle it won't be me experiencing that.
Zvi your existence was always scheduled to end. Whether slowly or all at once, it's eternal oblivion presumably. That was the fate of all of your ancestors and currently is the scheduled fate of all of your descendants.
Everyone only lives an eyeblink of cosmic time.
This is why the possibility of AI doom shouldn't seem that scary. Because of the expected value. You lose, you die like your already expected fate. But there are lots of routes to "wins" even if some of them involve you personally or your descendants living for thousands of years in essentially a zoo. All those outcomes have +EV.
Not necessarily. Exotic or unknown physics could mean that computation could go on for eternity.
But even assuming a limited budget: Thousands of years of mere humanity with no power is a detestable pity prize compared to what is actually out there to use in the lightcone, if we aestivated until the universe cooled enough for very efficient computation. It is hard to describe just how large 10^55-75 or so subjective human years of experience is. 10^55 is like if every grain of sand on Earth was another Earth, and then you spent a full year getting to know each grain of sand on each one of those Earths, done a billion times.
Correct it's a pity prize. Sometimes I think of the natural result of unrestrained capitalism combined with a cost reduction mindset. Life in a mega tower where every resident gets a pod bunk.
But again, even in that hellish environment, there's full VR and a robo clinic on a floor where the ai doctors regulate your biological age to 18. Not for your benefit per say but as a cost savings. (Vastly lower healthcare costs, and this is a privilege that can be revoked if you riot). All the food is processed algae.
Even that is a better fate than your life today, or all of your ancestors.
>Exotic or unknown physics could mean that computation could go on for eternity.
I've seen the proposals for infinite computation that are compatible with thermodynamics (you asymptotically approach equilibrium but never achieve it). But I thought these were incompatible with quantum mechanics. If the universe is finitely divisible, then you can't get infinite computation with a finite amount of matter constrained by thermodynamics.
I keep hearing these kinds of Pascal-flavoured arguments, and on top of natural skepticism to matching that pattern, have to question that a) it's not possible to vanquish Death some other way that doesn't involve running so close to various cliffs, and b) that an intelligence powerful enough to defeat the final foe doesn't create more such cliffs than it possibly averts us from. Strict utilitarianism and betting the house on pure +EV often leads to dumb mistakes like ignoring the geometric mean, or at least that's one of the rationalist messages I've absorbed over the years anyway.
(Also the unstated assumption c, that we need a deus ex machina to save us from our various foot-shootings in bleak late capitalist hellworld. Tried and found wanting vs found hard and not tried - this too I contest! Life's pretty great, often getting better, and the problems are often not so extraordinary that extraordinary measurse are necessary. Moloch hasn't won yet.)
(1) there have been no measurable lifespan gains using human healthcare or other efforts since the 1970s. Please look at a life expectancy chart
(2) The implicit assumption for acceleration faction is individuals in it want to live indefinitely personally, but barring that, they want to go to their deaths knowing their children and friends may get indefinite lifespan.
So those are the constraints : 0 measurable progress for 50 years, you don't consider it a win without meaningful life extension within 50 years from right now. (That's enough time if you are age 20 you can expect that you or your friends won't die of aging)
I personally see that within these constraints there is nothing really to discuss, there is no factual or rational way to "agree to disagree", that given what I consider to be overwhelming evidence acceleration to ASI is the only rational choice.
Habryka et al argues it's selfish to want to remaining living (or my family and friends if it isn't in the cards for myself).
I don't think habryka or even zvi dispute the reality that without ASI, biological death is their certain personal fate. They just want it built "carefully". (And probably too slowly for many people)
I have not seen a convincing argument for anything else so far. To say "let's just pause for 6 months and then impose government regulations that slow things down by an order of magnitude" - a typical doomer faction ask - is PROBABLY the death of every person currently alive.
I find myself aesthetically attracted to the anti-AI fight but selfishly I often think along these lines. I want my grandparents to be around as long as possible — life would be so much better if they would stay — but in their mid 80s, nothing except maybe the singularity seems to give them even a 1% chance of a good end where they could be around long enough to know their great-grandchildren. Death is slowly eating away at everyone I know and love, and the default path moves us endlessly towards its open maw... the correct answer is probably not the world's riskiest Hail Mary, but what else is there?
Aging is extremely complex and hard and current scientists only know it's probably not wear and tear from yamacka factor experiments (which will kill you horribly if blindly given) and tampering with mTor and a few other regulatory pathways. There are tens of thousands of interconnected biological pathways.
It probably isn't stochastic as much as it appears now. Most humans wouldn't be able to survive to a median age of 70 something or have 5 fingers and a brain and usually working vision if nature wasn't tightly adhering to programs that can be manipulated.
So it's pretty clear to me that a solution is possible. Even if you have to learn every single damn pathway in embryology - every past protein and receptor - and print entire replacement torsos and limbs organ by organ then assembling them with the right growth factors for scarless joins.
But it's just too complex for any living single human mind, or even clusters of thousands of them working together, to solve this reliably. You need a mind that can consume ALL scientific papers, and the raw data of ALL robotic experiments you would do to develop the capabilities to print torsos.
Before you can print torsos you need to print organs that work. Before that sheets of cells that function as expected, one for every possible sheet or tubular structure in the human body. Before that you need to manipulate countless trillions of individual cells and small groupings, trying many possible new genes and working through exactly what each regulatory molecule really does - for all of them. There cant be a single gene that is being expressed or variant where you don't know what it is and exactly what it does.
You also need to ratchet through different states. You should be able to manipulate individual cells to any age, 9 days to 99 years, etc.
See you don't need magic. I just described scientific experiments that at today's costs would be trillions and the problem is that every scientist doing them doesn't know what every other scientist already did. So it doesn't really scale - science just gets caught in loops, especially as shoddy data gets published preventing accurate conclusions.
You can do it with ASI though. Not necessarily even all that powerful an ASI - the machine doesn't need the kind of capabilities doomers fear, and you wouldn't use one single ASI but swarms of thousands of them of a diverse set of brands.
I think this is what makes the anti-AI fight somewhat difficult to organize among selfish actors. Most things that are Bad, you can convince people to react to the extent that it will be bad for them (or at least not impede the progress of altruists since it doesn't make their life worse). But if we assume that AI has any reasonable chance of solving aging, rational adults who don't want to die will probably look at the fact that they and every other adult they care about are pretty much guaranteed to die in the next 50-70 years unless we do something very drastic at a different rate than anything we've ever managed before. For adults with parents, that's shortened to a max ~40 year budget, and for adults with grandparents it can be low double digits as an expected maximum.
I would like to prioritize slowing AI until we have any idea what the hell we're doing from an altruistic standpoint. But fully selfishly, I'd at least seriously consider rolling the dice if the chances were something like 5% everyone lives as long as they want / 95% near-instant painless human extinction, because the EV of that bet seems positive to me (I'll get more than just a couple years with the people I love who are near their upper limit).
Currently, my selfish & less selfish wants have sort of bumbled to a standstill such that I'm not really trying to move the needle either way (especially since I need to live in harmony with the — presumably — vast majority of people who'd consider that potential trade a horrific violation of all their strongest values), but I do wish I could find my way to a clear-cut opinion.
You're perceiving the tradeoffs correctly. And my only other comment is that even ASI may be nowhere near enough. I pencil it in as something like "you have many variations of general ASI AND enormous numbers of robots built exponentially using thr technology AND at least 10 linear years, using over a billion robots, of the physical effort".
So if it takes 10 years from physical AI that doesn't suck, 10 years to do the R&D, and 5 years to get to physical AI from 2025...well a lot of us will lose a great many loved ones.
And see, there's no room for any delays. Any sort of wasted time could mean you or your grandchildren are dead of aging - 25 years is kind of a median to optimistic case, there are many outcomes where the problem may be harder than it appears. (And some "Early LEV" outcomes where say 10 years from right now, early ASI uses the evidence already discovered by humans to reverse engineer the true aging clocks and rationally design a simple and easily manufactured set of small molecule drugs that add 30 percent to human lifespan)
1) Scott heavily contests this Hansonian claim, and I found Hanson et al's rebuttals deeply disingenous, which makes me more skeptical than I was originally. I'm familiar with the RAND study, among others, and think there's probably a lot of truth to "large portions of healthcare are low EV or even negative". The difficulty is identifying those parts and actually cutting them without getting a pitchfork mob at your door. Furthermore, life expectancy is an average, and we all know the negatives dragging those numbers down. Protip: healthcare addresses some of those (advances on cancer, cardiovascular, obesity), some of those are intractable political problems (firearms, demographic poverty), and some are historical flukes burning themselves out (drug deaths leveling off because you can only die once, phasing out lead). This is like the whole "people only lived to 40 in the past" thing, where it's mostly a statistical artefact of childhood mortality...which, again, ordinary measures like healthcare and sanitation largely solved. There are still lots more ordinary wins to rack up, too - like it's insane that air pollution still shaves years off of life expectancy, even after we've come a long way from the smog-choked cities of the past.
2) That's nice for accs. Have they considered the infinite negative utility of possibly killing everyone if they fail? (Let's round off the epsilon of those who don't actually want immortality, although they too should get a say.) Given a large enough x-risk factor, the expected outcome converges on cancelling out, which is the same doomed outcome as the null hypothesis anyway. Except worse because we did it on purpose despite ample warning, instead of dying out to the slings and arrows of outrageous fortune. So then it comes back to debating the size of |x|. I don't buy all the "doomer" arguments, but they've successfully convinced me the risk is much greater than epsilon, and the unknown-unknowns are hard to bound (unlike other risks like bio, which are eminently addressable, just...boring and unsexy, so no one bothers). Even if it were some low p(doom) like, oh, 5%...that's not actually a small number! Nat 1s happen all the time!
unstated 3) that even a short pause would set us back generations, as Scott illustrated in "Ars Longa, Vita Brevis" - this Proves Too Much. Gonna be late to work if I keep workshopping the comment, so I'll end on a note that not everyone who disagrees with the accelerationist position is a 20-year-old pause advocate. In 50 years on the AI-fizzle timeline, I'll be 20 years into retirement, and with any luck, enjoying it a lot longer and in better health than my (grand)parents. That's enough of a progress win for me.
1. Aging is an exponential process. (This is why I say the evidence is overwhelming, my pov here has so much evidence there is little reason to discuss it. Please ask an AI model if you dispute point 1 further. Make sure you paste in everything I said and ask for factual sources that confirm)
2. From e/acc perspective, people you won't live to meet have 0 value, + or -. If you are dead there is nothing. Similarly your own death is 0 EV. Not negative.
With 0 that makes the math work out where the positive outcomes - where instead of peeing yourself at age 90 in the fizzle world you have received a newly printed body and are more capable than you were originally - Is hugely positive EV.
Negative scenarios have to be scenarios you live to see, like the rich hoard all wealth or ASI aging therapy is available but you can't personally afford it.
Right...as Zvi says, when people tell you who they are, believe them. I don't think it would be productive for either of us to engage further with such strongly held priors. Thanks for playing at any rate. I do wish other acc avatars like Jezos would similarly bother to engage with anything besides flippancy; it'd make me take their views more seriously.
Thank you, Zvi, for posting this! We need more of this kind of thing: love and pep talk at a time of the greatest crisis ever.
And thank you even more for being my #1 source of crucial information on day-to-day developments on AI. There's a growing number of us doing what we can for our right not to be murdered by rogue AI, but your contribution really stands out, so hang in there and keep up your excellent work!
Here's hoping we're in a good timeline, we make it to a good world on the other side, and Netflix puts out a good docudrama of it all, where this is a pivotal speech. <3
Wouldn't it be polite to attribute the quotation "I don't think we're going to make it," to its author, James Barrat? I only mention it because big tech runs roughshod over copyright and attribution, bigly. And I'm close to the author, who wrote that sentence in the prologue to his recent book, The Intelligence Explosion. BTW I appreciate your work, Zvi.
Perhaps I'm missing important context, but isn't that just, like, a normal English phrase? I'm sure I've said it before myself and heard others say it too. I might as well say that you should be attributing "I only mention it because" to somebody else.
I see your point, but there’s a difference between using a phrase in conversation and committing it to print. Because I put it in my book prologue, interviewees kept bringing this idea back to me, saying it neatly captures the “doomer” point of view. Once it was printed, I had effectively “coined a phrase,” and it spread in a meme-like way.
I don’t consider that a stupendous achievement - I’d have forgotten about it if the media hadn’t picked it up - but I do feel entitled to say I coined it, just as I say I originated the acronym ASI (Artificial Superintelligence) in my 2013 book, Our Final Invention. If someone used it earlier, I'd like to know. Neither of these is an earth‑shattering accomplishment, but as a writer I can’t help noticing when a term I’ve put into print becomes part of the lingua franca. Concurrently, due to generative AI, ideas of attribution and copyright are in danger of becoming eroded into nonexistence, and I like to push back.
Well written and inspiring. Thanks for doing all that you do.
Thanks, I needed this.
We are living through the ultimate underdog story. If we win, it'll be bigger than Greece 2004, bigger than Liverpool 2005, bigger even than Leicester 2016. If we win, Peter Drury will obliterate Shakespeare in the art of the spoken English word. Humanity's "remontada"? We will be there.
https://www.youtube.com/watch?v=C5PIY7GY9E0
Thinking of the GOAT: https://youtu.be/M_Ld8JGv56E
I don’t think the Comet King is a good role model. His mantra, “Somebody has to and nobody else will”, appealed for me for a while but then I realized how bitter and resentful it is. If your mission is to protect the world and those you love, you can do better than having disdain for everyone else.
Much better would be “I’m honored to have a vital role” or “I will protect those who cannot protect themselves” (that second one is from Brandon Sanderson).
I’m still workshopping alternatives to the Comet King’s slogan to find something as catchy, and would love to hear ideas from others.
Going hard on selflessly helping others didn't work out so great for Emiya, at least in most possible worlds, but I consider that story an inspiration anyway. "The thought of wanting to help can't be wrong." I similarly like the Comet King's mantra because it's an imperative: one *must* do this duty, it's not supererogatory. Bitter, yes - would that others would step up, oftentimes it only takes one, and not even a hero at that. But that's the key point to my mind, that anyone can be that somebody. You don't need a magic sword or an angelic dad or Providence, RI to walk the Lawful path. Just the conviction to unflinchingly face the myriad opportunities each day to Do The Right Thing rather than the easy thing, rather than reflexively looking away. (Part of why making the right thing easy is a crucial part of good system design!) This doesn't even mean getting them all right, just seeing the branches at all versus passively going along towards the default Bad End. If it were easy, somebody else would have, and nobody would have to. And I'd rather be somebody than nobody. One must imagine Jalaketu happy, at the end of all things.
In some worlds we win, in others we lose. We will only experience the worlds in which we win.
The question is not whether we will win or lose. It's what proportion of worlds we win in.
I usually think among similar lines. Every day is a fight to gain a few more possible worlds among the many, and maybe that is enough.
The day that Dario renames the lab to Anthropics is when I'll know to pack it up, pack it in. Although by the same principle it won't be me experiencing that.
Nicely done, wizard. :)
I'm thankful for your work
Zvi your existence was always scheduled to end. Whether slowly or all at once, it's eternal oblivion presumably. That was the fate of all of your ancestors and currently is the scheduled fate of all of your descendants.
Everyone only lives an eyeblink of cosmic time.
This is why the possibility of AI doom shouldn't seem that scary. Because of the expected value. You lose, you die like your already expected fate. But there are lots of routes to "wins" even if some of them involve you personally or your descendants living for thousands of years in essentially a zoo. All those outcomes have +EV.
Not necessarily. Exotic or unknown physics could mean that computation could go on for eternity.
But even assuming a limited budget: Thousands of years of mere humanity with no power is a detestable pity prize compared to what is actually out there to use in the lightcone, if we aestivated until the universe cooled enough for very efficient computation. It is hard to describe just how large 10^55-75 or so subjective human years of experience is. 10^55 is like if every grain of sand on Earth was another Earth, and then you spent a full year getting to know each grain of sand on each one of those Earths, done a billion times.
Correct it's a pity prize. Sometimes I think of the natural result of unrestrained capitalism combined with a cost reduction mindset. Life in a mega tower where every resident gets a pod bunk.
But again, even in that hellish environment, there's full VR and a robo clinic on a floor where the ai doctors regulate your biological age to 18. Not for your benefit per say but as a cost savings. (Vastly lower healthcare costs, and this is a privilege that can be revoked if you riot). All the food is processed algae.
Even that is a better fate than your life today, or all of your ancestors.
>Exotic or unknown physics could mean that computation could go on for eternity.
I've seen the proposals for infinite computation that are compatible with thermodynamics (you asymptotically approach equilibrium but never achieve it). But I thought these were incompatible with quantum mechanics. If the universe is finitely divisible, then you can't get infinite computation with a finite amount of matter constrained by thermodynamics.
I keep hearing these kinds of Pascal-flavoured arguments, and on top of natural skepticism to matching that pattern, have to question that a) it's not possible to vanquish Death some other way that doesn't involve running so close to various cliffs, and b) that an intelligence powerful enough to defeat the final foe doesn't create more such cliffs than it possibly averts us from. Strict utilitarianism and betting the house on pure +EV often leads to dumb mistakes like ignoring the geometric mean, or at least that's one of the rationalist messages I've absorbed over the years anyway.
(Also the unstated assumption c, that we need a deus ex machina to save us from our various foot-shootings in bleak late capitalist hellworld. Tried and found wanting vs found hard and not tried - this too I contest! Life's pretty great, often getting better, and the problems are often not so extraordinary that extraordinary measurse are necessary. Moloch hasn't won yet.)
(1) there have been no measurable lifespan gains using human healthcare or other efforts since the 1970s. Please look at a life expectancy chart
(2) The implicit assumption for acceleration faction is individuals in it want to live indefinitely personally, but barring that, they want to go to their deaths knowing their children and friends may get indefinite lifespan.
So those are the constraints : 0 measurable progress for 50 years, you don't consider it a win without meaningful life extension within 50 years from right now. (That's enough time if you are age 20 you can expect that you or your friends won't die of aging)
I personally see that within these constraints there is nothing really to discuss, there is no factual or rational way to "agree to disagree", that given what I consider to be overwhelming evidence acceleration to ASI is the only rational choice.
Habryka et al argues it's selfish to want to remaining living (or my family and friends if it isn't in the cards for myself).
I don't think habryka or even zvi dispute the reality that without ASI, biological death is their certain personal fate. They just want it built "carefully". (And probably too slowly for many people)
I have not seen a convincing argument for anything else so far. To say "let's just pause for 6 months and then impose government regulations that slow things down by an order of magnitude" - a typical doomer faction ask - is PROBABLY the death of every person currently alive.
I find myself aesthetically attracted to the anti-AI fight but selfishly I often think along these lines. I want my grandparents to be around as long as possible — life would be so much better if they would stay — but in their mid 80s, nothing except maybe the singularity seems to give them even a 1% chance of a good end where they could be around long enough to know their great-grandchildren. Death is slowly eating away at everyone I know and love, and the default path moves us endlessly towards its open maw... the correct answer is probably not the world's riskiest Hail Mary, but what else is there?
Aging is extremely complex and hard and current scientists only know it's probably not wear and tear from yamacka factor experiments (which will kill you horribly if blindly given) and tampering with mTor and a few other regulatory pathways. There are tens of thousands of interconnected biological pathways.
It probably isn't stochastic as much as it appears now. Most humans wouldn't be able to survive to a median age of 70 something or have 5 fingers and a brain and usually working vision if nature wasn't tightly adhering to programs that can be manipulated.
So it's pretty clear to me that a solution is possible. Even if you have to learn every single damn pathway in embryology - every past protein and receptor - and print entire replacement torsos and limbs organ by organ then assembling them with the right growth factors for scarless joins.
But it's just too complex for any living single human mind, or even clusters of thousands of them working together, to solve this reliably. You need a mind that can consume ALL scientific papers, and the raw data of ALL robotic experiments you would do to develop the capabilities to print torsos.
Before you can print torsos you need to print organs that work. Before that sheets of cells that function as expected, one for every possible sheet or tubular structure in the human body. Before that you need to manipulate countless trillions of individual cells and small groupings, trying many possible new genes and working through exactly what each regulatory molecule really does - for all of them. There cant be a single gene that is being expressed or variant where you don't know what it is and exactly what it does.
You also need to ratchet through different states. You should be able to manipulate individual cells to any age, 9 days to 99 years, etc.
See you don't need magic. I just described scientific experiments that at today's costs would be trillions and the problem is that every scientist doing them doesn't know what every other scientist already did. So it doesn't really scale - science just gets caught in loops, especially as shoddy data gets published preventing accurate conclusions.
You can do it with ASI though. Not necessarily even all that powerful an ASI - the machine doesn't need the kind of capabilities doomers fear, and you wouldn't use one single ASI but swarms of thousands of them of a diverse set of brands.
I think this is what makes the anti-AI fight somewhat difficult to organize among selfish actors. Most things that are Bad, you can convince people to react to the extent that it will be bad for them (or at least not impede the progress of altruists since it doesn't make their life worse). But if we assume that AI has any reasonable chance of solving aging, rational adults who don't want to die will probably look at the fact that they and every other adult they care about are pretty much guaranteed to die in the next 50-70 years unless we do something very drastic at a different rate than anything we've ever managed before. For adults with parents, that's shortened to a max ~40 year budget, and for adults with grandparents it can be low double digits as an expected maximum.
I would like to prioritize slowing AI until we have any idea what the hell we're doing from an altruistic standpoint. But fully selfishly, I'd at least seriously consider rolling the dice if the chances were something like 5% everyone lives as long as they want / 95% near-instant painless human extinction, because the EV of that bet seems positive to me (I'll get more than just a couple years with the people I love who are near their upper limit).
Currently, my selfish & less selfish wants have sort of bumbled to a standstill such that I'm not really trying to move the needle either way (especially since I need to live in harmony with the — presumably — vast majority of people who'd consider that potential trade a horrific violation of all their strongest values), but I do wish I could find my way to a clear-cut opinion.
You're perceiving the tradeoffs correctly. And my only other comment is that even ASI may be nowhere near enough. I pencil it in as something like "you have many variations of general ASI AND enormous numbers of robots built exponentially using thr technology AND at least 10 linear years, using over a billion robots, of the physical effort".
So if it takes 10 years from physical AI that doesn't suck, 10 years to do the R&D, and 5 years to get to physical AI from 2025...well a lot of us will lose a great many loved ones.
And see, there's no room for any delays. Any sort of wasted time could mean you or your grandchildren are dead of aging - 25 years is kind of a median to optimistic case, there are many outcomes where the problem may be harder than it appears. (And some "Early LEV" outcomes where say 10 years from right now, early ASI uses the evidence already discovered by humans to reverse engineer the true aging clocks and rationally design a simple and easily manufactured set of small molecule drugs that add 30 percent to human lifespan)
1) Scott heavily contests this Hansonian claim, and I found Hanson et al's rebuttals deeply disingenous, which makes me more skeptical than I was originally. I'm familiar with the RAND study, among others, and think there's probably a lot of truth to "large portions of healthcare are low EV or even negative". The difficulty is identifying those parts and actually cutting them without getting a pitchfork mob at your door. Furthermore, life expectancy is an average, and we all know the negatives dragging those numbers down. Protip: healthcare addresses some of those (advances on cancer, cardiovascular, obesity), some of those are intractable political problems (firearms, demographic poverty), and some are historical flukes burning themselves out (drug deaths leveling off because you can only die once, phasing out lead). This is like the whole "people only lived to 40 in the past" thing, where it's mostly a statistical artefact of childhood mortality...which, again, ordinary measures like healthcare and sanitation largely solved. There are still lots more ordinary wins to rack up, too - like it's insane that air pollution still shaves years off of life expectancy, even after we've come a long way from the smog-choked cities of the past.
2) That's nice for accs. Have they considered the infinite negative utility of possibly killing everyone if they fail? (Let's round off the epsilon of those who don't actually want immortality, although they too should get a say.) Given a large enough x-risk factor, the expected outcome converges on cancelling out, which is the same doomed outcome as the null hypothesis anyway. Except worse because we did it on purpose despite ample warning, instead of dying out to the slings and arrows of outrageous fortune. So then it comes back to debating the size of |x|. I don't buy all the "doomer" arguments, but they've successfully convinced me the risk is much greater than epsilon, and the unknown-unknowns are hard to bound (unlike other risks like bio, which are eminently addressable, just...boring and unsexy, so no one bothers). Even if it were some low p(doom) like, oh, 5%...that's not actually a small number! Nat 1s happen all the time!
unstated 3) that even a short pause would set us back generations, as Scott illustrated in "Ars Longa, Vita Brevis" - this Proves Too Much. Gonna be late to work if I keep workshopping the comment, so I'll end on a note that not everyone who disagrees with the accelerationist position is a 20-year-old pause advocate. In 50 years on the AI-fizzle timeline, I'll be 20 years into retirement, and with any luck, enjoying it a lot longer and in better health than my (grand)parents. That's enough of a progress win for me.
1. Aging is an exponential process. (This is why I say the evidence is overwhelming, my pov here has so much evidence there is little reason to discuss it. Please ask an AI model if you dispute point 1 further. Make sure you paste in everything I said and ask for factual sources that confirm)
2. From e/acc perspective, people you won't live to meet have 0 value, + or -. If you are dead there is nothing. Similarly your own death is 0 EV. Not negative.
With 0 that makes the math work out where the positive outcomes - where instead of peeing yourself at age 90 in the fizzle world you have received a newly printed body and are more capable than you were originally - Is hugely positive EV.
Negative scenarios have to be scenarios you live to see, like the rich hoard all wealth or ASI aging therapy is available but you can't personally afford it.
Right...as Zvi says, when people tell you who they are, believe them. I don't think it would be productive for either of us to engage further with such strongly held priors. Thanks for playing at any rate. I do wish other acc avatars like Jezos would similarly bother to engage with anything besides flippancy; it'd make me take their views more seriously.
Thank you, Zvi, for posting this! We need more of this kind of thing: love and pep talk at a time of the greatest crisis ever.
And thank you even more for being my #1 source of crucial information on day-to-day developments on AI. There's a growing number of us doing what we can for our right not to be murdered by rogue AI, but your contribution really stands out, so hang in there and keep up your excellent work!
Amazing ❤️
When art imitates life but then life imitates art, you can’t trust anything. It becomes worse than a bad photocopy.
I don't know if I've ever commented here, but I must have read 100+ of your posts by now.
Thank you for your work.
Here's hoping we're in a good timeline, we make it to a good world on the other side, and Netflix puts out a good docudrama of it all, where this is a pivotal speech. <3
Wouldn't it be polite to attribute the quotation "I don't think we're going to make it," to its author, James Barrat? I only mention it because big tech runs roughshod over copyright and attribution, bigly. And I'm close to the author, who wrote that sentence in the prologue to his recent book, The Intelligence Explosion. BTW I appreciate your work, Zvi.
James Barrat
Perhaps I'm missing important context, but isn't that just, like, a normal English phrase? I'm sure I've said it before myself and heard others say it too. I might as well say that you should be attributing "I only mention it because" to somebody else.
I see your point, but there’s a difference between using a phrase in conversation and committing it to print. Because I put it in my book prologue, interviewees kept bringing this idea back to me, saying it neatly captures the “doomer” point of view. Once it was printed, I had effectively “coined a phrase,” and it spread in a meme-like way.
I don’t consider that a stupendous achievement - I’d have forgotten about it if the media hadn’t picked it up - but I do feel entitled to say I coined it, just as I say I originated the acronym ASI (Artificial Superintelligence) in my 2013 book, Our Final Invention. If someone used it earlier, I'd like to know. Neither of these is an earth‑shattering accomplishment, but as a writer I can’t help noticing when a term I’ve put into print becomes part of the lingua franca. Concurrently, due to generative AI, ideas of attribution and copyright are in danger of becoming eroded into nonexistence, and I like to push back.