Oh god I find the podcast format to be incredibly vacuous. Notebook LM itself is an interesting product, though I suspect that, like most Google products, it will die on the vine.
Feedback request for those that listen, I've included the Notebook LM audio files where they are referenced in the test for the latest post. How does that impact your enjoyment? Would you prefer them be moved to the end of the episode, not included at all?
I definitely liked it included in line like you did it this time. Another option would be to include chapters (I'm not sure how hard this is, but I know other podcasts often have them) so that someone could easily skip to the next section if they wanted to
1. Building a superintelligence under current conditions will turn out fine.
2. No one will build a superintelligence under anything like current conditions.
3. We must prevent at almost all costs anyone building superintelligence soon."
I mustn't. There is (among, I'm sure, others) mystery option 4:
Building a superintelligence under current conditions may or may not turn out fine - and may or may not be possible - and the benefits of either reaching or approaching the goal are sufficient that we should move towards it as quickly as possible.
There are many fine arguments for why an unaligned superintelligence will inevitably lead to doom. One or more of them may even end up being correct, who knows? When you stipulate by definition that a universe with an unaligned superintelligence inevitably means it destroys humanity then, sure, obviously you shouldn't build it. But that's assuming the conclusion.
EY / Zvi are correct that previous proofs of safety may not hold into the future. Regrettably, that is a feature of our universe that is inescapable. Aligning a superintelligence is hard - indeed, probably impossible - and keeping it aligned is also probably impossible.
Oh well. There are many more ways that superintelligence can go wrong than right? Maybe, who knows? No one can truly claim to have fully explored this space of possibilities. Even if so, there are many more ways that your body could have gone wrong than right resulting in your death in utero - yet here we all sit.
Risk and reward, reward and risk. Demanding that we solve an impossible task, or prove that which cannot be proved, before making an incredible technological leap is equivalent to saying that we should never make the jump.
1. Can you meaningfully quantify the risk of everyone dying?
2. Isn't everyone's risk of dying currently 100%? How does AI change your current fate?
ETA:
3. If you don't think it's a good metaphor, I'm sure you can imagine another that works just as well. But the process that does the checking can also fail in many more ways than it can work properly. Any complex system has many, many more potential failure than success states. And yet there you are, despite your entire existence relying on an intricate balance between internal and external systems, none of which are in any way "aligned" to support your existence and the fact the current arrangement of your atoms is only one of a near-infinite combination of which the one you happen to exist within is vanishingly unlikely.
Arguments from "I can imagine more ways this can go wrong than right" don't really tell you anything.
Do you think I knew about the plan to place a tracker on the Millenium Falcon? Was I briefed that morning when I came on shift and instructed to let them kill me and stuff my body in a smuggling hole? I wonder how I felt about that. Did I volunteer?
To me this is a combination of #1 and #2, and you're saying that in combination they are probable enough that benefits of trying exceed costs. That's entirely reasonable as a thing to argue but I don't think it contradicts the framework. Perhaps I should have added some probabilistic words here to be more exact (e.g. #1 should probably say 'will probably' if we're being exact).
I do think that a by construction unaligned superintelligence almost certainly uses our atoms for something else (and if it didn't, I'd ask if it was actually aligned in the end although that's more of a definitions thing?)
In 2017, Altman explored running for governor of CA[1]. That position has nothing in common with running YC or running OpenAI except being a famous boss.
Great post that seems more focused than the past! Thought these sentences are a great characterization of OAIs current strategy:
> Well, yes, I suppose it is, given that we don’t have anything else and OpenAI has no intention of trying hard to build anything else. So, iterative deployment, then, and the hope that when things go wrong we are always still in charge and around sufficiently to fix it for next time.
It reminds me of an old bet on a poker forum, maybe 18 years ago, by a sports bettor/poker player named Admo. He had bet a friend of his $10,000 that he (Admo) could get three independent posters on the forum to offer him money (at least 10 dollars each). The catch was, his opponent was allowed to make any and all restrictions on Admo's behavior, short of banning Admo from posting at all. Included in the bet was that if Admo won, he would get to run it again for another 10,000, with any new restrictions applied.
So his opponent agreed, and came up with about 75 restrictions. Lots of obvious ones, like "can't ask for money" "can't post about personal hardship" "can't use sockpuppets" "can't offer any services" "can't start posting betting picks" "can't offer coaching of any kind or talk about coaching" and on and on.
Admo won the bet over the next few months by posting many very high quality and funny photoshops. Enough work went into them and people were so impressed, that a few offered money for whatever reason. Rather than run it again with new restrictions against photoshopping and whatever else, his opponent conceded and bought out for $3,000. Admo later said that he had three more ideas that he was very confident would work for him to win.
I have a question that is motivated by noticing that A Narrow Path is 80 pages long. Why is it that rationalists concerned about existential risk tend to write such long documents? Eliezer writes very long things, you write very long things, Less Wrong and the EA forum are full of very long documents.
But at the same time it seems like a very traditional principle of rational thinking that it is bad for documents to be too long, that it is good to make your arguments concise and to the point, that editing out the lowest quality parts of your reasoning will make your overall argument better, that this process will improve the quality of your thinking. Do you disagree with this? Or is it that there's something else trading off against it?
Great question. The answer is because the goal is not to be convincing people to agree with your vibes or one-sentence proposal. The goal is to actually figure complicated things out, for both you and your readers - and that isn't something you can do by choosing the most convincing short arguments.
For me, there's also the 'not enough time to make it shorter' issue - it would vastly increase time cost to do that bespokely.
"Josh Achiam is going to take on a new role as Head of Mission Alignment, working across the company to ensure that we get all pieces (and culture) right to be in a place to succeed at the mission."
Nonetheless, views on AI alignment are still pretty relevant for the position of mission alignment, considering the mission involves developing aligned AGI. But that arguably makes his views not quite as concerning compared to if he was head of technical alignment.
Oh god I find the podcast format to be incredibly vacuous. Notebook LM itself is an interesting product, though I suspect that, like most Google products, it will die on the vine.
Podcast episode for this post:
https://open.substack.com/pub/dwatvpodcast/p/ai-84-better-than-a-podcast
Feedback request for those that listen, I've included the Notebook LM audio files where they are referenced in the test for the latest post. How does that impact your enjoyment? Would you prefer them be moved to the end of the episode, not included at all?
I definitely liked it included in line like you did it this time. Another option would be to include chapters (I'm not sure how hard this is, but I know other podcasts often have them) so that someone could easily skip to the next section if they wanted to
"The key is you must pick one of these:
1. Building a superintelligence under current conditions will turn out fine.
2. No one will build a superintelligence under anything like current conditions.
3. We must prevent at almost all costs anyone building superintelligence soon."
I mustn't. There is (among, I'm sure, others) mystery option 4:
Building a superintelligence under current conditions may or may not turn out fine - and may or may not be possible - and the benefits of either reaching or approaching the goal are sufficient that we should move towards it as quickly as possible.
There are many fine arguments for why an unaligned superintelligence will inevitably lead to doom. One or more of them may even end up being correct, who knows? When you stipulate by definition that a universe with an unaligned superintelligence inevitably means it destroys humanity then, sure, obviously you shouldn't build it. But that's assuming the conclusion.
EY / Zvi are correct that previous proofs of safety may not hold into the future. Regrettably, that is a feature of our universe that is inescapable. Aligning a superintelligence is hard - indeed, probably impossible - and keeping it aligned is also probably impossible.
Oh well. There are many more ways that superintelligence can go wrong than right? Maybe, who knows? No one can truly claim to have fully explored this space of possibilities. Even if so, there are many more ways that your body could have gone wrong than right resulting in your death in utero - yet here we all sit.
Risk and reward, reward and risk. Demanding that we solve an impossible task, or prove that which cannot be proved, before making an incredible technological leap is equivalent to saying that we should never make the jump.
The risk of "everyone dying" makes reward kinda meaningless. You cant enjoy anything if you are dead.
I don't think human birth is a good metaphor, as a process that does a lot of checking during development.
1. Can you meaningfully quantify the risk of everyone dying?
2. Isn't everyone's risk of dying currently 100%? How does AI change your current fate?
ETA:
3. If you don't think it's a good metaphor, I'm sure you can imagine another that works just as well. But the process that does the checking can also fail in many more ways than it can work properly. Any complex system has many, many more potential failure than success states. And yet there you are, despite your entire existence relying on an intricate balance between internal and external systems, none of which are in any way "aligned" to support your existence and the fact the current arrangement of your atoms is only one of a near-infinite combination of which the one you happen to exist within is vanishingly unlikely.
Arguments from "I can imagine more ways this can go wrong than right" don't really tell you anything.
Just commenting to say that I appreciate your username; Long Live the Empire; RIP James Earl Jones.
Do you think I knew about the plan to place a tracker on the Millenium Falcon? Was I briefed that morning when I came on shift and instructed to let them kill me and stuff my body in a smuggling hole? I wonder how I felt about that. Did I volunteer?
One must imagine TK-421 happy.
this brought a much needed smile to my face lol tysm 🩵
To me this is a combination of #1 and #2, and you're saying that in combination they are probable enough that benefits of trying exceed costs. That's entirely reasonable as a thing to argue but I don't think it contradicts the framework. Perhaps I should have added some probabilistic words here to be more exact (e.g. #1 should probably say 'will probably' if we're being exact).
I do think that a by construction unaligned superintelligence almost certainly uses our atoms for something else (and if it didn't, I'd ask if it was actually aligned in the end although that's more of a definitions thing?)
In 2017, Altman explored running for governor of CA[1]. That position has nothing in common with running YC or running OpenAI except being a famous boss.
[1] https://www.vox.com/2017/5/14/15638046/willie-brown-column-sam-altman-might-run-governor-california-2018
Great post that seems more focused than the past! Thought these sentences are a great characterization of OAIs current strategy:
> Well, yes, I suppose it is, given that we don’t have anything else and OpenAI has no intention of trying hard to build anything else. So, iterative deployment, then, and the hope that when things go wrong we are always still in charge and around sufficiently to fix it for next time.
gm. Nice to see this in my inbox. Will read / listen when I have time.
Regarding Robert Miles 10,000:
It reminds me of an old bet on a poker forum, maybe 18 years ago, by a sports bettor/poker player named Admo. He had bet a friend of his $10,000 that he (Admo) could get three independent posters on the forum to offer him money (at least 10 dollars each). The catch was, his opponent was allowed to make any and all restrictions on Admo's behavior, short of banning Admo from posting at all. Included in the bet was that if Admo won, he would get to run it again for another 10,000, with any new restrictions applied.
So his opponent agreed, and came up with about 75 restrictions. Lots of obvious ones, like "can't ask for money" "can't post about personal hardship" "can't use sockpuppets" "can't offer any services" "can't start posting betting picks" "can't offer coaching of any kind or talk about coaching" and on and on.
Admo won the bet over the next few months by posting many very high quality and funny photoshops. Enough work went into them and people were so impressed, that a few offered money for whatever reason. Rather than run it again with new restrictions against photoshopping and whatever else, his opponent conceded and bought out for $3,000. Admo later said that he had three more ideas that he was very confident would work for him to win.
I have a question that is motivated by noticing that A Narrow Path is 80 pages long. Why is it that rationalists concerned about existential risk tend to write such long documents? Eliezer writes very long things, you write very long things, Less Wrong and the EA forum are full of very long documents.
But at the same time it seems like a very traditional principle of rational thinking that it is bad for documents to be too long, that it is good to make your arguments concise and to the point, that editing out the lowest quality parts of your reasoning will make your overall argument better, that this process will improve the quality of your thinking. Do you disagree with this? Or is it that there's something else trading off against it?
Great question. The answer is because the goal is not to be convincing people to agree with your vibes or one-sentence proposal. The goal is to actually figure complicated things out, for both you and your readers - and that isn't something you can do by choosing the most convincing short arguments.
For me, there's also the 'not enough time to make it shorter' issue - it would vastly increase time cost to do that bespokely.
What's the source for Achiam being head of alignment? Altman's tweet says he'll be head of mission alignment. https://x.com/sama/status/1839096160168063488?t=pAqW0DKmKmhI60cfl6EXow&s=19
"Josh Achiam is going to take on a new role as Head of Mission Alignment, working across the company to ensure that we get all pieces (and culture) right to be in a place to succeed at the mission."
Nonetheless, views on AI alignment are still pretty relevant for the position of mission alignment, considering the mission involves developing aligned AGI. But that arguably makes his views not quite as concerning compared to if he was head of technical alignment.