61 Comments

Oh god I find the podcast format to be incredibly vacuous. Notebook LM itself is an interesting product, though I suspect that, like most Google products, it will die on the vine.

Expand full comment

Podcast episode for this post:

https://open.substack.com/pub/dwatvpodcast/p/ai-84-better-than-a-podcast

Feedback request for those that listen, I've included the Notebook LM audio files where they are referenced in the test for the latest post. How does that impact your enjoyment? Would you prefer them be moved to the end of the episode, not included at all?

Expand full comment

I definitely liked it included in line like you did it this time. Another option would be to include chapters (I'm not sure how hard this is, but I know other podcasts often have them) so that someone could easily skip to the next section if they wanted to

Expand full comment

I liked it but would have preferred a warning

Expand full comment

"The key is you must pick one of these:

1. Building a superintelligence under current conditions will turn out fine.

2. No one will build a superintelligence under anything like current conditions.

3. We must prevent at almost all costs anyone building superintelligence soon."

I mustn't. There is (among, I'm sure, others) mystery option 4:

Building a superintelligence under current conditions may or may not turn out fine - and may or may not be possible - and the benefits of either reaching or approaching the goal are sufficient that we should move towards it as quickly as possible.

There are many fine arguments for why an unaligned superintelligence will inevitably lead to doom. One or more of them may even end up being correct, who knows? When you stipulate by definition that a universe with an unaligned superintelligence inevitably means it destroys humanity then, sure, obviously you shouldn't build it. But that's assuming the conclusion.

EY / Zvi are correct that previous proofs of safety may not hold into the future. Regrettably, that is a feature of our universe that is inescapable. Aligning a superintelligence is hard - indeed, probably impossible - and keeping it aligned is also probably impossible.

Oh well. There are many more ways that superintelligence can go wrong than right? Maybe, who knows? No one can truly claim to have fully explored this space of possibilities. Even if so, there are many more ways that your body could have gone wrong than right resulting in your death in utero - yet here we all sit.

Risk and reward, reward and risk. Demanding that we solve an impossible task, or prove that which cannot be proved, before making an incredible technological leap is equivalent to saying that we should never make the jump.

Expand full comment

The risk of "everyone dying" makes reward kinda meaningless. You cant enjoy anything if you are dead.

I don't think human birth is a good metaphor, as a process that does a lot of checking during development.

Expand full comment

1. Can you meaningfully quantify the risk of everyone dying?

2. Isn't everyone's risk of dying currently 100%? How does AI change your current fate?

ETA:

3. If you don't think it's a good metaphor, I'm sure you can imagine another that works just as well. But the process that does the checking can also fail in many more ways than it can work properly. Any complex system has many, many more potential failure than success states. And yet there you are, despite your entire existence relying on an intricate balance between internal and external systems, none of which are in any way "aligned" to support your existence and the fact the current arrangement of your atoms is only one of a near-infinite combination of which the one you happen to exist within is vanishingly unlikely.

Arguments from "I can imagine more ways this can go wrong than right" don't really tell you anything.

Expand full comment

The current default path is AI takeover/loss of control.

Currently my children would not die and have full lives without AI. Therefore, AI significantly changes their fate and hugely impactful.

Hope is not a plan.

Expand full comment

So that's a no on being able to quantify the risk. You're trying to serve a cake baked entirely of assumptions.

You assume the default path is AI takeover/loss of control.

You assume AI takeover/loss of control means a worse outcome for humans. It's very easy to posit many scenarios where AI takeover is better for humans than the status quo.

Expand full comment

We can also see the consequences of animals losing control: powerlessness results in habitat destruction and extinction.

There is no path for benevolence and present actions by Moloch is nonbenevolent.

Expand full comment

Is there a well-founded reason to assume that an ASI will invariably be Moloch-ian?

Expand full comment

Your children (and even ourselves) are definitely not guaranteed to have full lives without AI. We're still one matchstick away from a nuclear war, which would definitely impact you if you live in the US or Europe.

Expand full comment

Not guaranteed, but humanity and definitely biological life would survive. Otoh, AI guarantees their lives are going to be inhuman, if not dead.

Expand full comment

We've had a tiny glimpse into what that "survival" would look like during COVID and the disruptions from the war in Ukraine. Now scale that up 1000x and see if that's still a world you'd enjoy living in, assuming you're lucky enough to survive WWIII in the first place.

Expand full comment

This seems like the anthropic fallacy. If we didn't exist, we wouldn't be here to say we don't exist.

You don't have to quantify the risk of humanity losing control. You just have to admit that we don't know what will happen. There's no good reason to take on that level of risk.

Expand full comment

What level of risk? And why is humanity losing control default bad? How many of your life outcomes do you think you have full control over?

Expand full comment

This seems to basically lead to "why is dying bad?"

Expand full comment

Begging the question is a logical fallacy.

Expand full comment

Looks like you are sane. Let me know if you join #PauseAI, I'll like to coordinate.

Expand full comment

"despite your entire existence relying on an intricate balance between internal and external systems, none of which are in any way "aligned" to support your existence"

These various internal and external systems are internally aligned as the result of evolutionary adaptive processes to ensure reproductive success (with a mesa-optimization of organism survival) and externally aligned inasmuch as they were either created by humans or such evolutionary optimization took place in response to the environment.

The whole point is that what counts for "wellbeing" for an AI (e.g. minimizing its loss function, making more paperclips, whatever) has no natural overlap with, and is by default antagonistic to[1], human survival.

[1] due inter alia to competition for resources with no natural upper bound on such competition, and a better-adapted, smarter adversary than humans competing with us for said resources means that humans lose.

Expand full comment

[1] is completely assumed and presumes that because no upper bound exists the competition will immediately and inexorably extend to the upper bound. No justification for that assumption. Many possible arguments to the contrary and no way to know.

Expand full comment

Why would it not? All you need is a maximizing goal like "Make as many paperclips as possible." Given that loss-function minimization is the nature of AI training (and the ease of giving maximizing goals to an AI) that ought be our default assumption as to the outcome instead of justifying some sort of agnosticism.

Expand full comment

I think the forefront of Safety focused thought has moved beyond a simplistic paperclipping argument. You may be behind the times.

But let's return to your [1]. Given the size of the universe and what we know of how life develops, it's almost certain that intelligent life exists or will exist elsewhere in the universe. Due inter alia to competition for resources with no natural upper bound on such competition, we should therefore assume that we will be in an existential death match with other forms of life, and there's no reason to assume that it will not be smarter and more capable than us.

Therefore, it's only reasonable for us to immediately cease any activity that sends any detectable signals or industrial activity that modifies our atmosphere in a way they could detect, lest they come for our atoms. Humanity can survive just fine at a lower technological level - we did for most of our existence - and to do otherwise is playing Russian roulette for $20. Right?

Expand full comment

Just commenting to say that I appreciate your username; Long Live the Empire; RIP James Earl Jones.

Expand full comment

Do you think I knew about the plan to place a tracker on the Millenium Falcon? Was I briefed that morning when I came on shift and instructed to let them kill me and stuff my body in a smuggling hole? I wonder how I felt about that. Did I volunteer?

One must imagine TK-421 happy.

Expand full comment

this brought a much needed smile to my face lol tysm 🩵

Expand full comment

To me this is a combination of #1 and #2, and you're saying that in combination they are probable enough that benefits of trying exceed costs. That's entirely reasonable as a thing to argue but I don't think it contradicts the framework. Perhaps I should have added some probabilistic words here to be more exact (e.g. #1 should probably say 'will probably' if we're being exact).

I do think that a by construction unaligned superintelligence almost certainly uses our atoms for something else (and if it didn't, I'd ask if it was actually aligned in the end although that's more of a definitions thing?)

Expand full comment

No, not exactly. My objection to the framework is that you're limiting the options based around your assumption that an unaligned superintelligence will be hostile to human life by default. It creates an insurmountable burden while framing the debate to lead to your conclusions.

When I say "may or may not" I don't mean "will probably" or "will probably not". I mean "may or may not". It is not an answerable question.

Arguing that an unaligned superintelligence might use our atoms for something else is reasonable enough - saying that it almost certainly will projects an unwarranted certainty. There is simply no way to know, estimate the possibility, or even understand the space of potential outcomes / motivations. We can swap campfire tales about what we think will happen until we run out of smores. Means nothing.

Expand full comment

The only reason I can think of for *not* assuming that ASI would use our atoms for something else would be an upper bound on its resource needs by hypothesis. But why would we ever expect that?

Expand full comment

No clue. How well do you think you can predict the behavior of an ASI? Or the form it might take? Or any of the intermediate forms that we can create short of ASI, should that prove to be impossible?

Expand full comment

In my opinion, that's not a reasonable position. I don't think one needs to argue that unaligned superintelligence necessarily leads to doom to say that we should avoid it, just that we don't know what the outcome would be. And as you say, we clearly don't know. We can just continue enjoying the benefits of regular, non-ASI progress in the meantime. There's no particular reason that we need to take the leap until we can get closer to scenario 1 - we actually understand and can manage the risks involved.

Just because we didn't die in utero doesn't mean we should play Russian roullette because someone offered us $100.

Expand full comment

You're making wild assumptions of both sides of the risk reward equation. Do you have any reason to think we have a 1/6 chance of death for only $100 stakes? Even those opposed to ASI tend to agree that the potential benefits are generally higher than a rapidly depreciating c-note.

Also, many safety / pause proponents are trying to stifle all AI progress, not just AGI or ASI and push policies that would have the same effect.

Expand full comment

Also, your daily reminder that you (and everyone else reading this) have at most 50-70 years left to live, of which 20-40 would be in great health. AGI is the only thing that can realistically prevent that from happening.

Expand full comment

In 2017, Altman explored running for governor of CA[1]. That position has nothing in common with running YC or running OpenAI except being a famous boss.

[1] https://www.vox.com/2017/5/14/15638046/willie-brown-column-sam-altman-might-run-governor-california-2018

Expand full comment

Great post that seems more focused than the past! Thought these sentences are a great characterization of OAIs current strategy:

> Well, yes, I suppose it is, given that we don’t have anything else and OpenAI has no intention of trying hard to build anything else. So, iterative deployment, then, and the hope that when things go wrong we are always still in charge and around sufficiently to fix it for next time.

Expand full comment

gm. Nice to see this in my inbox. Will read / listen when I have time.

Expand full comment

Regarding Robert Miles 10,000:

It reminds me of an old bet on a poker forum, maybe 18 years ago, by a sports bettor/poker player named Admo. He had bet a friend of his $10,000 that he (Admo) could get three independent posters on the forum to offer him money (at least 10 dollars each). The catch was, his opponent was allowed to make any and all restrictions on Admo's behavior, short of banning Admo from posting at all. Included in the bet was that if Admo won, he would get to run it again for another 10,000, with any new restrictions applied.

So his opponent agreed, and came up with about 75 restrictions. Lots of obvious ones, like "can't ask for money" "can't post about personal hardship" "can't use sockpuppets" "can't offer any services" "can't start posting betting picks" "can't offer coaching of any kind or talk about coaching" and on and on.

Admo won the bet over the next few months by posting many very high quality and funny photoshops. Enough work went into them and people were so impressed, that a few offered money for whatever reason. Rather than run it again with new restrictions against photoshopping and whatever else, his opponent conceded and bought out for $3,000. Admo later said that he had three more ideas that he was very confident would work for him to win.

Expand full comment

I have a question that is motivated by noticing that A Narrow Path is 80 pages long. Why is it that rationalists concerned about existential risk tend to write such long documents? Eliezer writes very long things, you write very long things, Less Wrong and the EA forum are full of very long documents.

But at the same time it seems like a very traditional principle of rational thinking that it is bad for documents to be too long, that it is good to make your arguments concise and to the point, that editing out the lowest quality parts of your reasoning will make your overall argument better, that this process will improve the quality of your thinking. Do you disagree with this? Or is it that there's something else trading off against it?

Expand full comment

Great question. The answer is because the goal is not to be convincing people to agree with your vibes or one-sentence proposal. The goal is to actually figure complicated things out, for both you and your readers - and that isn't something you can do by choosing the most convincing short arguments.

For me, there's also the 'not enough time to make it shorter' issue - it would vastly increase time cost to do that bespokely.

Expand full comment

I feel like both sides claim that their writing style is better for figuring out complicated things. For example Paul Graham on essay-writing:

https://paulgraham.com/best.html

On the intellectual value of the editing process:

"The prize for being strict with what you've written is not just refinement. When you take a roughly correct answer and try to make it exactly right, sometimes you find that you can't, and that the reason is that you were depending on a false assumption. And when you discard it, the answer turns out to be completely different."

I'm not sure what to conclude. It's just a curious stylistic correlation to me, between the x risk community and long document writing.

Expand full comment

What's the source for Achiam being head of alignment? Altman's tweet says he'll be head of mission alignment. https://x.com/sama/status/1839096160168063488?t=pAqW0DKmKmhI60cfl6EXow&s=19

"Josh Achiam is going to take on a new role as Head of Mission Alignment, working across the company to ensure that we get all pieces (and culture) right to be in a place to succeed at the mission."

Nonetheless, views on AI alignment are still pretty relevant for the position of mission alignment, considering the mission involves developing aligned AGI. But that arguably makes his views not quite as concerning compared to if he was head of technical alignment.

Expand full comment

About the NotebookLM podcast, you write that it’s “not information dense and no way to get informed.” You’re obviously absolutely right, but I think there was no intention for it to be information dense. If anything, it’s designed to be the opposite. I see the use case as taking something dense and difficult, and teasing it apart into less information density and more easy analogies.

Separately, or maybe not, try putting the EU AI bill in and see what you get out, I’d be curious to see what these imaginary podcasters think.

Expand full comment

my retort to "the narrow path" would be "you would not be able to get buy in from China, because commentators like leo, among others, have more or less precommitted the West to 'defecting' from the desired outcome of cooperation on safety, in a game theoretic sense, and have fallen far short of doing the necessary political work of reaching out to what safetyist elements exist in China and convincing them cooperation is still possible and hawks won't secretly defect (good luck with this one)"

thus, the secret 4th thing, "it's already too late to leisurely 'pick' among 1-3"

Expand full comment

I'm biased here, but I really do think you're misunderstanding/mischaracterizing a couple things.

>In other words, the new head of AI alignment at OpenAI is on record lecturing EAs that misalignment risk from AGI is not real.

(1) head of mission alignment != head of AI alignment

(2) believing P(Misaligned AGI doom by 2032): <1e-6% does not at all imply one believes misalignment risk is not real

For what it's worth, I agree with Josh that there is nearly zero change of misaligned AGI doom by 2032. And yet I consider AI misalignment one of the most important challenges for our species. I just think it's a bit naive/underconfident to think it's going to happen in the next 8 years.

Expand full comment

I wouldn’t be surprised if the entry-level engineer market salary was also due to overseas hiring. A number of startups are solving the sourcing and quality issue that made this more of a hassle in the past.

Expand full comment

Examples?

Expand full comment

When you learn how much SV loves the Sovereign Individual and James Burnham's Managerial Revolution, so much of their attitudes towards any idea of regulation or limitation makes sense.

Expand full comment

“e.g. empowering bad human actors by helping them accomplish a significant bioterrorist operation, with scale of impact smaller than Covid19): maybe ~3%?”

A literal interpretation of smaller than includes “much smaller than” … in which case, I think 3% is way, way too low. Surely bad actors will achieve some small scale terrorism.

If “smaller than” is meant to mean “nearly the same scale, but not very much bigger” … I think anything with a 3% risk of covid19 level casualties needs to be shut down right now.

Expand full comment

The genie we have let out of the bottle with covid1 lab leak theories is now every ambitions terrorist group knows that it is possible to kill tens of millions of people on a very modest budget.

Expand full comment

>None of this, of course, is going to save you if somebody marches troops across the bridge, and the rhythm of their marching feet is close to a resonant frequency of the bridge.

> (Eg: Angers Bridge, 1850, 226 deaths.)

From reading the Wiki article it seems like this was taken into account by the army:

> As usual in crossing that bridge, the soldiers had been ordered to break step and to space themselves farther apart than normal.

The actual root cause was the wind, combined with the increased load on the bridge. There was no (intentional) marching going on. I get EY's point but as is often with fun stories, reality is usually disappointing and not that fun.

Expand full comment