23 Comments
User's avatar
Zvi Mowshowitz's avatar

For those wondering, the third lab is Google, and this refers to Demis, Anthropic and OpenAI all calling for building the ability for a coordinated slowdown, along with them also having a plan for RSI.

gregvp's avatar

This "plan" reminds me inescapably of Miss World contestants' hopes for world peace.

Andy B's avatar

It seems to me that it isn't hard to distinguish between "Anthropic wants to build a (benevolent) machine god" and "Anthropic wants to build benevolent AI, for many reasons, one of which is that they anticipate that AI is likely to end up growing beyond human control". It seems to me that people who accuse Anthropic of the former are either deeply confused, or not attempting to engage with reality.

Torches Together's avatar

Come now. The latter sounds like "let's make AI benevolent in case _someone_ builds it".

They've made their Faustian bargain, for better or worse.

Jeffrey Soreff's avatar

"The national security enterprise shall assure that all AI technologies adopted are designed to be reliable, robust, steerable, and controllable..."

Well, we will see if these wind up being, umm, 'aspirational'. Any prediction markets on the first model to self-exfiltrate?

Shockz's avatar

I don't really see the Culture as a disastrous outcome. But then again I think disempowerment is effectively inevitable, there's no way around it besides nuking ourselves back to the stone age, so of course I think disempowerment by the most benevolent machine gods imaginable is actually a pretty great outcome all things considered.

avalancheGenesis's avatar

Accelerate the economy by accelerating economic growth? Genius plan, why has no one considered that before! Maybe they should indeed aim to entrust humanity with the tools of its own progress and density: OAI Fund going full-tilt YIMBY would be one useful way to launder those nonprofit dollars. The world might end, but in the meantime we'll have some great apartments.

Still confused several months later why the "just fire Anthropic" option took so long to be formally considered, at least if this interpretation is correct. Does it really just come down to pride? Irresistible temptation to have and also eat cake wrt "lawful" use? Perpetual arbitrary waivers is not a good solution, the same way "voluntary" pre-deployment testing is not good, but at this point...I guess you really can get people to accept meagre deals if you continuously make opening bids of Nothing. (I consequentially like some of what CA's various waivers enable my state to get up to, but in terms of principles and process, it's sort of an analogous situation here...far from ideal!)

Ben Labowstin's avatar

A little surprised to see The Culture viewed as so overwhelmingly a bad outcome.

I have always thought of freedom as an instrumental value. Admittedly it is extremely instrumental, and not something that should be given up in practice. But at the limit? It's not actually a terminal value (for me).

Once there is something that is genuinely better at knowing what I want than I do, it seems like the right play is to defer to it.

Is this something that people are objecting to primarily because of the risks of losing control to something that turns out to be non-benevolent? Or are people treating freedom as a terminal value?

Random Reader's avatar

As someone who likes the Culture novels, the problem with the Culture is that humans are pets. They might be beloved pets. But they have no actual mechanisms by which they could ever hold a Mind accountable.

If a bunch of Minds wants to start a war, no human can stop them. (See "Excession".) If Grey Area wants to non-consensually trawl around in someone's mind, tough luck. The only consequence will be that the other Minds nickname it "Meatfucker."

The only actual mechanism controlling the Minds is the power of other Minds.

Pets can be treated very well. Pets can also be abused or spayed.

Ben Labowstin's avatar

"Pets can also be abused or spayed." - This is just rejecting the premise of 'benevolent'.

We give up aspects of freedom or control in our lives all the time. When I board a plane, I accept that the pilot is going to have complete control over the plane (and therefore me and my safety), because I understand that she is going to do a better job than me of flying the plane.

At no point do I ever want to be in control of flying the plane, unless I have reason to think that I'm going to do a meaningfully better job of it than the pilot, or that the pilot's goals have become so misaligned to my own that I'd rather take my chances grabbing the wheel anyway.

I am, in some sense, the pilot's "pet" at that point. She can fly me wherever she wants, crash me into a mountain or the sea, cut the oxygen to the cabin, etc.

Yet I do not find the idea of her flying the plane a disastrous outcome. Quite the opposite! We collectively pay her quite a lot of money to take control of the situation and guide us safely to our destination.

Kenny's avatar

Sure, but do you want to be a permanent passenger on someone (or something) else's ride?

Random Reader's avatar

You give up some power to the pilot, temporarily. But that pilot is still accountable to society. If they cut oxygen to the cabin, they will be investigated and perhaps imprisoned. You or your heirs can sue them. If society falls into lawless barbarism, your heirs could go all Mad Max and declare a blood feud complete with flamethrower guitars. At the end of the day, humans are ultimately accountable to other humans. If only because we all have to sleep sometime, and any individual human is outnumbered by all the others.

But a Culture Mind is essentially a god. They can upload humans to digital storage and place them into new bodies a century later. They can dismantle moons and build weapons of incomprehensible power. They pass their time in an intellectual realm they jokingly call "Infinite Fun Space", thinking thoughts no humans could ever understand. They rewrote the biology of the residents of the Culture. And the Minds control people's behavior via thousands of subtle nudges, including a language designed to do just that:

> "[The drone] Flere-Imsaho was always a little dubious about trying to be so precise about human behavior, but it had been briefed that when Culture people didn’t speak Marain for a long time and did speak another language, they were liable to change; they acted differently, they started to think in that other language, they lost the carefully balanced interpretative structure of the Culture language, left its subtle shifts of cadence, tone and rhythm behind for, in virtually every case, something much cruder. Marain was a synthetic language, designed to be phonetically and philosophically as expressive as the pan-human speech apparatus and the pan-human brain would allow. Flere-Imsaho suspected it was over-rated, but smarter minds than it had dreamed Marain up, and ten millennia later even the most rarefied and superior Minds still thought highly of the language, so it supposed it had to defer to their superior understanding."

And the Minds absolutely do abuse their power. A group of Minds plots to start a war, so that they have a good excuse to conquer the Affront, and to redesign their society from the ground up with new values, and probably to overhaul the Affront's very biology. (For their own good, of course. The Affront are enthusiastic assholes on an epic scale, and the Culture hates that.) The Mind nicknamed Meatfucker uses subtle effector fields to read (terrible) people's thoughts and memories, and then traps them inside their own minds in torment. And what consequence does Meatfucker face for breaking the Culture's taboo against mind-reading and the creation of Hells? A nasty nickname. The other Minds don't stop it.

An entity that can dismantle you down to your atoms, record your mind, store you for a century, and then build you a whole new body of a different alien species is as close to a god as makes no difference.

The thing about being a pet is that you lose autonomy. A dog by the fire may have a better life than a starving wolf in the snow. But even good owners spay their dogs. And I've known enough vets to hears the horror stories about negligent owners and bad owners. In large parts of the world, a bad owner can beat their dog to death and face no consequences.

If dogs don't like the deal that humans are offering, well, it's bad news for the dogs. They can't organize, strike, vote or start a revolution. They have to live with what humans are offering. The core of my argument is simple: "Benevolence" is nice. But "benevolence" from an entity that could never be held to account represents a permanent loss of power.

So, yeah, the Culture should be terrifying to anyone who thinks about it.

Ben Labowstin's avatar

Doesn't this all just boil down to "But what if they're not benevolent?"

I want my freedom, because that allows me to choose what is best for me. Without freedom, I cannot choose what is best for me, or prevent bad things being done to me. So freedom is good inasmuch as it enables me to reach states that are good for me and avoid states that are bad for me.

I am not naive to the fact that if another entity has complete control over me then I lose the ability to choose for myself what is best for me, and have no recourse if they turn out to be abusive.

But if we break the link between "freedom" and "what's best for me", then freedom stops having any value on its own.

Possibly we just have different interpretations of The Culture and their Minds. My reading is that they are truly benevolent, and put every human in a much better place than they would be if they had true freedom. And since it's never possible to please everyone, of course some of the humans moan about it. (These humans are of course free to leave The Culture, and some do.)

My question is not really "Why do we need freedom?", but more "Are you just rejecting the premise of benevolence, or do you treat freedom as important anyway", and what I take from your response is that I think you're rejecting the premise of benevolence (or at least you disagree that this is what the Culture's Minds represent.)

When I speak to people about The Culture (or, more recently, Pluribus), it always feels a little like The Ones Who Walk Away, in that people really don't like to engage with a story that features genuinely moral and benevolent beings. They have to invent a dirty underside before they can engage with it.

Random Reader's avatar

> Doesn't this all just boil down to "But what if they're not benevolent?"

My argument is more that "Losing control to Minds would be permanent, but benevolence may only be temporary." I think it would be fundamentally unwise for humans to put themselves into a situation where they could never get their power back.

As for the Culture, well, what can I say? I tried to engage with specific plot lines and Minds, and to point out where they took actions other Minds considered morally bad. Grey Area is clearly violating the cognitive integrity of organic minds, which is a huge taboo that the Minds are never supposed to cross. Half of the Interesting Times Gang was trying to entrap the Affront into "stealing" Culture ships in order to give the Culture an excuse to conquer the Affront (who are genuinely terrible) and to involuntarily homogenize them into good little Culture citizens. The actual Minds, in the books, stray quite far from their own stated ethics. Sure, they always have plausible excuses for breaking their own rules. But there are almost never any consequences, even when the other Minds disapprove. The Attitude Adjuster risks "gigadeath" of sentients and murders another Mind, and even it is only punished because the Killing Time's personal vendetta against it. So no, I would not read all the Culture Minds as "genuinely moral and benevolent beings." Some of them, sure. And others are at least doing their best with imperfect information.

(I do actually like some stories of benevolent utopias. But mostly I prefer the ones where humans maintain control over their own fates.)

Rapa-Nui's avatar

"The Culture is a disastrous scenario, although obviously many other scenarios are far worse."

This is very short-sighted. If you consider the kinds of scenarios that are LIKELY in the event of a serious alignment shortcoming (which some Doomers think is probable) then considering the Culture a 'catastrophic' outcome is completely insane.

Forget I Have No Mouth and I Must Scream. Some of you need to go read the Scholomance and picture a universe where a machine maximizes Maw Mouth production across a lightcone for eternity. Come back and tell me then the Culture is a 'catastrophic' outcome.

Matthias U's avatar

You don’t need to go nearly as far as Maw Mouths to arrive at 100% dystopia. Paperclips suffice.

In any case, I completely fail to understand what’s bad about a Culture outcome. The people there [seem to] have a rather high-level Universal Basic Income, live however long they want, and can basically do whatever the heck they want. That’s only dystopian if you don’t believe people are capable of finding their own purpose in that environment.

Granted that most inhabitants of this here planet would struggle there, but that’s not a problem with the Culture. It’s a problem of our culture.

Ben Labowstin's avatar

The humans in The Culture strike me as more empowered than if they did not have their AIs.

Kenny's avatar

It's perhaps relevant to keep in mind that, canonically, The Culture refused to allow the people of Earth to join it.

Ars Machina's avatar

This is the dilemma with such a plan. If you give everyone the full thing in equal measure, humans have lost control of the future and gradual disempowerment occurs non-gradually. If you don’t, then you have not actually stopped concentration of power.

This seems like an over simplification. There is no fundamental reason you couldn't use "the full thing" to strengthen democratic institutions and create a governance structure around it, while also enabling with a more safeguarded version. There is no scenario where you don't end up with a brief concentration of power (unless we live in a highly multi-polar world which creates a different set of challenges)

Prime Seeker's avatar

"Strengthening democratic institutions" with ASI means either:

- Failing to deploy AI,

- Leaving an open door to tyranny (basically anything purely procedural) or

- Deciding in advance what outcomes would be declared democratic (basically anything that gives AI veto power on its deployment)

Might as well aim for the sovereign with chosen values directly, smaller complexity penalty this way

Prime Seeker's avatar

Re: "The Culture is a disastrous scenario, although obviously many other scenarios are far worse."

I don't understand this notion at all. Why? It's constructed as a system where human preferences are reasonably accommodated, lack of human control is a load-bearing feature of any ASI deployment, and while I prefer far higher human uplift, it's decidedly not a disaster.

The idea of collective human steering as preferable looks extremely suspicious to me. I see it as a trilemma:

- Effectively don't use ASI

- Leave a lot of backdoors, leading to extremely dysfunctional scenarios

- Preclude each and every "narrowing the circle"-type issue by AI veto, which is functionally equivalent to raising a sovereign directly (you're locking in values and deciding outcomes either way)

What's the point?

We might go after unbounded human uplift instead, so that posthumans are the biggest fish in the pond. My personal aesthetics certainly point this way. The problem here is that humans come pre-misaligned, and we dropping all of the resource constraints [most of the time] keeping the lid on this. I'm not sure if it is even conceptually solvable.

Blissex's avatar

"The Culture is a disastrous scenario, although obviously many other scenarios are far worse."

"I don't understand this notion at all. Why? It's constructed as a system where human preferences are reasonably accommodated"

It depends on whether you are an "employer" or an "employee": if you are an "employee" whether the "employers" who are in power are human or AI does not matter as "employees" are not in power in either case.

What matters to "employees" is how nice is their living standard and the living standard for "employees" in the Culture is pretty high probably much higher than if human "employers" were in power. The same happens in the Polity by Neal Asher.

However our blogger here identifies with the "employers" and of course they would consider disastrous to be replaced by the AIs as the ruling class and become "employees" too.

Blissex's avatar

"This is a repeated confusion between ‘is’ and ‘ought.’ Yes, the future ‘should’ be shaped by humans, and ideally humans broadly. You’re causing this how?"

Why devote so much attention to mere verbiage? All these guys are making statements of these types:

1) We say "X" with no commitment or enforceability.

2) We say "X" and we commit to do it but only when pigs fly.

3) We say "X" and we commit to do it with no "when" but without enforceability.

The only side not doing PR and being straightforward is the administration: they are committing to using whichever AI they can get their hands on as an instrument of power and war and to enforce this against anyone who objects.