- Kai Greshake (Donovan): Lot of details wrong here. In short: You can absolutely hook an LLM up to a closed-loop network that has regular material imported from the outside; you just have a sanitization process, plus like seventeen other safety protocols. It's not 100%, but neither is spam blocking, and we all use email.
- Larry Summers (insiders): Better phrased as "insiders don't criticize other insiders IN PUBLIC." In private, all the time.
AI doesn't make the final kill decisions, a human is in the loop. The DOD is discussing removing humans from the loop entirely, which sounds like a very bad idea.
There are no serious proposals removing humans from the loop. There ARE discussions about to what level a human can be involved and it still be considered in the loop. We currently are at "Human authorizes engagement, algorithm chooses specific methodology" in many cases.
The board history timeline is pretty unusual. This board is churning like crazy! And never revealing the reasons for anything. In 2023 three different people left the board for undisclosed reasons. Or in Reid Hoffman’s case, it was later reported that he didn’t really think he had a conflict, but he stepped down because Sam wanted it.
It’s no real surprise that Sam wanted control of the board. That’s exactly how Zuck, the Google founders, and Steve Jobs operated. Maybe he just pushed a little too hard. Oops.
Personally I have confidence in Bret Taylor to be a good board member. He stood up to Elon Musk and handled himself in an incredibly stressful job while selling Twitter.
Bill Gates called himself a “minor wizard” for being resistant to Steve Jobs’s reality distortion field. I think Bret Taylor is also a minor wizard.
Shouldn't the resolution of the OpenAI back-and-forth as being "basically primate dominance games" cause us to update more towards p(doom)? It's like the quintessential archetype of the source of basic coordination problems, and they performed *less* well than most for-profit corporations do today, and at much higher stakes.
Because it sure doesn't fill me with warm fuzzies that the smart people on either side of the dust-up have "AI don't kill everyone-ism" priorities straight.
Great as ever. Not sure about your understanding of the insiders rule - yes, as an insider you theoretically get to choose whether to listen to the outsiders' perspective, but if you're an insider in a non-scenescent organism/organisation then the further in you get the more likely it is that you have been chosen because you won't do so, and moulded so that you don't do so, and had the information that reaches you filtered in such a way that it would be difficult to do so. Insiders are as crippled as outsiders, just in different ways.
I’m starting to feel like “artificial intelligence” as a phrase is preventing some people from taking doom seriously, because it’s too abstract, and they can’t engage with it. I wonder if saying “digital intelligences” or “intelligent software” would effect a change in those people’s perception of concern.
re: the Loopt story about hiring people to pretend to be employees:
I can't find it mentioned online, but I recall hearing a story about how Palantir had gov't folks over at their offices to vet them early on. They were very small then so they went to Fry's to buy a bunch of computers and set them up to make it seem like Palantir had a lot more employees. Then they set the meeting for first thing in the morning, so they could posture like engineers never come to work before noon anyway, etc.
Not a value judgment for or against the Loopt thing, but the fact that Palantir did it is interesting considering that it was, you know, to get a defense contract.
> machine-based systems that can, for a set of defined objectives, make predictions, recommendations, or decisions influencing real or virtual environments
Am I missing something, or how does this not apply to a f---ing thermostat?
Re: 17 (EA ?= totalitarian), I have to push back a little against your dismissal. The reason that EA proposals are not "completely ordinary regulatory regimes" is that they require cooperation of all world powers, something which (IIRC) EY has advocated that we should be willing to enforce with threats of bombing datacenters and so on.
A typical/boring law decided by the US govt is not totalitarian (e.g. some change to the tax code or whatever). Enforcement is directed inwards, and since the value of the law doesn't depend on absolute compliance, there's some slack on the margins.
In contrast, consider something like copyright law. The value of such laws depends in relatively large part on people outside the US going along with them. So we get a bunch of bullshit like US laws being imported into the EU in non-democratic ways, as riders on free trade agreements and so on. This already feels a lot more totalitarian, especially viewed from outside the US.
A more extreme example would be a law on gain-of-function research. A single breach can infect the world in a month, so 99% worldwide compliance is about as good as 0% worldwide compliance. We are aiming for 99.999%+ compliance. So, either (1) essentially all world governments agree on this, (2) we go to war or (3) we shrug and live with the risk of GoF for now, while slowly working on pushing consensus towards (1).
Now, where on this spectrum is AI regulation according to EAs? Seems pretty clear to me that it's another step beyond GoF research. Higher stakes. A harder to contain adversary. Option (1) is not the reality we live in; (3) is out per typical timelines of EAs. That leaves (2). A call that we need worldwide effective AI regulation *right now*, whatever the cost, *is* a call for totalitarianism.
(reposting here as I had trouble getting the comment through on wordpress)
I checked, dunno what happened at WP, it wasn't in the spam filter or the approval queue.
As for the argument, it is good of you to make the argument more explicit here. I would still say that the core claim is that 'ordinary regulatory regime enforced everywhere upon one particular industry' constitutes 'requires totalitarianism.' I would respond that, again, that is not what totalitarianism means. Claiming this is a call for totalitarianism is centrally highly misleading or deeply confused at best.
E.g. "Totalitarianism is a political system that seeks to control all aspects of human life, including political, economic, social, and cultural spheres. It is characterized by strong central rule, often under the control of a single individual or party, that employs all-encompassing campaigns in which propaganda is broadcast by state-controlled mass media in order to control the citizenry."
This is... not that. And when you make a claim 'X calls for totalitarianism' they are very much thinking literally the above. If the argument is 'involves use of realpolitik to ensure compliance by potential rogue states' then... that's just not the same thing at all.
Fair enough, there's a broader and stricter interpretation. But I stand by the point that AI-focused EAs are operating under the assumption that total victory must be achieved, and there's no such thing as partial victory. That's the right stance to hold *if* the beliefs of that group are correct. This is a unique feature of the unfriendly AGI problem domain.
In other problem domains, incrementalism works fine. Incrementalism is also less threatening to people outside the group, more compatible with democratic and multi-cultural norms (i.e., how to get along when we disagree on the direction we should be heading in). Make small changes. Analyze the consequences. Have ongoing dialogue. Repeat. There are very good reasons why developed societies only allow change through such soft, gradual processes. People who refuse to respect this way of doing things are often deluded and/or sociopathic, and end up inflicting harm (both direct and indirect "social fabric" stuff) that dwarfs the benefits of their preferred policies.
The upshot is that AI-EAs will always feel threatening. I'm EA-adjacent and I take AI x-risk quite seriously (even though I disagree on feasibility/timelines). There are very few things worth throwing democracy out the window for, but imminent annihilation of the human race is one of them. I know it's a horrible conclusion, but it's the obvious endgame of utilitarism, being bullet-biters and "taking ideas seriously". I don't think there are any EAs currently gearing up for a worldwide coup (it's a terrible idea for all sorts of moral and practical reasons). But I consider this a quite serious PR problem for the movement because we are, in the abstract, exactly the kind of people who would throw a coup if the circumstances were extreme enough. That sounds crazy to an outsider, but obviously there's a bunch of uncertainty, risk-aversion and some amount of non-utilitarian common sense wrapped in "extreme enough". Normies' imaginations don't stretch that far. So I'm tempted to propose setting our survival odds to 0% in the imminent/fast takeoff worlds out of principle, so that we can be a more trustworthy group in the other worlds.
It doesn't matter if the average EA is virtuous as fuck by the way. It's the worst EA, the next SBF, who might ruin the movement over this.
I do not believe democracy will need to be thrown out the window, unless you count e.g. the current United States or EU as doing so. I would be shocked if there were not broad public support, both in the USA/UK/EU and worldwide, when the time comes, for limits on the development of frontier AI models.
I took Ilya's alignment comments on that podcast to be an intentional simplification, but now that I read this I want to go back and re-listen. My guess is still on that though.
As you are A LOT smarter than me, especially as it pertains the subject of AI, I had a few questions about the pessimistic (or realistic, if you prefer) position if you don't mind:
1. While a lot of experts are worried about AI killing everyone, it seems those closest to the frontier are not as worried as their fellow experts. Shouldn't that update us in an optimistic direction?
2. Assuming we reject any Pascal's mugging-type notion of infinite negative or positive utility, what would the perceived p(doom) need to be for you to agree to "take the plunge" on ASI? And does your answer weight good and bad outcomes equally?
3. Given the possible worlds we could be in, isn't it a good thing that the people at the world's (current) two leading AI companies take AI safety somewhat seriously? Shouldn't that be an additional cause for optimism?
Apologies if you've answered these questions before, and feel free to deny any of the premises if you think they are false.
1. The decision to be close to the frontier is a function of your lack of worry - if you think something will get us all killed you might choose not to do it. And also many of those very much at the frontier ARE worried and doing it anyway.
2. Details matter on what you expect as counterfactuals and positive outcomes, and I don't think precision here is useful, but I will say 1% is go, 10% is not so fast unless you think it only gets worse by waiting, is something like my 'zone of reasonable disagreement.'
3. See #1? Yes, it is a cause for optimism that Demis/Sam take it somewhat seriously. It would be very very bad if they didn't. They also don't take it as seriously as I'd like, or in the exact ways that I'd like, but they're much better than replacement-level choices here (e.g. the CEOs of Google/Microsoft are much much worse, Meta exists etc). It's priced in.
Did you see that Ezra Klein gave you a shout-out in the NY Times?
I did, and I appreciate it.
"win-win deal to diffuse" → "defuse"
"Executive Order could also does not" → extra "could"
"a bit three" → "a big three"
- Kai Greshake (Donovan): Lot of details wrong here. In short: You can absolutely hook an LLM up to a closed-loop network that has regular material imported from the outside; you just have a sanitization process, plus like seventeen other safety protocols. It's not 100%, but neither is spam blocking, and we all use email.
- Larry Summers (insiders): Better phrased as "insiders don't criticize other insiders IN PUBLIC." In private, all the time.
If my spam blocker fails, no one dies. A 99% SLA isn't acceptable for software that gets to make kill/no-kill decisions.
Air defense systems already use AI to make engagement decisions, and they are definitely not 100%.
AI doesn't make the final kill decisions, a human is in the loop. The DOD is discussing removing humans from the loop entirely, which sounds like a very bad idea.
There are no serious proposals removing humans from the loop. There ARE discussions about to what level a human can be involved and it still be considered in the loop. We currently are at "Human authorizes engagement, algorithm chooses specific methodology" in many cases.
Thank you for writing these. Always impressively informative and fascinating!
The board history timeline is pretty unusual. This board is churning like crazy! And never revealing the reasons for anything. In 2023 three different people left the board for undisclosed reasons. Or in Reid Hoffman’s case, it was later reported that he didn’t really think he had a conflict, but he stepped down because Sam wanted it.
It’s no real surprise that Sam wanted control of the board. That’s exactly how Zuck, the Google founders, and Steve Jobs operated. Maybe he just pushed a little too hard. Oops.
Personally I have confidence in Bret Taylor to be a good board member. He stood up to Elon Musk and handled himself in an incredibly stressful job while selling Twitter.
Bill Gates called himself a “minor wizard” for being resistant to Steve Jobs’s reality distortion field. I think Bret Taylor is also a minor wizard.
Excellent as always Zvi, cheers!
Shouldn't the resolution of the OpenAI back-and-forth as being "basically primate dominance games" cause us to update more towards p(doom)? It's like the quintessential archetype of the source of basic coordination problems, and they performed *less* well than most for-profit corporations do today, and at much higher stakes.
Because it sure doesn't fill me with warm fuzzies that the smart people on either side of the dust-up have "AI don't kill everyone-ism" priorities straight.
Great as ever. Not sure about your understanding of the insiders rule - yes, as an insider you theoretically get to choose whether to listen to the outsiders' perspective, but if you're an insider in a non-scenescent organism/organisation then the further in you get the more likely it is that you have been chosen because you won't do so, and moulded so that you don't do so, and had the information that reaches you filtered in such a way that it would be difficult to do so. Insiders are as crippled as outsiders, just in different ways.
I’m starting to feel like “artificial intelligence” as a phrase is preventing some people from taking doom seriously, because it’s too abstract, and they can’t engage with it. I wonder if saying “digital intelligences” or “intelligent software” would effect a change in those people’s perception of concern.
re: the Loopt story about hiring people to pretend to be employees:
I can't find it mentioned online, but I recall hearing a story about how Palantir had gov't folks over at their offices to vet them early on. They were very small then so they went to Fry's to buy a bunch of computers and set them up to make it seem like Palantir had a lot more employees. Then they set the meeting for first thing in the morning, so they could posture like engineers never come to work before noon anyway, etc.
Not a value judgment for or against the Loopt thing, but the fact that Palantir did it is interesting considering that it was, you know, to get a defense contract.
> machine-based systems that can, for a set of defined objectives, make predictions, recommendations, or decisions influencing real or virtual environments
Am I missing something, or how does this not apply to a f---ing thermostat?
Re: 17 (EA ?= totalitarian), I have to push back a little against your dismissal. The reason that EA proposals are not "completely ordinary regulatory regimes" is that they require cooperation of all world powers, something which (IIRC) EY has advocated that we should be willing to enforce with threats of bombing datacenters and so on.
A typical/boring law decided by the US govt is not totalitarian (e.g. some change to the tax code or whatever). Enforcement is directed inwards, and since the value of the law doesn't depend on absolute compliance, there's some slack on the margins.
In contrast, consider something like copyright law. The value of such laws depends in relatively large part on people outside the US going along with them. So we get a bunch of bullshit like US laws being imported into the EU in non-democratic ways, as riders on free trade agreements and so on. This already feels a lot more totalitarian, especially viewed from outside the US.
A more extreme example would be a law on gain-of-function research. A single breach can infect the world in a month, so 99% worldwide compliance is about as good as 0% worldwide compliance. We are aiming for 99.999%+ compliance. So, either (1) essentially all world governments agree on this, (2) we go to war or (3) we shrug and live with the risk of GoF for now, while slowly working on pushing consensus towards (1).
Now, where on this spectrum is AI regulation according to EAs? Seems pretty clear to me that it's another step beyond GoF research. Higher stakes. A harder to contain adversary. Option (1) is not the reality we live in; (3) is out per typical timelines of EAs. That leaves (2). A call that we need worldwide effective AI regulation *right now*, whatever the cost, *is* a call for totalitarianism.
(reposting here as I had trouble getting the comment through on wordpress)
I checked, dunno what happened at WP, it wasn't in the spam filter or the approval queue.
As for the argument, it is good of you to make the argument more explicit here. I would still say that the core claim is that 'ordinary regulatory regime enforced everywhere upon one particular industry' constitutes 'requires totalitarianism.' I would respond that, again, that is not what totalitarianism means. Claiming this is a call for totalitarianism is centrally highly misleading or deeply confused at best.
E.g. "Totalitarianism is a political system that seeks to control all aspects of human life, including political, economic, social, and cultural spheres. It is characterized by strong central rule, often under the control of a single individual or party, that employs all-encompassing campaigns in which propaganda is broadcast by state-controlled mass media in order to control the citizenry."
This is... not that. And when you make a claim 'X calls for totalitarianism' they are very much thinking literally the above. If the argument is 'involves use of realpolitik to ensure compliance by potential rogue states' then... that's just not the same thing at all.
Fair enough, there's a broader and stricter interpretation. But I stand by the point that AI-focused EAs are operating under the assumption that total victory must be achieved, and there's no such thing as partial victory. That's the right stance to hold *if* the beliefs of that group are correct. This is a unique feature of the unfriendly AGI problem domain.
In other problem domains, incrementalism works fine. Incrementalism is also less threatening to people outside the group, more compatible with democratic and multi-cultural norms (i.e., how to get along when we disagree on the direction we should be heading in). Make small changes. Analyze the consequences. Have ongoing dialogue. Repeat. There are very good reasons why developed societies only allow change through such soft, gradual processes. People who refuse to respect this way of doing things are often deluded and/or sociopathic, and end up inflicting harm (both direct and indirect "social fabric" stuff) that dwarfs the benefits of their preferred policies.
The upshot is that AI-EAs will always feel threatening. I'm EA-adjacent and I take AI x-risk quite seriously (even though I disagree on feasibility/timelines). There are very few things worth throwing democracy out the window for, but imminent annihilation of the human race is one of them. I know it's a horrible conclusion, but it's the obvious endgame of utilitarism, being bullet-biters and "taking ideas seriously". I don't think there are any EAs currently gearing up for a worldwide coup (it's a terrible idea for all sorts of moral and practical reasons). But I consider this a quite serious PR problem for the movement because we are, in the abstract, exactly the kind of people who would throw a coup if the circumstances were extreme enough. That sounds crazy to an outsider, but obviously there's a bunch of uncertainty, risk-aversion and some amount of non-utilitarian common sense wrapped in "extreme enough". Normies' imaginations don't stretch that far. So I'm tempted to propose setting our survival odds to 0% in the imminent/fast takeoff worlds out of principle, so that we can be a more trustworthy group in the other worlds.
It doesn't matter if the average EA is virtuous as fuck by the way. It's the worst EA, the next SBF, who might ruin the movement over this.
I do not believe democracy will need to be thrown out the window, unless you count e.g. the current United States or EU as doing so. I would be shocked if there were not broad public support, both in the USA/UK/EU and worldwide, when the time comes, for limits on the development of frontier AI models.
I took Ilya's alignment comments on that podcast to be an intentional simplification, but now that I read this I want to go back and re-listen. My guess is still on that though.
As you are A LOT smarter than me, especially as it pertains the subject of AI, I had a few questions about the pessimistic (or realistic, if you prefer) position if you don't mind:
1. While a lot of experts are worried about AI killing everyone, it seems those closest to the frontier are not as worried as their fellow experts. Shouldn't that update us in an optimistic direction?
2. Assuming we reject any Pascal's mugging-type notion of infinite negative or positive utility, what would the perceived p(doom) need to be for you to agree to "take the plunge" on ASI? And does your answer weight good and bad outcomes equally?
3. Given the possible worlds we could be in, isn't it a good thing that the people at the world's (current) two leading AI companies take AI safety somewhat seriously? Shouldn't that be an additional cause for optimism?
Apologies if you've answered these questions before, and feel free to deny any of the premises if you think they are false.
1. The decision to be close to the frontier is a function of your lack of worry - if you think something will get us all killed you might choose not to do it. And also many of those very much at the frontier ARE worried and doing it anyway.
2. Details matter on what you expect as counterfactuals and positive outcomes, and I don't think precision here is useful, but I will say 1% is go, 10% is not so fast unless you think it only gets worse by waiting, is something like my 'zone of reasonable disagreement.'
3. See #1? Yes, it is a cause for optimism that Demis/Sam take it somewhat seriously. It would be very very bad if they didn't. They also don't take it as seriously as I'd like, or in the exact ways that I'd like, but they're much better than replacement-level choices here (e.g. the CEOs of Google/Microsoft are much much worse, Meta exists etc). It's priced in.