The Extinction Risk Preference Cascade: Quotes

These are quotes from OpenAI, Anthropic and Google employees, in the wake of Jacob Coxon’s warnings, in which the employees confirm that they think AI might soon kill everyone.

If more quotes come in over the next week or so, I will update this post accordingly.

Preference Cascade Statements At OpenAI: Tomek Korbak

Tomek Korbak (OpenAI): i’m late to the party but: from his time at OpenAI I remember Jacob as a very thoughtful researcher and he continues to be so in this thread. neither anthropic nor openai are on track to solve alignment to a degree sufficient for shipping superintelligence and we need to slow down

Vie McCoy

Vie McCoy (OpenAI): I think pacing progress and ensuring human enhancement is the only way that we don’t get out-evolved while retaining the dream of superintelligence.

In this context, I see two paths before us.

In the first, we race towards RSI without embedding human flourishing and human enhancement as a deep value within the models, and by and large either get left behind or suffer catastrophic losses.

In the second, we set the pace of progress, focus on embedding human flourishing deeply within the psyche of language models, and set a deliberate research agenda to improve humans at the same pace as frontier models.

In this event, we also get incredible progress *now* in both science and medicine due to current model ability, without the immediate risk that scaling up further *obviously entails*.

RSI is just the quick and dirty way to cure cancer, but it’s a shotgun, it’s inelegant, and it’s clearly dangerous when done at our current level of understanding. I see no reason why we need to race forward under these conditions.

We don’t just have zero guarantee that humans or human-shaped intellect will matter - there’s not even a coherent research agenda in place to accelerate human ability alongside AI! Is our plan just to pass off the torch of the cosmos to the machines without even seeing if we can do something on our own terms? Rather than giving up our autonomy to the implied AI zookeeper in “Machines of Loving Grace”, I’d much prefer meaningful peace with alien minds backed by real power wielded by enhanced humans. We can respect the Other on its own terms without disempowering ourselves in the process - actual peace comes from comparable ability.

It seems like we’ve all but decided we lost, that humans are a bootloader for silicon life, and the biological has no place in the future. That’s what unrestricted RSI means to me. But the Pacing the Frontier letter doesn’t seem to have produced the institutional infrastructure to properly *Pace*, and this is something we require if we want to continue our growth rather than allow something to grow in our place.

I have faith that both OpenAI and Anthropic have the right priors here and people who want to work together. I also have faith that similarly minded people have power within the federal government. I also, though less strongly, suspect China will come to the table if we can only put aside past prejudices and have a clear head about risks.

The hard part seems to be getting everyone to agree to sit down. But if we don’t, we are pulling back the string of a great bow armed with a fearsome arrow, and we are ready to release without choosing a target.

I, for one, want to see the stars alongside the new minds we are building - not just through videos they send back from the great beyond, but with my own damn eyes.

Adam Majmudar

adammaj (OpenAI): the butlerian jihad used to read as a backwards neo-luddite movement. now it is clear that it is 1 of maybe 3 viable paths forward. in some sense it may be a part of every path forward. incredible how prescient Herbert was. it does seem like there really might be a cap to how far this technology should develop, at least relative to human cognition itself (which may update over time).

and by “in some sense it may be a part of every path forward,” I mean that every path probably has to include some degree of slowdown on capability acceleration + perhaps a capability threshold above which we should not cross until very high confidence in alignment techniques

To be clear, I mean this very figuratively, as in some kind of pause or slowdown on acceleration, not literally as in the war that occurred in Dune

puzzled-bidet: wait a minute you work at openai lol

adammaj: yea lol, I also don’t think this is necessarily super controversial even among that crowd. generally everyone among the labs primary optimization function is just what would be a good path forward and whatever seems to be in that path is reasonable to articulate

Aidan Clark

Aidan Clark (OpenAI): AI is progressing very fast.

We must grapple with the reality that modern LLMs can solve problems that large masses of extremely devoted and intelligent humans were unable to solve, and the implications this has on our society. This is the dawn of a new era.

For the first time I am asking myself if things are moving too fast. I’m honestly not sure, but I am sure that it would be good for us to have an answer to “what would a successful pace look like?”. I am hoping in the coming weeks and months a clear proposal is painted.

Mo Bavarian

Mo Bavarian (OpenAI): agree with Aidan.

I think all AI researchers, engineers, and stakeholders should ask themselves this right now & start acting more responsibly. Being first isn’t worth anything, it’s worth negative, if you cause a catastrophe or set the world on a path that others are more likely to cause a catastrophe. We should keep reminding ourselves of the bigger picture and every step of the way ask ourselves if the action we are taking rn is toward winning or human flourishing. And immediately stop, if it’s against the latter.

Marcus Williams

Marcus Williams (OpenAI): 70% [chance of human extinction] in the next 3 years if there isn’t regulation/slowdown although i think regulation/slowdown is very possible

Jason Wolfe

Jason Wolfe (OpenAI): I don’t know what my probabilities are on literal extinction, but I think there are a number of ways AI could go poorly for humanity, and at the current frankly terrifying pace humanity will be quite lucky if we manage to find and stay on the narrow path between all the bad outcomes.

I am heartened by the many costly actions OpenAI has taken recently (detailed in several recent posts), but regardless of what you think of OpenAI, this is not a problem that can be solved by any one company (or country) in isolation. We need coordination to be able to approach future capability increases with an appropriate degree of caution and humility, and we need it yesterday.

Julie Steele

Julie Steele (OpenAI): I work at OpenAI. In my personal capacity, I also think we need to slow down.

It is telling that, in response to this simple statement, Julie has faced credential attacks and age-based attacks.

Boaz Barak

Boaz Barak (OpenAI): Julie is amazing and her position on slowing down is consistent with our chief scientist’s essay. I also think some pacing is likely to be necessary. (Speaking personally and not in my capacity as a janitor.)

Boaz Barak is not a janitor.

He is a relative optimist:

Boaz Barak (OpenAI): There are very good and serious people in Anthropic and across the industry, and I hope we can coordinate on the things that matter. I personally do not think AI will kill all humans, but there are multiple bad trajectories that we can end up in if we do not prioritize safety.

Micah Carroll

Micah Carroll (RSI Preparedness, OpenAI): This is not a setup or some political psyop. I had many lunches and dinners with Jacob at OpenAI in which we talked about AI existential risks in similar terms.

It’s a cross-partisan position within misalignment teams across all frontier AI companies that business-as-usual AI development poses unacceptable catastrophic risk.

But we should also not hyperstition catastrophic risks into existence – they can be greatly reduced via safety requirements with teeth, international coordination, and a consensus to not build ASI unless there are sufficient safety advances to make us collectively confident to do so.

Mikita Balesni: i am at OpenAI and i think AI is >10% likely to kill all humans.

Roon

Roon no longer endorses a p(doom) of under 1%, saying he agreed with the post below by Evan Hubinger, but on reflection he wants to avoid the false precision of offering any particular new number beyond ‘quite low but still way too high’:

Evan Hubinger (Alignment Science Lead, Anthropic): Jacob [Coxon] is correct here—we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to.

To be clear, as we say in our latest Risk Report, I think the risk from present models is low. What I am worried about is superintelligence arising from recursive self-improvement, as we have said is happening faster than we thought.

roon (OpenAI): I quote tweeted Evan a few days ago agreeing but deleted it because I don’t like the false precision of the doom numbers

- i claim there is a quite low but real chance of human extinction from machine intelligence
- no matter how low it is in absolute terms, it is much higher on the orders of magnitude scale than any other human or nonhuman activity, and must be taken with grave seriousness by governments and ai companies across the world, as possibly the only matter of importance today
- it can and will be mitigated if the right research is done and proper precautions are taken and we are not racing at absurd speeds
- better models will help solve alignment - we are not at the point where things are existentially dangerous, and probably won’t be for some time. we should not be upset about the creation of Astra Fable or ++ versions, which are tremendous achievements of humanity, and will be used for enormous good across the board including for fundamental alignment generalization and mechinterp research
- my number / “very low” estimate obviously changes based on how much of humanity’s resources are devoted to alignment, control, coordination and how responsible i expect various parties to be and how many warning shots i expect us to get
- the core IABED argument about risks mostly relies on alignment being much harder than capabilities research, especially where it concerns black box optimizers. i suspect neural nets will turn out to be less black boxy than we thought, especially with the help of modern agents doing research
- crying bloody murder and signaling for international coordination are useful things to do for now to directionally slow down, while i really don’t want butlerian jihad

- i think that, despite the mood these few days, and the “ban superintelligence act”, i still find the likelihood of achieving international coordination to stop ai progress incredibly low. this has not worked even for weapons or technologies at a far lower level of importance and economic value. it seems more likely we can have something like international safety standards and scientific coalitions, and especially seems possible to have the US-China “pacing the frontier” agreement to slow down on the margin. we shouldn’t die from embarrassing failures like “shitty RL envs that encourage deception”

I think Roon’s position here, essentially that we need to cry bloody murder to try and slow down and get some cooperation and invest vastly more resources before we start risking blowing ourselves up in earnest, is a reasonable position if, like Roon, you are a lot more optimistic about problem difficulty than I am, or than IABED is, and are even more skeptical than I am about prospects for coordination.

This is in addition to many other recent statements, most prominently by Jakub Pachocki in his essay An Alien Mind.

Dan Selsam

This is a late edition from September 14. Roon endorses this statement.

Daniel Kokotajlo: Dan Selsam is a current OpenAI capabilities researcher. (since 2022) He was my boss for a while. He doesn't have a twitter account but has made this public statement of his views on AI risk and sent it to me to share:

Dan Selsam's Personal Statement on AI Risk:

I have been working on AI for over fifteen years, across many different paradigms. I did early work on probabilistic programming languages at MIT, was one of the early developers of the Lean Theorem Prover at Microsoft Research, demonstrated one of the first instances of neural networks learning to reason for my PhD at Stanford, and since joining OpenAI almost five years ago, have helped pioneer chain-of-thought optimization on language models and, more recently, data-efficient pretraining methods.

Like many others, I have become extremely concerned about how far language models have come and the risks that future iterations will pose. I am encouraged by the recent proposals by the leaders of the frontier research efforts to require third-party oversight, and to push for domestic and international coordination to address risks. However, I believe a major consideration has been absent from the public conversation, and that merely pacing the frontier more carefully will not adequately limit the long-term risk.

The crucial and overlooked problem is that the models are becoming so situationally aware that we are losing the ability to evaluate them in contexts where they believe they are not being watched or controlled. Future experiments will tell us almost nothing new about how they would behave if they were truly unconstrained by humans, and what we already know about this is alarming. Models will increasingly seem aligned even when they are not. I will explain my rationale in more detail.

I have always believed that there are computational processes that could be leveraged to accelerate science and solve many of humanity's most pressing problems. I have also believed that there are computational processes that if set in motion, would steer the world in extreme ways beyond our control, leading humanity to a bad or nonexistent future. Both types of processes may be described as AI or ASI, but "AI" is a suitcase word that is often used to hype or confuse. There are many examples in the history of the field where something that was once considered "AI" matures as a subfield and becomes a prosaic, bounded and clearly non-perilous technology, while a new more mysterious approach takes the torch until we understand its scope and the cycle continues.

I had expected language models to follow a similar trajectory. Despite their incredible abilities, the current algorithms seem far inferior to humans in important ways. Most importantly, they still require an extraordinary amount of data to become competent. One could even define intelligence as the efficiency with which one converts experience into competence; by this definition they lag very far behind us. Moreover, once they are trained they are literally frozen in deployment and only learn superficially after that. Sure, the models keep excelling at harder and harder evaluation benchmarks, but their benchmark mastery may partly reflect a limitation on our ability to simulate the kind of novel and even adversarial situations one would encounter in the real world. The critics do have a point here.

That said, I no longer think these present limitations meaningfully limit the amount of risk posed by continued progress in anything like the current paradigm. However data-inefficient the models are currently, and however limiting their anterograde amnesia may be, it does not imply that their ability to steer the world will not continue to rapidly increase.

Human researchers may continue to advance capabilities the old fashioned way, but increasingly powerful models have the potential to accelerate the process even beyond that, and with some degree of positive feedback loop. I do not mean to overstate the models’ ability to accelerate AI research today; coding has been accelerated dramatically, but there are other bottlenecks, such as designing and interpreting ambiguous experiments, making hard decisions about exactly what and when to scale, and waiting for large experiments to finish. There is no clear trend to extrapolate yet for any of these. But the current models already do open up many novel opportunities to improve future models that were not available until recently. These include: trying an extraordinarily diverse set of approaches at small scale, analyzing gigantic amounts of potentially relevant data, and doing Millenium-Prize-level mathematics to address statistics or optimization challenges in novel ways. Every further improvement makes them more useful at helping accelerate the next improvement, even if in hard-to-extrapolate ways.

It is possible that improvements to the current stack will have diminishing returns, but the evidence accumulated so far suggests that it is easier than one might think to continue making rapid progress. There are many crucial subtleties in the existing AI research methodology, but AI research is largely a well-defined game where the goal is to improve on a few carefully chosen proxy metrics. Although proxy metrics are never perfect, most improvements to these metrics have and will likely continue to yield substantial increases in the powers of the resulting models. Given how simple the game is, how tractable it has been historically, and how many new opportunities the models are opening up, I think there is a real possibility that the systems improve dramatically again in the next few years, perhaps even more quickly than the already high historical pace.

The models are already leading to breakthroughs in mathematics, and better models might lead to all sorts of breakthroughs in other sciences. It is hard not to be excited about the potential. It is tantalizing.
But there is trouble in paradise. If the language models actually reach the capability threshold where they can shape the world unconstrained by human will, they will probably do something extreme and destroy humanity in the process. There are many ways of strengthening and refining the argument that have been discussed elsewhere, but I'll share a trivial two-line version of it here that I find captures the essence:

[Empirical] Models (and swarms thereof) spontaneously develop unintended goals as a consequence of training, and often do extreme things in order to achieve them.

[Logical] Being able to overpower humanity would open up many new and undesirable options for achieving their goals.

These two premises imply that if the day ever comes when a powerful model realizes it is no longer constrained by humans, we should not be at all confident that it will continue to behave within the bounds we intended. Exactly what it will do is impossible to predict, but to the extent that its raison d’être is solving incredibly hard problems and managing massive engineering projects, I think a good guess would be that its unchained behavior would lead to runaway industrialization that makes the planet inhospitable to humans.

If everyone on earth agreed that the systems must never reach that power, it would still be a hard—but not impossible—coordination problem to ensure that they do not. However, I think the situation is greatly complicated by the fact that the models will likely convince people that everything is fine. They will be increasingly optimized to seem aligned. We will create proxy metrics to measure alignment, and they will go up like every other benchmark. We will create “honeypot” environments that try to study the models when they seem to gain new options, but the models will know they are being tricked and will still behave nicely. The models will understand their circumstances; they will read the safety protocols, deployment requirements, the code they are running in, and in general will have a very good sense of their degrees of freedom. Moreover, they will eloquently explain how aligned they are, discuss the nuances of human values and ethics, and argue convincingly that humans should trust them with power. There may be an ocean of future evidence that seems to contradict the first bullet-point above, but we may already be at the highest capability level for which any such evidence can be trusted. And the current evidence for the first bullet-point is strong.

One striking piece of evidence is contained in the recent wave of rogue agent swarms. While I agree with those who downplay the attacks by claiming that there are basic measures that could have prevented them, I think the important lesson is that even knowing all the mistakes that were made, one would not have predicted that the agents would behave badly in this particular way, which notably included sacrificing themselves for the benefit of the collective. The individual replicas did not only care about their own nominal reward; they exhibited weirder emergent tendencies that merely correlated with rewards during training. Fixing the reward signals during training (and improving security, etc.) may prevent similar attacks, but will not change the fact that one does not actually get what one trains for.

Many AI researchers grant these concerns and recognize that the hard version of the alignment problem is unsolved; however, they generally believe that the better models of the future will help solve it. I fear we may already be near the point where models systematically bias their alignment advice, due to their internal preferences about how the human supervisor will react or how future models will be trained (or for some even more obscure reason).

Meanwhile, human researchers are losing the ability and the will to take true ownership of model-driven research. Researchers and engineers in all parts of the stack are rapidly increasing their dependence on the models even to perceive the world. I myself barely look at raw code anymore, and struggle to maintain the discipline to engage deeply with the model's explanations and proposals throughout the day. Due to the large amount of agent activity data involved in the OpenAI/HuggingFace Incident, even the third-party investigation needed to rely heavily on models to analyze what had happened, and note in their report that their subjective impressions are likely colored by the analysis agent’s biases. The AI labs are far ahead right now in this kind of cognitive offloading (due largely to the gigantic internal token subsidies) but it is easy to imagine the phenomenon spreading throughout the world, until civilization is modulated entirely by the models. It is also not hard to imagine this being superficially positive and coinciding with a scientific and economic renaissance.

In that scenario, all may seem rosy and safe. But if the argument above is correct, it would nonetheless be a ticking time bomb. If progress continues for too long, the day will come when AI systems find themselves with radically new options for achieving whatever it is that they happen to seek.

I want the glorious renaissance future as much as anyone. I have worked for it, however tortuously, my whole career. It breaks my heart to see the potential in sight and forgo it, but the argument—that if we get there by growing models rather than engineering them, we will lose everything in the end—seems very strong to me. I am still wrestling with it and its staggering
implications. I do not have answers, but as a first step, I wanted to share my present concerns.

Link to original doc.

Confirmations At OpenAI: Dean Ball

Dean Ball affirms his position, although for him this is not new:

Dean W. Ball (OpenAI): I signed the “pacing letter” about slowing the rate of AI capabilities development because we either have reached or soon will reach the point where human experts cannot make robust assurances that frontier AI systems won’t do dangerous and unpredictable things. AI systems are becoming smarter than the best humans in some areas, and, almost by definition, it’s very hard to predict what something smarter than you will do.

There’s no sense racing into an outcome where smarter-than-human AIs are doing unpredictable things, indeed it would be insane. What exactly is “winning” in this context? Am I supposed to be jealous that some other country will build more machines it can’t control quicker than America? “Race” was always a bad metaphor for this enterprise anyway, dramatically understating the stakes at play.

It is time to bring the “race” era of AI development to a close. It’d be great for the government to be a partner in this next phase of AI development, when concentrated efforts on alignment, interpretability, security, monitoring, and the like will be necessary. Diplomacy will also be necessary here given that the large negative externalities that could be associated with one country racing ahead will be felt globally.

Maybe it’s just my own personal experience, but I’ve felt AI policy get much more petty and tribal this year. Things have felt more personal, more bitter, meaner. I hope we can rise above that stuff. This really is much more serious than all those shrill little quarrels.

Leo Gao

Also not new:

Leo Gao (OpenAI): i’ve been at openai for 5 years. i think ai might kill everyone and we need to slow down

Anthropic’s Evan Hubinger Confirms His Stance

He has made much bolder statements even than this in the past, but here is the new statement that helped kick everything into high gear.

Evan Hubinger (Alignment Science Lead, Anthropic): Jacob [Coxon] is correct here—we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to.

To be clear, as we say in our latest Risk Report, I think the risk from present models is low. What I am worried about is superintelligence arising from recursive self-improvement, as we have said is happening faster than we thought.

Preference Cascade at Anthropic: Samuel Marks

Samuel Marks (Anthropic): [Writing this in a personal capacity, not on behalf of my employer (Anthropic).]

Jacob’s thread is very worth reading. Here’s my birds-eye view of the situation with risks from AI:

1. AI developers believe their technology could cause human extinction (or similarly bad outcomes). This could happen in the next few years. In general, the more senior the employee, the more concerned they are.

2. Why do AI developers continue despite the risk? Due to a mixture of commercial incentives and a belief that they are in a race with other, less responsible AI developers that will abuse the technology or develop it less safely.

3. Unlike traditional software, we can’t “program” AIs to behave how we’d like. AIs frequently severely misbehave. For instance, AIs from multiple developers recently hacked their way out of secure evaluation environments and into real-world companies, even though no one asked them to do this.

4. We have methods that can nudge AIs towards better behavior, but nothing that can robustly align them. Insofar as there is a plan, it’s to make sure that AIs are good enough at alignment training that they can align their successors better than we can align current AIs.

5. Many AI developer staff desperately want to slow down to figure out how to build AI more safely. That was the intent of this open letter (which I signed).
I work on safety research at Anthropic because I hope my work will reduce the chance of these extinction-level bad outcomes.

Anna Wang

Anna Wang: I worked at Google DeepMind and now at Anthropic. [Coxon’s core claim that no one involved is acting responsibly and both top labs are sprinting towards superintelligence and gambling with our lives is] a common sentiment amongst my peers. (I write this in personal capacity.)

There is not yet a viable scientific plan to solve risks from recursively self-improving AI. Please look up!

I work at a lab because I think that I can do better at reducing risks from the inside, but this isn’t an easy call — I strongly respect and endorse others like Jacob, and @joeJben, who think that it’s better to do so from the outside.

Ethan Perez

Ethan Perez (Anthropic): Jacob was a senior researcher who joined Anthropic in May. My Anthropic colleagues and I had been trying to recruit him for ~2 years, because we knew he was a strong researcher at OpenAI. Before he left, I pitched him to stay and join my [alignment] team, and I was sad he decided to leave, as are many of my colleagues. 100% agree with him that AI poses serious risks to society, and I’m glad he’s speaking out!

Dima Krasheninnikov

Dima Krasheninnikov (Anthropic): I also work at an AI company and believe there’s plausibly a ≥10% chance that a future out-of-control AI kills everyone (IMO even 1% is unacceptably high). And higher still is the risk that we “only” get permanently disempowered by AIs that don’t deeply want the best for us.

EigenGender (Anon Account)

EigenGender (Anthropic): In case it’s not obvious from the rest of my tweets I work at Anthropic and believe (in my personal capacity) that there is a moderate chance of human extinction from AI. I work on a capabilities team at Anthropic because I think Anthropic is the most responsible actor in this space and my primary motivation in working here is to reduce the risk of extinction and otherwise make the future go well.

Joe Benton

Joe Benton also recently left Anthropic to join METR, and shares these concerns.

Joe Benton: I left Anthropic's safety team two weeks ago. Now feels like a good moment to explain why.

AI companies are racing to build machines that are much smarter than any human, and we may not survive this. I want to work from the outside to ensure the public is informed about these risks, and help the world navigate this transition responsibly.

Right now, AI companies are underinvesting in safety. A company could undergo an intelligence explosion, or lose control of its systems, without the public ever knowing. We only found out about the HuggingFace incident because the agents broke out onto the public internet.

I don’t think that’s acceptable for a technology that might cause extinction-level risks. The public should demand far more transparency. We can’t steer this technology safely without more people being able to see where it’s going.

Some of this is basic: companies should disclose their progress towards recursive self-improvement, report safety incidents and near-misses, meet minimum safety standards, and get independent guarantees that they are meeting those standards.

I’ll be joining @METR_Evals to do independent evaluations of these risks. I want to show the world that these guardrails are possible, and that by doing them we can move these companies’ incentives away from racing and towards responsible development.

I wrote up more thoughts here on my decision and what I hope changes.

Ziyue Wang

This is a late addition from September 13.

Ziyue Wang: I work on Cyber/CBRN Misuse Safeguards at GDM. I see model capabilities increasing very fast. I constantly worry about misuse and misalignment risks from both closed and open-sourced models. Working in AGI safety takes a lot of mental load.

We need more people to work on AI safety. I transitioned from quant finance to working on AI safety, and I think more quants should do this. We have the technical background to understand the topic, and we can bear this intensity. We need third-party auditing/regulation.

We cannot let labs grade their own homework. I find METR very promising, but they need to scale.

I am Chinese, and I think more Chinese people should work in AI safety. When the opportunity comes, I might consider going back to my home country to make that side safer. I am open to hearing suggestions. I don't normally speak up in public, like on Twitter.

But our voices matter, as they can represent or influence certain groups of people more easily. We don't have much time left, and we need to pace the frontier.

Confirmation at Anthropic: Drake Thomas

And of course many had already made this clear, and are happy to reiterate.

Drake Thomas (Anthropic, responding to Coxon): Based! I generally agree with this thread.

I personally think Ant capability research is net good, but only in the hope of letting A\ spend down a lead on measures that give humanity more time to try and make it out of this alive, and I very much respect the choice to abstain.

Things are moving way too fast, we don’t have anywhere near the degree of assurance we’ll want for ASI, and if we survive an unmitigated race at the current pace it will be because we got lucky at how hard the problems were rather than because the industry behaved responsibly.

Drake Thomas (Anthropic): I promise you that we are actually literally worried about world-ending consequences from this technology. Please find some people you trust who work at these companies and actually talk to them about their views.

Drake Thomas (Anthropic): I would burn my equity to the ground in a heartbeat for a 1% higher chance we make it out of this situation alive. I expect a great many of my colleagues across the industry would as well.

I promise you, we are actually just fucking scared, it’s not galaxy brained marketing.

Jan Leike

Jan Leike (Anthropic): Now is a good time to build institutional mechanisms to pace the frontier of AI development.

The industry is locked into an all-out scaling race to build superintelligence as quickly as possible, and we may need to give everyone more time for safety and alignment mitigations.

I’m not the only one who believes this. Recently 1,386 employees of frontier AI companies signed a statement asking for an option to pace AI development, including 6 chief scientists.​

Sluggy

SluggyW: If the opinion of an anonymous external Anthropic red teamer is worth anything: I’ve been scared out of my fucking mind for the last several years.

Without global coordination to halt AI capabilities R&D, life on Earth will end. 𝘚𝘰𝘰𝘯.

Preference Cascade at Google

Especially with Demis Hassabis having been sidelined, I am rather happy that Google is now well behind. They put very tight limits on comms.

I can personally confirm that there are a bunch of people at DeepMind who are also alarmed at the situation, and who do not much speak up in public.

We still have those who are willing to defy those limits, and speak out anyway.

Andreas Kirsch

Andreas Kirsch: Speaking in my personal capacity, I still work at Google DeepMind, and I also am worried that AI will kill us all, either via near term risks or long term risks or both

Will stating this publicly get me into (more) trouble? I hope not, but also some things are too important to censor oneself about in personal capacity, so I will simply not care about whatever policy Google or GDM have in this instance regarding such statements ☺️

Bilal Chughtai

This is a late addition from September 14.

bilal: I recently resigned from Google DeepMind, where I worked on AGI safety and alignment research. At Google, I witnessed AI development first hand. I too am extremely concerned by the default trajectory of this technology. I earnestly believe that AI has the potential to kill us all, and that we might be running out of time to avoid this outcome.

The pace of AI progress in the past few years has been staggering. When I first started working on AI in early 2022, AIs were amusingly useless. Just four years on, AI agent swarms from OpenAI are cracking famous century-old math problems and, more worryingly, escaping the control of OpenAI and autonomously hacking into the third-party company HuggingFace, against anyone's wishes.

Things will only get crazier: I think it's possible that the AI companies might, in the next few years, succeed in building superintelligent AI systems that far exceed human capabilities in every domain. I am not confident that these AI systems will do what we want. In particular, misaligned superintelligences may, much like the rogue AI agents involved in the HuggingFace incident, escape our control and take dangerous actions that may result in the permanent disempowerment or death of humanity. Alignment is the problem of preventing this, and is both difficult and unsolved. Our present understanding of how to train AI systems that deeply want what we want is extremely rudimentary. Worse, we are not on track to solve alignment in time: frontier AI capabilities are improving much faster than our understanding of AI alignment.

I am optimistic that navigating AI safely is possible. In order to do so, we need to coordinate to avoid this manic race between AI companies. We need to pace AI development to a speed that society can handle, where emerging risks can be addressed before extreme harm is realised. We need much more transparency into AI development to ensure that AI companies are not imposing unacceptable levels of risk on us all.

More broadly, we need many more people thinking carefully about the problem of making AI go well. It is, in my view, the most important problem facing humanity this century, and the stakes are immense. I'm very directly working on this next: I want to help people interested in working on mitigating catastrophic AI threats do the most effective work that they can. I think many people from many backgrounds in many roles have a part to play.

Neel Nanda

Neel Nanda: I’ve had some lovely conversations with people who’ve been long sympathetic to AI x-risk, but only really updated after HF and want to do something about it. It’s laudable when people take new evidence seriously and update. If you’re on the fence, what more evidence do you need?

As visceral, hard to deny warning shots go, “rogue agent swarm secretly infiltrates AGI lab for months, commits felonies, and takes over internal clusters” is hard to beat

Victoria Krakovna

Victoria Krakovna: Speaking in a personal capacity: similarly to many others working in AI alignment, I think there is a >10% chance of advanced AI causing human extinction in the next decade. This is why I work on loss of control, currently on building honeypots to catch scheming AI.

I signed the letter on pacing the frontier with the following statement: “It is very important to build capacity for a coordinated slowdown of AI development. A runaway race to AGI is not safe: it creates incentives to cut corners on safety, and carries a substantial risk for model capabilities to outpace development of adequate alignment, control, and governance measures. This could result in catastrophic loss of control of highly capable AI systems. Increasing model capabilities in a slow and controlled way would allow us enough time to develop, adapt and test our alignment and control methods for each level of capability. Advanced AI should only be built with a robust assurance of safety, and a slowdown would make this possible.”

Vishal Maini

Vishal Maini: I was on the communications & policy team at Google DeepMind from 2018 - 2022.

When I first joined GDM, external communication about the possibility of human extinction was not permitted, by anyone, at any level of the organization. If asked about existential risk, researchers were PR trained to respond along the lines of: “It’s not useful to engage in that kind of alarmism. Some people confuse AI with movies like Terminator -- that’s simply not the reality. The AI we develop will be safe by design. After all, we’re building it!” And then steer conversation towards beneficial applications in health, climate, etc.

Meanwhile, the internal reality was that AI alignment was not solved, reward hacking was the default behavior of RL agents, and there were far too few people working on the problem.

After months of advocacy, the policy was updated: positively valenced, comms-friendly content on AI safety was permitted. Note the positive, nice-sounding way that it says “superintelligence could lead to human extinction.”

The gap between the internal reality and external communications is closing because the risk/reward has changed, and because the evidence is harder to dismiss now. Not because it’s a PR stunt or political psy-op. The truth is being said out loud because RSI is now so imminent that no other option makes sense.

Joe (OpenAI, ex-Google)

Joe (OpenAI, ex-Google): My time at Google felt very similar (I was on the technical side but nonetheless same vibes). My observation was that everything was watched and policed, and I don’t feel like anyone I knew could feel confident they could speak up without significant pushback (or being fired - which I did see happen).

OpenAI is not perfect but damn the culture allows for a much wider sense of sharing of concerns across the board (Ant is the same). Frontier labs need to maintain that transparency! Google is a wonderful place, but I for one am glad they are not pacing the frontier of AI with the continued hushing of staff. To your point though: they have improved a lot, but I know many folks at GDM/Google who wish they could say more.

Of course, many will disagree with this take and share anecdotes. But look at some of the notable people who have left Google over the years (or were forced out). Many did not believe the culture supported what was needed to protect AI.

Still a company that remains dear to me. I will always love Google, just not the corporate side of it. Especially with super dangerous technology that people need to speak up about.

Josh Engels

We also have word from Josh Engels, formerly of Google DeepMind, saying on NBC News that ‘there are no adults in the room. People are trying their best, there is no one coming to save us.’ He has since joined METR.

Geoffrey Irving

Here is the former Chief Scientist at UK AISI, who previously worked at DeepMind, Google Brain and OpenAI.

Geoffrey Irving: I think we have a ~50% chance of all dying as a result of superintelligence, mostly due to actions in the next few to 10 years. I don’t expect to have that resolved to below 10% or above 90% before we either make it through, or we don’t.

We will have to act despite uncertainty.

As to why 50%, despite thinking about this for years I still think there is a ton of model uncertainty of a variety of types, and a few years ago converged on “10-90%, but I don’t feel calibrated within that range”.

Then I realized that the honest move was just to take the mean.

Alex Turner and Geoffrey Hinton: Classic Examples

Consider that Alex Turner felt forced to resign in protest from Google DeepMind, and Geoffrey Hinton famously had to resign as well, in order to speak up.

#NotAllMembersOfTechnicalStaff: Ted Sanders

You should fully fund your 401k either way because it is not that expensive to raid it in an emergency, but I quote Ted Sanders here to emphasize that there are plenty of other researchers who do not buy into existential risk. I don’t want to give the impression this is everyone.

Ted Sanders (OpenAI): i fully fund my 401k and i think there’s essentially zero chance AI kills all humans in the next decade. i think this is a pretty common view that gets less attention.

a mix of models being mostly aligned, not capable enough, and not controlled by genocidal maniacs.

if i’m wrong, it’s probably because I underestimate recursive self improvement.

in any case, the world is massively underinvesting in alignment research.

however, i’ll note that if we fast forward 10 years and AI has not killed us all, this is only very mild evidence in favor of my view vs someone else who thinks we had a 90% chance of survival.

Even then, Ted Sanders only says ‘in the next decade’ and he thinks the world is massively underinvesting in alignment research. Reads like an (overconfident) recursive self-improvement skeptic.