Demis Hassabis on the New Coming Age

Google CEO Demis Hassabis offered us a first rate second rate essay, A Framework for Frontier AI and the Dawning of a New Age. I’ll go over that essay and various responses to it in Part 1.

Part 2 of this post then covers Alex Turner’s resignation, and his story about how he tried and failed to prevent Google from signing up to allow the Department of War to use its models for essentially whatever the government wants, including autonomous weapons.

Demis Hassabis sold DeepMind to Google on condition that something like this would not happen. Yet here it is, happening. A cautionary tale.

I will cover Kimi K3 tomorrow. I am hoping to know more by then. Please do share any reactions or info about it in the comments here.

The Core Statement and Request

He saying we are standing in the foothills of the singularity.

His ask is a Frontier AI Standards Body within the US Government, similar to FINRA, that would govern ‘frontier labs,’ defined as any company that produces a frontier model based on various technical benchmarks. Evaluations would be updated regularly, and vulnerabilities would be addressed, both before and after release.

He is excellent about stating that this is big, really big, no bigger than that, it be big.

Demis Hassabis: I’ve spent my whole life working on AGI because I’ve always had a deep conviction that, if built and deployed responsibly, it would prove to be one of the most beneficial and transformative technologies ever invented. AGI cannot be compared to standard technological breakthroughs, not even ones as consequential as the internet or mobile - it is much more akin to the discovery of electricity or fire. If you stop to think about it, we’ve essentially found a way to make sand think. It’s miraculous.

The magnitude of this technology’s impact will be unprecedented, perhaps 10x of the Industrial Revolution at 10x the speed. It will help us solve some of the biggest problems society faces from accelerating drug discovery to developing new clean energy sources to creating novel advanced materials. We could even reach a point where resources are no longer the limiting factor for human progress, leading to an amazing new era of abundance.

Things Left Unsaid

There is definitely a ‘don’t say the thing’ aspect of this, where he won’t name what the downside risks actually are. When Demis says ‘experts disagree’ he is rather avoidant about the way in which they disagree here.

Nate Soares (MIRI): I’m glad Demis acknowledges that this is a “pivotal moment in human history” during an “extremely intense” race. I’m disappointed that his proposed solution is a “standards body” to evaluate whether models are dangerous, with no plan for what to do once they are.

I’m glad he acknowledges that “experts disagree.” I’m annoyed that he glosses past how the disagreement is about whether there’s a ~5% or ≥50% chance of total catastrophe. We’ve gotta do better.

Aaron Scher: Glad to see AI CEOs speaking publicly about their views on AGI. I think Demis is wrong about his policy prescription: it’s far too little too late. When he says the experts disagree, he means that some think 5% this tech kills literally everybody, some at 40%, some at 90%.

Clearly this is strategic, but if you don’t already know, or are looking to not realize, it is very easy to come away thinking that Demis does mean the effect on jobs, even though when he says ‘safely’ he very much does not (primarily) mean that.

The Proposal

Demis Hassabis: … On the horizon, we will need robust safeguards to maintain control of increasingly agentic, recursively self-improving systems - and tackle unknown issues that will only become clearer over time.

… I’ve always believed in the power of human ingenuity and creativity to solve any problem. I’m confident that mitigating the technical risks related to AI is a challenge we can collectively address, but only if we give ourselves the time and space to get this next crucial step right. Currently, as a field and as a wider society, we aren’t doing that.

He makes clear part of this is about giving us options, including for a slowdown.

The strength of this approach is it would be technically focused, while at the same time supporting innovation and incentivising responsible behaviour. It is designed to keep up with the field’s acceleration and adapt to the biggest risks as they are identified, and could be ratcheted up if the seriousness of the situation demands, including coordinating a slowdown in development among the Frontier Labs if deemed necessary.

Demis keeps it short, not offering many details. To the extent that he has laid out a proposal, it seems to be a good one. It is definitely an improvement on the margin.

Jack Clark (Anthropic): At this point, everyone at the frontier of AI agrees that third-parties should test out AI systems and use these to develop standards to feed into policy - excellent to see @demishassabis laying out a framework to do this!

Samuel Hammond: It is striking to see leadership at Google, Anthropic, OpenAI and Microsoft all fairly independently sounding warning alarms about an imminent technological acceleration.

Thus I file this post and its ask, as high praise, under ‘the least you could do.’

A Good Start But Insufficient

I agree with Peter Wildeford that while better than nothing FINRA is not a great model here, with heightened risk of regulatory capture, and not a substitute for full government action. You need an SEC to your FINRA. That doesn’t mean don’t make the FINRA. It does mean you still need the SEC.

Would such a (at least partly) voluntary regime, only for models intended for release, and without a related binding intentional agreement, be sufficient to solve the problem? No, again it’s just way better than doing nothing, as Peter Wildeford and many others noted.

You do not need to believe, as Aaron Scher and Connor Leahy do below, that only a full halt would be sufficient here, to know we have a long way to go. Demis’s statements here, if you know what they actually mean, imply a level of danger and urgency that is not reflected in the proposal.

Eli Tyre: > Initially, Frontier Labs would voluntarily share models with the Standards Body for review up to 30 days before release.

Is this proposal only intended to address risks from models that companies plan to release? If a company develops a frontier model and never releases it, only deploying it internally to develop even more powerful AI capabilities, are they thereby exempt from this oversight scheme?

Connor Leahy: While @demishassabis is right that we need urgent action to address risks as we approach AGI (and superintelligence, I’d add), the correct response to the threats is not a ‘self-regulatory organization’.

We need to prohibit superintelligence, not give industry regulatory power.

Aaron Scher: … The extinction threat, the “only a few short years”, the “10x the Industrial Revolution”—these aren’t indicators that point to “let’s evaluate models to understand their capabilities and have voluntary safety standards”. We need to back off, we need to halt the creation of ASI.

Point 2: I agree with the attached quote that we need more time. But I think Demis’s optimism is a vibe, not a trustworthy basis for predictions. Rob Miles says it best in this video, if an asteroid we’re headed earth’s way 200 years ago, we’d just die 🤷

Point 3: As others have pointed out, it’s not clear that this proposal would reduce risks from internal deployment (it seems to focus on public deployment and pre-deployment testing), but internal deployment is where much of the risk is.

Point 4: I don’t think the proposed body could actually enact, verify, and enforce a slowdown; there’s ambiguity about what’s voluntary. Again, I think we need a long-term international treaty and to actually back off, not just to slow down a little.

Skeptics Of Future AI Capabilities

Timothy Lee dissents on the theory that we have already discovered more than 10% of available technological value from superior intelligence and information by 1945, as weighted by living standards. I don’t think that is the only way to measure impact. If it is, then it depends on if you mean at a given point in time or over a lifetime, and how ‘still human’ you are restricting the experience to being.

In terms of moment-to-moment lived experience as a basically human entity, measured the way we would for the quality per year in a QALY measurement, if you assume a mostly fixed lifespan and path of aging, I think it is likely we are more than 10% of the way to optimal. However, I would take a cure for aging and disease, plus the internet (and computers and phones) which people underestimate bigly, plus not having to work a job like those in 1945, over everything that came before, by a very wide margin, let alone full transhumanist experiences. Something about heaven and Earth and your philosophy, even if things stay relatively ‘normal’ afterwards.

Whereas if we’re just looking at pure GDP numbers? That seems super doable.

Noah Smith: The Second Industrial Revolution lasted roughly 100 years -- 1870 to 1970. In that time, living standards in the U.S. quintupled.

10x the Industrial Revolution in 1/10th the time would be a 50x increase in living standards in 10 years -- a 48% annual GDP growth rate.

The stupid dissent, espoused here by Coinbase CEO Brian Armstrong, is ‘oh AI is like software so we don’t even need an SRO, existing laws and incentives are fine.’

DeepMind On Bioresilience

DeepMind also released this three page statement on their approach to bioresilience, which can serve as an example test of their ability to self-govern.

The program has three pillars:
● Prevent threat actors from misusing our models.
● Detect new outbreaks quickly.
● Respond decisively to these outbreaks.

They are using a variety of methods to prevent misuse via defense-in-depth, of which they are (at least somewhat appropriately) vague.

At least one of these four must be true:

1. Their methods don’t work.

2. Their methods only work because Gemini is not good enough.

3. Anthropic is going nuts with its classifiers for no reason.

4. Anthropic’s countermeasures are very incompetent in an expensive way.

The second pillar, detecting outbreaks, is something the world should be doing anyway, but it mostly isn’t. DeepMind is looking for ways to help, but not on the level of ‘we are taking this responsibility on ourselves.’ That’s not an entirely reasonable thing to ask even of Google, but I’d look into it.

The third pillar is again important, a place where the world is not prepared, and not exactly Google’s department. They are trying to help somewhat via use of AI, but most of what must be done at this capability level is ordinary prevention and preparedness work that our government has decided not to do.

Thus, Google suggests that policymakers get off their collective asses and do the second and third things via various new laws and actions. Without looking into their proposed details I am confident we should at least do these things.

They also call for a federal framework to judge risk-benefit tradeoffs in bio for frontier models, which would presumably be a complement to Demis’s proposal. Good.

Part 2: DeepMind Folds To The Department of War

After heroic efforts to stop Google from signing an ‘all lawful use’ deal with the Department of War, Alex Turner resigns from Google DeepMind because they broke their promise and sold AI to the military with no restrictions against killer robots or mass spying. Lots of people did not raise their voice and justified it because they said it would not happen. Then it happened. So far only Alex Turner is quitting in response.

One thing I want to be clear on here is that, even though Jeff Dean failed to use his full leverage on this issue, Jeff Dean went farther than most anyone else. I don’t want anyone to focus their blame on Jeff Dean for ultimately not opposing this more strongly, as opposed to all those who did nothing at all.

DeepMind Leadership Failed Us

I do think Demis Hassabis is blameworthy here. He continues to insist that DeepMind’s principles are unchanged and unviolated. This is not the case.

If Hassabis was overruled by Google CEO Sundar Pichai or otherwise had no choice, perhaps due to other leverage brought against Google, that happens and Demis Hassabis is not a king. I would have accepted that, although it would have showed that his strategy of steering Google was failing. Even then, it does not excuse pretending that nothing happened.

Andreas Kirsch speaks plainly and offers a similar read. Andreas works at DeepMind, and has made clear that all the safeguards promised in 2014 have now been erased. More than 600 colleagues signed an open letter asking Google not to sign this contract, and then Google went ahead and still signed the contract. As a lawyer pointed out and as should be clear to anyone who follows the issue, Google’s contract terms here are completely ineffective and the government can do whatever it wants.

Andreas Kirsch: None of this is new. DeepMind’s governance record:

2014: independent ethics board — reportedly a condition of the sale
2015: one informal meeting; then effectively abandoned
2018: AI Principles exclude weapons & surveillance
2025: exclusions dropped

2026: Pentagon contract

Demis Hassabis then made a different bet: “Take the energy that was going into the trustless negotiation and put it into creating real trust.”

And: “Safety isn’t about governance structures.”

Trust instead of governance. The Pentagon contract is the litmus test of that bet.

You can read the Twitter thread version, or the 25-page full version of what happened. The whole post is telling, but long post is long. I will quote from the thread.

Alex Turner: What hurt most was watching AI ethics leaders do nothing. When the Pentagon threatened Anthropic, I was at a conference held by the International Association for Safe and Ethical AI. I lobbied IASEAI leaders to act.

What hurt most was watching AI ethics leaders do nothing.

When the Pentagon threatened Anthropic, I was at a conference held by the International Association for Safe and Ethical AI. I lobbied IASEAI leaders to act.

On stage, IASEAI leadership announced a member vote on making a statement supporting Anthropic. ~Unanimous show of hands supported it. But the vote vanished. IASEAI went silent, ignoring two months of opportunities to sway Google.

Inside Google: I organized a petition to Chief Scientist Jeff Dean asking him to fight. Over 250 GDM employees signed. I also asked Jeff to sign an amicus brief backing Anthropic. He did, which was awesome.

… I proposed that Jeff head the potential Review Body, but he didn’t champion the proposal. In the end, I don’t think he used his leverage. I think he could have stopped the deal but didn’t. As a last attempt, I directly messaged Google DeepMind’s CEO, Demis Hassabis. He told me to send my proposal to two senior policy staff. They let the proposal wilt unattended until Google signed the deal.

In 2018, Demis, Jeff, & DeepMind itself pledged to neither “participate in nor support the development [..] or use of lethal autonomous weapons.”

Google signed, yet these pledgers remain. I see three honest paths: explain, renounce the pledge, or quit. Silence isn’t one of them.

Here’s a path Demis has chosen: deny anything has changed. He claims “nothing’s changed about our principles”, yet he co-authored the post removing the weapons prohibition from Google’s AI Principles. Both cannot be true.

Many hope that when things get bad enough, someone powerful will say “no.” I tested that for months. Anthropic defended its red lines, but most did not. Pledges of conscience often vaporize on contact with power.

More specifically:

The way I saw it was: I don’t need to get Jeff to agree to quit over this. In a sense, he’d already agreed to quit over it. He had signed a pledge in 2018:

Lethal Autonomous Weapons Pledge

Signed by 5,218 people

The decision to take a human life should never be delegated to a machine. There is a moral component to this position, that we should not allow machines to make life-taking decisions for which others⁠—or nobody⁠—will be culpable. There is also a powerful pragmatic argument: lethal autonomous weapons, selecting and engaging targets without human intervention, would be dangerously destabilizing for every country and individual. […]

By removing the risk, attributability, and difficulty of taking human lives, lethal autonomous weapons could become powerful instruments of violence and oppression, especially when linked to surveillance and data systems. […]

We, the undersigned, call upon governments and government leaders to create a future with strong international norms, regulations and laws against lethal autonomous weapons. These currently being absent, we opt to hold ourselves to a high standard: we will neither participate in nor support the development, manufacture, trade, or use of lethal autonomous weapons.’

Signers include: Google DeepMind (the organization), Demis Hassabis, Shane Legg (cofounder, now Chief AGI Scientist), Raia Hadsell (VP of Research), Jay Yagnik (VP and Engineering Fellow at Google, leading large parts of Google AI) and Jeff Dean.

Alas, no, that did not translate into willingness to quit.

This Was a Failure We Must Learn From

Many at Google DeepMind made various efforts to avoid this outcome. They failed.

The mechanisms of self-government of DeepMind failed. Its employees spoke up, some prominently, but it was not enough, and those with the highest leverage did not use it. The promises made when they sold to Google have been broken.

In the long run, this will impact DeepMind’s ability to recruit, motivate and retain the best talent. Google and DeepMind are losing to OpenAI and Anthropic, despite Google’s numerous large advantages. Inability to recruit and retain and properly use the best talent is a lot of why. They have some great talent in leadership, but those leaders cannot do it alone.

Demis Hassabis is to be applauded for his statement, but calls on others to do things, without any consequences following, risks becoming only cheap talk.

My hope is that it can become more than cheap talk, and move the conversation forward. It won’t be easy, but we have an opportunity. Let’s make the most of it.