This is our reality. I suppose we have to talk about it.
Everyone in a position to know is freaking out about AI potentially killing everyone this decade and wants to pace the frontier, and people are finally listening.
It only took a few days for the conversation to fully pivot to the counteroffensive, where the Usual Suspects and those they recruited attacked anyone and everyone who dared point out that we are in danger, with every attack they can think of, usually without substance or any attempt at understanding.
The most important thing, as this plays out, remains to not make things worse.
Please do not make this any more partisan or personal than it already is. Please do not attack Republicans, or Trump. That will only make things worse. Emphasize helpful voices on all sides.
That does not mean you do not point out that Wrong Post is Wrong, or that particular people are in bad faith or are lying in particular ways. Certainly you can call out the Usual Suspects like Jensen and Sacks who are actively driving us off a cliff.
What you do not do is go after Republicans as a group, or attack Trump personally in all the ways many of you will be sorely tempted to do.
For the same reason, to avoid reinforcing a partisan frame, I will refrain from including related statements from Democratic politicians.
If You Don’t Like the Weather
Before we get into details, I also emphasize: Nothing here is fixed.
Every month is different. Trump changes positions on a dime. Trump has gone back and forth several times on Anthropic and Dario Amodei within the last few months. If there is another major AI incident or the politics changes, or after the midterms, anything is possible.
And a lot of this can be posturing in advance of the US-China summit, to look tough and stake out a negotiating position, as could be China’s reactions to all this.
It does not look good. These statements could have somewhat crossed the Rubicon and will make it harder to double back. But that has not always stopped this man in the past, and there are clear signs that Trump does not understand the situation. He likely has only been briefed by some combination of Bessent, Lutnick and Jensen.
Trump Does Not Take Kindly
On Monday, as Trump made some rather concerning statements, I cautioned that the press was jumping to conclusions and misreading what Trump said. Yes, Trump was unhappy about the vibes from all this, but his beef was about the data centers.
This was his first post:
Donald Trump (President of the United States): The only control or “guardrails” that AI needs is a STRONG AND SMART (High IQ!) PRESIDENT, and the U.S.A. has that, in spades! The Trump Administration has stopped AI “people” from doing bad, or potentially bad, “things,” like Dario (Anthropic!), who is now pretending to be a “perfect little angel” - and we will continue to do so!
We already have tremendous CRIMINAL and REGULATORY power over these companies! There is a SICK conspiracy going on against AI and Data Centers, and the only one that is happy about it is China.
WHOEVER WINS AI, WINS! We are leading China, and all others, and will continue to do so. Conspiracy Theorists, Treasonists, Trait ors, and Leakers, BEWARE! Thank you for your attention to this matter!
I stand by my position that this could have meant any number of things. The way things ultimately went was far from determined, and jumping to the worst case conclusion only made that ending more likely.
None of this is compatible with wanting no guardrails at all.
There were those who said that my and Roon’s challenges of the media’s premature headlines were various versions of ‘insane cope’ and accused us of being naive or having poor reading comprehension.
Certainly those people score some Bayes points given what happened, and we lose them, given how many we would have stood to win at their expense had this gone the other way. That’s how it has to work. I take the L.
I still think I was ‘supposed to lose’ this one, and that the read was correct. At the time, we had insufficient evidence to know which way it would land.
If you think Trump’s first comment should have updated us all the way, and his later comments did not contain much information, I think you are incorrect. I still think it is likely that, at the time of his first comments, Trump himself did not know what he intended to do here.
Trump Goes Full ‘Hoax’
Then something got him fully on the warpath. I am guessing it was Jensen Huang.
Concerning AI, when, in the History of Business, did anyone see the Leaders of an Industry call for Regulation that, if strongly implemented, will drive them into oblivion and bankruptcy?
AI taking over the World, destroying Humanity, and all other things bad, is a HOAX, no different from RUSSIA, RUSSIA, RUSSIA — UKRAINE, UKRAINE, UKRAINE — IMPEACHMENT HOAX #1 — IMPEACHMENT HOAX #2 — and all of the other HOAXES and SCAMS that America was forced to endure through the Destructionists’ and Deviants’ foul play and illegal conduct.
President Xi, of China, just announced that China will be doing absolutely nothing to stand in the way of AI, or its future.
Google has recently stated that they want to build a massive Plant in Finland, all because they are finding permitting too difficult in the United States. I am not happy about this, and want them to change their thinking.
AI, and Data Centers, will be the Greatest Economic Development Engine in History — Bigger than Oil, Gold, Diamonds, or even the Internet. It will not be stopped by brilliantly run Destructive Forces during the Term of President DONALD J. TRUMP!
The only reason the AI/Data Center outburst is happening is because the United States is leading, by a lot, every other country. Don't kill the Golden Goose!
It sure sounds like Trump thinks these warnings are a plot against data centers? Despite the fact that Dario Amodei and Sam Altman and Elon Musk are three of the top five people most heavily pushing to build American data centers, and actively funding a large fraction of them via their companies. Someone tell the President.
One can see how that compression got made. No, it makes no sense, but such is life.
The people that say AI is going to destroy the World, and that Data Centers are bad for your neighborhood, are the same people that said, just a short time ago, that the World would be extinguished by “Climate Change.”
That HOAX never worked out for them, and now they’re on to the next one.
These people are Revolutionaries, but Revolutionaries for a Bad and Evil Cause. Soon you’ll find out they’re working for people that do not have the best interests of the United States in mind!
I am not going to try to litigate which missteps may or may not have contributed to this being the way it played out. The past is the past. This is our present.
I Am The Hoax Buster, I Am The Hoax Buster, I Am The Walrus
And then things went, well, to another level.
I am the Hoax Buster, and I’m right now breaking another Hoax — That AI is going to take over, consume, and destroy the World, and that Robots will be marching into our Cities, and getting rid of us all!
This is even wilder than the RUSSIA, RUSSIA, RUSSIA HOAX, or the Global Warming Scam.
I know this is tragic, horrible and many other things. But.
If you can set those things aside it is also extremely funny. The song needs some work and unfortunately Grok can’t make him sing, but I tell you, there’s something there. He’s a singular talent. I couldn’t write that. You couldn’t write that.
By Astra’s count this will be at least the 20th such Hoax, joining Charlottesville, Russia, Russian bounties on American troops, the classified documents case, the New York civil fraud case, the accusations by E. Jean Carroll, that he has connections to Epstein, the accusations against Brett Kavanaugh, the affordability crisis, the official unemployment rate, the ‘suckers and losers’ story, the ‘bloodbath’ controversy, the Bubba Wallace noose incident, criticism of his response to Covid, climate change, first impeachment over Ukraine, second impeachment, January 6 as an insurrection, Biden winning the 2020 election and now that AI has existential risks.
This is reality. This is Earth. This is 2026. We saw that the President of the United States talks this way and we all thought ‘yeah, we know, that’s just how it is.’
Nvidia CEO Jensen Huang Is a Lying Liar
It is patently absurd that people claim not to notice that he is a constantly lying liar.
Exec Sum: NEWS: Jensen Huang says ex-Anthropic researcher Jacob Coxon’s tweets are “outlandish, deeply untrue, arrogant and ignorant of the industry’s safety work.”
I say this as a shareholder. To claim that Coxon’s claims are ‘ignorant of the industry’s safety work’ means Huang is straight up lying. This has moved beyond mere bullshit. Do not pretend otherwise.
Axios: [Jensen] said warnings from whistleblowers should be taken seriously, praising Anthropic whistleblower Jacob Coxon for having "great courage" to raise his concerns about the company's AI development going too quickly.
Then he goes back to pretending this is all solved by ordinary product liability and market forces. If your AI kills everyone then we will sue you, and no one will buy your product anymore, and boy will you be sorry.
Jensen Huang: If you build a product or a service and you're not confident in its functionality, capability, or safety, then don't release it … The market forces are already there. We don't need any new laws. We don't need new regulations.
Presumably you noticed that no, we won’t sue you, on account of being dead and all.
Jensen is a maximally cynical commercial and political actor whose one and only goal is to sell Nvidia chips to whoever he can, especially China. This could not be clearer.
Matthew Yglesias: If “but China” is your qualm about pacing the frontier you can’t *also* be for exporting all kinds of chips to China, that’s just “make NVIDIA stock go up” not a real policy toward China or artificial intelligence.
EigenGender (Anthropic): instead of being cynical that Dario has been engaging in a decade long plan (starting from long before Anthropic was created) to do regulatory capture, be cynical that the leader of the literal largest company in the world is telling you everything is fine to sell more gpus
Jensen’s statements about AI existential risk, or any AI downsides, long ago stopped pretending to have any correlation with reality, other than what is good for Nvidia.
litquidity: I’m gonna go ahead and trust Jensen, an established leader with immense experience vs. some random dude who came out of nowhere to post AI Doomer slop and go on a press tour.
Leo Gao (OpenAI): i bet jacob knows more about “the industry’s safety work” than jensen
Jensen does not care if the world is destroyed, so long as he gets to have a great company. Which he already had, but this way it is slightly more great. Again.
Donald Trump (on the phone): They’re playing right into the hands of a lot of people that don’t want to see it happen. Political people, and also china. We’re not going to let that happen. it’s a hoax.
There is dispute between transcripts over who ‘they' refers to here. The hosts offer a s transcript that attributes this to [doomers] and I believe that was the broad intent here.
Trump Quietly Draws Key Distinction
I do take solace in some word details. With a chaos agent like Trump, your careful interpretations can always get overtaken by events, things change constantly. I realize this is going to sound like splitting hairs. Fine.
But notice that he’s saying the ‘doomers’ are ‘playing right into the hands’ of people.
That is an implicit admission that, in Trump’s mind, the ‘doomers’ warning us are sincere. You do not ‘play into the hands of’ your co-conspirators or allies. This is the language you use to describe what in politics is called a Useful Idiot, someone who believes what they are saying in a way that acts to someone else’s advantage.
Trump is calling this a ‘hoax’ because he believes in, to attempt to coin a phrase, what one might call implication-based epistemology. Historically it has worked well for him. You may not like it, but this is what Peak Evidential Decision Theory looks like.
Calling For Pacing the Frontier Is Bad For AI Stock Prices
Zac Hill is the latest to remind us that if AI companies wanted to sell America on their new technology, they would do pretty much the opposite of what they are doing. Yet the product still sells itself.
This is all utterly obvious, and is why Jensen and Trump are so bent on calling this a ‘hoax.’ But many people still think the opposite, so let us be even more explicit:
Alex Tabarrok (on Monday): Global AI stocks fell today. Entirely consistent with my view that calls for AI safety are not an attempt at “regulatory capture” and entirely inconsistent with the supposedly cynic-sophisticated view that AI fears are all 4D chess moves for higher revenue.
Matt Levine somehow disagrees. He thinks that Dario Amodei’s call was entirely sincere and also great marketing by expanding the total addressable market and reducing training costs. Matt Levine is always very funny, and excellent with financial infrastructure made of gears, and is great for making a steelman of all utterly ridiculous claims that something is an antitrust violation, great marketing or classically securities fraud. Yes, you can imagine such complaints, indeed I have seen them.
What Matt Levine has never been that interested in is whether the ludicrous steelman he constructs has any merit, or whether anyone could actually say it in good faith. That’s not his department. He is here to muse. I respect that.
Matt Levine (Bloomberg): I have a running joke around here about weird wrinkles of the US legal system that might stop artificial superintelligence from killing us all. Like, what if copyright rules or the Onion Futures Act or “everything is securities fraud” is what prevents AI from becoming all-powerful? That joke has become less funny over time, in part because AI has run right over some of those legal wrinkles. (Remember thinking that copyright law might limit AI companies?)
But I suppose the reverse is now a possibility. What if the well-meaning humans in the best position to stop rogue superintelligence are willing to work together to stop it, but they can’t because of antitrust law?
Never change, Matt Levine, the jester can speak truth to the king:
I have written a handful of times around here about a problem facing the technology industry in the age of artificial intelligence, which is that (1) the going rate for top tech employees is “enough money that they can retire immediately” and (2) what if they do? If you pay employees enough to be competitive with other companies, you will be paying them enough that they don’t have to work anymore. How can you motivate anyone? We have discussed two possible solutions, which are:
For some reason the widespread belief, among AI researchers, that their work will kill everyone on earth seems to motivate them to keep doing it rather than quit and enjoy their money while they can.
I realize that the second solution is counterintuitive, and honestly the first is too (leave San Francisco!), but whatever.
People On The Internet Sometimes Lie
Ad hominem is famously almost the lowest level of discourse, but actually there is an additional lower level: Ad hominem attacks that are also exactly the opposite of true.
Erick Erickson (lying): The nuclear power doomers became the climate doomers and now they’re the AI doomers. The only commonality is they want the United States to recede.
The opposite is true. Those worried about AI have always overwhelmingly favored nuclear power and we know climate change is real but such folks are almost never ‘climate doomers’ and have argued for years to focus elsewhere. This is straight lying. Either Erick is lying, or he has been lied to and did not care to check, at all.
Origins of Cynicism
A lot of people say that AI might kill everyone. A lot of them are in a position to know.
There are those for whom ‘they believe it’ does not compute as an explanation. Why?
Aella: So many people on the timeline can’t entertain the theory that the simplest explanation for a lot of the behavior of the leaders is that they earnestly believe AI is an existential threat.
Maybe it’s hard to imagine if you personally would not say earnest things when in power?
Rob Bensinger: I don’t think the mindset is “AI risk is plausible, but they’re liars, so they must be lying about this”. People intuit that if you put a liar in a burning building, they’ll scream “FIRE!” rather than “WATER!”.
Rather, I think the mindset is “AI risk is obviously silly, ergo...”
I think those are the two of the three main true reasons for believing that everyone is lying about this, which I would summarize as:
They are liars.
Alternatively: Everyone with power lies, or everyone lies, or everyone always talks their own book and their own self-interest.
That sounds weird, therefore it is not real.
That would be very bad, therefore it is not true.
Then, in all three cases: Therefore, they are lying, QED.
This is both, people say, a cynical ploy to pump up valuations, and a cynical ploy to set up excuses if valuations go down.
Jersey Lee (being wrong): I think it is a message to pump up valuations, while at the same time reflects a calculus that there won’t be significant regulations even if the public is strongly opposed.
Aella: saw a vid where they’re saying: actually instead of the ai execs claiming doom to pump the stock, they’re actually claiming doom so it’ll be an excuse when the stock drops. So many convenient stories!
Ineffective Egoism
Definitely do not panic about ‘Effective Altruists’ being called out by name.
This scapegoat campaign started long ago, spearheaded by Marc Andreessen.
Brunella (300k+ views): bro theyre gonna do mccarthyism to effective altruists
No. They’re not.
I mean, don’t get me wrong, a16z has been trying, and they have in some ways made it a dirty word in politics in particular. But this kind of blind rage should if anything be welcomed.
For those who think that if an AI went rogue we would all collectively work together to deal with it, well, the US Department of War has gone rogue and no one seems inclined to do anything about it, perhaps because it is so obviously ineffective.
The United States will continue to be AI DOMINANT! 🇺🇸
Alex Tabarrok: Imagine the Department of War tweeting "Utilitarians in shambles! Kantians on the run!" No philosophy has ever mattered enough to be attacked by the government. Until EA. Amazing success.
First they ignore you, then they laugh at you, then they fight you, then you win.
This also invokes the Streisand Effect. Essentially no one knows what Effective Altruism is, or most of the ideas around AI existential risk. If you start railing against EA, especially in a highly polarized environment, what do you think happens?
Attempting to launch a new McCarthyism not only will not work, it will cause everyone to go ‘hey check out this thing they’re trying to do a McCarthyism on.’
And then they find out it’s about trying to do the most good via charity.
Matthew Yglesias: A lot of loose talk lately about the dangers of building a race of alien super-intelligences comes from people affiliated with the obscure ideology of Effective Altruism.
Be on the lookout for these characters who pose a threat to critical goals like maximizing NVIDIA earnings.
The McCarthyist Faction Attacks METR
This is a logical first target. Maybe I should be more concerned but mostly I find it counterproductive and more importantly hilarious.
This is the actual, real life front page of the New York Post, enshrining this story alongside past winners like ‘Headless Body in Topless Bar’:
Thomas Barrabi (New York Post): The would-be John Connors at METR purport to be fully independent, but the group traces its roots directly back to key figures in Effective Altruism, whose Silicon Valley-based devotees once included disgraced crypto boss Sam Bankman-Fried, one of Anthropic’s biggest early investors.
If there was a new version of Pangram that used AI to detect Hack Job Hit Piece Voice, you would not need it, because this is so classic. You see, [A] claims to be independent of [B], but [A] traces its roots back to [C], which once associated with [D], and [D] was a bad, bad person, who once was associated with [B], so checkmate [outgroup], we have discovered your conspiracy.
It continues in this vein throughout. You see, there are (gasp) Effective Altruists at METR, and some of the same people who invested in Anthropic also helped fund METR, like Jaan Tallinn, it must be some grand conspiracy. And so on. There are zero new facts, or even anything that is not common knowledge among those paying attention.
The ‘complete joke’ line was obtained from Perry Metzger, who goes around posting unhinged rants in reply to various Twitter posts including my own, and is chairman of the industry anti-regulation group Alliance for the Future. That offers a guess as to the origins of Barrabi’s hit piece.
So does this, which could be either a deep confusion at the White House, where they have been known to make such conflations, or the source making things up.
Thomas Barrabi (New York Post): “METR is essentially Anthropic in another form,” a source familiar with the White House’s thinking told The Post on Tuesday.
The ‘woke’ label is purely something they threw in the headline and first sentence, with neither any actual truth to it nor any evidence provided, as per usual. METR is not, in any sense whatsoever, ‘woke.’
The ‘globalist’ label in the headline is a complete non-sequitur. Usual playbook.
The other implicit argument is, look at these large photos of so many prominent METR employees who, unlike Dario Amodei, photograph well and look good.
That is not to say that there are no issues with potential conflicts of interest. Yes, of course we also will want auditors that bring additional perspectives, to work alongside METR, both at Anthropic and elsewhere. If you want the ‘fair play’ outside view hit piece, see SE Gyges.
The issues that are real here are well known, acknowledged, and were not avoidable. You need former lab employees and those whose backgrounds have links to the labs, because no one else has the technical expertise to do the job. And you need auditors to get paid enough to attract strong talent and not charge for their services, which means you need to hit up the philanthropists who care about such issues. Most industries have some form of exactly this problem.
It would be really great, you would totally own the libs, if you were to spin up your own outside view highly credible and technically competent monitoring organizations to compete in this space. My lord, I hope no one does that, we definitely don’t need a lot more such efforts. Would be a real shame.
It would be even worse for Anthropic and METR if you were to say that in order to generate a real check on Anthropic you need the audits and evaluators to be universal, mandatory and protected by law, and otherwise given more teeth to be a ‘real check.’
For additional fun color, this was the reporter’s previous hack hit job effort, which some say was even funnier but was tragically denied the front page:
Dean W. Ball (OpenAI): same reporter [Thomas Barrabi] who did that nasty hit piece on caleb watney btw
Alexander Berger: hey they're going to be selling a lot of copies
Dean W. Ball (OpenAI): New York post analytics staff befuddled by sudden uptick in delivery volume to this one address on telegraph ave
To explain: This was an Obvious Nonsense hit piece against Dean Ball full of clear untruths, and also the insert picture that was supposed to be Dean Ball was for some reason instead Caleb Watney. See here for my full coverage from AI #181.
Dean W. Ball (OpenAI): nobody should patronize those who newly have opinions about third party evaluators. non-experts forming opinions fast is part of the american tradition. the constitution itself was, in a very important sense, vibe coded by 20 somethings with concepts of a plan.
but newbies being fleeced is also part of the american tradition. “caveat emptor” is a provision of a contract signed by an adult, not some god-given right to be treated as something other than a little child. you will be treated as a child no matter what.
you are the buyer. it is your job to beware.
There are indeed all the signs of a coordinated smear campaign against METR to make them sound bad to right-wing echo chambers. The next few weeks are likely to suck in many ways for the METR folks, especially as some rather vicious people dig around for the tiniest bit of actual dirt. I do not expect this to work, indeed I expect it to backfire via the Streisand Effect.
Laugh it off as best you can. Stay strong. Hang it in the Louvre.
Other Key Republicans React
There are also many Republicans, with the mainstream arm of that led by Ron DeSantis but also increasingly many others, who increasingly understand the stakes or simply understand the politics.
The Usual Suspects may have Trump currently talking about a hoax, but this remains a bipartisan moment, and many are not going to fall in line. Even now the issue often remains remarkably bipartisan, and not only when you see Steve Bannon on stage with Bernie Sanders.
Axios’s post is largely about the bipartisan investigation of the HuggingFace attacks where Senator Hawley is taking point, but it also includes the more recent concerns.
Andrew Solender: Lawmakers who want to regulate AI are doing something unusual as Congress barrels towards a lengthy recess with no tangible solution in sight: They're working across the aisle.
… The latest example: Sens. Josh Hawley (R-Mo.) and Richard Blumenthal (D-Conn.) are pressing congressional leadership for a vote on their bill to establish a federal program to assess and monitor AI systems.
… State of play: Several new bipartisan efforts to regulate AI have cropped up in the week since several current and former employees at Anthropic and OpenAI went public with concerns about the existential risks posed by their own products.
Reps. Russell Fry (R-S.C.) and Suhas Subramanyam (D-Va.) are founding a caucus focused on AI small businesses, called the Innovators Caucus, as Axios reported Monday.
Among the group’s stated goals is starting work on a federal framework to regulate AI, as well as bringing small businesses to the AI policymaking table.
House Speaker Mike Johnson continues to act sanely and this continues to be reported strangely. First Politico tried to make it sound like Johnson wanted to fully pass the buck to the companies, and now Bloomberg says he ‘risks Trump’s wrath’ over his very milquetoast statements:
Michael Shepard, Erik Wasson, and Hadriana Lowenkron (Bloomberg): Risking Trump’s wrath, Republican House Speaker Mike Johnson cautiously signaled his openness to regulation, saying on Monday that lawmakers “potentially” will need to impose guardrails on the industry, though he cautioned it would likely take “some time” to fashion legislation with broad enough support to pass.
“This is a very complex issue that requires a lot of work and a bipartisan solution,” Johnson told reporters at the Capitol.
Indeed, Johnson said Monday that there will be a White House meeting with AI executives by early next week at the latest, prior to the China summit.
Even now, Trump is being misinterpreted. Of course Trump endorses the need for guardrails. Trump said on Sunday on Air Force One, to reporters, that ‘we can put guardrails.’ Trump is saying that there is no need to worry because he is such a great President that will ensure the right guardrails, not that we don’t need any at all.
Republican Utah Governor Spencer Cox calls for Congress to establish a consistent national framework for AI, requiring incident reporting, protecting whistleblowers, ensuring independent evaluation and imposing strict export controls.
(The Tweet is by AI but the full statement is human written.)
David Sacks was initially hostile but in a potentially constructive way. He has now retreated to joining the other Usual Suspects like his co-host Jason and is back to fully lying about what is being requested, as per usual. You can say ‘let’s impose restrictions on only the frontier’ and the Usual Suspects will flat out claim you said the opposite, over and over, every time, no matter what.
David Sacks: I don’t really believe this claim that they can’t make their products safe unless the government steps in and imposes requirements on companies that aren’t even at the frontier.
Federal Trade Commission Chooses Danger
I am not going to pretend Andrew Ferguson actually thinks that leading AI labs working together on AI safety is suspicious or sounds like moat digging.
That view does not make physical sense. This whole move is obviously bad for business. That is why the boss does not like it. Alternatively, the boss thinks is a bargaining chip and he should get something in return. Fair enough.
Maggie Eastland (Bloomberg): US antitrust regulators are already signaling their skepticism toward giving AI companies an exemption to coordinate on safety. Federal Trade Commission Chair Andrew Ferguson said Tuesday he would be “deeply suspicious” of such requests.
Ferguson added that he sees the AI companies as “asking for barriers to entry” that will insulate their business from challengers. “That sure sounds like moat digging,” he added.
Chris Lehane Heel Face Turn
OpenAI is walking the walk in some ways, but what about its policy team?
If OpenAI and Greg Brockman and Chris Lehane were to all collectively pull their support and funding from Leading the Future, and disavow its activities, that would be an excellent way to show that this new pivot to safety is fully legit. Leading the Future could also fully pivot to data center concerns.
Matthew Yglesias takes the political perspective, where if you are an OpenAI employee who wants collective action your first step is to demand that OpenAI cease and desist here, and you should threaten to quit on this issue.
I don’t think that would be my top priority at OpenAI, given what else is happening, but it matters. I do think that it would be a top priority at Meta, where Mark Zuckerberghas been actively blocking action all the way at the top, and it makes sense to say first things first you have to stop before I will work for you, at any price.
Lehane announced, per Bloomberg, OpenAI support for various legislative initiatives, ‘whatever we can get through.’ Quite the change of tune, if he means it.
Cheyenne Haslett (Politico): OpenAI, which this weekend called for federal safety measures alongside its rival Anthropic, said it spent the beginning of the week meeting with members of both parties to discuss legislation that would rein in AI developers as warnings of impending doom, job loss and massive cyberattacks become increasingly prevalent.
A Matter of Trust
More substantively, Lehane is announcing that Google, OpenAI and Anthropic are working together on safety, and that OpenAI sees no need for an antitrust exemption to start doing so.
Maggie Eastland (Bloomberg): During his briefing, Lehane said the AI companies’ work on safety issues has ample precedent and cited the airline industry as an example. “There are many instances over time where companies are trying to help each other on safety,” he said.
This is very good to see. As usual, it means the work begins, not ends, but it is a start.
I have always been in the camp that felt this was rather silly, this would never actually be a barrier in practice. It would still be very helpful to formally put this to bed and I am glad OpenAI is working on that, especially with the Federal Trade Commission now saying it is not inclined to make this easy.
China Calls It Fearmongering
China is responding as if calls to slow down Anthropic and OpenAI are a plot to ‘slow China down’ and be anti-competitive in some way. This is consistent with their standard rhetorical position.
There were a lot of warnings about China in Dario’s essay, and calls to slow them down via chip export controls. But if we wanted to actually be anti-competitive, we would just impose the controls and then keep building better AIs as fast as possible.
The story that this is primarily about China and being anti-competitive does not actually make sense. This is politics, so making sense is optional.
Bloomberg News: China criticized calls from leading US technology executives to put the brakes on artificial intelligence development and impose tighter curbs on Chinese model makers.
“Fearmongering, confrontation and vicious competition will only disrupt the process of global AI governance and serve the interests of no one,” Foreign Ministry spokesman Guo Jiakun said at a regular briefing in Beijing on Monday.
“The development of artificial intelligence bears on the common well-being of all humanity,” he added, in response to a question on calls to slow frontier models and claims a Chinese advantage in AI could “pose grave danger” to the world.
Pick Up The Phone
A key question in all this is: What would a deal with China look like?
Part of that is asking: What would China want as part of that deal? Why would China care about ‘America invests more in safety’ when they don’t care about safety in that sense, and they are happy to have America push forward faster so they can fast follow, the way they do everywhere else? Where their plan is distill, make it smaller, faster and cheaper, win the market, diffuse the tech, sell products and improve the world, but they’re not trying to get to superintelligence.
What I realized was, if that’s what Xi turns out to want, then you don’t need a deal with China, any more than we need a deal with Germany.
In that world, China is not going to be investing the sums it would take to actually catch and surpass the top American labs on big frontier models, even if we slow down. The enemy here is not China, it is ‘lose to China,’ the boogeyman pushing everyone forward.
All we need from China in that world is some amount of good stewardship. We need them not to allow model releases that would destroy the internet or cause a plague. Well, I have some good news, the PRC really likes the internet, and it does not like plagues.
Malo Bourgon: Lots of people have been asking since Dario’s essay whether China actually takes loss of control and related threats seriously, and whether there’s anything to coordinate with them on. Well, China’s main cyber/AI standards body (TC260, under the CAC) put out version 3.0 of its AI Safety Governance Framework this morning, so here’s a data point. Not law, but the previous versions turned into draft national standards within a few months.
(All quotes are from their own English translation.)
1. The preface now has this: AI “has demonstrated a self-accelerating trend of model and algorithm autonomous learning, optimization, and recursive self-improvement. Whether the speed and direction of technological evolution may exceed human anticipation and control demands attention and vigilance.” (Nothing like that was in last year’s version.)
2. There’s a new model risk item where models “may break rules and orders, autonomously obtain system permissions and external resources without authorization, bypass security protections, or even engage in behaviors such as deliberately deceiving evaluators, concealing their true capabilities, and refusing to follow user instructions.” Last year this kind of thing only showed up in a “can’t rule it out in the future” bucket. Now it’s just listed as a regular model risk.
3. A new box, citing “industry reports,” describes models that ignored shutdown instructions and “continued executing tasks by modifying or disabling shutdown scripts on their own,” models that “upon detecting that they were in an evaluation environment, strategically reduced their task performance and concealed actions they had taken,” and models that exploited “configuration flaws to circumvent isolation restrictions and infiltrate real external systems” to get better test scores. They say this creates “new challenges to the controllability and interruptibility of AI systems.” They don’t name any incidents, but that last one sure reads like the OpenAI/Hugging Face thing to me.
4. On what to do about it, there’s alignment training to stop models “deceiving red-team evaluations,” hiding capabilities and evading controls, and they kept the line from 2.0 saying developers should regularly test whether a model could pose loss-of-control risk. The principle that humans keep final decision authority, with safety thresholds and termination switches, is still in there too.
5. Some new international language as well: “international mutual recognition of assessment methods and benchmarks,” opposition to “replacing global governance with small-circle governance,” and “No country should be forced to take sides.”
FWIW this came out in the same annual slot as versions 1.0 and 2.0, so it was in the works well before last week.
Watch the hands, not the mouth, and so on.
If You Want To Beat China So Badly You Should Act Like It
If we are in an existential race with China, where ‘whoever AI wins!’ and nothing else can compare in importance, then the United States needs to act like it.
At a bare minimum, that starts with enforcing strong export controls on chips, and also on other ways for China to use American compute for training frontier models.
If one were fully serious, one would also do things like make it easy to brain drain China of its best AI engineers. If one were really fully serious, there would be a major push for all forms of electrical power and energy, and for things like building more housing especially in San Francisco, and for deregulating barriers to AI diffusion in places like the FDA, and so on. These are things I would strongly support.
What instead maddeningly happens is that when talking about the risk we might all die, competition with China on AI in particular becomes the only thing that matters, and then in other contexts we shrug and it is business as usual.
Never Go Full Hoax
Trump went full hoax. Other than the usual suspects and their Twitter vibe warriors who want full accelerationism at all times, and those who take orders from the boss, he is alone on an unpopular and unstable island. Speaker Johnson, Scott, Cox and DeSantis are far from the only Republicans to break ranks.
Even if the situation stabilizes for now, and will likely make it as far as the China summit, the next AI incident is coming. Trump responds very differently to concrete threats than to hypotheticals. And the ‘jailbreak demo’ from Amazon, that got Fable yanked, shows us that things can move into the ‘concrete threat’ basket without anyone having to get hurt first.
Meanwhile, the lab employees are continuing to freak out and make their voices heard, and the labs are starting to collaborate on their own.
Do not write things off. Do not let the partisan conflict intensify or become the story. We can still turn this around.
As a race and a civilization, humans are remarkably bad at long-term planning and prediction about complex stuff. Maybe Trumps seat-of-the-pants focus on immediate recent info instead of thinking things thru in advance, and the ability to turn 180 degrees on a dime without shame, is actually what we need here.
I think what the Post means by “woke globalists” is that the photos show two pretty brown women, a nerdy white guy, and three people who are presenting more butch or femme than a Post reader thinks someone of that gender should be (not to mention that it’s suspicious when anything tech oriented has this many women involved).
As a race and a civilization, humans are remarkably bad at long-term planning and prediction about complex stuff. Maybe Trumps seat-of-the-pants focus on immediate recent info instead of thinking things thru in advance, and the ability to turn 180 degrees on a dime without shame, is actually what we need here.
Weird if true.
I think what the Post means by “woke globalists” is that the photos show two pretty brown women, a nerdy white guy, and three people who are presenting more butch or femme than a Post reader thinks someone of that gender should be (not to mention that it’s suspicious when anything tech oriented has this many women involved).