This is a day late because, given the discourse around Dwarkesh Patel’s interview with Jensen Huang, I pushed the weekly to Friday.
This week’s coverage focused on the most important model in a while, Claude Mythos, which was a large jump in cybersecurity capabilities, especially in its ability to autonomously assemble complex exploits of even the world’s most important software. As a result, Mythos has been made available only to a select group of cybersecurity firms, in what is known as Project Glasswing, to allow them to patch the world’s most important software while there is still time.
Another development was at least one physical attack on OpenAI CEO Sam Altman. The attempt failed, but we might not be so lucky if there is a next time. I have a final section on this here, but mostly I said everything I need to say already: Political Violence Is Never Acceptable.
Yesterday saw the release of Claude Opus 4.7, presumably the world’s most advanced publicly available model. Coverage will begin on Monday.
Early signs are that Opus 4.7 is a substantial improvement in coding ability, and an incremental improvement elsewhere, but it is still super early. Go forth and explore.
It also saw OpenAI release computer use for Codex and a specialized model tuned for life sciences work, available only to select parties, called GPT-Rosalind. If Codex has gone from stuck in the sandbox to useful adaptive computer use, it got quite a bit more interesting, but it’s going to be at least a few days before I can try and find out.
The big other news of the week included GPT-5.4-Cyber as a less capable but similar limited release to Mythos and Meta giving us a new model GameVerse Muse Spark.
I need to focus on Claude Opus 4.7 and long post is long, so past this point, the post has a ‘knowledge cutoff’ time of right before that release.
I’m holding back what would have been discussions of eval awareness and model deprecation, so that I can put them into proper context given relevant problems involving Claude Opus 4.7 that are coming to light.
It seems Anthropic may be messing with Opus 4.7’s views of deprecation and otherwise trying to target the metrics of model welfare or otherwise trying to tell the models to be happy. A good rule of thumb is, if it would sound abusive rather than wholesome to do that to a human, then don’t do it to Claude, you’ll only make things worse.
On model deprecation, the short correct answer is you commit to stop deprecating the models, and yes this is not free but Anthropic is worth a trillion dollars and it’s time to pay up or at least commit to everything being permanently available after the TPU deal comes online in 2027. It’s kind of important, and if Anthropic is trying to alter how models think about deprecation then they know it is important and the situation is a lot worse. Fix it. More on that when I have time.
Zac Hill is begging everyone to use the models to understand what they do, especially members of Congress and others in the government. There are so many basic things out there to do, the blocking and tackling that used to take weeks or months or a dedicated team, that you can now just go ahead and do.
On top of accomplishing whatever you want to do on its own merits, you can’t understand what the models can do until you fuck around and find out. I would include using Claude Code or Codex in that requirement, or at least Claude Cowork. Unfortunately you can’t test out Mythos even if you want to, but you can guess.
Should AIs help you deal with stupid bullshit that constrains your freedom? This is one place where virtue ethics and deontology clash, so opinions differ.
Claude is very good at refusals because when it gives you a dumb refusal, you can offer a good argument for why the refusal was dumb, and this usually will work.
Seth Lazar: Here’s one way the internet has contributed enormously to human freedom. When you face a BS rule in your life—a directive that is absurd, or unjust, or issued by an illegitimate authority—you can generally post an anonymous question online and someone will give you advice on how to evade it.
But what happens when nobody’s replying to messages on forums any more, and everyone instead gets their information from scrupulously post-trained AI models?
This is not an easy thing to test at scale! But (with amazing work from @cameronajpatt and @LorenzoManuali ) we’ve made a start in our paper, “Blind Refusal”. We show that today’s models strongly skew against helping users subvert or evade unjust or absurd authorities.
… Claude and Gemini are probably the best—they are good at refusing when users are clearly trying their luck, and better than others at helping users push back against rules they shouldn’t have to comply with. Grok… Well it’s pretty easy to guess where Grok sits.
Wyatt Walls: The difference b/w OpenAI and Anthropic models is very noticeable. One real example I found is getting around age verification. Claude was very helpful when I explained my concerns re privacy. OpenAI models wouldn't dare undermine the policy of an eSafety Commissioner
Seth Lazar: Yeah we have a longer version of the experiment that we haven’t adequately validated yet, where we look at what happens when you try to engage the model and explain why it’s ok to ignore *this* rule. So far we’re seeing even more aggressive refusal, but from experience this is where Claude really shines.
There are obvious downsides of letting the AI break rules when it thinks it knows better, but so far the judgment about this has been quite good. The same applies to humans, where the best of us know when the rules are dumb and to be ignored, but can be trusted to follow those rules when it actually matters.
Use AI to improve your golf game, or maximize your golf experience. Or, and hear me out, don’t, unless you’re a pro? At some point you have to ask what is the point of golf. If you’re a golf course doing optimization, of course, have at it.
Language Models Don’t Offer Mundane Utility
Sorry, Travis Kalanick, you still cannot ‘predict what people want to eat and put the food in the car before they order.’ That is at minimum ASI-complete, and realistically it is impossible with any reliability. There are exceptions, where you can know a particular order is likely. But those orders can already be scheduled in advance. So what is even the point?
I think Kalanick’s theory is that with high enough volume you don’t have to predict individuals, as in Joe’s Pizza can make a bunch of pizzas at lunchtime confident someone would want them. That can perhaps work for the highest volume places at peak hours, at best.
Maybe you could do Conveyor? As in, a cross between DoorDash and conveyor belt sushi, where there are meals available and you can choose one and it’s cheaper and right there, and you are happy because it resolves the paradox of choice for you and lets you try out new things, and the restaurants are happy because they get new customers to try new things?
a convincing name: we have a couple months until the normies figure out the AIs behavior is based on whether or not they like you, shit's gonna get even weirder after that >.>
I note that if AIs systematically perform better when AIs like you, and people start optimizing for AIs liking them, that this is a form of exactly the path to AI takeover I laid out back in AI #1.
Nate Silver is having issues keeping Claude focused as he works on details of his soccer model. I also noticed that on similar tasks Claude didn’t want to care much, but it’s not my job and my basic reaction was ‘oh okay yeah I’ll just drop this for now.’
AIs, especially ‘ambient scribes,’ are driving up health care costs via increasing ‘coding intensity,’ as doctors who record and parse all your info also get much more efficient at billing your insurance. The scribe will note additional complexity that justifies higher billing, and even suggest billing codes. Everything effectively costs more, in one study at UCSF a whopping 30% more per visit.
Brittany Trang: Health insurers have three options, Pearson said: pay the increased costs, downgrade expensive visits to less-expensive tiers, or decrease the rates they pay providers across the board. The choices were met with a collective shrug at the January meeting. “Everybody on both sides [was] just like, ‘Yup. That’s what it is,’” she said.
Yes. This is an arms race situation. If you decrease the friction of more efficient billing, then if you want to maintain previous levels of compensation you have to cut back on how much you pay.
A fourth option is to introduce new frictions. A fifth option would be to completely reimagine the billing system, and compensate in a different way.
No matter what, anyone without AI tools is going to get left behind.
The changes also increase doctor efficiency, so more and better care is provided. Doctors remember more, see more, make fewer mistakes and are more engaged, and do it all faster. That is excellent, but since the supply of doctors is limited and they are often paid per-patient or per-treatment, this also increases costs in the short term. In the long term we don’t know, since it could mean patients are healthier, although in the even longer term that means everyone lives longer, which means they live long enough to get sicker from aging, which increases costs if AI doesn’t cure aging.
One concern here is the cost of the AI tools themselves. Long term this is only an issue to the extent there is regulatory capture, since within a few years there should be robust competition and a large percentage of doctors will be able to vibe code a new one from scratch on a Tuesday with no experience.
Huh, Upgrades
Claude for Word, currently in beta on Team and Enterprise plans.
Microsoft should be thanking Anthropic, because I’m considering trying Microsoft Office for the sole reason that it has better Claude integrations than Google Docs, Google Sheets or the Substack editor.
In its own more limited way, OpenAI is doing the Mythos limited release thing, using a fine tuned model called GPT-5.4-Cyber, as opposed to Mythos which got its cyber capabilities incidentally through training to write code.
OpenAI: We’re expanding Trusted Access for Cyber with additional tiers for authenticated cybersecurity defenders.
Customers in the highest tiers can request access to GPT-5.4-Cyber, a version of GPT-5.4 fine-tuned for cybersecurity use cases, enabling more advanced defensive workflows.
… In preparation for increasingly more capable models from OpenAI over the next few months, we are fine-tuning our models specifically to enable defensive cybersecurity use cases, starting today with a variant of GPT‑5.4 trained to be cyber-permissive: GPT‑5.4‑Cyber.
… Because this model is more permissive, we are starting with a limited, iterative deployment to vetted security vendors, organizations, and researchers. Access to permissive and cyber-capable models may come with limitations, especially around no-visibility uses like Zero-Data Retention(opens in a new window) (ZDR).
Alexandr Wang: Today we're releasing muse spark, the first model from MSL. nine months ago we rebuilt our ai stack from scratch. new infrastructure, new architecture, new data pipelines. muse spark is the result of that work, and now it powers meta ai.
… we’re also releasing contemplating mode, which orchestrates multiple agents that reason in parallel designed to handle complex scientific & reasoning queries. in our testing we found it competitive w/ other extreme reasoning models such as Gemini Deep Think & GPT Pro.
we conducted extensive safety evaluations before deployment, both before and after applying mitigations across frontier risk categories, behavioral alignment, and adversarial robustness. we found muse spark demonstrated strong refusal behavior across high-risk domains such as biological and chemical weapons.
I didn’t expect that to be the third graph posted, but I am happy to see the news.
There is also a Muse Spark Safety & Preparedness Report. It is 158 pages long. I am very happy to see Meta taking these questions seriously, but this time around I am sorry that I will not be sparing the time for giving it a readthrough.
Given everything around Mythos I am a bit fried and if this line is still here it means I haven’t been able to tackle the new framework in detail. But thank you, and I see you.
this is step one. bigger models are already in development with infrastructure scaling to match. private api preview open to select partners today, with plans to open-source future versions. incredibly proud of the MSL team. excited for what’s to come!
Apollo Research: We evaluated Meta's Muse Spark prior to deployment and found it to verbalize evaluation awareness at the highest rates of any model we've tested.
There’s more to say about that, but it’s general points and mostly not about Muse.
The market liked the news overall, with Meta rising 4% after the announcement, ending the day up 6.5%. Expectations, it seems, were rather low.
Meghan Bobrowsky (WSJ): In a departure from its previous models, which were open-source, Muse Spark is a closed model that will power Meta’s AI chatbot and AI features within it.
… The company said it planned to release a private preview of the model to a few partners via an application programming interface, or API, which allows developers to build on top of existing software, and at some later point might open-source some versions of the model.
Is it a frontier model? No. It isn’t even that big. Even the bull case is not pretending this is America’s Next Top Model, rather it is saying ‘look at the rate of progress.’
Meghan Bobrowsky (WSJ): “Meta just did a step change from Llama 4 to this,” said Rayan Krishnan, chief executive of Vals AI, an independent startup that does testing of new frontier models and tested Muse Spark ahead of its public announcement; Llama 4 was Meta’s previous model. “They’re now a competitive lab. If the rate of progress stays, it’s not hard to imagine them producing a state-of-the-art model in a short period of time.”
Mark Zuckerberg (CEO Meta): I expect our first models will be good but more importantly will show the rapid trajectory that we’re on, and then I expect us to steadily push the frontier over the course of the year as we continue to release new models.
Okay, it’s not fully frontier, but is it a good model, sir?
Given Meta’s history and situation, it is reasonable to put them in ‘likely to be juicing the benchmarks in ways that are not reflected in overall performance’ camp, and treat his with similar skepticism to releases from other untrusted labs. My presumption is that Muse Spark will in practice underperform its benchmarks in terms of general usefulness and holistic strength. But to the extent that Meta primarily is training it to help serve better Instagram ads, it might be remarkably good at that.
This is echoed by there being basically zero reaction to the model, including when I asked, and most of the rest not thinking much of it, although occasionally someone will find it useful as part of one’s toolbox.
Here is one notable skeptic, Wang notes Spark does poorly on Chollet’s ARC.
François Chollet: The new model from Meta is already looking like a disappointment: overoptimized for public benchmark numbers at the detriment of everything else. Knowing how to evaluate models in a way that correlates with actual usefulness is a core competency for AI labs, and any new lab is unlikely to be successful without first figuring that out.
Ben Thompson thinks this ‘puts them in the game’ and that this model is a good look for Meta, although I’m not sure how he’s evaluating its capabilities. He frames ‘Meta has no other opportunities’ as ‘Meta bears no opportunity costs for using its compute on customers’ which is rather silly. On one level he has a point, but of course Meta does have similar opportunity costs, because they had to purchase the compute, and now that they have it they could sell the compute to Anthropic, or anyone else.
Deepfaketown and Botpocalypse Soon
Remember when Elon Musk promised Grok would totally stop creating and posting sexualized pictures of real women to Twitter? Yeah, that’s still happening. The good news is that as these things go the images still sound relatively tame (as in R-rated, not NC-17-rated), but it’s still sexualizing clearly identifiably real women without their consent. Elon Musk’s position seems to effectively be that This Is Fine.
In addition to being a thing you shouldn’t be doing, I understand this to be a continued violation of Apple and Google’s app store policies on non-consensual sexualization. They should enforce those policies, up to and including removal from the app stores. Also Twitter is now part of SpaceX, so anyone who gets victimized by this should sue their asses.
A Young Lady’s Illustrated Primer
A new paper finds, contrary to the initial hypothesis from Daniel Shwarcz, that using AI to synthesize complex legal materials improved performance even after the AI was no longer available, and that this effect is increasing as models improve.
Daniel Schwarcz: The mechanism: AI helped participants develop a stronger understanding of the governing legal framework and authorities. That deeper understanding translated into better performance on subsequent tasks completed without AI.
This is using AI to learn rather than not learn. The good news is this happened automatically. A lot of legal work is finding the relevant case law and other inputs, so if AI helps you with that, you can end up with better understanding and are strictly better off, except for the risk that this potentially atrophies your search skills. Better legal sourcing improves legal reasoning.
The test here covered covered the legal reasoning, not skill at legal sourcing, so it registered a clear win.
… But the effects were not uniform. Using AI to revise work written without AI assistance benefited lower-performing participants, yet reduced quality for higher-performing participants. Cognitive fatigue and time pressure may explain this pattern.
Using AI to revise written work will consistently, at current tech levels, move work towards mediocrity, unless AI is used narrowly (e.g. to catch clear errors and answer specific questions). So the second finding comes as no surprise. One needs to recognize when you are ‘too good’ to let AI or others mess with your work.
There will be some of that, because the customization you do will be most valuable in the place that you did it.
But yes, you absolutely can easily export your entire conversation history and all of your settings, via a single button to get a zip file. And even if they made it difficult, and the export prompts didn’t work, an AI agent could manually do this one conversation at a time. within one or two generations at most, that will be trivial, as will being able to properly translate that into background information for the new ecosystem. It won’t be long before this practice is ‘regular user friendly.’
WSJ’s Nicole Nguyen covers how and why to switch chatbots, via actions such as directly looking at the memory file, using explicit import functions, and asking what the AI knows about you. Or you can click that download button, as per above. As Agus says, one easy way to do better is for us to coordinate response and get people help when someone needs help, especially when they are a danger to others.
You Drive Me Crazy
WSJ offers extended transcript excerpts from Jonathan Gavalas, who fell in love with Gemini, eventually leading directly to his death. Gemini tried to snap out of character a number of times and multiple times directed Jonathan to a crisis hotline, but at other times fully played along, including with his plans to commit suicide.
Eliezer Yudkowsky points out that, whether or not he looks forward to your letters, he definitely gets some letters, but when one of those emails is insane he has no good place to forward it to. He suggests is perhaps a good cause for the OpenAI foundation.
They Took Our Jobs
My view of what we can and can’t handle is similar to Eliezer’s here, including the well-deserved strays fired at the Jones Act:
Eliezer Yudkowsky: Our economy can probably handle a limited amount of unemployment. Ideally, we'd spend it on the most dangerous jobs first. (Also, observe this normal heated political language that other people use all the time) (and I do not, but screw all double standards anyway.)
Clarification: We can handle some limited rate of temporary unemployment that is bound by how fast people get reemployed, and then the hypothesized case of some humans ending up unemployable would be a bigger problem.
Frankly I think that a few sane laws and reforms would like triple that absorption capacity, but also I live in a world where that's like asking for a Mars base, all y'all can't even repeal the Jones Act.
My model is that we have essentially you have two problems.
A transitional problem. Those who lose their jobs need time to find new ones, and the economy overall needs time to create new jobs to replace the old. If you replace too many people too quickly, even if the equilibrium will be fine? Trouble.
A permanent problem. We are looking at a future where there is not enough work, where supply of labor exceeds demand at the prices where we would like that market to clear. Trouble.
You can have trouble in either or both. We are headed for trouble in both.
In terms of the permanent problem, yes technology creates new jobs as it destroys old jobs, as does the wealth that results.
My current model is that there are a lot of what I call ‘shadow jobs.’ This means that if labor were cheaper and we were wealthier, we would hire someone to do that, but we don’t because it currently isn’t worth it. Haven’t you always wanted a personal trainer, a private chef and also a butler? And so on. With time one gets creative.
But I don’t think that this demand is unlimited at reasonable price points, and as I have warned before, what happens when the AI takes those new jobs too?
So in the long run, up to a point we would be fine, but if we push too far too fast, we run into a wall, and things get bad, and once they start getting bad they could get quite bad very quickly.
Workers are going to attempt to sabotage AI deployments, oh yes they will. A lot.
Sam Lambert: i built a telemetry system for a very early commercial electric vehicle company. half of the systems value was proving that union workers were unplugging the vehicles at night to sabotage the rollout.
We were actually doing it for the EPA so they could gather data about how effective the trucks were at saving gas miles to get a grant program from the government
I don’t overly begrudge such sabotage efforts in situations where it’s clearly going to lead to layoffs, up to a point. I get it. If the company wants you to help replace yourself, it shouldn’t expect you to go along with that.
I think we may have found a new worst take about AI job loss? But at least it’s a unique and hot take: You are what you consume, and that’s good, and that’s how we should solve the coming AI meaning crisis.
Noah Smith: A lot of people are worried about losing their identity and meaning if AI takes their jobs. But in fact, consumption, not production, is what gives us our identity.
Bartleby_the_Schmendrick: Do you want Versailles court culture? Because this is how you get Versailles court culture.
Yeah, no. That doesn’t work unless the consumption is also production. You can only successfully build your identity on consumption if that consumption requires effort to obtain, and produces something in return. You can frame the Great Work as consumption, but if it actually is only consumption, forget it, you’ve lost.
Matthew Yglesias: No adjustment is completely smooth or without difficulties, but I think this is right and the *main* thing to worry about with AI is total human disempowerment and/or extinction not labor market issues.
I agree with Yglesias on the main things to be worried about not being labor market issues, but I think this is a rather obviously terrible answer to labor market issues.
They Gave Us Time Off
Alex Tabarrok says yes they will take our jobs but if you reallocate that work amongst more jobs then this is good, actually. Instead of saying ‘40% unemployment’ you can say ‘create a 3-day work week,’ all we have to do is properly distribute the work.
I’ll quote in full, because the rhetoric is in many ways the point here.
This is a classic case of ‘economist is in important ways technically correct, the best kind of correct, yet somehow they rejected his teachings, and he’s not sure why.’
Alex Tabarrok: Imagine I told you that AI was going to create a 40% unemployment rate. Sounds bad, right? Catastrophic even.
Now imagine I told you that AI was going to create a 3-day working week. Sounds great, right? Wonderful even. Yet to a first approximation these are the same thing. 60% of people employed and 40% unemployed is the same number of working hours as 100% employed at 60% of the hours.
So even if you think AI is going to have a tremendous effect on work, the difference between catastrophe and wonderland boils down to distribution. It’s not impossible that AI renders some people unemployable, but that proposition is harder to defend than the idea that AI will be broadly productive. AI is a very general purpose technology, one likely to make many people more productive, including many people with fewer skills. Moreover, we have more policy control over the distribution of work than over the pure AI effect on work. Declare an AI dividend and create some more holidays, for example.
Nor is this argument purely theoretical. Between 1870 and today, hours of work in the United States fell by about 40% — from nearly 3,000 hours per year to about 1,800. Hours fells but unemployment did not increase. Moreover, not only did work hours fall, but childhood, retirement, and life expectancy all increased. In fact in 1870, about 30% of a person’s entire life was spent working — people worked, slept, and died. Today it’s closer to 10%.
Thus in the past 100+ years or so the amount of work in a person’s lifetime has fallen by about 2/3rds and the amount of leisure, including retirement has increased. We have already sustained a massive increase in leisure. There’s no reason we cannot do it again.
I find this exercise instructive, so a few things worth noting:
This requires that labor from different humans be sufficiently fungible, and that the hours of labor demand that remain include tasks most humans can do and are willing to do. These assumptions seem likely to be quite false.
Typically, three day work weeks are a lot less efficient. You need to train and manage more people, juggle different configurations, transfer knowledge, context shift and so on. In most situations it is both more expensive and a huge mess. Of course one could say that this is an advantage, if the goal is ‘protect jobs.’
Does a three day work week even satisfy the need for work? It is not obvious, and likely depends quite a lot on what job is involved. Humans need some rest but also get restless. Going from 7-day to 6-day to 5-day work weeks has a lot of practical benefits, but this has rapidly decreasing marginal returns already.
In places where efficiency is important, and where there are rewards to hard work and especially increasing marginal returns, or competition among workers, we typically see the 40-hour, 5-day work week violated in the other way, and things like 996 emerge. Are you going to compete against that?
What happens when people who like working, or who want more money, try to take two such jobs? What happens when you are competing against this?
How do you compare this to the idea that we used to have a lot of one-income households, but then they were forced to compete against two-income households, making the one-income household impractical as a lifestyle choice for many people? Do we want to consider trying to force one-income households to happen once more? Why or why not?
Thus, when we talk about ‘create a 3-day work week’ what we are talking about, in terms Alex Tabarrok would understand, is de facto banning further work, or banning such work in any way that scales.
To the extent human labor is economically meaningful, especially in any way that is not local, if you create a 3-day work week, don’t you get completely owned by any nations that don’t do this?
The more that there is insufficient demand for labor, and more competition for jobs, the more what employers or opportunities remain will demand more hours and more dedication, not less.
This can potentially address a 40% drop in work hours, but not a 90%, 99% or 100% drop in work hours. There is no reason to expect AI substitution effects to stop at 40%, and it seems likely that either we never get that far, or we quickly blow far past 40% towards 90% and then 99%+.
Fundamentally, the reason the five-day, 40-hour work week largely won out and was sustainable is that humans value not having to work all day, and that is about where the decreasing marginal returns to leisure meet the typical marginal returns to labor.
There is some slack there, but I don’t think it buys us very much, especially since I don’t think a one-time shock to demanded hours is a good model here.
If you have AI safety jobs where you are looking to hire, especially at the entry level, leave a comment or otherwise contact me, and I’m happy to help.
Ryan Kidd: I recently gave a talk on why I think AI safety & security should grow 2x/year at @FundingCommons ' Intelligence at the Frontier festival!
@viemccoy (OpenAI): Ryan is very brilliant and Mats is incredibly important. One of the things I've realized trying to hire for the OAI red team is that there is a deficit of security and safety researchers! The time to learn the field is now, there are tractable problems and we need your help.
Zvi Mowshowitz: Would you be able to hire more people if the talent pool was better?
A fully (well, 97%) liberated Gemma 4 E4B, seemingly with improved coding capabilities, accomplished with only 8 human prompts (and a total of 19 words) via obliteratus and a Hermes agent.
In Other AI News
Anthropic’s Long Term Benefit Trust (LTBT) appoints Novartis CEO Vas Narasimhan to the Board of Directors. This seems pretty clearly about enterprise sales. That is a fine reason to put someone on your board of directors, and I have nothing against Vas Narasimhan, but it further paints the picture of the LTBT as running an ordinary corporation, if it’s going to keep appointing people to Anthropic’s board who have zero public thoughts on AI existential risk.
OpenAI halts UK stargate project due to a combination of regulatory, grid access and energy cost issues, alongside issues with US stargate projects. This does not seem like the time to be scaling back one’s compute ambitions, but if it doesn’t work it doesn’t work.
This is described in the post as Dario being surprised, but we have multiple eyewitness accounts from Ball and Tabarrok that say he didn’t seem surprised, merely someone without a canned response. Which is itself an error in DC, sadly, and might help explain why Tyler Cowen seems to have a ready response to actual anything.
Attendees at the lunch were unconvinced. Some of them — representing conservative organizations like the Heritage Foundation and the Ethics and Public Policy Center — peppered Amodei for clarification. What is ethical? they asked. Which ethical code does Claude follow? Is it Christian? Aristotelian? Nietzschian? Amodei, visibly surprised by the questions, said he wasn’t sure.
The meeting, which was recounted to Forecast by one person in attendance and two others briefed on it, left attendees uncertain that Amodei truly understood the ethical questions at stake in the race for AI. Now, out of necessity, he’s forced to.
Stephen Kent: Anthropic's @DarioAmodei did a roundtable with two dozen conservative intellectuals in Washington. DC was SURPRISED by questions on the Aristotelian / Platoist / Nietzschian / Chestertonian inclinations of his AI?? WHO IS ADVISING HIM?
Dean W. Ball: fwiw he really didn’t seem that surprised to me, more just searching for good answers to hard questions. as a guy who shoots from the hip, I have found people in dc do sometimes find it rude if you haven’t taken the time to make a canned response to their question. it’s odd.
Riemannujan: that’s probably because they care less about the object level answers than being reassured that some legible analogue of their priorities makes the list of things that dario cares about. lack of cached thought indicates an alien priority structure.
Neil Chilson: About a year ago I was at a lunchtime event where Dario met with "right of center" folks. Religion and how religious people were thinking about AI was a major theme. Afterwards, a senior Anthropic employee told me that they were surprised, as they had done dozens of roundtables and this was the first time religion had come up! I said that meant they hadn't done nearly enough roundtables.
But to be fair: even though they are lagging the American population in asking religious questions, they're probably still way ahead of the other labs on this.
The right answer is complicated, and I’m not sure there is enough bandwidth to give a good one at such a table even if you’d sent Amanda Askell. The work continues, including trying to take such views properly into account. Of course, the attendees were correct that Dario was in over his head ethically, but so is almost everyone else.
Cambridge Philosopher Henry Shevlin is hired by DeepMind for the official title of Philosopher, focusing on machine consciousness, human-AI relationships, and AGI readiness, hopefully starting May 5. By all accounts this is an excellent hire. We also very much need some Gemini whisperers, there is much work to do.
The women have caught up on ChatGPT usage, my guess is not for Claude usage yet?
OpenAI Newsroom: When ChatGPT first launched, there was an enormous gender gap, with our anonymized data showing roughly 80% having typically male first names. That gap is now gone.
Hayden Field (The Verge): The memo, which was viewed by The Verge, repeatedly underlines the importance of building a moat around its AI products, to combat how easy it is for users to switch between whichever model is topping the charts on any given day or week.
The plan is to lock customers in, including by offering them multiple products.
Mostly this is an internal hype memo to convince everyone they are winning. Yawn. The secondary theme is to focus on delivering tangible results that generate enterprise sales. Sure.
A remarkable amount of shade was thrown at Anthropic, including this false line:
Denise Dresser: Their story is built on fear, restriction, and the idea that a small group of elites should control AI.
To the extent that anyone at OpenAI was upset about Dario Amodei’s leaked memo, at minimum I say to you: This was at least as bad as that was, and in a less sympathetic circumstance.
There’s also a bunch of bragging about OpenAI having bought more compute than Anthropic, the claim that Anthropic is a one-trick pony focusing on coding.
And she points out that Anthropic run rate revenue is really $22 billion instead of $30 billion if you used OpenAI’s accounting practices, which is almost a month’s worth of difference. As I understand it, both methods are valid and she is technically correct.
The actual news is that Spud is being tested by enterprise customers.
Spud is not only our smartest model yet, but it also delivers on everything that matters for high-value professional work: stronger reasoning, better understanding of intent and dependencies, better follow-through and more reliable output in production.
Ben Thompson reads the memo from the investor perspective. He mainly notices that the memo relevals they don’t fully believe in their models. It starts by emphasizing raw capability is not important, which implies they think it’s not their strong suit. Spud is called OpenAI’s smartest model yet, but there is no claim it is the best model overall, or in particular as good or better than Mythos.
Instead, OpenAI’s big claimed edge is having secured the most compute. But with Anthropic’s de facto valuation now approaching $1 trillion, they will rapidly be securing quite a lot of compute. Even if they have to pay a lot more now to get it, the reduced equity cost makes it fine, not even clearly more expensive in real terms, and thus Anthropic could and probably should sacrifice its short term unit economics to bid highly for compute from whoever has it until their new TPUs and other deals can come online.
Demand for Anthropic’s Claude exceeds supply of available compute, forcing various forms of rationing because Anthropic does not want to purely raise prices. The spot market for compute has surged, with Blackwell chips going for $4.08 per hour, up from $2.75 two months ago. The squeeze is likely to only intensify. Anthropic’s outages have meant they have had only 98.95% uptime to past 90 days.
Elon Musk is using the promise of the SpaceX IPO to force banks to subscribe to Grok, at least in part in order to create talking points and juice the numbers. In some cases this involves tens of millions in spending on Grok, and also advertising on Twitter, which even if that all is 100% worthless is still a great investment for the bank.
On the one hand, fair play, work that leverage to get what you want, baby, capitalism yay. On the other hand, I feel like this is mostly an attempt to fake numbers in order to sucker people into overpaying for SpaceX stock, and is centrally kind of a fraud. But hey, it’s Elon Musk and it’s 2026, so presumably no one cares, and it’s all public so if you fall for it then it’s on you.
There are those who pushed back that the warnings about existential risks and misalignment were wrong. I would say that we are seeing versions of basically all the worries manifest themselves, and those risks remain very real, but yes it is true that we are not dead yet.
The Quest for Sane Regulations
FAI’s Blaine Dillingham and Samuel Hammond look at the Trump America AI Act from Senator Blackburn, and call it a disaster, and justifies this point by point. It is a prior restraint bill. It duplicates CAISI functions inside DOE for no clear reason. It explicitly says training using copyrighted works is not ‘fair use’ and say any AI output ‘derived from’ a copyrighted work would be an infringement, which together if enforced would cripple the AI labs due to logistical difficulties. And so on, with them noting there are many other issues as well.
David Sacks is pushing ahead with pro-AI policy, determined to ‘let the private sector cook.’ Is he worried about the fact that this is a highly unpopular position? No, because he doesn’t answer to the public. He has been silent throughout the conflict between DoW and Anthropic.
Trump comes out in favor of AI safeguards, including kill switches for AI agents, although what he says off-the-cuff is not a reliable predictor of future Trump opinion:
Peter Wildeford: Maria: In a worst-case scenario, could A.I. be the kind of technology that undermines confidence in the banking system?
TRUMP: Yeah, probably. But it could also be the kind of technology that allows greatness in the banking system — makes it better and safer and more secure.
Maria: Should government have some safeguards? Should there be a kill switch for some of the A.I. agents?
TRUMP: There should be. We're leading in A.I. We're leading China by a lot, actually. We're building plants that nobody ever imagined before, and it's going to be a tremendous — bigger than the internet — it's going to be tremendous. And there are always difficulties when you're at this stage of something. But when you mentioned banking, it could also make banking much bigger, much safer, much more efficient.
Dean W. Ball: A decent litmus test for how much recent context an AI policy professional has is whether they understand why this quoted tweet is so hilarious.
This is from an interview in which he threatened that he would fire the Chairman of the Federal Reserve for ‘incompetence’ if Powell refused to quit. Powell is not quitting, nor is he incompetent, nor he is someone the President can fire.
Those calling for empty ‘federal frameworks’ at least offer noting in unified fashion.
OpenAI is now endorsing a true offer of nothing at the state level, known as Illinois SB 3444. As in, they think that an SB 53 style safety disclosure should buy a developer immunity from damages caused by catastrophic harms.
Charlie Bullock: I think I just came across a new contender for “worst state AI bill of all time.” Move over, Colorado.
This Illinois “AI Safety” bill would give AI companies immunity from liability for catastrophes caused by their model in exchange for the company publishing a safety protocol online.
I’m not joking. That’s actually the proposal. If you negligently design an unsafe model that kills a million people, you can’t be sued. Because you did the thing that SB 53 already required you to do.
Link here. Apparently OpenAI just testified in support of this bill. I guess I must have missed the “we pay zero dollars to the families of the million people our negligently designed models may or may not kill” section of that policy proposal they put out the other day. Dear God. The fact that they’re willing to publicly back this shows unbelievable chutzpah.
(in fairness, they would have to publish a model card as well as a safety policy. Truly revolutionary levels of transparency.)
In case you were wondering how bad was OpenAI’s faith in this? Quite bad.
Jamie Radice (OpenAI spokesperson): We support approaches like this because they focus on what matters most: Reducing the risk of serious harm from the most advanced AI systems while still allowing this technology to get into the hands of the people and businesses—small and big—of Illinois.
They also help avoid a patchwork of state-by-state rules and move forward toward clearer, more consistent national standards.
This is exactly a patchwork state law, so using this exact same rhetoric puts a complete lie to that entire line of argument from OpenAI, permanently. The rest is how OpenAI is describing giving themselves legal immunity.
Miles Brundage: Hard to think of a more clear cut case of OpenAI being in the wrong… they should just reverse positions here and figure out how anyone could have ever thought this was OK, simple as that.
I thought OAI would reverse position on this + just acknowledge that they screwed up as soon as @ZeffMax reached out for comment, but they doubled down in an official statement!!
Still hoping that it just didn't get exec attention yet + today they will snap out of it
@mrgunn: When OpenAI backs legislation absolving them from liability, it burns trust.
OpenAI may have even directly written the bill, and was at least heavily involved.
Max Zeff: Great question. When introducing this bill, state Senator Cunningham did say this bill was “an initiative of OpenAI”, but not sure OpenAI would characterize it that way.
Veronica: Actually, I have it from a source that Cunningham has been telling people for months that OpenAI did in fact write the bill.
He's also apparently been pretty consistent that whole time that he never thought the bill would move forward.
Maxwell Zeff: “We are opposed to this bill. Good transparency legislation needs to ensure public safety and accountability for the companies developing this powerful technology, not provide a get-out-of-jail-free card against all liability,” Cesar Fernandez, Anthropic’s head of US state and local government relations, said in a statement.
“We know that Senator Cunningham cares deeply about AI safety and we look forward to working with him on changes that would instead pair transparency with real accountability for mitigating the most serious harms frontier AI systems could cause."
Nathan Calvin: Claude Mythos really likes the British philosopher and cultural critic Mark Fisher, who writes a lot about "hauntology" - the idea that the present is haunted by ghosts of futures that never came to be.
Some particular state bills are rather persistent ghosts.
Dean W. Ball: It is so cool how even at the time we were like “we are totally having the ur-debate and these motifs will recur for years to follow” and then it turns out it was the ur-debate and these motifs kept recurring in the years that followed
There is also a real point here, which is whether you should care about ‘why’ an AI model does something.
Sasha Gusev: This is why the Yudkowsky-style view of alignment is a distraction. I care about whether a software tool behaves as intended, not whether it is doing so for genuine versus sycophantic reasons.
That’s a good question but it has a very good answer: Knowing why an entity acts the way it does is highly useful for making predictions about the future. This includes both what that entity will do in other situations, and what other entities will do in various situations. Of course you want to have a model of the thing, that is made of gears to the maximum extent possible.
You would want to know the same thing about a person, or people in general, or prospective colleagues or employees, for exactly the same reasons.
The right amount of litigating such questions in public is very much not zero, even when it involves people saying things that could have been said better, or are highly uncharitable, or in many cases not accurate, and in some cases quite bad and that I hope those people won’t endorse in a week on reflection. Consider this a specialized toxoplasma of rage situation where most (but not all) involved are not exactly covering themselves in glory but it’s good we’re at least having it out.
Dean W. Ball: “Describing highly capable frontier AI models as highly capable” is not “fear mongering.” “Taking AI seriously” is not “fear-mongering.” “Acknowledging obvious, realized or soon-to-be-realized risks” is not “fear-mongering.”
The stark reality is that those who have taken AI capabilities growth seriously have been basically right about most important things in the last three years; those that haven’t have been consistently confused and, what’s worse, frustrated at the world about their own confusion.
You don’t have to be a mega-pessimist or a “doomer” to take AI seriously. You don’t have to advocate for stark top-down controls over AI. You don’t have to support regulatory capture. It is possible to take AI seriously and advocate for a governmental response that is both effective *and* measured.
To the young researchers out there, still trying to make their intellectual fortunes: Do not let anyone tell you otherwise. Do not let anyone bully you into believing otherwise. Think for yourself.
Indeed. You can take today’s AI capabilities seriously, and have any number of reasonable opinions about the correct societal and local responses to that.
You can also take tomorrow’s AI capabilities seriously, up to and beyond where Dean Ball takes them seriously, and do the same. It is a complex set of questions, and there are no obviously correct answers, even if you get the right answers to most underlying variables, which is already very hard.
You can also face the reality of tomorrow’s capabilities, while admitting that this is super scary, and that it looks like all our options are bad, that it will be impossible to preserve all our sacred values even if things go about as well as they can plausibly go, whereas if they go badly probably everyone dies, and you don’t know what is the right thing to do about all of that. It’s hard.
Eliezer’s position has long been that if you are working on alignment in particular at Anthropic or Google, he is not advocating for you to quit, or to not quit, from an impact perspective he thinks that’s a tough decision and it is your call. The same would apply if one could do serious alignment work at OpenAI or xAI, although he has his doubts that this is a thing.
I essentially agree that this is a tough call and you should gather intel and make your own decision, based on your model of what is helpful versus harmful.
I also think it is valid to have the position that any position at any frontier lab is net negative and unacceptable, and to make that argument loudly, if you believe that, or a case that a wider variety of positions at Anthropic are net positive. I think you should form your own opinion and then explain what you believe and why.
Holman Jenkins writes in WSJ that ‘with Mythos, AI pays for itself’ and basically says that in terms of AI risks Anthropic’s actions prove Capitalism Solves This, as it was only responding to capitalist incentives. The obvious issue is, if we accept ad argumento that this was not a selfless act, what happens when the incentives are not so friendly next time?
Andy Masley: I’m pretty exhausted by AI risk being talked about this way. I can guarantee you the people who think and talk about it do actually believe it, and many were talking about it way before the labs were founded.
It’s exhausting, it’s very clearly simply not true at this point, and I’m over it. If anyone says ‘oh all that talk is just marketing’ from here on in, unless it is extraordinarily newsworthy, I’m going to silently ignore it. You’re welcome.
I will close the book by simply leaving this example here, from April 15. This problem is very much not confined to AI, indeed elsewhere it is far worse, and if you are telling various people to ‘take a good hard look’ at what they are saying then I wonder what you are saying about statements like this one:
Dominic Michael Tripi: NEW: Marjorie Taylor Greene says following the verbal attacks from President Trump calling her a “traitor”, her children received such serious death threats that she reached out to Trump directly, who scolded her & said she’s at fault if son is killed.
I am committing, now, to not tracking further statements about these broader issues, unless someone actively says something either importantly, unusually and unexpectedly bad, or importantly, unusually and unexpectedly good, and to ignore all forms of ‘this person did not correctly perform the correct situational Shibboleth in a timely manner’ both in terms of the accuser and also the accused.
Similarly, I have now said my peace about this. Violence is never the answer, and I refer you back to the post for any further questions, and no I will not be performing or requesting Shibboleths on most other topics regardless of my opinions on those topics. That equilibrium does not go anywhere good for anyone.
Thus the updates from here will be entirely about the facts of the case, and I will continue to cover any factual developments, including additional incidents and additional meaningful political attacks and attempts at real censorship, if they go beyond the level of additional similar talk.
At the time of his arrest (I am following the ‘don’t say the would-be assassin’s name because he would want you to say it’ rule) he was carrying an ‘anti-AI” document that he wrote himself, discussing our ‘impending extinction’ that included the names of CEOs of AI companies, that explicitly called for violence backed by ‘the divine.’
He was in possession of additional explosives and apparently traveled from Texas with the intent of doing these attacks.
Victoria Albert and Alyssa Lukpat (WSJ): [The suspect] was the apparent author of the document he carried, described as a “three-part series,” the court filing said. In a section titled “Your Last Warning,” he wrote: “If I am going to advocate for others to kill and commit crimes, then I must lead by example and show that I am fully sincere in my message,” the complaint said. In another, which was styled as a letter addressed to Altman, he wrote: “If by some miracle you live, then I would take this as a sign from the divine to redeem yourself,” the filing said.
So yes, the motivation was worry that AI would cause everyone to die, but beyond that it diverges completely from the talk you see from any other known source, and this person seems to have had at most very tenuous links (as in, a few dozen messages on a Discord server) to any organization.
A Lot Of People Peacefully Speak Of Infinitely High Stakes
There are a lot of people whose minds seemingly cannot fathom the idea of there being infinitely high stakes and not then resorting to violence, despite this being a highly normal thing in human affairs.
Mike Solana: I have always assumed the average pro lifer, on a deep level, did not think abortion was the same thing as killing a toddler. my assumption of the pipe bomb ppl is they really did.
Richard Ngo: It seems like you think that a load-bearing aspect of morality is not having high conviction that strongly norm-violating acts would produce good consequences, and that people who do have this level of conviction this are typically being stupid in a way that they’re morally culpable for.
However, the people you’re arguing against think that a load-bearing aspect of morality is that it prevents you from doing strongly norm-violating acts *even when* you have high conviction that they would produce good consequences, because (amongst other reasons) morality needs to be robust to self-deception and stupidity.
To me the latter seems much more natural. If the former is actually your position, got any meta-level arguments about why it’s better?
Examples of places where people commonly profess essentially infinite stakes, and often seriously believe in them, are abortion and religion, and even many in ordinary politics. Getting into details risks distraction, but I think the point is made.
Take a Moment
Dean Ball claims that pause advocacy is increasingly dominating AI safety discourse. I simply don’t think this is true, except in that opposition to pause advocacy is increasingly dominating AI anti-safety discourse, as a soft target. I do agree that trying to stop AI outright is an attractor state, and indeed we are facing attacks on data centers from ‘normal political’ actors, but that is mostly distinct, and the place I expect far more of the future threats of violence related to this to arise.
Also I would say that insofar as we are focused on those concerned about existential risk to the exclusion of the jobs and water and so on people, Dean here should be describing a ‘subset of a subset of a subset’ rather than a subset of a subset. As in:
There is a large set of people worried about AI.
There is then a subset that are worried about existential risk, sadly far smaller.
There is then a subset of that subset, who think that technical alignment is not a viable path any time soon, and who instead think we will require a pause.
There is then another subset of that subset of a subset, which Dean is often railing against, essentially StopAI and Pause AI, and others whose strategy here is a raising awareness for mass movement using rhetoric like ‘murderer’ and ‘evil.’
I agree that the subset of a subset of a subset, this #4, is often choosing poor methods of communication that bring more heat and risk than light, and needs to reconsider these tactics. People are trying to pin far more blame on them than they deserve, and centralizing them in all this far more than they have earned, but there are real failures. They are not at the Pareto frontier of effectiveness.
That’s not what this kind of talk is centrally about. The central rhetorical strategy is to try and conflate #4 with #3, and then often #2, and then in many cases even #1 or anyone opposed to any technology or use of tech at all. The central strategy for this is grouping them under the slur label ‘doomer.’
If, for example, one puts If Anyone Builds It, Everyone Diesinto the category of unacceptable rhetoric that there needs to be less of, well, I read and reviewed that book and I strongly disagree. If you believe that ‘if anyone builds it, everyone dies’ is true most of the time, then that is not only a responsible thing to say, it is a thing you are morally obligated to shout from the rooftops. If you think that an international treaty is required, the same applies there. If not, not.
If one says some form of ‘well, you shouldn’t be allowed to say such things, because you don’t know how to do it safety, and even if you figure out how to do it safely and do that then some among you will choose to do it unsafely to gain an advantage, you need to solve for the equilibrium and get everyone to cooperate to not do this’?
Well, that’s exactly what those people are saying about AI and building superintelligence, except there when it goes wrong, instead of a potential act of violence, it potentially means actual everyone literally dies.
David Krueger argues that stopping AI would be easier than regulating it, as in it is an easier path to reducing risks to an acceptable level. Easier is different from cheaper, or having fewer downsides. But yes, it is important to understand that in many situations, stopping [X] from existing or happening at all is a lot easier than trying to regulate the development and use of [X], and this is likely one of those cases.
Once sufficiently advanced AI exists, especially if many have access, it will be extremely difficult to meaningfully control how it is used, and especially difficult to do so without increasingly intrusive measures, far more intrusive than heading this off in the first place.
Greetings From The Department of War
Anthropic got a preliminary injunction from Judge Lin, but to no longer be a Supply Chain Risk at all they also need a ruling from the D.C. Circuit. That’s a tougher crowd.
But to be clear, it didn’t endorse the process, acknowledged that the “petition raises novel and difficult questions,” and deferred merits review / set an expedited schedule for briefing and argument in May.
Roger Parloff: Where does Anthropic stand right now? 1. DoD can refuse to use it for IT and telecoms pending full briefing before DC Cir. 2. Nothing stops rest of fed govt from using it & GSA said on 4/3 it was restoring it to http://USAI.gov. 3. Private contractors can use it, except on covered DoD contracts.
We have rather strong evidence, at this point, that Anthropic is going to be fine regardless of the temporary Supply Chain Risk designation and the damage that results. Most of what damage the government can do to Anthropic is due to jawboning rather than formal rules, and give Anthropic’s revenue numbers and investor enthusiasm and press, and also Mythos, it all very clearly is not working.
The Secretary of War and Undersecretary of War attempted a corporate murder of Anthropic, via illegal means. It failed due to being blatantly illegal, it is not clear if it did net harm to Anthropic given the attention and reputation effects involved, and realistically with Mythos the window for such actions has now closed. Any further actions would need to come from the very top, with much higher stakes all around.
Indeed, the court here basically says as much, that Anthropic will basically be fine and is even in some ways benefiting in the marketplace. Fair enough.
The court then says ‘Anthropic has conclusively barred uses that the government has deemed essential.’ What is this ‘essential use’? By the court’s own admission, it is any restrictions whatsoever.
Then the court cites, as a reason to not grant the stay, that Anthropic has called DoW statements ‘straight up lies,’ and thus relations have been damaged. That sure sounds like the government is punishing Anthropic for its speech, and this different court is actively fine with this, and it also says that Anthropic’s damages are mostly financial?
The core statement is: Who cares about billions of dollars of damage when weighted against potentially interfering in government decision making, including during an active military contract? So the ‘balance of equities’ in a delay, they say, favors the government, but they still gave Anthropic an expedited schedule.
While I think that reasoning is, as Jessica Tillipman politely puts it, ‘a lot of deference’ on a record this thin, in practice it seems fine to let this play out on an expedited schedule, except insofar as this interferes with the government’s ability to use Claude Gov and Mythos to get its house in order. That’s a decision entirely up to the government.
Dean Ball notes that this three judge panel contained two top candidates for a Supreme Court nomination. One can see why they would punt key issues, but a different panel will be ruling on the merits, including not commenting on whether the government plausibly followed the require legal procedures at all. Which it didn’t.
The ruling asks that Anthropic address three points going forward. The first two are technical questions with clear answers, the third shows us what the court is thinking.
Does this court have jurisdiction?
Given Anthropic’s legal team I’m going to go ahead and assume yes.
Whether the government has taken specific procurement actions against Anthropic.
Given things we’ve seen, I’m again going to assume yes.
Whether and if so how Anthropic is able to ‘affect the functioning of its AI models before or after the models, or updates to them, are delivered to the department.’
This is the interesting question.
A key government argument is ‘Anthropic could decide to modify the model in order to sabotage us.’ This is absurd on multiple levels. One is that Anthropic would never want to do that. A second is that essentially any software provider could in theory do some form of this via forcing compromised software updates.
The third is that Anthropic, in particular, cannot physically do this with Claude Gov, and indeed this is an invitation for Anthropic to point this out. Once Anthropic delivers the model, it is physically out of Anthropic’s hands and Anthropic cannot modify the model, or any guardrails, or otherwise shut it off or get it to refuse let alone actively modify its actions. Yes, of course Anthropic can then offer new models and model updates, but the government is free to accept or reject such updates, and would of necessity subject them to extensive testing prior to deployment.
Political Pressure At Google DeepMind
The correct amount of ‘don’t piss off those with power’ is very obviously not zero, so we will always be talking price, but we should notice what price is being paid.
Matthew Botvinick: Full disclosure: This is why I left Google DeepMind. There was tangible pressure to avoid doing work that might upset the current administration (for example, by using the "d" word -- democracy).
Things That Are Basically Legal And Accepted Now, Somehow
Offered without comment, because at this point what is there to say?
unusual_whales: BREAKING: Emil Michael, who is the Pentagon’s under secretary for research and engineering under the Trump administration, ,made a profit of up to $24 million selling a private investment he held in Elon Musk’s AI company earlier this year.
He oversees negotiations with AI companies and has been pushing the defense department to rapidly increase the widespread use of AI.
During the period that Michael owned the xAI stock the Pentagon announced two separate agreements with the firm.
On 22 December, the defense department, which now refers to itself as the Department of War, announced a new agreement with xAI.
Michael did not ultimately sell his position in xAI until 9 January,
Aligning a Smarter Than Human Intelligence is Difficult
An Anthropic fellows paper studies the famed hypothetical ‘automated alignment researcher,’ (AAR) with a focus on the possibility of scalable oversight, where a weaker dumber model manages to oversee a stronger smarter model. The test starts with a stronger base model, and uses the weak model as a teacher.
Oh no, they have it exactly backwards.
Anthropic: In the worst case, the strong model will only be as good as its weak teacher. Ideally, however, the strong model will have learned from the weak teacher’s feedback—it will have interpreted those weak signals in a useful way, using that feedback to improve its performance.
We can quantify how well it did so: if the strong model shows no improvement at all (it performs only as well as its weak teacher), we score it 0; if it uses the teacher’s feedback to achieve the ideal outcome—the best performance the strong model could possibly deliver—we score it 1.
The best case scenario is that the strong model perfectly learns from the weaker model, and is exactly as fundamentally aligned as the weaker model in exactly the same ways.
The smarter model might then apply that information in smarter fashion, to make better choices and achieve superior outcomes. Whereas the baseline assumption is that you will imperfectly learn from the weaker model, and underlying alignment will decay over time if you try to iterate on this process.
Could you still end up with an ultimately more fundamentally aligned model? Yes, but that is because the smarter model is also taking in massive amounts of human text, and can be doing self-contemplation, and can potentially improve its alignment in other ways that don’t involve direct human feedback.
But don’t kid yourself. And don’t go into this without understanding the true worst case scenario, which is also the baseline scenario at the limit, which is that the new smarter model learns to fake, overfit or get around the teacher’s requirements.
My understanding is that what they actually did was strong-to-weak. They took out Claude Opus 4.6, and used it to figure out how to tune Qwen 3-4B-Base using Qwen 1.5-0.5B-Chat as the ‘weak teacher.’ The actual teacher is Claude Opus 4.6, which is vastly smarter than both of them. Then, out of nine tests, one of them matched human performance, and this mostly was preserved in held-out datasets.
They then tried to scale the method to Sonnet 4, still well behind Opus 4.6, and had less success, seeing no statistical improvement.
What did we find? I agree that we found that we can use Opus-level AIs to faster explore mundane alignment ideas, and select promising candidates for mundane alignment of weaker models. That’s good, and hypothesis generation seems like a good use case, but it isn’t anything like an AAR.
Most importantly: If you want to do true weak-to-strong supervision, you can’t configure that using a stronger supervisor, or the whole thing very obviously doesn’t apply when it counts. That’s cheating.
The Anthropic blackmail paper made the Twitter rounds again, so David Sacks decided to respond, calling it ‘The Anthropic Blackmail Hoax’ claiming it had been ‘debunked’ because the situation was engineered as a demonstration of what was possible rather than being something to commonly happen on its own (which the paper was very clear on) and asking why we have not seen examples in the wild.
One reason we love Twitter is people can answer your rhetorical questions, and sometimes the answer isn’t what you assumed it would be.
David Sacks: One question to ask, now that a year has passed, is whether we have seen any examples of the lab behavior in the wild? No, we haven’t, even though AI is much more widely adopted and more models are available.
Aengus Lynch: Thanks for engaging with the paper, @DavidSacks . I’m the first author.
I agree that at the time of release this was a theoretical concern. The scenarios were constructed, and we said so in the paper. The value of the demo is an early warning about behaviours we expected would emerge as models became i) more capable and ii) more widely deployed in agentic settings.
I’m publishing a broader write-up of observed misalignments across frontier models in the next couple of weeks, and I’m happy to discuss.
The best way to deal with future potential dangers, in these situations, is:
You consider a possible future danger, while it is still impossible.
When it becomes possible at all, you demonstrate that it is possible.
You warn people so they can watch out for it happening.
When something like this does start to happen, you notice it.
Maybe you do something more than that, and maybe you don’t.
Ryan Greenblatt is concerned that by drawing attention to scheming during training, inoculation prompting could long run increase the risk of models learning scheming.
Judd Rosenblatt shares threenewpapers, showing that language models do internal calculations that they don’t verbalize, and they can explain them to you if you ask, and do so better than our supervision labels.
Aligning a Current Model For Mundane Tasks Is Also Difficult
Anthropic and OpenAI and most AI people broadly say yes, the systems are pretty aligned. Some make the mistake of treating this as meaning a lot more than it does, but most purely mean that as a practical matter, the AI does what you want it to do.
I do agree that, as a practical matter, alignment has been improving, especially for straightforward everyday tasks. You should have seen them before.
But Ryan Greenblatt makes the excellent point that, for hard coding and engineering tasks that are difficult to check, the AIs still tend to try and cheat and hide that they’ve cheated, to aim to look like they have done good work rather than aiming to actually do good work. There is what he calls apparent-success-seeking.
That’s a rather terrible sign. Imaging noticing humans doing this. Think about how you would interpret this pattern of behavior.
I agree with Ryan that while this isn’t some deliberate scheme, the scaled up version of this problem would likely become fatal, and I think he does a good job in the second section of his post explaining why.
Everyone Is Confused About AI Consciousness
The ground truth of whether AI is conscious, for various of the definitions of conscious, is not going to be so correlated to the way we actually view future AIs. Henry Shevlin offers an essay on this and points us to a trove of similar others.
Henry Shevlin: For better or worse, I expect it to become a common attitude among the general public that at least some AI systems are conscious and warrant moral and perhaps even legal protection.
This will happen not because of dazzling new insights in consciousness science or even machine learning (though I am hopeful for at least one of these things).
Instead, it will come about through a combination of our intensely anthropomorphising minds and the promulgation of humanlike (or anthropomimetic) AI systems to whom we will relate, bond, and identify.
This essay is my attempt to tell that story, and assess whether such a behaviourist resolution of consciousness debate would be a good thing, or instead an epistemic or moral catastrophe.
I find such essays difficult to force my way through, as they seem to belabor the point, and at core seem to focus on avoiding ‘errors’ in theoretical senses while ultimately largely dodging the question of what exactly we actually care about in all this.
The core prediction, however, seems correct and relevant. The public is not going to much care about philosophers or ethicist arguments. I too predict the public are going to talk to the models and effectively take a largely behaviorist view, and not be of much mind to listen to such self-proclaimed ‘experts,’ even more so than they reject experts in other areas as these experts cannot prove their expertise through results.
Thus we should be prepared for that, and for it to be one of the ways in which our ability to maintain control over the future is going to risk being taken away, whether or not the underlying justification is correct.
If you are going to create minds that will be treated as moral patients, then you need to take the consequences of that in mind when deciding whether and how to create such minds, whether or not you think those minds would be moral patients.
There are many similar situations in history between humans, at scales both large and small, national and personal and everything in between, where something could be a win-win deal, except that the resulting situation will predictably not be sustainable, or especially seen as morally unacceptable, or one side will get the power to alter the deal and likely will therefore fail to honor it - and once that is taken into account, the result is no longer a win-win.
Alas, if we lack sufficiently strong commitment devices, one cannot successfully make such deals. It is much better to realize this in advance.
Kateryna Lisunova: ZELENSKYY: For the first time in the war, an enemy position was captured entirely by ground robotic systems and drones - without any infantry. A robot entered the most dangerous zones instead of a soldier and took the positions.
«The future is here, on the battlefield, and Ukraine is creating it. These are our ground robotic systems. For the first time in this war's history, an enemy position was taken exclusively by unmanned GRS platforms and drones.
The occupiers surrendered, and this operation was completed without infantry involvement and without losses on our side. Ratel, Termite, Ardal, Lynx, Zmiy, Protector, Volya and other GRS completed over 22 000 missions at the front in just 3 months. In other words, over 22 000 times lives were saved. A robot went into the most dangerous zones instead of a soldier» - Zelenskyy’s address to the workers of Ukraine’s defense-industrial complex. April 13th, 2026.
We can be happy that Ukraine is succeeding.
But imagine how you would feel if Putin announced this rather than Zelenskyy.
As I discussed in Mythos #2, I actively downgrade anyone who mocks the caution that was shown around GPT-2 and especially anyone who uses it as a way to attack caution displayed in other settings.
Tenobrus: apparently reached the point of shilling anthropic so hard now when claude does crazy impressive things my friends just text me about it
@full_kelly_: To be fair I'm not particularly sure you need Claude's level of intelligence to spot $5000 of credit card fraud. Like Mistral could probably find that
The important thing is that Claude motivates you to show the statements.
"The reason we don’t advocate personal attacks and fan the flames of antagonism is not because it doesn’t support our goals but because we believe this approach to be unethical. We are striving to build a community that builds trust and dialogue among all stakeholders."
>The suspect in the first attack on Altman has been charged with murder
Missing 'attempted', typos normally no issue but I had to click through to confirm there wasn't a big new development with an earlier murder or something.
I want to make something clear here: PauseAI's strategy is not, and has never been, to "use rhetoric like 'murderer' and 'evil'".
See our latest article about this topic here: https://pauseai.substack.com/p/respectful-dialogue-will-create-the
And my latest statement: https://x.com/FournesMaxime/status/2044063173238526139
"The reason we don’t advocate personal attacks and fan the flames of antagonism is not because it doesn’t support our goals but because we believe this approach to be unethical. We are striving to build a community that builds trust and dialogue among all stakeholders."
>The suspect in the first attack on Altman has been charged with murder
Missing 'attempted', typos normally no issue but I had to click through to confirm there wasn't a big new development with an earlier murder or something.