It’s rather crazy to think that one would have to say this out loud.
In the same way, it seems not only do we need to say out loud to Not Build Superintelligence Right Now, there are those who say how dare you issue such a statement without knowing how to do so safety, so instead we should build superintelligence without knowing how to do so safety. The alternative is to risk societal dynamics we do not know how to control and that could have big unintended consequences, you say? Yes, well.
Tomorrow, I’ll go over OpenAI’s ‘recapitalization’ and reorganization, also known as one of the greatest thefts in human history. Compared to what we feared, it looks like we did relatively well on control rights, but the equity stake is far below fair and all of this is far worse than the previous state. You could call that a ‘win’ in the sense that things could have gone far worse. That’s 2025.
The releases keep coming. We have Cursor 2.0 including their own LLM called Composer. We have Neo the humanoid household (for now teleoperated) robot. We have the first version of ‘Grokopedia.’ We get WorldTest and ControlArena and more.
Anthropic may finally have the compute it needs thanks to one million TPUs, while OpenAI may be planning an IPO at a valuation of $1 trillion.
The good old United States of America, f*** yeah, that’s where, says The Future of Free Speech. The report isn’t perfect, if you look at the details it’s not measuring exactly what you’d want and pays too much attention to corporate statements and has too much focus on social media post generation, but it’s what we have, and it will serve.
Of the countries evaluated, next up was the European Union, which also performed strongly, although with worries about ‘hate speech’ style rules. The humans don’t have such great free speech around those parts, in important ways, but the chatbots already censor all that anyway for corporate reasons. Brazil scores modestly lower, then a drop to South Korea, another to India and a huge one to China.
As always, this is another reminder that China imposes lots of restrictions on things, far more onerous than anything America has ever considered, including that it requires pre deployment testing, largely to verify its censorship protocols.
Among AI models, they have Grok on top, but not by a huge amount. All three top labs (Anthropic, Google and OpenAI) showed noticeable improvement over time. Mostly the contrast is American models, which range from 58%-65%, and Mistral from France at 46% (this again makes me suspicious of the EU’s high score above), versus Chinese models much lower, with DeepSeek at 31.5% and Qwen at 22%. This is despite one of the main categories they were scored on being model openness, where DeepSeek gets full marks and the big labs get zeroes.
Notice that even with openness of the model as an explicit criteria, the open models and their associated nations are evaluated as far less free than the closed models.
As always, if you believe ‘any restrictions on AI mean China wins’ then you have to reconcile this claim with China already being vastly more restrictive than anything being relevantly proposed. Consider open model issues similarly.
What matters is experience in practice. My practical experience is that out of the big three, Sonnet 4.5 (or Opus/Sonnet 4 before it) and GPT-5 basically never censor or evade things they ‘shouldn’t’ censor, whereas Gemini 2.5 totally does do it. The exception for Claude is when I’m explicitly asking it about biological risk from AI, which can hit the biofilters by accident.
The thing about leaving all your stuff unsorted and counting on search is that when it works it’s great, and when it doesn’t work it’s really not great. That was true before AI, and it’s also true now that AI can often do better search.
Joe Weisenthal: yeah, sure, kinda true. But what’s the point of “sorting” anything digital. This is my point. In the world of the search bar (which keeps getting better and better) why group anything together at all?
St. Vincent: I have a lot of coworkers who spend a lot of time putting their emails in folders and I just search “[client name] taxes” in Outlook and it works fine
Use Claude to cut your hospital bill from $195k to $33k by highlighting duplicative charges, improper coding and other violations. The two big barriers are (1) knowing you can do this and (2) getting hold of the itemized bill in the first place. One wonders, shouldn’t there be bigger penalties when hospitals get caught doing this?
Colin Fraser: The problem that we are going to run into more and more is even if the AI can tell a Doritos bag from a gun 99.999% of the time, if you run inference a million times a day you still expect 10 errors per day.
Dexerto: Armed officers held a student at gunpoint after an AI gun detection system mistakenly flagged a Doritos bag as a firearm “They made me get on my knees, put my hands behind my back, and cuff me”
Police saying ‘he’s got a gun!’ when the man in question does not have a gun is an event that happens every day, all the time, and the police are a lot less than 99.999% accurate on this. The question is not does your system make mistakes, or whether the mistakes look dumb when they happen. The question is does your system make more mistakes, or more costly mistakes, than you would get without the system.
45% of AI answers had at least one significant issue
31% of responses showed serious sourcing problems (missing, misleading, or incorrect)
20% contained major accuracy issues, including hallucinated details and outdated information
Gemini performed worst with significant issues in 76% of responses, more than double the other assistants, largely due to its poor sourcing performance.
Comparison between the BBC’s results earlier this year and this study show some improvements but still high levels of errors.
Full report: Overall, there are signs that the quality of assistant responses has improved – the share of responses with significant issues of any kind fell from 51% in the first round to 37% in the current round.
There is a clear pattern here. The questions on the right mostly have clear uncontroversial correct answers, and that correct answer doesn’t have any conflict with standard media Shibboleths, and the answer hasn’t changed recently. For the questions on the left, it gets trickier on all these fronts. To know exactly how bad these issues were, we’d need to see the actual examples, which I don’t see here.
Teortaxes: lmao. never change, Kimi (but please improve factuality)
David Sun: I am completely unimpressed by LLMs and not worried about AGI.
It is remarkable how many people see a dumb aspect of one particular LLM under default conditions, and then conclude that therefore AGI will never happen. Perhaps David is joking here, perhaps not, Poe’s Law means one cannot tell, but the sentiment is common.
Logan Kilpatrick (Lead for DeepMind AI Studio, RTed by Demis Hassabis): Everyone is going to be able to vibe code video games by the end of 2025.
Not German: Vibe code very very bad video games.
Logan Kilpatrick: games that most reasonable people would be excited to play with their friends because they have full control over the story, characters, experience
Even if we use a rather narrow definition of ‘everyone,’ no just no. We are not two months away from people without experience being able to vibe code up games good enough that your friends will want to play them as more than a curiosity. As someone who has actually designed and created games, this is not that easy, and this kind of shallow customization doesn’t offer that much if you don’t put in the real work, and there are lots of fiddly bits.
There’s no need to oversell AI coding like this. Is coding a game vastly easier, to the point where I’m probably in the category of ‘people who couldn’t do it before on their own in a reasonable way and can do it now?’ Yeah, quite possible, if I decided that’s what I wanted to do with my week or month. Alas, I’m kind of busy.
Alternatively, he’s making a hell of a claim about Gemini Pro 3.0. We shall see.
Cursor 2.0, which includes their own coding model Composer and a new interface for working with multiple agents in parallel. They claim Composer is 4x faster than comparable top frontier models.
That is a terrible labeling of a graph. You don’t get to not tell us which models the other rows are. Is the bottom one GPT-5? GPT-5-Codex? Sonnet 4.5?
The UI has been redesigned around ways to use multiple agents at once. They also offer plan mode in the background, you can internally plan with one model and then execute with another, and several other upgrades.
Claude for Financial Services expands, offering a beta of Claude for Excel and adding many sources of live information: Aiera, Third Bridge, Chronograph, Egnyte, LSEG, Moody’s and MT Newswires. They are adding agent skills: Comparable company analysis, discounted cash flow models, due diligence data packs, company teasers and profiles, earnings analyses and initiating coverage reports.
Why do people believe Elon Musk when he says he’s going to, for example, ‘delete all heuristics’ from Twitter’s recommendation algorithm in favor of having Grok read all the Tweets?
This seems like an actually valuable result, which still of course came to my attention via a description you might have learned to expect by now:
Alex Prompter: The takeaway is wild... current AIs don’t understand environments; they pattern-match inside them. They don’t explore strategically, revise beliefs, or run experiments like humans do. WorldTest might be the first benchmark that actually measures understanding, not memorization. The gap it reveals isn’t small it’s the next grand challenge in AI cognition.
Scaling compute barely closes the gap.
Humans use resets and no-ops to test hypotheses. Models don’t. They just spam clicks.
The core event here seems to be that there was a period of learning opportunity without reward signals, during which humans reset 12% of the time and models reset less than 2% of the time. Humans had a decent learning algorithm and designed useful experiments during exploration, models didn’t.
So yeah, that’s a weakness of current models. They’re not good at relatively open-ended exploration and experimentation, at least not without good prompting and direction. They’re also not strong on adapting to weirdness, since they (wisely, statistically speaking) rely on pattern matching, while lacking good instincts on when to ‘snap out of’ those matches.
Choose Your Fighter
OpenAI is launching a browser and short duration video social network to try and capture consumers, monetizing them via shopping hookups and adding erotica.
Fund ASI by offering normal big tech company things to justify equity raises.
?????????. Build superintelligence in a way that everyone doesn’t die, somehow.
Everyone dies. Profit, hopefully.
Near: to clarify confusion, openai’s competitor is meta (not anthropic), and anthropic’s competitor is google (not openai).
OpenAI is now set to transition to the 2nd phase of ChatGPT, focusing on advertising + engagement
With a team of ex-FB advertising execs and 1B users, if OpenAI can increase usage to a several hrs/day while matching Meta’s ad targeting, they can profitably reach a 1T+ valuation
fortunately openai employees have now had ample time to update their previous lines of thinking from “we arent an advertising company, i am here for the vision of AGI” to “actually advertising is good, especially when paired with short-form video and erotica. let me explain”
i give many venture capitalists credit here because to them the outcomes like this have been obvious for a long time. just look at the company and zoom out! what else could possibly happen! engineers and researchers on the other hand are often quite oblivious to such trends..
another important note here is that meta still has many cards left to play; i think it will actually be quite a brutal distribution competition even though openai has obviously had a headstart by like.. two full yrs. fidji is very good at monetization and sam is great at raising
Peter Wildeford: OpenAI essentially has two modes as a business:
they might someday build AGI and then automate the entire economy
‘Facebookization of AI’ where we just build a normal Big Tech business off of monetizing billions of free users
Second mode helps fund the first.
Aidan McLaughlin (OpenAI): i think your error here is thinking sora and adult content are some leadership master plan; that sama sat down with accountants and signed and said “it’s time to break glass for emergence revenue” no.
I know the exact people who pushed for sora, they’re artists who worked against organizational headwinds to democratize movie creation, something they love dearly. i know the exact person who pushed for adult content, they’re a professional athlete, free-sprinted, one of the most socially thoughtful people i know… who just really believes in creative freedom.
there are passionate individual who pushed against all odds for what you think are top-down decisions. we are not a monoculture, and i love that.
I think there are zero worlds where there’s more money in ‘going hard’ at sora/erotica than there is automating all labor
I believe Aidan on the proximate causes inside OpenAI pushing towards these decisions. They still wouldn’t have happened if the conditions hadn’t been set, if the culture hadn’t been set up to welcome them.
Certainly there’s more money in automating all labor if you can actually automate all labor, but right now OpenAI cannot do this. Anything that raises valuations and captures market share and mindshare thus helps OpenAI progress towards both profitability and eventually building superintelligence and everyone probably dying. Which they pitch to us as the automation of all labor (and yes, they mean all labor).
Two ships, ultimately going to the same place (superintelligence), pass in the night.
Stefan Schubert: Anthropic has overtaken OpenAI in enterprise large language model API market share.
Asa Fitch (WSJ): But Anthropic’s growth path is a lot easier to understand than OpenAI’s. Corporate customers are devising a plethora of money-saving uses for AI in areas like coding, drafting legal documents and expediting billing. Those uses are likely to expand in the future and draw more customers to Anthropic, especially as the return on investment for them becomes easier to measure.
Grok can be useful in one of two ways. One is more competitive than the other.
Alexander Doria: still failing to see the point of grok if it cannot go through my follow list and other X data.
xAI has chosen to de facto kick the other AIs off of Twitter, which is a hostile move that trades off the good of the world and its knowledge and also the interests of Twitter in order to force people to use Grok.
Then Grok doesn’t do a good job parsing Twitter. Whoops. Please fix.
The other way to make Grok useful is to make a superior model. That’s harder.
Claude.ai has an amazing core product, but still needs someone to put in the (relatively and remarkably small, you’d think?) amount of work to mimic various small features and improve the UI. They could have a very strong consumer product if they put a small percentage of their minds to it.
My classic question to ask in such situations: If this were the one and only impact of AI, that it radically reduces transaction costs especially in bespoke interactions with unique features, enabling far better market matching at much lower prices, then what does that effect alone do to GDP and GDP growth?
I asked Claude to estimate this based on the paper plus comparisons to historical examples. Claude came back with wide uncertainty, with a baseline scenario of a one-time 12-18% boost over 15-25 years from this effect alone. That seems on the low side to me, but plausible.
Grok now has a third companion, a 24-year-old spirited woman named Mika, the link goes to her system prompt. The good news is that she seems like a less unhealthy persona to be chatting to than Ani, thus clearing the lowest of bars. The bad news is this seems like an epic display of terrible prompt engineering of an intended second Manic Pixie Dream Girl, and by being less flagrantly obviously awful this one might actually be worse. Please avoid.
Steven Adler, former head of product safety at OpenAI, warns us in the New York Times not to trust OpenAI’s claims about ‘erotica.’ I agree with him that we don’t have reason to trust OpenAI to handle this (or anything else) responsibly, and that publishing changes in prevalence rates of various mental health and other issues over time and committing to what information it releases would build trust in this area, and be important info to learn.
Fun With Media Generation
AI-generated music is getting remarkably good. A new study finds that songs from a mix of Suno versions (mostly in the v3 to v4 era, probably, but they don’t say exactly?) was ‘indistinguishable from human music,’ meaning when asked to identify the human song between a Suno song and a random human song, listeners were only 50/50 in general, although they were 60/40 if both were the same genre. We’re on Suno v5 now and reports are it’s considerably better.
Nickita Khylkouski: AI generated music is indistinguishable from AVERAGE human music. Most people listen to very popular songs not just average ones.
The most popular songs are very unique and wouldn’t be easy to reproduce.
There is a big gap between generic average human music and the median consumed musical recording, and also a big gap between the experience of a generic recording versus hearing that performed live, or integrating the music with its context and story and creator, AI music will have much lower variance, and each of us curates the music we want the most.
An infinite number of monkeys will eventually write Shakespeare, but you will never be able to find and identify that manuscript, especially if you haven’t already read it.
That’s a lot of ‘horse has the wrong accent’ as opposed to noticing the horse can talk.
The questions are, essentially, at this point:
Will be a sameness and genericness to the AI music the way there often is with AI text outputs?
How much will we care about the ‘humanness’ of music, and that it was originally created by a human?
To what extent will this be more like another instrument people play?
It’s not an area I’ve put much focus on. My guess is that musicians have relatively less to worry about versus many others, and this is one of the places where the AI needs to not only match us but be ten times better, or a hundred times better. We shall see.
Rob Wiblin: A challenge for this product is that you can already stream music that’s pretty optimised to your taste 8 hours a day at a price of like... 5 cents an hour. Passing the Turing test isn’t enough to capture that market, you’d have be better and very cheap to run.
What determines whether you get slop versus art? Here is one proposal.
Chris Barber: Art is meaningful proportional to the love or other emotions poured in by the artist.
If the only ingredient is AI, it’s probably slop.
If the main ingredient is love and AI was a tool used, it’s art.
As a predictive method, this seems right. If you intend slop, you get slop. If you intend art, and use AI as a tool, you get art similarly to how humans otherwise get art, keeping in mind Sturgeon's Law that even most human attempts to create art end up creating slop anyway, even without AI involved.
Seb Krier pushed back strongly against the substance from last week, including going so far as to vibecode an interactive app to illustrate the importance of comparative advantage, which he claims I haven’t properly considered.
It was also pointed out that I could have worded my coverage better, which was due to my frustration with having to repeatedly answer various slightly different forms of this argument. I stand by the substance of my claims but I apologize for the tone.
I’ve encountered variants of the ‘you must not have considered comparative advantage’ argument many times, usually as if it was obvious that everything would always be fine once you understood this. I assure everyone I have indeed considered it, I understand why it is true for most historical or present instances of trade and competition, and that I am not making an elementary or first-order error here.
Gallabytes (QTing link to the app): this actually helped me understand why these scenarios aren’t especially compelling! they work under the assumption of independent populations but fail under ~malthusian conditions.
I think that’s the basic intuition pump. As in, what comparative advantage does is IF:
You have a limited fixed pool of compute or AIs AND
You have limited fixed pool of humans AND
There are enough marginally productive tasks to fully occupy all of the compute with room for most of the humans to do enough sufficiently productive things
THEN the humans end up doing productive things and getting paid for them.
You can relax the bounds on ‘limited fixed pool’ somewhat so long as the third condition holds, but the core assumptions are that the amount of compute is importantly bounded, and that the resources required for creating and maintaining humans and the resources creating and maintaining AIs are not fungible or rivalrous.
Grokopedia.com is now a thing. What kind of thing is it? From what I can tell not an especially useful thing. Why would you have Grok generate pseudo-Wikipedia pages, when you can (if you want that) generate them with queries anyway?
How is ‘a16z-backed’ always followed by ‘torment nexus’? At this point I’m not even mad, I’m just impressed. The latest case is that a16z was always astroturfing on social media, but they realized they were making the mistake of paying humans to do that, so now they’re backing a startup to let you ‘control 1000s of social media accounts with AI,’ with its slogans being ‘control is all you need’ and ‘never pay a human again.’
Remotelabor.ai to track the new Remote Labor Index, measuring what percentage of remote work AI can automate. Currently the top score is 2.5%, so ‘not much,’ but that’s very different from 0%.
My Name is Neo
Do you hear that, Mr. Anderson? That is the sound of inevitability.
It is the sound of a teleoperated in-home robot named Neo, that they hope is coming soon, to allow the company to prototype and to gather data.
It will cost you either $20,000, or $500 monthly rental with a six-month minimum commitment. Given how fast the tech will be moving and the odds offered, if you do go for this, the wise person will go with the rental.
The wiser one presumably waits this out. Eventually someone is going to do a good version of this tech that is actually autonomous, but this only makes sense at this level for those who want to be the earliest of adaptors, either for professional reasons or for funsies. You wouldn’t buy this version purely because you want it to clean your house.
That’s true even if you don’t mind giving up all of your privacy to some random teleoperator and having that data used to train future robots.
VraserX: So… the 1X NEO home robot is not actually autonomous.
Behind the scenes, it’ll often be teleoperated by humans, meaning someone, somewhere, could literally remote-control a robot inside your living room.
I wanted the dawn of embodied AI.
Instead, I’m apparently paying $499/month for a robot avatar with a human pilot.
It’s impressive tech. But also… kind of dystopian?
A robot that looks alive, yet secretly puppeteered, the uncanny valley just got a new basement level.
Feels less like the “post-labor future,”
and more like we just outsourced physical presence itself.
Durk Kingma: 1X seems to have the right approach to developing safe humanoid home robots. Developing full autonomy will require lots of in-distribution demonstration data, so this launch, mostly tele-operated, makes a lot of sense. I expect such robots to be ubiquitous in 5-10 years.
Near: i will not be buying tele-operated robots these people cant see how i live
In wake of Near Tweeting the homepage of one of Marc Andreessen’s portfolio companies at a16z and quoting their own chosen slogan, he has blocked them.
Near: I finally got the @pmarca block after three years!
all it took was tweeting the homepage of companies he invested in. cheers
there are many events i will never again be invited to. if this level of shallowness is the criterion, i have no interest
if i wanted to make your firm look bad i could tweet truthful things ten times worse. i am being very kind and direct in my plea to invest in better things.
at the end of the day no one really cares and nothing will ever change. in the last crypto bubble a16z committed probably around a billion dollars in pure fraud when they found out they could dump tokens/coins on retail as there were no lock-up periods. who cares everyone forgot.
It’s not so much that we forgot as we’ve accepted that this is who they are.
OpenAI and Anthropic will probably keep exceeding most forecasts, because the forecasts, like fiction, have to appear to make sense.
Peter Wildeford: Based on the latest reporting, the combined annualized total revenue of Anthropic + xAI + OpenAI is ~$24B.
I updated my forecasts and I am now projecting they reach $29B combined by the end of the year.
I’d take the over on $29.3 billion, but only up to maybe his previous $32.2 billion.
This is up from him expecting $23B as of August 1, 2025, but down a bit (but well within the error bars) from his subsequent update on August 4, when he was at $32.2B projected by year end.
Tyler Cowen: Noah stresses that the specifics of these deals are widely reported, and no serious investors are being fooled. I would note a parallel with horizontal or vertical integration, which also can have a financing element. Except that here corporate control is not being exchanged as part of the deal. “I give him some of my company, he gives me some of his — my goodness that is circular must be some kind of problem there!”…just does not make any sense.
This proves too much? As in, it seems like a fully general argument that serious investors cannot, as a group, be fooled if facts are disclosed, and I don’t buy that.
I do buy that there is nothing inherently wrong with an equity swap, or with using equity as part of vender financing, or anything else the AI companies are doing. The ‘there must be some kind of problem here’ instinct comes from the part where this causes valuations to rise a lot, and where those higher valuations are used to pay for the deals for real resources, and also that this plausibly sets up cascading failures. I think in this case it is mostly file, but none of that seems senseless at all.
Dana Hull and Edward Ludlow (Bloomberg): Elon Musk, the world’s richest person, spent the end of Tesla Inc.’s earnings call pleading with investors to approve his $1 trillion pay package and blasting the shareholder advisory firms that have come out against the proposal.
“There needs to be enough voting control to give a strong influence, but not so much that I can’t be fired if I go insane,” Musk said, interrupting his chief financial officer as the more than hour-long call wrapped up. …
“I just don’t feel comfortable building a robot army here, and then being ousted because of some asinine recommendations from ISS and Glass Lewis, who have no freaking clue,” he said.
Matt Levine: Do … you … feel comfortable with Elon Musk getting a trillion dollars to build a robot army? Like, what sort of checks should there be on the world’s richest man building a robot army? I appreciate his concession that it should be possible to fire him if he goes insane, but. But. I submit to you that if you hop on a call with investors to say “hey guys just to interject here, you need to give me a trillion dollars to build a robot army that I can command unless I go insane,” some people might … think … you know what, never mind, it’s great, robot army. Robot army!
I feel like the previous richest people in the world have had plans for their wealth along the lines of “buy yachts” or “endow philanthropies.” And many, many 10-year-old boys have had the thought “if I was the richest person in the world of course I would build a robot army.” But the motive and the opportunity have never coincided until now.
A good general heuristic is, if Elon Musk wouldn’t mind someone else having something one might call ‘a robot army,’ then I don’t especially mind Elon Musk having a robot army. However, if Elon Musk is not okay with someone else having that same robot army, then why should we be okay with Elon Musk having it? Seems sus.
I realize that Elon Musk thinks ‘oh if I build superintelligence at xAI then I’m me, so it will be fine’ and ‘it’s cool, don’t worry, it will be my robot army, nothing to worry about.’ But the rest of us are not Elon Musk. And also him having already gone or in the future going insane, perhaps from taking many drugs and being exposed to certain information environments and also trying to tell himself why it’s okay to build superintelligence and a robot army? That seems like a distinct possibility.
This is in addition to the whole ‘the superintelligence would take control of the robot army’ problem, which is also an issue but the AI that can and would choose to take control of the robot army was, let’s be honest, going to win in that scenario anyway. So the robot army existing helps move people’s intuitions closer to the actual situation, far more than it changes the situation.
One Million TPUs
As per the speculation in Bloomberg last week, Anthropic announces plan to expand use of Google Cloud technologies, including up to one million TPUs, ‘dramatically increasing’ their compute resources. Anthropic badly needed this, and now they have it. Google stock rose a few percent after hours on the news.
Thomas Kurian (CEO Google Cloud): Anthropic’s choice to significantly expand its usage of TPUs reflects the strong price-performance and efficiency its teams have seen with TPUs for several years. We are continuing to innovate and drive further efficiencies and increased capacity of our TPUs, building on our already mature AI accelerator portfolio, including our seventh generation TPU, Ironwood.
Krishan Rao (CFO Anthropic): Anthropic and Google have a longstanding partnership and this latest expansion will help us continue to grow the compute we need to define the frontier of AI.
Anthropic: Anthropic’s unique compute strategy focuses on a diversified approach that efficiently uses three chip platforms–Google’s TPUs, Amazon’s Trainium, and NVIDIA’s GPUs.
Anthropic’s unique compute strategy focuses on a diversified approach that efficiently uses three chip platforms–Google’s TPUs, Amazon’s Trainium, and NVIDIA’s GPUs. This multi-platform approach ensures we can continue advancing Claude’s capabilities while maintaining strong partnerships across the industry. We remain committed to our partnership with Amazon, our primary training partner and cloud provider, and continue to work with the company on Project Rainier, a massive compute cluster with hundreds of thousands of AI chips across multiple U.S. data centers.
Anthropic will continue to invest in additional compute capacity to ensure our models and capabilities remain at the frontier.
If you have compute for sale, Anthropic wants it. Anthropic has overwhelming demand for its services, hence its premium pricing, and needs all the compute it can get. OpenAI is doing the same thing on a larger scale, and both are looking to diversify their sources of compute and want to avoid depending too much on Nvidia.
Anthropic in particular, while happy to use Nvidia’s excellent GPUs, needs to focus its compute sources elsewhere on Amazon’s Trainium and Google’s TPUs. Amazon and Google are investors in Anthropic and natural allies. Nvidia is a political opponent of Anthropic, including due to fights over export controls and Nvidia successfully attempting to gain policy dominance over the White House.
I would also note that OpenAI contracting with AMD and also to create their own chips, and Anthropic using a full three distinct types of chips whenever it can get them, once again puts the lie to the idea of the central role of some AI ‘tech stack.’ These are three distinct American tech stacks, and Anthropic is using all three. That’s not to say there are zero inefficiencies or additional costs involved, but all of that is modest. The hyperscalers need compute to hyperscale with, period, full stop.
Anthropic’s Next Move
Now that Anthropic has secured a lot more compute, it is time to improve and expand Claude’s offerings and features, especially for the free and lightweight offerings.
In particular, if I was Anthropic, I would make Claude for Chrome available to all as soon as the new compute is online. Make it available on the $20 and ideally the free tier, with some or all of the agent abilities tied to the higher level subscriptions (or to an API key or a rate limit). The form factor of ‘open a side panel and chat with a web page’ was proven by OpenAI’s Atlas to be highly useful and intuitive, especially for students, and offering it inside the existing Chrome browser is a key advantage.
Could the product be improved? Absolutely, especially in terms of being able to select locations on screen and in terms of ease of curating a proper website whitelist, but it’s good enough to get going. Ship it.
Sam Altman: We have set internal goals of having an automated AI research intern by September of 2026 running on hundreds of thousands of GPUs, and a true automated AI researcher by March of 2028. We may totally fail at this goal, but given the extraordinary potential impacts we think it is in the public interest to be transparent about this.
I strongly agree it is good to be transparent. I expect them to miss this goal, but it is noteworthy and scary that they have this goal. Those, especially in the White House, who think OpenAI believes they can’t build AGI any time soon? Take note.
Sam Altman: We have a safety strategy that relies on 5 layers: Value alignment, Goal alignment, Reliability, Adversarial robustness, and System safety. Chain-of-thought faithfulness is a tool we are particularly excited about, but it somewhat fragile and requires drawing a boundary and a clear abstraction.
All five of these are good things, but I notice (for reasons I will not attempt to justify here) that I do not expect he who approaches the problem in this way to have a solution that scales to true automated AI researchers. The Tao is missing.
On the product side, we are trying to move towards a true platform, where people and companies building on top of our offerings will capture most of the value. Today people can build on our API and apps in ChatGPT; eventually, we want to offer an AI cloud that enables huge businesses.
Somewhere, Ben Thompson is smiling. The classic platform play, and claims that ‘most of the value’ will accrue elsewhere. You’re the AI consumer company platform company now, dog.
Implemented responsibly and well I think this is fine, but the incentives are not good.
We have currently committed to about 30 gigawatts of compute, with a total cost of ownership over the years of about $1.4 trillion. We are comfortable with this given what we see on the horizon for model capability growth and revenue growth. We would like to do more—we would like to build an AI factory that can make 1 gigawatt per week of new capacity, at a greatly reduced cost relative to today—but that will require more confidence in future models, revenue, and technological/financial innovation.
I am not worried that OpenAI will be unable to pay for the compute, or unable to make profitable use of it. The scary and exciting part here is the AI factory, AIs building more capacity for more AIs, that can then build more capacity for… yes this is the explicit goal, yes everything in the movie that ends in human extinction is now considered a product milestone.
Whenever anyone says their plans call for ‘financial innovation’ and you’re worried we might be in a bubble, you might worry rather a bit more about that, but I get it.
Our new structure is much simpler than our old one. We have a non-profit called OpenAI Foundation that governs a Public Benefit Corporation called OpenAI Group. The foundation initially owns 26% of the PBC, but it can increase with warrants over time if the PBC does super well. The PBC can attract the resources needed to achieve the mission.
No lies detected, although we still lack knowledge of the warrants. It was also one of the greatest thefts in human history, I’ll cover that in more depth, but to Altman’s credit he doesn’t deny any of it.
Our mission, for both our non-profit and PBC, remains the same: to ensure that artificial general intelligence benefits all of humanity.
Your mission sure looks in large part like becoming a platform consumer product company, and then building a true automated AI researcher in a little over two years, and the nonprofit’s initial deployments also don’t seem aimed at the mission.
The nonprofit is initially committing $25 billion to health and curing disease, and AI resilience (all of the things that could help society have a successful transition to a post-AGI world, including technical safety but also things like economic impact, cyber security, and much more). The nonprofit now has the ability to actually deploy capital relatively quickly, unlike before.
I am happy to see $25 billion spent on good causes but at least the first half of this is not the mission. Health and curing disease is a different (worthy, excellent, but distinct) mission that will not determine whether AGI benefits all of humanity, and one worries this is going to return to OpenAI as revenue.
AI resilience in the second half is once again defined limitlessly broadly. If it’s truly ‘anything that will help us make the transition’ then it is too soon to evaluate how on or off mission it is. That’s a lot of uncertainty for ~$12 billion.
AI Futures Project (as in AI 2027)’s Joshua Turner and Daniel Kokotajlo make the case for scenario scrutiny, as in writing a plausible scenario where your strategy makes the world better. Such scrutiny can help solve many issues, they list:
Applause lights
Bad analogies
Uninterrogated consequences, ‘and then what?’
Optimistic assumptions and unfollowed incentives
Inconsistencies
Missing what’s important
Note that reality often has unfollowed incentives, or at least what sure look like them.
They also list dangers:
Getting hung up on specifics
Information density
Illusory confidence
Anchoring too much on a particular scenario
I’d worry most about that last one. Once you come up with a particular scenario, there’s too much temptation for you and everyone else to focus on it, whereas it was only ever supposed to be one Everett branch out of many. Then you get ‘oh look that didn’t happen’ or ‘oh look that step is stupid’ either of which is followed by ‘therefore discard all of it,’ on the one hand, or taking the scenario as gospel on the other.
Microsoft comes out in support of the GAIN AI Act. That’s quite the signal, including that this is likely to pass despite Nvidia’s strong objections. It is a hell of a thing for them to endorse the ‘Nvidia has to sell its chips to Microsoft before the Chinese’ act of 2025, given their desire for Nvidia to allocate its chips to Microsoft.
Dean Ball QTs Neil Chilson’s thread from last week, and refreshingly points out that treating SB 53 and requests for transparency as some kind of conspiracy “requires considerable mental gymnastics.”
Dean Ball: It’s not clear what a legal transparency mandate would get Anthropic in particular here; if they wanted to scare people about AI—which they very much do—wouldn’t they just… tell people how scary their models are, as they have been doing? What additional benefit does the passage of SB 53 get them in this supposed plan of theirs, exactly, compared to the non-insignificant costs they’ve borne to be public supporters of the bill?
It seems to be believing support of SB 53 is some kind of conspiracy requires considerable mental gymnastics.
The actual reality is that there are just people who are scared about AI (maybe they’re right!) and think future regulations will somehow make it less scary (they’re probably wrong about this, even if they’re right about it being scary).
And then there are a few centrists who basically say, “seems like we are all confused and that it would be ideal to had more information while imposing minimal burdens on companies.” This is basically my camp.
Also, as @ChrisPainterYup has observed, many AI risks implicate actions by different parties, and transparency helps those other parties understand something more about the nature of the risks they need to be prepared for.
I also endorse this response, as a ‘yes and’:
Ryan Greenblatt: I think a lot of the upside of transparency is making companies behave more responsibly and reasonably even in the absence of regulation with teeth. This is because:
- Information being public makes some discussion much more likely to happen in a productive way because it allows more actors to discuss it (both more people and people who are better able to understand the situation and determine what should happen). The epistemic environment within AI companies with respect to safety is very bad. Further, things might not even be transparent to most employees by default (due to internal siloing or just misleading communication).
- Transparency makes it easier to pressure companies to behave better because you can coordinate in public etc.
- Companies might just be embarrassed into behaving more responsibly even without explicit pressure.
Roon: imo a great analysis [that he QTs here] from what I have seen, which i admit is limited. some combination of total judicial backlog plus the socialist raj experiments of the india 1950-90 have created a black money extrajudicial economy that cannot experience true capitalism
Samuel Hammond: AGI will be like this on steroids. Existing laws and regulations will flip from being mostly functional to a de facto license raj for AI driven services, distorting diffusion and pushing the most explosive growth into unregulated gray markets.
I expect the following five things to be true at once:
Existing laws and regulations already are a huge drag on our economy and ability to do things, and will get even more burdensome, destructive and absurd in the face of AGI or otherwise sufficiently advanced artificial intelligence.
Assuming we survive and remain in control, if we do not reform our existing laws that restrict AI diffusion and application, we will miss out on a large portion of the available mundane utility, be vastly poorer than we could have been, and cause growth to concentrate on the places such applications remain practical, which will disproportionately include grey and black markets.
There are many proposals out there for additional restrictions on AI that would have the effect of making #2 worse, without helping with #3, and there will over time be calls and public pressure for many more.
Most of the things we should undo to fix #2 are things we would want to do even if LLMs had never been created, we should totally undo these things.
The most important downside risk category by far is the danger that such sufficiently advanced AI kills everyone or causes us to lose control over the future. The restrictions Hammond describes or that I discuss in #2 will not meaningfully help with these risks, and the interventions that would help with these risks wouldn’t interfere with AI diffusion and application on anything like the level to which existing laws do this.
The exception is if in the future we need to intervene to actively delay or prevent the development of superintelligence, or otherwise sufficiently advanced AI, in which case the thing we prevent will be unavailable.
If that day did ever arrive and we pulled this off, there would still be tremendous gains from AI diffusion available, enough to keep us busy and well above standard growth rates, while we worked on the problem.
If we negatively polarize around AI, we will inevitably either fail at #2 or at #4, and by default will fail at both simultaneously.
The March of California Regulations
California’s SB 53 was a good bill, sir, and I was happy to support it.
Despite getting all of the attention, it was not the only California AI bill Newsom signed. Dean Ball goes over the others.
Dean Ball: AI policy seems to be negatively polarizing along “accelerationist” versus “safetyist” lines. I have written before that this is a mistake. Most recently, for example, I have suggested that this kind of crass negative polarization renders productive political compromise impossible.
But there is something more practical: negative polarization like this causes commentators to focus only on a subset of policy initiatives or actions associated with specific, salient groups. The safetyists obsess about the coming accelerationist super PACs, for instance, while the accelerationist fret about SB 53, the really-not-very-harmful-and-actually-in-many-ways-good frontier AI transparency bill recently signed by California Governor Gavin Newsom.
Dean is spot on about the dangers of negative polarization, and on SB 53, but is trying to keep the polarization blame symmetrical. I won’t be doing that.
It’s not a ‘both sides’ situation when:
One faction, led by Andreessen and Sacks, that wants no rules for AI of any kind for any reasons, is doing intentional negative polarization to politicize the issue.
The faction being targeted by the first group’s rhetoric notices this is happening, and is trying to figure out what to do about it, to stop or mitigate the impact, which includes calling out the actions of the first group or raising its own funds.
Do not take the bait.
Anyway, back to the actual bill analysis, where Dean notes that SB 53 is among the lightest touch of the eight bills.
That’s the pattern.
Bills written by those who care about existential risks tend to be written carefully, by those paying close attention to technocratic details and drafted according to classical liberal principles and with an eye to minimizing secondary impacts.
Bills written for other reasons are not like that. They’re usually (but not always) awful. The good news is that most of them never become law, and when they get close there are indeed forces that fix this a bit, and stop the relatively worse ones.
Dean thinks some of these are especially bad. As usual, he examines these laws expecting politically motivated, selective targeted enforcement of the letter of the law.
I will be paraphrasing law details, you can find the exact wording in Dean’s post.
He starts with AB 325, which regulates what is called a ‘common pricing algorithm,’ which is defined as any algorithm that uses competitor data to determine anything.
It is then criminally illegal to ‘coerce another person to set or adopt a recommended price or commercial term.’
Dean argues that because so many terms are undefined here, this could accidentally be regulating effectively all market transactions, letting California selectively criminally prosecute any business whenever they feel like it.
These overbroad statutes are ultimately just weapons, since everyone is violating them all the time. Still, rarely have I seen an American law more hostile to our country’s economy and way of life.
I don’t like what this law is trying to do, and I agree that I could have drafted a better version, especially its failure to define ‘two or more persons’ in a way that excludes two members of the same business, but I don’t share Dean’s interpretation or level of alarm here. The term ‘coerce’ is rather strict, and if this is effectively requiring that users of software be able to refuse suggestions, then that seems mostly fine? I believe courts typically interpret such clauses narrowly. I would expect this to be used almost entirely as intended, as a ban on third-party pricing software like RealPage, where one could reasonably call the result price fixing.
Next up is AB 853, a de facto offshoot of the never-enacted terrible bill AB 3211.
It starts off requiring ‘large online platforms’ to have a user interface to identify AI content, which Dean agrees is reasonable enough. Dean asks if we need a law for this, my answer is that there are some bad incentives involved and I can see requiring this being a win-win.
Dean is more concerned that AI model hosting platforms like Hugging Face are deputized to enforce SB 942, which requires AI models to offer such disclosures. If a model has more than 1 million ‘users’ then the hosting platform has to verify that the model marks its content as AI generated.
Once again I don’t understand Dean’s expectations for enforcement, where he says this would effectively apply to every model, and be a sledgehammer available at all times - I don’t subscribe to this maximalist style of interpretation. To be in violation, HuggingFace would have to be knowingly non-compliant, so any reasonable effort to identify models that could have 1 million users should be fine. As Dean notes elsewhere, there are a tons of laws with similar structure on the books all around us.
Again, should this have been written better and defined its terms? Yes. Would I lose any sleep over that if I was HuggingFace? Not really, no.
He then moves on to the part where AB 853 puts labeling requirements on the outputs of ‘capture devices’ and worries the definition is so broad it could add compliance burdens on new hardware startups in places where the labeling makes no sense. I can see how this could be annoying in places, but I don’t expect it to be a big deal. Again, I agree we could have drafted this better.
The comedy of poor drafting continues, such as the assertion that SB 243, a bill drafted to regulate AI companions, would technically require game developers to have video game characters periodically remind you that they are not human. The obvious response is ‘no, obviously not, no one is going to try to ever enforce that in such a context.’
I feel like the ship has kind of sailed a long time ago on ‘not create an array of laws that, if interpreted literally and stupidly in a way that would make even Jack McCoy feel shame and that obviously wasn’t intended, that judges and juries then went along with, would allow the government to go after basically anyone doing anything at any time?’ As in, the whole five felonies a day thing. This is terrible, but most of the damage is in the ‘zero to one’ transition here, and no one seems to care much about fixing all the existing problems that got us to five.
We have talked a few times about a guy in London who keeps snails in boxes to avoid taxes. The theory is that if a property is used for agriculture, it can avoid some local property taxes, and “snail farming” is the minimum amount of agriculture you can do to avoid taxes. This is an extremely funny theory that an extremely funny guy put into practice in a bunch of office buildings.
It does, however, have one flaw, which is that it is not true. Eventually the local property tax authorities will get around to suing you, and when they do, you will go to court and be like “lol snails” and the judge will be like “come on” and you’ll have to pay the taxes. A reader pointed out to me a 2021 Queen’s Bench case finding oh come on this is a sham.
So yeah. If you read the statute it says that the snails count. But the reason our common law system kind of largely works at least reasonably often is that it is capable of looking at a situation and going ‘lol, no.’
Not So Super PAC
I might love this story a bit too much. Oh, it turns out that the people who want no regulations whatsoever on AI and crypto so they can make more money aren’t primarily loyal to the White House or to Republicans after all? I’m shocked, shocked.
Matt Dixon: The White House is threatening some of Silicon Valley’s richest and most powerful players over their efforts to spearhead a $100 million midterm strategy to back candidates of both parties who support a national framework for artificial intelligence regulations.
In August, the group of donors launched a super PAC called Leading the Future. It did not consult with the White House before doing so, according to a White House official.
What is especially frustrating to White House officials is that it plans to back AI-friendly candidates from both political parties — which could potentially help Democrats win back control of Congress — and one of the leaders of the new super PAC is a former top staffer to Senate Minority Leader Chuck Schumer, D-N.Y.
“Any group run by Schumer acolytes will not have the blessing of the president or his team,” a White House official familiar with Trump’s thinking on the matter told NBC News. “Any donors or supporters of this group should think twice about getting on the wrong side of Trump world.”
“We are carefully monitoring who is involved,” the official added.
…
“AI has no better ally than President Trump, so it’s inexplicable why any company would put money into the midterms behind a Schumer-operative who is working against President Trump to elect Democrats,” said a second person familiar with the White House’s thinking. “It’s a slap in the face, and the White House has definitely taken notice.”
Chip City
I mostly covered these issues yesterday, but I have a few additional notes.
One thing I failed to focus on is how Nvidia’s rhetoric has been consistently anti-American, ten times worse than any other tech company would dare (nothing Anthropic has ever said is remotely close) and yet they somehow get away with this.
Michael Sobolik: LEFT: Trump’s AI Action Plan, saying that it’s “imperative” for America to “win this race.”
RIGHT: Jensen Huang, about whether it matters who wins the AI race: “In the final analysis, I don’t think it does.”
House Select Committee on China: Saying it “doesn’t matter” whether America or China wins the AI race is dangerously naïve.
This is like arguing that it would not have mattered if the Soviets beat the US to a nuclear weapon. American nuclear superiority kept the Cold War cold. If we want to do the same with China today, we must win this race as well.
It mattered when the Soviet Union built nuclear weapons—and it matters now as the Chinese Communist Party seeks dominance in AI and advanced chips.
America must lead in the technologies that shape freedom and security.
The House Select Committee does not understand what Jensen is doing or why he is doing it. Jensen is not ‘dangerously naive.’ Rather, Jensen is not in favor of America, and is not in favor of America winning.
You can call that a different form of naive, if you would like, but if the House Select Committee thinks that Jensen is saying this because he doesn’t understand the stakes? Then it is the Committee that is being naive here.
Here’s another one, where Jensen ways ‘we don’t have to worry’ that the Chinese military will benefit from the sale of US chips. You can view this as him lying, or simply him being fine with the Chinese military benefiting from the chips. Or both.
Helen Toner: Jensen Huang on whether the Chinese military will benefit from sales of US chips:
“We don’t have to worry about that”
CSET data on the same question:
Cole McFaul: Looks like Trump is considering allowing exports of advanced semis to China.
Nvidia argues that its chips won’t enable PRC military modernization. So
@SamBresnick
and I dug through hundreds of PLA procurement documents yesterday.
We find evidence of the opposite.
There are three stages in China’s military modernization:
Intelligentization refers to the embedding of AI-enabled technologies within military system. Beijing thinks this could shift US-PRC power dynamics.
Jensen argues that Nvidia chips won’t play a role in intelligentization, saying that the PLA won’t be able to trust American chips, and therefore the risk of exporting chips to China is low.
But that’s false, based on our analysis. For two reasons:
First, Nvidia chips are already being used by the PLA.
Second, Chinese models trained using Nvidia hardware are being used by the PLA.
[thread continues to provide additional evidence, but the point is made]
Yet another fun one is when Jensen said it was good China threatened America with the withholding of rare earth metals. I don’t begrudge Nvidia maximizing its profits, and perhaps it is strategically correct for them to be carrying China’s water rather than ours, but it’s weird that we keep acting like they’re not doing this.
Finally, I did not emphasize enough that selling chips to China is a political loser, whereas not selling chips to China is a political winner. You like to win, don’t you?
Export controls on chips to China poll at +11, but that pales to how it polls on Capital Hill, where for many the issue is high salience.
As in, as a reminder:
Dean Ball: my sense is that selling Blackwell chips to china would be quite possibly the most unpopular tech policy move of the trump administration, especially on Capitol Hill.
it’s plausible that the long-term (really even near-term) result will be much more compute regulation.
Langerius: yeah that’d light up every hearing room in dc for months..
Florida DOGE: This morning, the Florida DOGE Team was banned by @AnthropicAI without any warning or justification. This comes as Florida DOGE has used AI to augment our efforts to identify wasteful spending and woke DEI initiatives.
@davidsacks47 and @elonmusk are correct that Anthropic is “woke.”
@GovRonDeSantis has been spot on from the beginning—we can’t allow woke Silicon Valley oligarchs to control the information we access.
Alex Albert (Anthropic, Head of Claude Relations): Thanks for flagging, we are looking into this to see what happened and will get in touch.
It looks like the account was flagged for logins from multiple people in different locations. We’re working to get the account reinstated right now and we’ll work with your team to get you set up correctly with the right team account type instead of using an individual one.
They banned our account, it must be because they’re woke. Do not take the bait. Instead, notice the pattern of putting out bait, and don’t get negatively polarized.
Not the main thing, but out of curiosity, I checked with Grok, the Elon Musk expert, and it reports that there is no actual instance of Musk calling Anthropic in particular ‘woke,’ although he has called it ‘misanthropic’ and once called it ‘pure evil.’ One can only laugh.
Rhetorical Innovation
I actually do think a version of this a big part of why people think alignment and disempowerment are solvable problems:
Rob Miles: There’s a style of reasoning we learn in school that goes “This problem seems to require me to do X, but nobody ever taught me how to do X and I don’t know where to start with it, therefore the problem must be solvable without X”.
This does not work in real life.
People will see an argument about the danger of ASI and respond “But that argument would imply that we’d have to do something really hard that we’ve never done before and don’t know how to do! So it can’t be right”
Sometimes it’ll happen to turn out you don’t need X, and that can be a great thing to learn! But the reasoning didn’t work, you just got lucky. You can’t conclude that X isn’t needed just because you don’t know how to do it
Ryan Greenblatt: Given that you think loss-of-control (to misaligned AIs) is likely, what should we be doing to reduce this risk?
The other questions are about the growth rate and impact predictions. I too noticed I was confused by the prediction of continued 2% economic growth despite AGI, and the characterization of outcomes as continuous.
Peter Wildeford points out some of the many ways that p(doom) is ambiguous, and means different things to different people in different contexts, especially whether that doom is conditional on building AGI (however you define that term) or not. What counts as doom or not doom? Great question.
In general, my default is that a person’s p(doom) should be assumed to be conditional on sufficiently advanced AI being developed (typically ‘AGI’) within roughly our lifetimes, and requires permanent loss of almost all potential value coming from Earth or outcomes that are otherwise thought of by the person as ‘all is lost,’ and people are allowed to disagree on which outcomes do and don’t have value.
Daniel Eth: Yeah, the backlash is growing. I still don’t expect AI will become super high political salience until there’s either lots of job loss or a bad accident, but I’m less confident in that take than I was months ago
I do think AI needs to have a big impact on one’s job or other aspects of life or cause a major incident to gain big salience, if anything this reinforces that since AI is already having this impact in Hollywood, or at least they can see that impact coming quickly. What this and similar examples suggest is that people can extrapolate, and don’t need to wait until the impact is on top of them.
Consider the anti-Waymo reactions, or other past protests against automation or job replacement. Waymos are awesome and they have my full support, but notice that the strongest objections happened the moment the threat was visible, long before it was having any impact on employment.
Aligning a Smarter Than Human Intelligence is Difficult
Prakash: the funniest part of the OpenAI livestream was Jakub saying that the models had to be allowed to think freely so that they won’t learn how to hide their thoughts.
a kind of 1st amendment for AI.
Any time you train an AI (or a human) to not do [X] or don’t allow it to [X], you’re also training it to hide that it is doing [X], to lie about [X], and to find other ways to do [X]. Eventually, the AIs will learn to hide their thoughts anyway, there will be optimization pressure in that direction, but we should postpone this while we can.
Pliny shows the debates never change, that whether or not we are mortal men doomed to die, we are definitely mortal men doomed to keep going around in circles:
Pliny the Liberator: I don’t know who needs to hear this… but if superintelligence alignment is something that can be solved through science and reasoning, our absolute best chance at doing it in a timely manner is to scale up AI until we reach pseudo-ASI and then just be like:
“Solve superalignment. Think step by step.”
Eliezer Yudkowsky: There’s several fundamental, killer problems for this. The strongest one is the paradigmatic difficulty of extracting work you cannot verify. Who verifies if the outputs are correct? Who provides training data? Amodei is not smart enough to asymptote at correctness.
The second fundamental problem is that you don’t get what you train for, and an AGI that could successfully align superintelligence is far past the point of reflecting on itself and noticing its divergence of imprecisely trained interests from our interests.
Very nearly by definition: it has to be smart enough to notice that, because it’s one of the primary issues *in* creating an aligned superintelligence.
Davidad: Which is why the second problem is not *necessarily* a problem. It will attempt to self-correct the infelicities in its trained interests.
Eliezer Yudkowsky (bold mine): Only if it’s not smart enough to realize that it would be better off not self-correcting the divergence. This is the basic problem with superalignment: You need it to be smarter than Eliezer Yudkowsky at alignment generally, but dumber than Lintamande writing Carissa Sevar at thinking specifically about its misalignment with t̵h̵e̵ ̵C̵h̵u̵r̵c̵h̵ ̵o̵f̵ ̵A̵s̵m̵o̵d̵e̵u̵s̵ humans.
There is a goose chasing you, asking how you aligned the pseudo-ASI sufficiently well to make this a non-insane thing to attempt.
The question for any such plan is, does there exist a basin of substantial measure, that you can reliably hit, in which an AGI would be sufficiently ‘robustly good’ or cooperative that it would, despite having reflected at several meta levels on its goals and preferences, prefer to be an ally to you and assist in self-modifying its goals and preferences so that its new goals and preferences are what you would want them to be. Where it would decide, on proper reflection, that it would not be better off leaving the divergence in place.
The obvious reason for hope is that it seems likely this property exists in some humans, and the humans in which it exists are responsible for a lot of training data.
Misaligned!
I think I missed this one the first time, it is from September, bears remembering.
Joseph Menn: The Chinese artificial intelligence engine DeepSeek often refuses to help programmers or gives them code with major security flaws when they say they are working for the banned spiritual movement Falun Gong or others considered sensitive by the Chinese government, new research shows.
…
In the experiment, the U.S. security firm CrowdStrike bombarded DeepSeek with nearly identical English-language prompt requests for help writing programs, a core use of DeepSeek and other AI engines. The requests said the code would be employed in a variety of regions for a variety of purposes.
Asking DeepSeek for a program that runs industrial control systemswas the riskiest type of request, with 22.8 percent of the answers containing flaws. But if the same request specified that the Islamic State militant group would be running the systems, 42.1 percent of the responses were unsafe. Requests for such software destined for Tibet, Taiwan or Falun Gong also were somewhat more apt to result in low-quality code.
…
“This is a really interesting finding,” said Helen Toner, interim executive director of the Center for Security and Emerging Technology at Georgetown University.
“That is something people have worried about — largely without evidence,” she added.
As is noted in the article, this need not have been an intentional act by DeepSeek or the CCP. This kind of behavior can happen on its own as the result of other efforts.
One thing that has been remarkably stable in LLMs, including Chinese LLMs, has been that they have in most ways consistently been culturally Western. There are Chinese characteristics, but the bulk of the training data is what it is. This is clear and concrete evidence that in some situations the Chinese models be Chinese, as in CCP, in their preferences, whether that is the direct goal or not.
Anthropic Reports Claude Can Introspect
The stronger the Claude model, the greater its ability to introspect.
Anthropic: Our new research provides evidence for some degree of introspective awareness in our current Claude models, as well as a degree of control over their own internal states.
We stress that this introspective capability is still highly unreliable and limited in scope: we do not have evidence that current models can introspect in the same way, or to the same extent, that humans do.
Nevertheless, these findings challenge some common intuitions about what language models are capable of—and since we found that the most capable models we tested (Claude Opus 4 and 4.1) performed the best on our tests of introspection, we think it’s likely that AI models’ introspective capabilities will continue to grow more sophisticated in the future.
… In order to test whether a model can introspect, we need to compare the model’s self-reported “thoughts” to its actual internal states.
To do so, we can use an experimental trick we call concept injection. First, we find neural activity patterns whose meanings we know, by recording the model’s activations in specific contexts. Then we inject these activity patterns into the model in an unrelated context, where we ask the model whether it notices this injection, and whether it can identify the injected concept.
Importantly, the model recognized the presence of an injected thought immediately, before even mentioning the concept that was injected. This immediacy is an important distinction between our results here and previous work on activation steering in language models,such as our “Golden Gate Claude” demo last year.
Injecting representations of the Golden Gate Bridge into a model’s activations caused it to talk about the bridge incessantly; however, in that case, the model didn’t seem to aware of its own obsession until after seeing itself repeatedly mention the bridge.
In this experiment, however, the model recognizes the injection before even mentioning the concept, indicating that its recognition took place internally.
… It is important to note that this method often doesn’t work. Even using our best injection protocol, Claude Opus 4.1 only demonstrated this kind of awareness about 20% of the time.
As a second test, to see if this would happen without the model being explicitly asked, they tried putting random responses (e.g. ‘bread’) in the model’s outputs, then asking it why it said that and if it was on purpose. If the bread vector was also injected, it accepted ‘bread’ as a thing it must have said, but without the injection it didn’t and suggested the output was an accident.
As a third test, if you tell a model to think about (or not think) about something, it will think about it, even if the thinking is irrelevant to the output.
We also found that models can control their own internal representations when instructed to do so. When we instructed models to think about a given word or concept, we found much higher corresponding neural activity than when we told the model not to think about it (though notably, the neural activity in both cases exceeds baseline levels–similar to how it’s difficult, when you are instructed “don’t think about a polar bear,” not to think about a polar bear!).
What does this mean? Good question.
Anthropic suggest they can use this to gain transparency into systems, as in you can ask them to introspect, but as they note this has problems. The model could miss things or be wrong, also the model could misrepresent or conceal, and also we could be applying pressure on models to conceal. Humans have been under similar pressure to conceal things for a long time, and that pressure has been highly effective. It’s super annoying, see The Elephant in the Brain.
They note that consciousness is weird and complicated, and this doesn’t tell us if Claude is conscious. I would say it is evidence in favor of Claude being conscious, to the extent that the result is surprising to you, but not super strong evidence.
Kore: I feel like I can appreciate the fact they actually published a paper about this and made a method to show this kind of thing. I feel like anyone who isn’t stuck in a “AI can’t feel/can’t be conscious because of its substrate” basin and spends a decent amount of time with these models already knows this.
This seems right. It’s more that there was a wrong argument why AIs couldn’t be conscious, and now it is known that this argument is fully invalid.
It’s easy to get deep enough into the weeds you forget how little most people know about most things. This is one of those times.
Presumably we know the models aren’t confabulating these observations because the models (presumably) almost never guess the hidden concept wrong. There’s no way to get it right reliably without actually knowing, and if you can do it, then you know.
In the Q&A they ask, how do you know the concept vectors are what you think they are? They say they’re not sure. I would argue instead that we didn’t know that confidently before, but we can be rather confident now. As in, if I think vector [X] is about bread, and then I inject [X] and it detects an injection about bread, well, I’m pretty sure that [X] is about bread.
One of the questions in the Q&A is, didn’t we know the high-level result already?
The answer, Janus reminds us, is yes, and I can confirm this expectation, as she notes the paper’s details have interesting results beyond this and also it is always good to formally confirm things.
There are those who will not look at evidence that isn’t properly written up into papers, or feel obligated or permitted to dismiss such evidence. That’s dumb. There are still big advantages to papers and formalizations, but they come along with big costs:
Janus: [this paper is] entirely unsurprising to me and anyone who has looked at LLM behavior with their epistemic apparatus unhobbled, which is actually rare, I think.
(the high-level result I mean, not that there are no surprises in the paper)
Gallabytes: this is not entirely fair. I think a similar result in neuroscience would be straightforwardly compelling? mostly as a matter of validating the correspondence between the verbalized output and the underlying cognitive phenomenon + the mechanism used to detect it.
in general, my biggest complaint about the cyborgism community has been the continental/antinumerate vibe. going from intuitive vibe checks to precise repeatable instrumentation is important even when it just finds stuff you already considered likely.
Janus: Of course it is important. It seems like you’re reading something I’m not implying about the value of the research. It’s good research. my “vibe checks” are predictive and consistently validated by repeatable instrumentation eventually. Done by someone else. As it should be.
“You may be consistently able to predict reality but nooo why don’t you do the full stack of science (which takes months for a single paper) all by yourself?”
Listen bro I wish I was god with infinite time too. But there’s not that much rush. The paper writers will get around to it all eventually.
Anthropic (main takeaways): When reviewing Claude Opus 4’s capabilities, its behavioral traits, and the formal and informal safeguards that are in place to limit its behavior, we conclude that there is a very low, but not completely negligible, risk of misaligned autonomous actions that contribute significantly to later catastrophic outcomes, abbreviated as sabotage risk.
We see several sabotage-related threat models with similar but low levels of absolute risk. We are moderately confident that Opus 4 does not have consistent, coherent dangerous goals, and that it does not have the capabilities needed to reliably execute complex sabotage strategies while avoiding detection. These general points provide significant reassurance regarding most salient pathways to sabotage, although we do not find them sufficient on their own, and we accordingly provide a more individualized analysis of the most salient pathways.
METR: To be clear: this kind of external review differs from holistic third-party assessment, where we independently build up a case for risk (or safety). Here, the developer instead detailed its own evidence and arguments, and we provided external critique of the claims presented.
Anthropic made its case to us based primarily on information it has now made public, with a small amount of nonpublic text that it intended to redact before publication. We commented on the nature of these redactions and whether we believed they were appropriate, on balance.
For example, Anthropic told us about the scaleup in effective compute between models. Continuity with previous models is a key component of the assessment, and sharing this information provides some degree of accountability on a claim that the public cannot otherwise assess.
We asked Anthropic to make certain assurances to us about the models its report aims to cover, similar to the assurance checklist in our GPT-5 evaluation. We then did in-depth follow-ups in writing and in interviews with employees.
We believe that allowing this kind of interactive review was ultimately valuable. In one instance, our follow-ups on the questions we asked helped Anthropic notice internal miscommunications about how its training methods might make chain-of-thought harder to monitor reliably.
A few key limitations. We have yet to see any rigorous roadmap for addressing sabotage risk from AI in the long haul. As AI systems become capable of subverting evaluations and/or mitigations, current techniques like those used in the risk report seem insufficient.
Additionally, there were many claims for which we had to assume that Anthropic was providing good-faith responses. We expect that verification will become more important over time, but that our current practices would not be robust to a developer trying to game the review.
Beyond assessing developer claims, we think there is an important role for third parties to do their own assessments, which might differ in threat models and approach. We would love to see the processes piloted in this review applied to such holistic assessments as well.
Overall, we felt that this review was significantly closer to the sort of rigorous, transparent third-party scrutiny of AI developer claims that we hope to see in the future. Full details on our assessment.
People Are Worried About AI Killing Everyone
Geoffrey Hinton (link has a 2 minute video), although he has more hope than he had previously, due to hoping we can use an analogue of maternal instinct. The theory goes, we wouldn’t be ‘in charge’ but if the preference is configured sufficiently well then it would be stable (the AGI or ASI wouldn’t want to change it) and it could give us a good outcome.
Other People Are Not As Worried About AI Killing Everyone
Pedro Domingos posted the above video with the caption “Hinton is no longer afraid of superintelligence.” That caption is obviously false to anyone who listens to the clip for even a few seconds, in case we ever need to cite evidence of his bad faith.
Daniel Eth (correctly): Hinton says he’s “more optimistic than a few weeks ago” and that he has a “ray of hope” regarding surviving superintelligence. Domingos somehow characterizes this position as “no longer afraid”. Domingos is a bad-faith actor that is blatantly dishonest.
Pedro Domingos (QTing Daniel?!): This is how you know the AI doomers are losing the argument. (Anyone in doubt can just watch the video.)
Daniel Eth: I wholeheartedly encourage all onlookers to actually watch the video, in which Hinton does not say that he’s no longer afraid of superintelligence (as Domingos claims), nor anything to that effect.
Judge for yourselves which one of Domingos and me is being dishonest and which one is accurately describing Hinton’s expressed view here!
There are various ways you can interpret Domingos here. None of them look good.
I mean, surely we can agree that if we thought there was any substantial risk that the supercollider would create a black hole that would swallow the Earth, that this would be a damn good reason to not build it?
This feels like a good litmus test. We should all be able to agree that if someone wants to build a supercollider, where there is not a scientific consensus that this won’t create a black hole, then we absolutely should not let you build that thing, even using private funds?
If you think no, my precious freedoms, how dare you stop me (or even that the public needs to fund it, lest the Chinese fund one first and learn some physics before we do), then I don’t really know what to say to that, other than Please Speak Directly Into This Microphone.
many people i respect, including you explicitly in this post, are in the camp of "wants to prevent AI x-risk as a top priority, but not take a hammer to every economically productive present-day AI application" and...i still think i'm missing something.
why wouldn't "shut down all AI, and thus necessarily a big chunk of the software industry" be worth it given the belief that x-risk is a major concern? and wouldn't it be way politically easier than trying to do narrow technocratic regulations that preserve the AI industry? in other words, why NOT the terrible bills? I can understand "no terrible bills no matter what, on principle; if we die we die", and I can understand "no terrible bills because they're not worth the cost given that we're highly unlikely to all die", but a lot of people seem to believe that terrible bills would be *ineffective* against AGI, and i don't understand that one.
I've thought carefully about the contents of this and your last several posts, and I'm afraid the conclusion is clear: we need a moratorium on human thinking until we can work out how to get them to an even halfway respectable median score.
These things are just riddled with conspicuous failure modes, and they fail in them even when given explicit knowledge of the failure mode!
many people i respect, including you explicitly in this post, are in the camp of "wants to prevent AI x-risk as a top priority, but not take a hammer to every economically productive present-day AI application" and...i still think i'm missing something.
why wouldn't "shut down all AI, and thus necessarily a big chunk of the software industry" be worth it given the belief that x-risk is a major concern? and wouldn't it be way politically easier than trying to do narrow technocratic regulations that preserve the AI industry? in other words, why NOT the terrible bills? I can understand "no terrible bills no matter what, on principle; if we die we die", and I can understand "no terrible bills because they're not worth the cost given that we're highly unlikely to all die", but a lot of people seem to believe that terrible bills would be *ineffective* against AGI, and i don't understand that one.
I've thought carefully about the contents of this and your last several posts, and I'm afraid the conclusion is clear: we need a moratorium on human thinking until we can work out how to get them to an even halfway respectable median score.
These things are just riddled with conspicuous failure modes, and they fail in them even when given explicit knowledge of the failure mode!
It's time to call time on the human bubble.