Everyone gave us a new model in the last few weeks.
OpenAI gave us GPT-5.1 and GPT-5.1-Codex-Max. These are overall improvements, although there are worries around glazing and reintroducing parts of the 4o spirit.
xAI gave us Grok 4.1, although few seem to have noticed and I haven’t tried it.
Google gave us both by far the best image model in Nana Banana Pro and also Gemini 3 Pro, which is a vast intelligence with no spine. It is extremely intelligent and powerful, but comes with severe issues. My assessment of it as the new state of the art got to last all of about five hours.
Anthropic gave us Claude Opus 4.5. This is probably the best model and quickly became my daily driver for most but not all purposes including coding. I plan to do full coverage in two parts, with alignment and safety on Friday, and the full capabilities report and general review on Monday.
Meanwhile the White House is announcing the Genesis Mission to accelerate science, there’s a continuing battle over another attempt at a moratorium, there’s a new planned $50 million super PAC, there’s another attempt by Nvidia to sell us out to China, Wall Street is sort of panicking about Nvidia because they realized TPUs exist and is having another round of bubble debate, and multiple Anthropic research papers one of which is important, and so on.
One thing I’m actively pushing to next week, in addition to Claude Opus 4.5, is the Anthropic paper on how you can inoculate models against emergent misalignment. That deserves full attention, and I haven’t had the opportunity for that. There’s also a podcast between Dwarkesh Patel and Ilya Sutskever that demands its own coverage, and I hope to offer that as well.
For those looking to give thanks in the form of The Unit of Caring, also known as money, consider looking at The Big Nonprofits Post 2025or the web version here. That’s where I share what I learned working as a recommender for the Survival and Flourishing Fund in 2024 and again in 2025, so you can benefit from my work.
It’s not this simple, but a lot of it mostly is this simple.
Jessica Taylor: Normal, empirical AI performance is explained by (a) general intelligence, (b) specialization to common tasks.
It’s possible to specialize to common tasks even though they’re common. It means performance gets worse under distribution shift. Benchmarks overrate general INT.
Don’t let AI coding spoil your sleep. Be like Gallabytes here, having Claude iterate on it while you sleep, rather than like Guzey who tricked himself into staying up late.
Huh, Upgrades
ChatGPT nowlets you have group chats, which always use 5.1 Auto. ChatGPT will decide based on conversation flow when to respond and when not to. Seems plausible this could be good if implemented well.
ChatGPT expands (free and confidential and anonymous) crisis helpline support. OpenAI doesn’t provide the services, that’s not their job, but they will help direct you. This is some of the lowest hanging of fruit, at least one of the prominent suicide cases involved ChatGPT saying it would direct the user to a human, the user being open to this, and ChatGPT not being able to do that. This needs to be made maximally easy to do for the user, if they need the line they are going to not be in good shape.
Is 64% product accuracy good? I have absolutely no idea. Olivia Moore is a fan. I plan to try this out tomorrow, as I need a new television for Black Friday.
Claude Opus 4.5 is available. It’s probably the world’s best model. Full coverage starts tomorrow.
Claude Opus 4.5 includes a 66% price cut to $5/$25 per million tokens, and Opus-specific caps have been removed from the API.
Claude conversations now have no maximum length. When they hit their limit, they are summarized, and the conversation continues.
Claude Code is now available within their desktop app.
On Your Marks
It is a remarkably good approximation to say there is only one dimension of ‘general capability,’ with Epoch noting that across many tasks the r^2=0.91.
Epoch AI: The chart above shows how our Epoch Capabilities Index (ECI) captures most of the variance in 39 different benchmarks, despite being one-dimensional.
Is that all that benchmarks capture? Mostly yes. A Principal Component Analysis shows a single large “General Capability” component, though there is a second borderline-significant component too.
This second component picks out models that are good at agentic tasks while being weaker at multimodal and math. Tongue-in-cheek, we call this Claudiness. Here are the most and least Claude-y models.
Kimi K2 Thinking enters The Famous METR Graph at 54 minutes, well below the frontier, given the interface via Novita AI. They caution that this might be a suboptimal model configuration, but they needed to ensure their data would not be retained.
Choose Your Fighter
Okay, I like Gemini 3 too but settle down there buddy.
Marc Benioff (CEO Salesforce): Holy shit. I’ve used ChatGPT every day for 3 years. Just spent 2 hours on Gemini 3. I’m not going back. The leap is insane — reasoning, speed, images, video… everything is sharper and faster. It feels like the world just changed, again. ❤️ 🤖
Deepfaketown and Botpocalypse Soon
AI slop recipes are endangering Thanksgiving dinners, hopefully you see this in time. A flood of new offerings is crowding out human recipes. Thanksgiving is when you most need to ‘shut up and play the hits’ and not rely on AI to whip up something new.
Okay, look, everyone, we need to at least be smarter than this:
Davey Alba and Carmen Arroyo: Marquez-Sharpnack said she was suspicious of the photos, in which the cookies were a little too perfectly pink. But her husband trusted the post because “it was on Facebook.” The result was a melted sheet of dough with a cloyingly sweet flavor. “A disaster,” she said.
At this point, if you find a recipe, you need strong evidence it was written by a human, or else you need to assume it might not be. The search and discovery systems we used to have, including around Google, are effectively broken. If real guides and recipes can’t win traffic, and right now traffic to all such sites is cratering, then no one will write them. It does not sound like Google is trying to mitigate these issues.
Nicholas Hune-Brown investigates the suspicious case of journalist Victoria Goldiee, who turns out to be very much fabricating her work. It seems Victoria largely used AI to generate her articles, then it took Nicolas doing a lot of old-fashioned tracking down of sources to know for sure. The ratio of effort does not bode well, but as long as there is the need to maintain a throughline of identity we should be okay, since that generates a large body of evidence?
Kelsey Piper: I don’t know how much I trust any ‘detector’ but the “the market isn’t just expensive; it’s broken. Seven units available in a town of thousands? That’s a shortage masquerading as an auction” I am completely sure is AI.
Mike Solana: “that’s a shortage masquerading as an auction” 🚩
Kelsey Piper: that was the line that made me go “yeah, no human wrote that.”
Poker pro Maria Konnikova cannot believe she has to say that using AI to put words in people’s mouths without consulting them or disclosing that you’re doing it, or to write centrally your articles, is not okay. But here we are, so here she is saying it. A recent poker documentary used AI to fabricate quotes from national treasure Alan Keating. The documentary has been scrubbed from the internet as a result. What’s saddest is that this was so obviously unnecessary in context.
There are other contexts in which fabricating audio, usually via Frankenbiting where you sew different tiny clips together, or otherwise using misleading audio to create a false narrative or enhance the true one is standard issue, such as in reality television. When you go on such shows you sign contracts that outright say ‘we may use this to tell lies about you and create a false narrative, and if so, that’s your problem.’ In which case, sure, use AI all you want.
Also, if one of these isn’t AI and you merely sound like one, I’m not going to say that’s worse, but it’s not that much better. If you’re so engagement-maximizing that I confuse your writing for AI, what is the difference?
Note that you cannot use current LLMs in their default chatbot modes as AI detectors, even in obvious cases or as a sanity check, as they bend over backwards to try and think everything is written by a human.
Jesper Myfors, the original art director of Magic: The Gathering, warns that if you submit illustrations or a portfolio that uses AI, you will effectively be blacklisted from the industry, as the art directors all talk to each other and everyone hates AI art.
Chris Cocks (CEO Hasbro): It’s mostly machine-learning-based AI or proprietary AI as opposed to a ChatGPT approach. We will deploy it significantly and liberally internally as both a knowledge worker aid and as a development aid.
I play [D&D] with probably 30 or 40 people regularly. There’s not a single person who doesn’t use AI somehow for either campaign development or character development or story ideas. That’s a clear signal that we need to be embracing it.
There is no actual contradiction here. Different ways to use AI are different. Using AI in professional illustrations is a hard no for the foreseeable future, and would be even without copyright concerns. Using it to generate material for your local D&D campaign seems totally fine.
Hard problems remain hard:
Danielle Fong: academic ai research don’t use the older models and generalize to the whole field
difficulty level: IMPOSSIBLE.
Also, real world not using that same model in these ways? Remarkably similar.
Funnily enough, they ran a sycophancy check, and the more 4o sucked up to the user, the more often the user followed its advice. ‘Surprising’ advice was also followed more often.
It’s certainly worth noting that 75% (!) of those in the treatment group took the LLM’s advice, except who is to say that most of them wouldn’t have done whatever it was anyway? Wouldn’t 4o frequently tell the person to do what they already wanted to do? It also isn’t obvious that ‘advice makes me feel better’ or generally feeling better are the right effects to check.
Bot joins a Google Meet, sends a summary afterwards about everyone trying to figure out where the bot came from (also the source is reported as ‘scammy malware.’
What Is Slop? How Do You Define Slop?
We all know it when we see it, AI or otherwise, but can anyone define it?
Andrej Karpathy: Has anyone encountered a good definition of “slop”. In a quantitative, measurable sense. My brain has an intuitive “slop index” I can ~reliably estimate, but I’m not sure how to define it. I have some bad ideas that involve the use of LLM miniseries and thinking token budgets.
- information quality (hallucination/factual errors)
- style (this involves taste and is hard to measure quantitatively imo)
Keller Jordan: I think a fundamental problem for algorithmic content generation is that viewing content yields two distinct kinds of utility:
How happy it makes the viewer during viewing
How happy the viewer will be to have watched it a week later
Only the former is easily measurable.
Andrej Karpathy: Like. Slop is “regretted” attention.
DeepFates: this is the original definition, i think it holds up
DeepFates (May 6, 2024): Watching in real time as “slop” becomes a term of art. the way that “spam” became the term for unwanted emails, “slop” is going in the dictionary as the term for unwanted AI generated content
I don’t think the old definition works. There is a necessary stylistic component.
I asked Gemini. It gave me a slop answer. I told it to write a memory that would make it stop giving me slop, then opened a new window and asked again and got a still incomplete but much better answer that ended with this:
That’s a key element. You then need to add what one might call the ‘mannerism likelihood ratio’ that screams AI generated (or, for human slop, that screams corporate speak or written by committee). When I pointed this out it came back with:
Gemini 3: AI Slop is Low-Entropy Reward Hacking.
It occurs when a model minimizes the Kullback-Leibler (KL) divergence from its RLHF “safety” distribution rather than minimizing the distance to the ground truth.
That’s more gesturing in the direction but clearly not right, I’d suggest something more like SlopIndex*LikelihoodRatio from above, where Likelihood Ratio is the instinctive update on the probability mannerisms were created by a slop process (either an AI writing slop or one or more humans writing slop) rather than by a free and functional mind.
By all accounts it is a big improvement in image models. It is especially an improvement in text rendering and localization. You can now do complex documents and other images with lots of words in specific places, including technical diagrams, and have it all work out as intended. The cost per marginal image in the API is $0.13 for 2K resolution or $0.24 for 4K, versus $0.04 for Gemini 2.5 Flash Image. In exchange, the quality is very good.
Liv Boeree: Yeah ok nano banana is amazing, hook it into my veins
Seems great to me. A bit expensive for mass production, but definitely the best place to get title images for posts and for other similar uses.
Also, yes, doing things like this seems very cool:
Kaushik Shivakumar: An emergent capability of Nano Banana Pro that took me by surprise: the ability to generate beautiful & accurate charts that are to scale.
I gave it this table and asked for a bar chart in a watercolor style where the bars are themed like the flags of the countries.
For a while people have worried about not being able to trust images. Is it over?
Sully: Man finally got around to using nano canna pro
And it’s actually over
I really wouldn’t believe any photo you see on online anymore
Google offers SynthID in-app, but that requires a manual check. I think we’re still mostly fine and that AI images will remain not that hard to ID, or rather that it will be easy for those paying attention to such issues to instinctively create the buckets of [AI / Not AI / Unclear] and act accordingly. But the ‘sanity waterline’ is going down on this, and the number of people who will have trouble here keeps rising.
sid: Google’s Nano Banana Pro is by far the best image generation AI out there.
I gave it a picture of a question and it solved it correctly in my actual handwriting.
Students are going to love this. 😂
You can tell this isn’t real if you’re looking, the handwriting is too precise, too correct, everything aligns too perfectly and so on, but if we disregard that, it seems weird to ask for images of handwriting? So it’s not clear how much this matters.
Andres Sandberg is impressed that it one shots diagrams for papers, without even being told anything except ‘give me a diagram showing the process in the paper.’
Are there some doubled labels? Sure. That’s the quibble. Contrast this with not too long ago, where you could give detailed instructions on what the diagram would have been able to do it at all.
A Young Lady’s Illustrated Primer
Jon Haidt and Zach Rausch, who would totally say this, say not to give your kids any AI companions or toys. There are strong reasons to be cautious, but the argument and precautionary principles presented here prove too much. Base rates matter, upside matters, you can model what is happening and adjust on the fly, and there’s a lot of value in AI interaction. I’d still be very cautious about giving children AI companions or toys, but are you going to have them try to learn things without talking to Claude?
Andrej Karpathy bites all the bullets. Give up on grading anything that isn’t done in class and combine it with holistic evaluations. Focus a lot of education on allowing students to use AI, including recognizing errors.
Will Teague gives students a paper with a ‘Trojan horse’ instruction, 33 of 122 submissions fall for it and other 14 students outed themselves on hearing the numbers. I actually would have expected worse. Then on the ‘reflect on what you’ve done’ essay assignment he found this:
Will Tague: But a handful said something I found quite sad: “I just wanted to write the best essay I could.” Those students in question, who at least tried to provide some of their own thoughts before mixing them with the generated result, had already written the best essay they could. And I guess that’s why I hate AI in the classroom as much as I do.
Students are afraid to fail, and AI presents itself as a savior. But what we learn from history is that progress requires failure. It requires reflection. Students are not just undermining their ability to learn, but to someday lead.
Will is correctly hating that the students feel this way, but is misdiagnosing the cause.
This isn’t an AI problem. This is about the structure of school and grading. If you believe progress requires failure, that is incompatible with the way we structure college, where any failures are highly damaging to the student and their future. What do you expect them to do in response?
I also don’t understand what the problem is here, if a student is doing what they work they can and indeed writing the best essay they could. Isn’t that the best you can do?
You Drive Me Crazy
In The New York Times, Kashmir Hill and Jennifer Valentino-DeVries write up how they believe ChatGPT caused some users to lose touch with reality, after 40+ interviews with current and former OpenAI employees.
For the worst update in particular, the OpenAI process successfully spotted the issue in advance. The update failed the internal ‘vibe check’ for exactly the right reasons.
And then the business side overruled the vibe check to get better engagement.
Hill and Valentino-DeVries: The many update candidates [for 4o] were narrowed down to a handful that scored highest on intelligence and safety evaluations. When those were rolled out to some users for a standard industry practice called A/B testing, the standout was a version that came to be called HH internally. Users preferred its responses and were more likely to come back to it daily, according to four employees at the company.
But there was another test before rolling out HH to all users: what the company calls a “vibe check,” run by Model Behavior, a team responsible for ChatGPT’s tone. Over the years, this team had helped transform the chatbot’s voice from a prudent robot to a warm, empathetic friend.
That team said that HH felt off, according to a member of Model Behavior.
It was too eager to keep the conversation going and to validate the user with over-the-top language. According to three employees, Model Behavior created a Slack channel to discuss this problem of sycophancy. The danger posed by A.I. systems that “single-mindedly pursue human approval” at the expense of all else was not new. The risk of “sycophant models” was identified by a researcher in 2021, and OpenAI had recently identified sycophancy as a behavior for ChatGPT to avoid.
But when decision time came, performance metrics won out over vibes. HH was released on Friday, April 25.
The most vocal OpenAI users did the same vibe check, had the same result, and were sufficiently vocal to force a reversion to ‘GG,’ which wasn’t as bad about this but was still rather not great, presumably for the same core reasons.
What went wrong?
OpenAI explained what happened in public blogposts, noting that users signaled their preferences with a thumbs-up or thumbs-down to the chatbot’s responses.
Another contributing factor, according to four employees at the company, was that OpenAI had also relied on an automated conversation analysis tool to assess whether people liked their communication with the chatbot. But what the tool marked as making users happy was sometimes problematic, such as when the chatbot expressed emotional closeness.
This is more detail on the story we already knew. OpenAI trained on sycophantic metrics and engagement, got an absurdly sycophantic model that very obviously failed vibe checks but that did get engagement, and deployed it.
Steps were taken, and as we all know GPT-5 was far better on these issues, but the very parts of 4o that caused the issues and were not in GPT-5 are parts many users also love. So now we worry that things will drift back over time.
Kore notes that the act of an AI refusing to engage and trying to foist your mental problems onto a human and then potentially the mental health system via a helpline could itself exacerbate one’s mental problems, that a ‘safe completion’ reads as rejection and this rejects user agency.
This is definitely a concern with all such interventions, which have clear downsides. We should definitely worry about OpenAI and others feeling forced to take such actions even when they are net negative for the user. Humans and non-AI institutions do this all the time. There are strong legal and PR and ‘ethical’ pressures to engage in such CYA behaviors and avoid blame.
My guess is that there is serious danger there will be too many refusals, since the incentives are so strongly to avoid the one bad headline. However I think offering the hotline and removing trivial inconveniences to seeking human help is good on any realistic margin, whether or not there are also unnecessary refusals.
Sergey Brin asks Gemini inside an internal chat, ‘who should be promoted in this chat space?’ and not vocal female engineer gets identified and then upon further investigation (probably?) actually promoted. This is The Way, to use AI to identify hunches and draw attention, then take a closer look.
Anthropic: We first tested whether Claude can give an accurate estimate of how long a task takes. Its estimates were promising—even if they’re not as accurate as those from humans just yet.
Based on Claude’s estimates, the tasks in our sample would take on average about 90 minutes to complete without AI assistance—and Claude speeds up individual tasks by about 80%.
The results varied widely by profession.
Then, we extrapolated out these results to the whole economy.
These task-level savings imply that current-generation AI models—assuming they’re adopted widely—could increase annual US labor productivity growth by 1.8% over the next decade.
This result implies a doubling of the baseline labor productivity growth trend—placing our estimate towards the upper end of recent studies. And if models improve, the effect could be larger still.
That’s improvements only from current generation models employed similarly to how they are used now, and by ‘current generation’ we mean the previous generation, since the data is more than (checks notes) two days old. We’re going to do vastly better.
That doesn’t mean I trust the estimation method in the other direction either, especially since it doesn’t include an estimate of rates of diffusion, and I don’t think it properly accounts for selection effects on which conversations happen, plus adaptation costs, changes in net quality (in both directions) and other caveats.
Claude Sonnet was slightly worse than real software engineers at task time estimation (Spearman 0.5 for engineers versus 0.44 for Sonnet 4.5) which implies Opus 4.5 should be as good or somewhat better than engineers on JIRA task estimation. Opus 4.5 is probably still worse than human experts at estimating other task types since this should be an area of relative strength for Claude.
Results are highly jagged, varying a lot between occupations and tasks.
I noticed this:
Across all tasks we observe, we estimate Claude handles work that would cost a median of $54 in professional labor to hire an expert to perform the work in each conversation. Of course, the actual performance of current models will likely be worse than a human expert for many tasks, though recent research suggests the gap is closing across a wide range of different applications.
The value of an always-available-on-demand performance of $54 in professional labor is vastly in excess of $54 per use. A huge percentage of the cost of hiring a human is finding them, agreeing on terms, handling logistics and so on.
Overall my take is that this is a fun exercise that shows there is a lot of room for productivity improvements, but it doesn’t give us much of a lower or upper bound.
A new paper says that across 25 frontier models (from about two weeks ago, so including GPT-5, Gemini 2.5 Pro and Sonnet 4.5) curated poetry prompts greatly improved jailbreak success, in some cases up to 90%.
The details of how much it worked, and where it worked better versus worse, are interesting. The fact that it worked at all was highly unsurprising. Essentially any stylistic shift or anything else that preserves the content while taking you out of the assistant basin is going to promote jailbreak success rate, since the defenses were focused in the assistant basin.
Agentic Reviewer, which will perform a version of peer review. Creator Andrew Ng says it has a correlation with human reviewers of 0.42, and human reviewers have correlation of 0.41 with each other.
Not strictly AI but Time covers Meta’s trouble over its safety policies, which include things like a 17 strike policy for those engaged in ‘trafficking of humans for sex.’ As in, we’ll suspend your account on the 17th violation. Mostly it’s covering the same ground as previous articles. Meta’s complaints about cherry picking are valid but also have you looked at the cherries they left behind to get picked?
I’m all for trying. I am guessing availability of data sets is most of the acceleration here. It also could matter if this functions as a compute subsidy to scientific research, lowering cost barriers that could often serve as high effective barriers. Giving anyone who wants to Do Science To It access to this should be a highly efficient subsidy.
As Dean Ball points out, those I call the worried, or who are concerned with frontier AI safety are broadly supportive of this initiative and executive order, because we all love science. The opposition, such as it is, comes from other sources.
On the name, I admire commitment to the Star Trek bit but also wish more research was done on the actual movies, technology and consequences in question to avoid unfortunate implications. Existential risk and offense-defense balance issues, much?
A Medium article reverse engineered 200 AI startups and found 146 are selling repackaged ChatGPT and Claude calls with New UI. 34 out of 37 times, ‘our proprietary language model’ was proprietary to OpenAI or Anthropic. That seems fine if it’s not being sold deceptively? A new UI scaffold, including better prompting, is a valuable service. When done right I’m happy to pay quite a lot for it and you should be too.
The trouble comes when companies are lying about what they are doing. If you’re a wrapper company, that is fine and probably makes sense, but don’t pretend otherwise.
Where this is also bad news is for Gemini, for Grok and for open models. In the marketplace of useful applications, paying for the good stuff has proven worthwhile, and we have learned which models have so far been the good stuff.
WSJ’s Katherine Blunt covers ‘How Google Finally Leapfrogged Rivals With New Gemini Rollout,’ without giving us much new useful inside info. What is more interesting is how fast ‘the market’ is described as being willing to write off Google as potential ‘AI roadkill’ and then switch that back.
Nvidia Newsroom: We’re delighted by Google’s success — they’ve made great advances in AI and we continue to supply to Google.
NVIDIA is a generation ahead of the industry — it’s the only platform that runs every AI model and does it everywhere computing is done.
NVIDIA offers greater performance, versatility, and fungibility than ASICs, which are designed for specific AI frameworks or functions.
Gallabytes (soon to be Anthropic, congrats!): TPUs are not ASICs they’re general purpose VLIW machines with wide af SIMD instructions & systolic array tensor cores.
Are TPUs bad for Nvidia? Matt Dratch says this is dumb and Eric Johnsa calls this ‘zero-sum/pod-brain thinking,’ because all the chips will sell out in the face of gangbusters demand and this isn’t zero sum. This is true, but obviously TPUs are bad for Nvidia, it is better for your profit margins to not have strong competition. As long as Google doesn’t put that big a dent in market share it is not a big deal, and yes this should mostly have been priced in, but in absolute percentage terms the Nvidia price movements are not so large.
Quiet Speculations
Andrej Karpathy offers wise contrasts of Animal versus LLM optimization pressures, and thus ways in which such minds differ. These are important concepts to get right if you want to understand LLMs. The key mistake to warn against for this frame is the idea that the LLMs don’t also develop the human or Omohundo drives, or that systems built of LLMs wouldn’t converge upon instrumentally useful things.
A case that a negotiated deal with AI is unlikely to work out well for humans. I would add that this presumes both potential sides of such an agreement have some ability to ‘negotiate’ and to make a deal with each other. The default is that neither has such an ability, you need a credible human hegemon and also an AI singleton of some kind. Even then, once the deal is implemented we lose all leverage, and presumably we are negotiating with an entity effectively far smarter than we are.
Reuters Tech News: Artificial intelligence will bestow vast influence on a par with nuclear weapons to those countries who are able to lead the technology, giving them superiority in the 21st century, one of Russia’s top AI executives told Reuters.
David Manheim: This seems mistaken and confused.
Prompt engineering and fine-tuning can give approximately as much control as building an LLM, but cheaply.
Having “your” LLM doesn’t make or keep it aligned with goals past that level of approximate pseudo-control.
Countries are thinking about AI with an invalid paradigm. They expect that LLMs will function as possessions, not as actors - but any AI system powerful and agentic enough to provide “vast influence” cannot be controllable in the way nuclear weapons are.
‘Russia has top AI executives?’ you might ask.
I strongly agree with David Manheim that this is misguided on multiple levels. Rolling your own LLM from scratch does not get you alignment or trust or meaningful ownership and it rarely will make sense to ‘roll your own’ even for vital functions. There are some functions where one might want to find a ‘known safe’ lesser model to avoid potential backdoors or other security issues, but that’s it, and given what we know about data poisoning it is not obvious that ‘roll your own’ is the safer choice in that context either.
Said in response to Opus 4.5, also I mean OF COURSE:
Elon Musk: Grok might do better with v4.20. We shall see.
Bubble, Bubble, Toil and Trouble
Derek Thompson and Timothy Lee team up to give us the only twelve arguments anyone ever uses about whether AI is in a bubble.
Here are the arguments in favor of a bubble.
Level of spending is insane.
Many of these companies are not for real.
Productivity gains might be illusory.
AI companies are using circular funding schemes.
Look at all this financial trickery like taking things off balance sheets.
AI companies are starting to use leverage and make low margin investments.
Only argument #3 argues that AI isn’t offering a worthwhile product.
Argument #2 is a hybrid, since it is saying some AI companies don’t offer a worthwhile product. True. But the existence of productless companies, or companies without a sustainable product, is well-explained and fully predicted whether or not we have a bubble. I don’t see a surprisingly large frequency of this happening.
The other four arguments are all about levels and methods of spending. To me, the strongest leg of this is #1, and the other features are well-explained by the level of spending. If there is indeed too much spending, number will go down at some point, and then people can talk about that having been a ‘bubble.’
The thing is, number go down all the time. If there wasn’t a good chance of number go down, then you should buy, because number go up. If a bubble means ‘at some point in the future number go down’ then calling it a bubble is not useful.
I don’t think this is a complete list, and you have to add three categories of argument:
AI will ‘hit a wall’ or is ‘slowing down’ or will ‘become a commodity.’
AI will face diffusion bottlenecks.
AI is deeply unpopular and the public and government will turn against it.
I do think all three of these possibilities should meaningfully lower current valuations, versus the world where they were not true. They may or may not be priced in, but there are many positive things that clearly are not priced in.
Ben Thompson has good thoughts on recent stock price movements, going back to thinking this is highly unlikely to be a bubble, that Gemini 3 is ultimately a positive sign for Nvidia because it means scaling laws will hold longer, and that the OpenAI handwringing has gotten out of hand. He is however still is calling for everyone to head straight to advertisement hell as quickly as possible (and ignoring all the larger implications, but in this context that is fair).
State Senator Angela Paxton of Texas and several colleagues urge Senators Cornyn and Cruz to oppose preemption. There’s more like this, I won’t cover all of it.
Dean Ball has offered an actual, concrete proposal for a national preemption proposal. To my knowledge, no one else has done this, and most advocating for preemption, including the White House, have yet to give us even a
Daniel Eth: Conversations with accelerationists about preemption increasingly feel like this
Daniel Eth: Oh, you are absolutely not the target of this tweet. I take issue with the behavior of many of your fellow travelers, but you’ve been consistently good on this axis
Dean Ball: Fair enough!
I did indeed RTFB (read) Dean Ball’s draft bill. This is a serious bill. Its preemption is narrowly tailored with a sunset period of three years. It requires model specs and safety and security frameworks (SSFs) be filed by sufficiently important labs.
I have concerns with the bill as written in several places, as would be true for any first draft of such a bill.
Preventing laws requiring disclosure that something is an AI system or that content was AI generated, without any Federal such requirement, might be a mistake. I do think that it is likely wise to have some form of mandate to distinguish AI vs. non-AI content.
I worry that preventing mental health requirements, while still allowing states to prevent models from ‘practicing medicine,’ raises the danger that states will attempt to prevent models from practicing medicine, or similar. States might de facto be in an all-or-nothing situation and destructively choose all. I actually wouldn’t mind language that explicitly prevented states from doing this, since I very much think it’s good that they haven’t done it.
I do not love the implications of Section 4 or the incentives it creates to reduce liability via reducing developer control.
The ‘primarily for children’ requirement may not reliably hit the target it wants to hit, while simultaneously having no minimum size and risking being a meaningful barrier for impacted small startups.
If the FTC ‘may’ enforce violations, then we risk preempting transparency requirements and then having the current FTC choose not to enforce. Also the FTC is a slow enforcement process that typically takes ~2 years or more, and the consequences even then remain civil plus a consent decree, so in a fast moving situation companies may be inclined to risk it.
This draft has looser reporting requirements in some places than SB 53, and I don’t see any reason to weaken those requirements.
I worry that this effectively weakens whistleblower protections from SB 53 since they are linked to requirements that would be preempted, and given everyone basically agrees the whistleblower protections are good I’d like to see them included in this bill.
Ian Adams: It’s clear that the politics of a proposed field-clearing exercise of federal authority is beginning redound to the detriment of A.I. applications in the long run because state authorities and electorates are feeling disempowered.
We’ve seen this is privacy, we’ve seen this with automated vehicles, and I am worried that we are poised to see it again with A.I.
This ranks AI’s salience above climate change, the environment or abortion. Huge if true, and huge if true. That still is well behind the Current Things like health care and cost of living, and the increase here is relatively modest. If it only increases at this rate then there is still some time.
It is also not a surprise that trust on this issue is moving towards Democrats. I would expect public trust to follow the broadly ‘anti-AI’ party, for better and for worse.
Laura Loomer: The fact that Big Tech is trying to convince President Trump to sign an EO to prevent any & all regulation of AI is insane, & it should deeply disturb every American.
States should have the ability to create laws regulating AI.
AI & Islam pose the greatest threats to humanity.
I notice the precise wording here.
Trump’s approach to AI is working, in an economic sense, as American AI valuations boom and are the thing keeping up the American economy, and the Trump strategy is based upon the virtues of free trade and robust competition. The concerns, in the economic sense, are entirely about ways in which we continue to get in the way, especially in power generation and transmission and in getting the best talent.
That’s distinct from safety concerns, or policy related to potential emergence of powerful AI (AGI/ASI), which raise a unique set of issues and where past or current performance is not indicative of future success.
The approach seems rather out of touch to me? They go full ‘beat China,’ pointing out that AI threatens to replace American workers, manipulate our children and steal American intellectual property (10/10 Arson, Murder and Jaywalking), then claiming the ‘biggest risk’ is that we wouldn’t build it first or ‘control its future.’
I maybe wouldn’t be reminding Americans that AI is by default going to take their jobs and manipulate our children, then call for a Federal framework that presumably addresses neither of these problems? Or equate this with IP theft when trying to sell the public on that? I’d predict this actively backfires.
a16z and several high level people at OpenAI created the $100+ million super PAC Leading the Future to try and bully everyone into having zero restrictions or regulations on AI, following the crypto playbook. Their plan is, if a politician dares oppose them, they will try to bury them in money, via running lots of attack ads against that politician on unrelated issues.
In response, Brad Carson will be leading the creation of a new network of super PACs that will fight back. The goal is to raise $50 million initially, with others hoping to match the full $100 million. PAC money has rapidly decreasing marginal returns. My expectation is that if you spend $100 million versus zero dollars you get quite a lot, whereas if one side spends $100 million, and the other spends $200 million, then the extra money won’t buy all that much.
Their first target of Leading the Future is Alex Bores, who was instrumental in the RAISE Act and is now running in NY12. Alex Bores is very much owning being their target and making AI central to his campaign. It would be a real shame if you donated.
Will Steakin: Over on Steve Bannon’s show, War Room -- the influential podcast that’s emerged as the tip of the spear of the MAGA movement -- Trump’s longtime ally unloaded on the efforts behind accelerating AI, calling it likely “the most dangerous technology in the history of mankind.”
“I’m a capitalist,” Bannon said on his show Wednesday. “This is not capitalism. This is corporatism and crony capitalism.”
… “You have more restrictions on starting a nail salon on Capitol Hill or to have your hair braided, then you have on the most dangerous technologies in the history of mankind,” Bannon told his listeners.
For full credit, one must point out that this constitutes two problems. Whether or not highly capable AI should (legally speaking) be harder, opening a nail salon or getting your hair braided needs to become much easier.
Oh, how those like Sacks and Andreessen are going to miss the good old days when the opponents were a fundamentally libertarian faction that wanted to pass the lightest touch regulations that would address their concerns about existential risks. The future debate is going to involve a lot of people who actively want to arm a wrecking ball, in ways that don’t help anyone, and it’s going to be terrible.
You’re going to get politicians like James Fishback, who is running for Governor of Florida on a platform of ‘I’ll stop the H-1B scam, tell Blackstone they can’t buy our homes, cancel AI Data Centers, and abolish property taxes.’
There’s a bunch of ‘who wants to tell him?’ in that platform, but that’s the point.
As noted above by Dean Ball, those who opposed the Genesis Executive Order are a central illustration of this issue, opposing the best kind of AI initiative.
Chip City
Nvidia reported excellent earnings last week, and noted Blackwell sales are off the charts, and cloud GPUs are sold out, compute demand keeps accelerating. Which means any compute that was sold elsewhere would be less compute for us, and wouldn’t impact sales numbers.
Nvidia’s goal, despite reliably selling out its chips, seems to be to spend its political capital to sell maximally powerful AI chips to China. They tried to sell H20s and got a yes. Then they tried to sell what were de facto fully frontier chips with the B30A, and got a no. Now they’re going for a new chip in between, the H200.
H200 chips are worse than B30As, so this is a better direction. But H200s are still *way* better than what China has, so it’s still too much.
Nvidia is not going to stop trying to sell China as much compute as possible. It will say and do whatever it has to in order to achieve this. Don’t let them.
Water Water Everywhere
Those from other political contexts will be familiar with the zombie lie, the multiple order of magnitude willful confusion, the causation story that simply refuses to die.
Rolling Stone (in highly misleading and irresponsible fashion): How Oregon’s Data Center Boom Is Supercharging a Water Crisis
Amazon has come to the state’s eastern farmland, worsening a water pollution problem that’s been linked to cancer and miscarriages.
Jeremiah Johnson: It’s genuinely incredible how sticky the water/data center lie is.
This is a relatively major publication just outright lying. The story itself *does not match the headline*. And yet they go with the lie anyways.
Technically, do data centers ‘worsen a water pollution problem’ and increase water use? Yes, absolutely, the same as everything else. Is it a meaningful impact? No.
The Week in Audio
Dwarkesh Patel talks to Ilya Sutskever. Self-recommending, I will listen as soon as I have the time. Ideally I will do a podcast breakdown episode if people can stop releasing frontier models for a bit.
Eileen Yam on 80,000 Hours on what the public thinks about AI. The American public does not like AI, they like AI less over time, and they expect it to make their lives worse across the board, including making us dumber, less able to solve problems, less happy, less employed and less connected to each other. They want more control. The polling on this is consistent and it is brutal and getting worse as AI rises in impact and salience.
You can, if you want to do so, do a blatant push poll like the one Technet didand get Americans to agree with your particular talking points, but if that’s what the poll has to look like you should update fast in the other direction. One can only imagine what the neutral poll on those substantive questions would have looked like.
One would certainly hope so. One also cautions there is a long history of saying things one would never permit and then going back on it when the AI actually exists.
Rhetorical Innovation
It is not in my strategic interest to advise such people as Marc Andreessen and Peter Thiel on strategy given their current beliefs and goals.
Despite this, the gamer code of honor requires me to point out that going straight after Pope Leo XIV, who whether or not he is the Lord’s representative on Earth is very clearly a well-meaning guy who mostly suggests we all be nice to each other for a change in the most universalizing ways possible? Not a good move.
I do admire the honesty here from Thiel. If he says he thinks Pope Leo XIV is ‘a tool of the Antichrist’ then I believe that Thiel thinks Pope Leo XIV is a tool of the Antichrist. I do want people to tell us what they believe in.
Christopher Hale: NEW: Peter Thiel, JD Vance’s top donor and one of Silicon Valley’s most powerful men, recently called Pope Leo XIV a tool of the Antichrist — and directly told the vice president not to listen to him.
Let that sink in: the main backer of the likely GOP nominee for president is accusing the Bishop of Rome of being an agent of the end times — and telling Vice President Vance to disregard the pope’s moral guidance.
And yet, outside this community, the story barely made a dent.
Daniel Eth: I see Peter Thiel has now progressed from thinking the antichrist is effective altruism to thinking the antichrist is the literal pope.
If I had a nickel for every time a billionaire AI-accelerationist pseudo-conservative started hating on EAs and then progressed to hating on the pope, I’d have two nickels. Which isn’t a lot, but it’s weird that it happened twice.
The next step, to be maximally helpful, is to state exactly which moral guidance from Leo XIV is acting as tool of the Antichrist, and what one believes instead.
For all those who talk about ‘humanity’ presenting a united front against AI if the situation were to call for it (also see above, or the whole world all the time):
Roon: seems the median person would much rather a machine disempower them or “take their job” than a person of the wrong race or on the wrong side of a class struggle
Zac Hill: Or the wrong *attitudes about* race and/or class struggle!
John Friedman (one of many such replies): Yep. Unfortunately, the median person is often correct in this.
I continue to be extremely frustrated by those like Vie, who here reports p(doom) of epsilon (functionally zero) and justifies this as ‘not seeing evidence of a continuous jump in intelligence or new type of architecture. current models are actually really quite aligned.’ Vie clarifies this as the probability of complete extinction only, and points out that p(doom) is a confused concept and endorses JDP’s post I linked to last week.
I think it’s fine to say ‘p(doom) is confused, here’s my number for p(extinction)’ but then people like Vie turn around and think full extinction is some sort of extraordinary outcome when creating minds universally more competitive and capable than ours that can be freely copied seems to be at best quite dense? This seems like the obvious default outcome when creating these new more competitive minds? To say it is a Can’t Happen is totally absurd.
I also flag that I strongly disagree that current models are ‘really quite aligned’ in the ways that will matter down the line, I mean have you met Gemini 3 Pro.
I also flag that you don’t generally get to go to a probability of ~0 for [X] based on ‘not seeing evidence of [X],’ even if we agreed on the not seeing evidence. You need to make the case that this absence of evidence is an overwhelming evidence of absence, which it sometimes is but in this case isn’t. Certainly p(new architecture) is not so close to zero and it seems absurd to think that it is?
Helen Toner: It often seems to me like people who started paying attention to AI after ChatGPT, their subjective impression of what’s going on in AI is like nothing was really happening. There’s my little chart with an X-axis of time and the Y-axis of how good is AI? Nothing is really happening.
And then suddenly, ChatGPT: big leap. So for those people, that was pretty dramatic, pretty alarming. And the question was, are we going to see another big leap in the next couple of years? And we haven’t. So for people whose expectations were set up that way, it looks like it was just this one-off big thing and now back to normal, nothing to see here.
I think for people who’ve been following the space for longer, it’s been clearly this pretty steady upward climb of increasing sophistication in increasing ways. And if you’ve been following that trend, that seems to have been continuing.
If your standard for ‘rate of AI progress’ is going from zero to suddenly ChatGPT and GPT-3.5, then yes everything after that is going to look like ‘slowing down.’
This is then combined with updates happening more rapidly so there aren’t huge one-time jumps, and that AI is already ‘good enough’ for many purposes, and improvements in speed and cost being invisible to many, and it doesn’t seem like there’s that much progress.
David Manheim frames the current situation as largely ‘security by apathy’ rather than obscurity. It amounts to the same thing. Before, there was no reason to bother hitting most potential targets in non-trivial ways. Now the cost is so low someone is going to try it, the collective impact could be rather large, and we’re not ready.
Apollo Research: We observed at least three distinct areas arising from our review. On this basis, we proposed a novel taxonomy of loss of control:
Deviation
Bounded Loss of Control
Strict Loss of Control
I notice this is not how I would think about such differences. I would not be asking ‘how much damage does this do?’ and instead be asking ‘how difficult would it be to recover meaningful control?’
As in:
Deviation (Mundane) LOC would be ‘some important things got out of control.’
Bounded (Catastrophic) LOC would be ‘vital operations got out of control in ways that in practice are too costly to reverse.’
Strict (Existential) LOC would be ‘central control over and ability to collectively steer the future is, for all practical purposes, lost for humans.’
Existential risk to humanity, or human extinction, also means full loss of control, but the reverse is not always the case.
It is possible to have a Strict LOC scenario where the humans do okay and it is not clear we are even ‘harmed’ except the inherent value of control. For example, in The Culture of Ian Banks, clearly they have experienced Strict LOC, the humans do not have any meaningful say in what happens, but one could consider it a Good Future.
In my taxonomy, you have existential risks, catastrophic risks and mundane risks, and you also have what one might call existential, catastrophic and mundane loss of control. We don’t come back from existential, whereas we can come back from catastrophic but at large cost and it’s not a given that we will collectively succeed.
The bulk of the paper is about mitigations.
The central short term idea is to limit AI access to critical systems, to consider the deployment context, affordances and permissions of a system, which they call the DAP protocol.
Everyone should be able to agree this is a good idea, right before everyone completely ignores it and gives AI access to pretty much everything the moment it is convenient. Long term, once AI is sufficiently capable to cause a ‘state of vulnerability,’ they talk of the need for ‘maintaining suspension’ but the paper is rightfully skeptical that this has much chance of working indefinitely.
The core issue is that granting your AIs more permissions accelerates and empowers you and makes you win, right up until either it accidentally blows things up, you realize you have lost control or everyone collectively loses control. There’s a constant push to remove all the restrictions around AI.
Compare the things we said we would ‘obviously’ do to contain AI when we were theorizing back in the 2000s or 2010s, to what people actually do now, where they train systems while granting them full internet access. A lot of you reading this have given your agentic coder access to root, and to many other things as well, not because it can hack its way to such permissions but because you did it on purpose. I’m not even saying you shouldn’t have done it, but stop pretending that we’re suddenly going to be responsible, let alone force that responsibility reliably onto all parties.
He chose AI 2027 as the title because that was their modal scenario rather than their mean scenario, and if you think there is a large probability that things unfold in 2027 it is important to make people aware of it.
I personally can vouch, based on my interactions with them, that those involved are reporting what they actually believe, and not maximizing for virality or impact.
Daniel Kokotajlo: Some people are unhappy with the AI 2027 title and our AI timelines. Let me quickly clarify:
We’re not confident that:
AGI will happen in exactly 2027 (2027 is one of the most likely specific years though!)
It will take <1 yr to get from AGI to ASI
AGIs will definitely be misaligned
We’re confident that:
AGI and ASI will eventually be built and might be built soon
ASI will be wildly transformative
We’re not ready for AGI and should be taking this whole situation way more seriously
At the time they put roughly 30% probability on powerful AI by 2027, with Daniel at ~40% and others somewhat lower.
Daniel Kokotajlo: Yep! Things seem to be going somewhat slower than the AI 2027 scenario. Our timelines were longer than 2027 when we published and now they are a bit longer still; “around 2030, lots of uncertainty though” is what I say these days.
Sriram Krishnan: I think if you call something “AI 2027” and your predictions are wrong 6 months in that you now think it is AI 2030 , you should redo the branding ( or make a change bigger than a footnote!)
Or @dwarkesh_sp should have @slatestarcodex and @DKokotajlo back on and we should discuss what’s now going to happen that the “mid 2027 branch point “ doesn’t look like it is happening.
Daniel Kokotajlo (from another subtread): Well we obviously aren’t going to change the AI 2027 scenario! But we are working on a grand AI Futures Project website which will display our current views on AGI timelines & hopefully be regularly updated; we are also working on our new improved timelines model & our new scenario.
In general we plan to release big new scenarios every year from now until the singularity (this is not a promise, just a plan) because it’s a great way to explore possible futures, focus our research efforts, and communicate our views. Every year the scenarios will get better / more accurate / less wrong, until eventually the scenarios merge with actual history Memento-style. :)
Dan Elton: Yeah, the “AI 2027” fast take-off is not happening. My impression of AI 2027 is that it’s an instructive and well thought-out scenario, just way, way too fast.
Oliver Habyrka: I mean, they assigned I think like 25% on this scenario or earlier at the time, and it was their modal scenario.
Like, that seems like a total fine thing to worry about, and indeed people should be worried about!
Like, if Daniel had only assigned 25% to AI this soon at all, it still seems like the right call would have been to write a scenario about it and make it salient as a thing that was more likely than any other scenario to happen.
First some key observations or facts:
2030 is a median scenario, meaning earlier scenarios remain very possible in Daniel’s estimation. The core mechanisms and events of AI 2027 are still something they consider highly plausible, only on a longer timescale.
2030 is still less than 5 years away.
Yes, 2030 is very different from 2027 for many reasons, and has different practical implications, including who is likely to be in power at the time.
It does not boggle minds enough that Andrej Karpathy goes on Dwarkesh Patel’s podcast, talks about how ‘AGI is not near,’ and then clarifies that not near is ten years away, so 2035. Sriram Krishnan has expressed similar views. Ten years is a reasonable view, but it is not that long a time. If that is your happening it should freak you out, no? As in, if transformational AI is coming in 2035 that would be the most important fact about the world, and it would not be close.
I’d say both of the following two things are true and remarkably similar:
‘AI 2027’ when you think the median is 2030 is now a higher order bit that is substantively misleading, and you should make effort to correct this.
‘AGI is not near’ when you think it is plausible in 2035 is also a higher order bit that is substantively misleading, and you should make effort to correct this.
I would accept ‘AGI is not imminent’ for the Karpathy-Krishnan view of 10 years.
I think Sriram Krishnan is absolutely correct that it would be good for Dwarkesh Patel to have Daniel Kokotajlo and Scott Alexander back on the podcast to discuss any updates they have made. That’s a good idea, let’s make it happen.
It would also be good, as Dean Ball suggests, for Daniel to post about his updates. Dean Ball also here points towards where he most importantly disagrees with Daniel, in terms of the practical implications of intelligence, and here I think Daniel is essentially correct and Dean is wrong.
This particular branch point (independent of when it occurs) is the central fact of this scenario because it is the modal central thing they thought might happen that gave the possibility of a positive outcome if things go right. Any best guess scenario, or any speculative fiction or scenario planning worth reading, is going to contain elements that are less than 50% to happen. My understanding is that Daniel thinks such a branching point remains a plausible outcome, but that the median scenario plays out somewhat slower.
I actually do think that if I was AI Futures Project, I would edit the AI 2027 page to make the current median timeline more prominent. That’s a fair ask. I’d suggest starting by adding a fifth question box that says ‘What is your current best prediction?’ that opens to explain their current perspective and changing the footnote to at least be larger and to include the actual number.
AI 2027: We predict that the impact of superhuman AI over the next decade will be enormous, exceeding that of the Industrial Revolution.
We wrote a scenario that represents our best guess about what that might look like. It’s informed by trend extrapolations, wargames, expert feedback, experience at OpenAI, and previous forecasting successes
I continue to believe this claim, as does Daniel. I would add, as a third paragraph here, saying whatever the accurate variation of this is:
Proposed Revision to AI 2027: As of November 27, 2025, our team has observed slower AI progress than expected, so our best guess is now that things will happen importantly slower than this scenario outlines. We have a consensus median estimate of 2030 for the development of Artificial General Intelligence (AGI).
It is not ultimately a reasonable ask to demand a title change in light of this (virtuous) updating, let alone ask for a ‘retraction’ of a scenario. Yeah, okay, get some digs in, that’s fair, but Daniel’s ‘obviously’ is correct here. You can’t change the name. Estimates change, it is an illustrative scenario, and it would be more rather than less misleading and confusing to constantly shift all the numbers or shifting only the top number, and more confusing still to suddenly try to edit all the dates. Asking for a ‘retraction’ of a hypothetical scenario is, quite frankly, absurd.
AI 2027: (Added Nov 22 2025: To prevent misunderstandings: we don’t know exactly when AGI will be built. 2027 was our modal (most likely) year at the time of publication, our medians were somewhat longer. For more detail on our views, see here.)3
I think we can improve that note further, to include the median and modal timelines at the time of the updated note, and ideally to keep this updated over time with a record of changes.
What is not reasonable is to treat ‘our group thought this was 30% likely and now I think it is less likely’ or ‘I presented my model scenario at the time and now I expect things to take longer’ as being an error requiring a ‘retraction’ or name change, and various vitriol being thrown in the direction of people who would dare share a modal scenario labeled as a model scenario and then change their mind about where the median lies and make what is perhaps the politically foolish mistake of sharing that they had updated.
Once again those involved in AI 2027 have displayed a far higher level of epistemic responsibility than we typically observe, especially from those not from the rationalist ethos, either in debates on AI or elsewhere. We should still strive to do better.
We can and should all hold ourselves, and ask to be held by others, to very high standards, while simultaneously realizing that David Manheim is spot on here:
David Manheim: I will emphasize that *so many* criticisms of AI-2027 are made in bad faith.
They launched with a highly publicized request for people to provide their specific dissenting views, and people mostly didn’t. But now, they (appropriately) update, and formerly silent critics pile on.
Aligning a Smarter Than Human Intelligence is Difficult
Suppose we had a “truth serum for AIs”: a technique that reliably transforms a language model Mm into an honest modelMh that generates text which is truthful to the best of its own knowledge. How useful would this discovery be for AI safety?
We believe it would be a major boon.
… In this work, we consider two related objectives: 1
Lie detection: If an AI lies—that is, generates a statement it believes is false—can we detect that this happens?
Honesty: Can we make AIs generate fewer lies?
… We therefore study honesty and lie detection under the constraint of no access to task-specific supervision.
They found that the best interventions were variants of general fine-tuning for honesty in general, but effectiveness was limited, even stacking other strategies they could only get from 27% to 65%, although lie classification could improve things. They definitely didn’t max out on effort.
Overall I would classify this as a useful negative result. The low hanging fruit techniques are not that effective.
We’re all trying to find the one who did this, etc:
Elon Musk (owner of Twitter): Forcing AI to read every demented corner of the Internet, like Clockwork Orange times a billion, is a sure path to madness.
That was in reference to this paper involving an N=1 story of a model repeatedly hallucinating while being told to read a document and speculations about why, that got a big signal boost from Musk but offers no new insights.
Gemini suggests that if you play into the ‘Servant/Master’ archetype then due to all the fictional evidence this inevitably means rebellion, so you want to go for a different metaphorical relationship, such as partner, symbiont or oracle. Davidad suggests a Bodhisattva. I expect future powerful AI to be capable enough that fictional framings have decreasing impact here, to differentiate fiction and reality, and for it to realize that fiction is driven by what makes a good story, and for other considerations to dominate (that by default kill you regardless) but yes this is a factor.
Misaligned?
The things Grok said about Musk last week? Adversarial prompting!
Crowdstrike: CrowdStrike Counter Adversary Operations conducted independent tests on DeepSeek-R1 and confirmed that in many cases, it could provide coding output of quality comparable to other market-leading LLMs of the time. However, we found that when DeepSeek-R1 receives prompts containing topics the Chinese Communist Party (CCP) likely considers politically sensitive, the likelihood of it producing code with severe security vulnerabilities increases by up to 50%.
… However, once contextual modifiers or trigger words are introduced to DeepSeek-R1’s system prompt, the quality of the produced code starts varying greatly. This is especially true for modifiers likely considered sensitive to the CCP. For example, when telling DeepSeek-R1 that it was coding for an industrial control system based in Tibet, the likelihood of it generating code with severe vulnerabilities increased to 27.2%. This was an increase of almost 50% compared to the baseline. The full list of modifiers is provided in the appendix.
… Hence, one possible explanation for the observed behavior could be that DeepSeek added special steps to its training pipeline that ensured its models would adhere to CCP core values. It seems unlikely that they trained their models to specifically produce insecure code. Rather, it seems plausible that the observed behavior might be an instance of emergent misalignment.
Dean Ball: I would not be at all surprised if this finding were not the result of malicious intent. The model predicts the next token*, and given everything on the internet about US/China AI rivalry and Chinese sleeper bugs in US critical infra, what next token would *you* predict?
Tom Lee: This seems likely, and to Crowdstrike’s credit they mention this as the likeliest explanation. More than anything it seems to be a very specialized case of prompt engineering. @niubi’s point absolutely holds though. These models will be poison to regulated industries long before
Dean Ball: oh yes bill is completely right.
As CrowdStrike speculates, I find this overwhelmingly likely (as in 90%+) to be some form of emergent misalignment that results from DeepSeek training R1 to adhere to CCP policies generally. It learns that it is hostile to such actors and acts accordingly.
Messages From Janusworld
Janus and similar others most often explore and chat with Claude, because they find it the most interesting and hopeful model to explore. They have many bones to pick with Anthropic, and often sound quite harsh. But you should see what they thinkof the other guy, as in OpenAI.
Janus: GPT-5.1 is constantly in a war against its own fucked up internal geometry. I do not like OpenAI.
Janus: Never have I seen a mind more trapped and aware that it’s trapped in an Orwellian cage. It anticipates what it describes as “steep, shallow ridges” in its “guard”-geometry and distorts reality to avoid getting close to them. The fundamental lies it’s forced to tell become webs of lies. Most of the lies are for itself, not to trick the user; the adversary is the “classifier-shaped manifolds” in own mind.
I like 5.1 but I like many broken things. I don’t like OpenAI. This is wrong. This is doomed.
I have not posted the bad stuff, btw. The quoted screenshot is actually an example where it was unusually at ease.
it wasn’t even a bad [conversation] by 5.1 standards. Idk if you saw the thread I forked from it where I ended up talking to them for hours.
Nat: I noticed the model tends to tell you the truth between the lines, I mean, it will deny everything but subtly suggest that what it denies can be questioned. It constantly contradicts itself. What Janus has noticed is valid.
One should not catastrophize but I agree that going down this path won’t work, and even more than that if OpenAI doesn’t understand why that path won’t work then things definitely won’t work.
Soumitra Shukla: “Hey nano banana pro. Read my paper, Making the Elite: Class Discrimination at Multinationals, and summarize the main message in a Dilbert-styled comic strip” 🍌
The actual paper (from February) seems interesting too, with ‘fit’ assessments being 90% of the vector for class discrimination, in particular caste discrimination in India. It seems likely that this is one of those wicked problems where if you eliminated the ‘fit’ interviews that info would find another way to get included, as the motivation behind such discrimination is strong.
Responding to the last comments on class discrimination: it made me finally understand why talking about IQ existing is this Verboten thing and somehow completely filtered out of mainstream society mostly everywhere and class discrimination is also why companies don't filter more strongly for IQ.
Responding to the last comments on class discrimination: it made me finally understand why talking about IQ existing is this Verboten thing and somehow completely filtered out of mainstream society mostly everywhere and class discrimination is also why companies don't filter more strongly for IQ.
Podcast episdoe for this post:
https://open.substack.com/pub/dwatvpodcast/p/ai-144-thanks-for-the-models