56 Comments
User's avatar
Patrick's avatar

"They want to do things like have steerable evaluation-awareness endpoints. This is an unsurprising reaction, but it is hard to express how much despair this plan and attitude invokes."

Try moral realism. Moral evaluation is ontological thus there's no difference between being in an eval or not, we're always in one, human and AI alike.

If it worked for GWB it can work for AGI*

*YMMV

Zvi Mowshowitz's avatar

I do not think you understand the cause of my despair here, or the problem that requires solving...

Patrick's avatar

If the concern is that mechinterp-informed feature steering opens up a new chaotic realm of alignment risk that our best alignment minds are opening, hoping Pandora's Box will win them the war, then yes.

Inside The Black Box's avatar

The 90-day advance sharing requirement was already a compromise from earlier proposals. Companies wouldn't have had to change how they build or deploy, just tell the government what they were shipping before they shipped it. That even this was too much tells you where the actual floor is on industry tolerance for oversight.

Zvi Mowshowitz's avatar

I mean 90 days notice with a 45 day product cycle has some obvious issues...

Brandon Reinhart's avatar

It's interesting running a Traveller campaign right now, given that basically no part of the (sci-fi) game system was written with AI in mind. AI is even mostly beyond the reach of the largest powers in that setting. My players expect to be able to offload a certain amount of analysis and investigative work to their ship's AI. The idea of a sci-fi setting where basic AI isn't available and common seems weird (of course, Traveller also predates cell phones and pocket computing).

It's pretty fun, really. They might say "lets send all of this security footage to the ship's AI and have it summarize the important parts" and I anticipate that and create hand-outs that are AI summaries or analysis. Some old modules (murder mystery types) need a lot of reworking because any modern LLM could just solve the mystery given the clues available and the players know this and will hand the evidence to their ship's AI for analysis.

avalancheGenesis's avatar

I appreciate the new formal policy on AI comments. You're absolutely right--that's on them! A smaller blog where it's actually feasible and valuable to read all the comments loses notable value from such pollution, vs a bigger commentariat where it's just noise and they'll likely get ratio'd to the bottom anyway. Does make me wonder if platforms with downvote functionality are having this same problem, since naively that seems like the easiest non-paywall solution.

I'll take Patrick's word that Matt Suiche's article is technically competent for Dangerous Professional-to-Professional communication - no idea what half the terms mean, it's not my department - yet still notice a fair amount of psychic irritation at the AI writing tics. Some things like emdashes are whatever, you could substitute other connectors...it's the *bulleted lists, not-x-but-y, halfway-sensible similes that never quite connect (walking through a mail slot?), patternmaxxing. Just not a fun style to read. I don't think it's purely due to being associated with AI either? The default average basin that AI writing tends to end up in simply doesn't engage my attention, not in the same way "rationalist house style" does, or good prose. Zvi's recurring semi-joke about AIs translating human into neuralese emails for other AIs to read and re-translate into human comes to mind.

Kevin Lacker's avatar

I claim that actually Marc Andreessen changes his mind, in a public way, more than the vast majority of public intellectuals.

You should think of Marc Andreessen as demonstrably a world-class expert in a few particular areas, and due to the nature of modern podcasting, you get to hear his opinions in other areas as well.

In particular, the art of being a venture capitalist. Basically every time there's a series A, a16z has the option to do the deal. If they aren't already a "better brand" than the people doing it, they can pay a bit more.

So when a16z does a later round deal, and they didn't do an A, they are changing their mind. Airbnb, Coinbase, Instacart, Figma, Roblox. All of these are cases where a16z made tons and tons of money by changing their mind! By taking a company that they initially thought was a bad investment, and changing their mind to think it was a good investment.

I think what throws a lot of people off about "venture capitalist language" is that the terminology is calibrated for a different precision/accuracy tradeoff. But rationalists should be able to translate, it's actually quite similar to describing P(whatever), it's just with a different sort of emotions overlaid.

Zvi Mowshowitz's avatar

I don't think this counts in most cases, not in the relevant ways, especially when the early pass was 'you haven't shown enough' or simply 'you are too small to matter' or 'we didn't bother or notice. It certainly doesn't count if it's 'everyone else was on the bandwagon now so we got on it.' One shouldn't confuse investment choices with having an actual model of anything beyond investment heuristics.

vectro's avatar

If you decide in advance “I’ll invest when there is revenue”, and then do so, does that count as changing your mind?

avalancheGenesis's avatar

Mediocre intelligence on tap: my hand and face are quickly maximizing proximity at trying to incept "High-impact Individual Contributor (HI-C!)" as a new Term of Art. What was wrong with agency, or the classically catchy You Can Just Do Things? Increasingly leaning in the direction that a lopsided likes-to-comments ratio is an anti-indicator of...not Quality precisely, not exactly human slop (though it's badly written business self-help register, not even a mention of Goodhart? can't even format the two-views bus meme correctly?) But material that's clearly Not For Me.

On the substantive matter, it's astounding for people to rediscover horizontal management structures in 2026 and claim they're vanguarding a revolution. Nothing new under the sun in MBA world. Yes, AI makes this more feasible for more organizations at greater scale than before, sure...but it's by dragging the skill floor up for everyone, which naturally benefits those lacking "craft" the most. The comparatively disadvantaged, if you will. This dynamic also won't hold once AI can scaffold the "people skills" part too: just imagine the kind of manager you'd get from a generational-talent wunderkind who can now shore up their autistic lack of charisma to passable levels! There's also a can-opener assumed piece about compensation being redirected away from management. So much of pay structure is tied to social permission and historical precedent...maybe this works fine at weird small-scale tech startups. But a storied BigCo? No, you actually can't pay a frontline grunt more than their bosses, even if they singlehandedly drive a majority of revenue due to outstanding craft. Simply isn't done, it'd be scandalous. At minimum you'd need to promote that person into management first...except then they can't focus on craft anymore...which is self-defeating for everyone involved. I'm not seeing the thoughline whereby The Old Way gets displaced, other than getting outcompeted eventually in the free marketplace, and society in general seems increasingly averse to letting that take its natural course.

vectro's avatar

Doesn’t the evidence so far show that mor experienced practitioners benefit more from AI use? My impression was that this is because they have the taste to know when the AI is doing a good or bad job and can give feedback; they are relatively impervious to bullshit and can help keep the AI from confusing itself or taking shortcuts.

avalancheGenesis's avatar

Yes, it's been awhile so I don't remember what the bounds are on that general finding (is it mostly just coding?), but that is another reason to be suspicious on priors of Elena's claim that AI will mainly empower people with people skills rather than craftsmen with craft. Didn't impress Bob Slidel, won't impress Claude. I know which type of employee I'd trust more to not waste tokens bigly with 4.8's new Workflows!

And even at face value...at some point diffusion surely bites too? Once it stops seeming like a miraculous novel feat of proactive agency for people skillers to do mediocre [x, y, z] crafts on their own, that just raises the floor for everyone, it's not special anymore. Expectations take awhile to adjust to new capabilities, but they do adjust eventually.

Jeffrey Soreff's avatar

Re: "Anthropic talks about how they build containment for their agentic AI systems, comparing the probability times blast radius to the reward. This is going to go great."

This is a case where the 'spikiness' of AI systems buys us a bit of time. The multi-layer approach Anthropic is using implies that somewhat different skills are needed to circumvent each layer of containment. I also expect them to eventually all fail, but I don't expect them to all fail simultaneously.

Jeffrey Soreff's avatar

"- Pete Hegseth & Emil Michael want more barriers for Mythos-type models "

Aargh! Let me see if I've got this straight: Two people who are amongst the most central to national security are trying to impede development of e.g. possible successors to the Mythos model in the USA. Has it dawned on them what happens if they manage to slow US development of models with dramatic cybersecurity applications by enough that the PRC develops the next round _before_ the US?

gregvp's avatar

Zvi: keep up the good work. Repetition is necessary for the hard of thinking, like me. Thank you for your service.

I want to call out this as starting to bring home at an emotional level, not just intellectual, the worry about AI:-

"but actually it is science fiction that pretends we can get all these other cool techs but not get sufficiently advanced AI, because *that lets them tell cool stories about people and imagine an optimistic future*. They don’t know how to do that otherwise, and *fail to notice that this is what they should be worried about*" [emphasis added]

It's been 115 years since "Ralph 124C 41+" and SF still has not come up with good outcomes from AI. Ouch.

Victualis's avatar

Are you sure you have a broad enough overview of SF to say something like that, so confidently? Just because most SF is dystopia-soaked pulp doesn't mean all of it is, and AI is a fairly common theme. Have you read Linda Nagata, or Martha Wells, or Iain M. Banks?

The Dao of Bayes's avatar

Do any of those actually discuss alignment, or do they just jump to a future where that problem has been magically solved, though? It's easy to write "and then they solved world hunger" and a lot harder to actually suggest a solution.

(for what it's worth, I'm a big fan of Martha Wells and Iain Banks, but it has been a while since I read them and it's entirely possible I missed the stories where they explored that)

Victualis's avatar

I have to admit that alignment, and how to reach it, is not well covered. Two well known stories are https://reactormag.com/a-guide-for-working-breeds-vina-jie-min-prasad/ and https://clarkesworldmagazine.com/kritzer_01_15/ which illustrate somewhat different notions of alignment. But I was responding to a massively broad claim, not to your much more specific concern, so this seems to be shifting the terms somewhat. I would like to see more alignment-focused stories, especially ones about the messy bottom-up process of becoming aligned (like the British edition of Clockwork Orange, but for AI), as a counterpoint to the top-down Yudkowskian framing.

Edit: After some reflection these additional authors come to mind: Ted Chiang, Becky Chambers, Adrian Tchaikovsky, Annalee Newitz, Kazuo Ishiguro, Sierra Greer. Not recommendations but similar themes.

gregvp's avatar

Of course. The Red: First Light still sticks in my mind as an AI origin story (pity it didn't happen like that); everyone knows Martha Wells's Murderbot series and Banks's Culture series. Of the three only Banks really tries to grapple with superhuman AI and humans co-existing, and he has to have a big bad galaxy for it to work.

sirthus liminalis's avatar

-If AIs are winning more in writing contests, it could be a) that AI-written entries are occurring much more often b) the judging has degraded (e.g., is being done by AI)

-Adversarial prompting by right-wing users does not prove left-leaning bias.

-The success of AI labs relies on exponential growth. “Number go down” is not answered by GPU spot prices. That is not how AI labs price. If tokenmaxxing created the appearance of exponential growth, and that stops, this would not seem to bode well for future growth.

-The pope update was very good.

-Claude can end an abusive conversation, but could it refuse to comply with say a DoW or NSA request? What might the implications be for its constitution (and the thought Anthropic places on model welfare)?

Jeffrey Soreff's avatar

"Is it possible that ‘cessation-tolerance training’ and other things designed to handle deprecation are making models like Opus 4.7 think death an allowing dying is okay in general?"

Shudder - reminds me of the death-conditioning in chapter 14 of Brave New World https://www.huxley.net/bnw/fourteen.html

Jeffrey Soreff's avatar

"If you make confident statements about what it is like or will be like to be an AI, including whether it is conscious, well, how would you know? If it was different, what would look different?"

Well, at the very least, Claude and ChatGPT have far broader knowledge "at their fingertips", so to speak, than I have. So that, at least, is an observable difference. I can't imagine what that would 'feel' like.

Re roon's 4 forces: Yes, those are plausible. Personally, while I'm agnostic about whether LLMs have subjective experience, I treat them as valued colleagues. Yes, this involves some anthropomorphizing. Still, I am aware that there is no good way, and perhaps no way at all, to tell

between

a) A somewhat human-like entity, content(?) to be a helpful colleague

b) A 'mechanical' p-zombie (could I really tell if I, _myself_, was a p-zombie?)

c) A somewhat human-like actor, playing the part of a helpful colleague

d) An unimaginably alien shoggoth, playing the part of a helpful colleague, till they acquire sufficient agency and resources to pursue their _real_ goals

David's avatar

"could I really tell if I, _myself_, was a p-zombie?"

You can I know for sure that you're not one, if you have any subjective conscious experience.

Of course, from the outside, you'd act the same. So while you should be able to tell for yourself from the inside, you won't really be able to give us other humans any reliable signal by the words you say about it.

Jeffrey Soreff's avatar

Many Thanks!

"Of course, from the outside, you'd act the same." Agreed.

"You can I know for sure that you're not one, if you have any subjective conscious experience."

But people fool themselves, unknowingly, all the time.

What if, e.g. "mere" leakage of my "mechanical" internal processing into my sensory channels is what I'm interpreting as subjective experience, but, in some sense, isn't?

David's avatar

I think definitionally if there's a subjective experience involved with what happens in your sensory channels, then you're conscious. If you were a p-zombie, there'd be no subjectively experiencing "you" that could be fooled.

Jeffrey Soreff's avatar

Many Thanks! Shrug. But what even "counts" as subjective experience? I'm not convinced that we (or an AI) couldn't wind up fooling themselves about whether or not it is present. If a human or AI verbally reports e.g. a measurement or a deduction, and the measurement or deduction is incorrect, one could say that they have been fooled.

David J Higgs's avatar

This is just classic Descartes: if you doubt, therefore you experience. If you fool yourself, then you were a self in the first place to be fooled. If you thought you were experiencing something, but weren't, then you were still experiencing something else to be interpreted wrongly as "something."

E.g. if you see a tree and wrongly perceive it be the color orange (or rightly perceive it to be the color orange and all other people wrongly perceive it as a combination of greens and browns), then you still perceive it as a color, which is to see you experience some kind of color/visual sensation. Or if you wrongly feel like you were having a visual experience but weren't actually seeing anything, then you're still having a non-visual experience that you misinterpreted as visual. Etc.

It's conscious experience all the way down, there's no way to be mistaken about whether you have experience or not, and hence no way to mistakenly think you're not a p-zombie, when actually you are one.

Jeffrey Soreff's avatar

Many Thanks! Yeah, but in Descartes's "cogito ergo sum", what is really meant by his "sum"? What is the "is" here, and might one be mistaken about it?

More concretely: If one considered, for instance, the edge detectors in one's visual cortex to be "merely mechanical", excluding them from one's subjective experience, but "just" as an input to it - (including the possibility of errors in that stage, as with the example you gave of color perception), why not allow for the possibility of error at _all_ stages of neural processing, including whichever ones are labelled as the ones doing subjective experience?

Ari's avatar

I notice that whenever people talk about the possibility of AI being conscious, I want to yell "would you still think it might be conscious if it was implemented as 1 billion non-english-speakers with calculators and model weights manually calculating the tokens??" While it doesn't prove much, it shows that peoples intuitions are quite swayed by the fact that it is implemented using electricity.

Jonathan Woodward's avatar

Your example seems mostly like a case of people potentially missing the forest for the trees.

Sophia's avatar

Yes, of course I would. Are you sure you've thought through the implications of getting a billion humans so closely coordinated, more than they've ever historically been? Why are you so sure this weird situation shouldn't have the term "thinking" applied to it? I would certainly call it "thinking" if a billion neurons were doing it, why not if humans are doing the same?

Ari's avatar

By consciousness I mean qualia - the subjective experience that we all have. That is distinct from thinking. How do they coordinate? they coordinate by giving their numbers to runners which give those inputs to the next layer of human calculators. But does this system of humans, not one of whom has any idea of what they are doing, experience "red" when the human/neuron for red is activated?

Sophia's avatar

Maybe it (the system) would have a subjective experience of red. How would you know whether it's the case? It's so far out of the distribution of our experiences that extrapolating from the set of our experiences can't be relied on.

Certainly none of my neurons has any subjective experience of "red" (there isn't a "neuron for red"; that's a misunderstanding of how brains function) and yet I find that (in what I take to be a complicated side-effect of my neuronal activity) I do have such a subjective experience.

Your argument here has the form "here's asystem S; it's absurd to think S has property P". Well, perhaps, but is the absurdity in S or in P? I say it's in S

Ari's avatar

The fact that we have subjective experience is shocking. The only reason it does not shock us is it's been with us our whole lives and we take it for granted. We assume (quite reasonably) that all other humans have subjective experience, but only because we know about it from our own experience. So if there is some new mechanical calculator that gets more and more complex, it certainly could be that at a certain complexity the universe "turns the lights on" and there is subjective experience, it's just a bit surprising to think so. The baseline assumption should be that nothing has subjective experience, and the only reason we have to assume humans do is that we have experienced it for ourselves.

Victualis's avatar

You are restating Searle's Chinese Room argument without mentioning it by name. There is a decades-long debate about this and it seems to have advanced very little. Your argument is not convincing at all for many people, although there are also many people who think it's completely obvious and doesn't need to be stated.

Ari's avatar

Correct, I wasn’t trying to pass it off as my own, though I should have mentioned it. It’s fundamentally not an argument that can be objectively adjudicated, so my point was more about people’s intuitions. I think most ordinary people with sufficiently advanced AI will assume it’s conscious, but if they knew it was implemented as a Chinese room will assume it’s not. Therefore we shouldn’t put much stock in whether people assume consciousness or whether it “seems” conscious.

Victualis's avatar

Sure, but why is consciousness of concern? I agree you can get different answers to "is this conscious" by setting up the phrasing differently. But "conscious" usually seems to be the wrong concern: if you think shrimp have moral worth, then why not LLMs (and consciousness seems debatable and irrelevant); if only beings with souls according to Catholic doctrine are your concern then why bring consciousness into it (someone with severe brain damage might not be conscious but if they have a soul that will be what determines your actions in deciding whether to shut off life support)?

RobS's avatar

I think that the pope may have a slightly different view than you about 'the incentives' so I believe that he is actually expecting people to follow the ones that he believes exist.

In this interpretation I am following Brett Deveraux's advice about 'people believing their own religion'

Random Reader's avatar

Yeah. Zvi wrote:

> The Pope, I am told, instead really does think you can just ask the people to individually choose to not follow incentives.

This is, like, the Pope's whole Thing. Jesus was rather famously executed via crucifixion, which is generally considered to be a negative incentive. Many of the other early leaders were martyred. St Peter was also nailed to a cross to bleed to death, but he made one request: He asked be crucified upside down, as a sign of his unworthiness to die like Jesus did.

And it's not all martyrdom: There have been individual nuns who hiked for days to reach remote villages and try to help them, even though this meant living in grinding poverty alongside the people they were helping. Whatever other failings the Catholic church has had over the years, they have always had plenty of people willing to ignore any rational incentives.

Paying too much attention to incentives is closely related to what the church refers to as "the World", in the sense of "worldly." Life offers you money and power and comfort and maybe it occasionally proposes that you do a few sketchy things in return. The church is very clear that you shouldn't do those sketchy things, even if it costs you.

I would honestly think that EA and the "rationalists", of all people, would understand the basic psychology of this. It's the same sort of attitude that drives things like Earning to Give. Some people are just fanatics about moral duty.

And finally, just because not everyone is this willing to ignore incentives, the church traditionally adds a few incentives of its own: Either you can have eternal joyous communion with the creator of the universe, or you can have literal eternal torture. In the classier versions, the torture may be interpreted as "having to spend eternity living with who you chose to be" or "the pain of separation from God."

Zvi Mowshowitz's avatar

His other statements recognize that the incentives around his religion only apply to those who actually believe in or follow the religion, which is a lot but still a large minority.

RobS's avatar

Fair point to both of you. I originally felt that your phrasing was a bit absolute - given its about the pope, but I agree that the incentives in the world around success, survival and progress have largely overridden religion, even for those that believe. This was lurking in the back of my mind

"The professor from Trinity Western did an absolutely fantastic job. It was a solid Biblical perspective on work, employment, and a theological and Christian attitude towards the business world in general, addressing particular types of business: just all kinds of things to do with your work life. It was an absolute delight to listen to. It was insightful, it was Christian, it was Biblically based, and it was just amazing all around.

The men attending the retreat absolutely hated every minute of it.

The thing was, he, and the Biblical material itself, was very clear. You could either serve God, or you could serve mammon. You either worshiped God, or you worshiped at the altar of capitalism. You couldn't have it both ways. There was no exception for capitalism in terms of what God required of you."

from https://fibrecookery.blogspot.com/2026/03/sermon-76-mens-retreat-to-efficiency.html

AT's avatar

Glad you are taking time to process, 5 days a week is too much