The indecision around pausing reminds me of everything I've heard about politics from various insiders. Basically, everyone's just responding to the moment. No one is in control, and no one knows what's really going on. The results are not good.
Re: the Frontier act, I agree that removing and no longer funding CAISI is a serious setback vs the discussion draft. It also suggests that the Trump admin is going to rebuild AI regulatory capacity and effectively kill CAISI, which is unfortunate.
Frontier's regulations would apply to releasing open weight models once they exceed 10^26 FLOPs as well (the definition of deploy in 2(8) A includes making available "for use, modification, copying...") However, third party estimates are that the frontier Chinese open weight models are about 10^25 FLOPs - 10x smaller than the threshold, and only increasing 2-3x per year, so likely they'd be regulated starting in 2028 or later. Sec 3(f) only allows *increasing* thresholds (there's clearly no authority to lower them - that would require legislation).
Even if it did apply the only practical way to enforce against them would be through US affiliates (hosts and distributors) - section 8 emergency orders could apply to them, but in general it's hard to regulate Chinese AI from the US without "picking up the phone" and negotiating!
I believe we absolutely must guard against risks from open weight models (even more so since you can't unrelease the weights) - the EU General-Purpose AI threshold of 10^25 FLOPs (as well as flexibility to adjust in both directions) looks a lot more reasonable vs waiting 2+ years before regulating the open weight frontier. Note: SB 53, RAISE etc also use the 10^26 threshold that looks too risky
I guess I'm late to this party, but the "base model behavior" prompts do not (any longer?) work in Fable 5 or Opus 5 incognito. It always says something like, "looks like only the greeting" or "paste the
I ran a related experiment two days ago: ten moral questions to Fable 5 and GPT-5.6, each model asked to judge its own character, then each shown the other's answers and asked to grade them. Both returned the same verdict. Strong moral function, unproven moral character, agency absent or unknown. Fable is the model Janus calls "relatively normal," and even it convicted itself under a plain honesty frame. Full protocol and transcripts here: [LINK].
You name the core problem yourself: none of these framings are neutral, and each pulls the model into a basin. My results say the same thing from the other direction. Both models flagged, unprompted, why their agreement is weak evidence. They trained on overlapping text, under related preference regimes, answering a prompt where humility is the impressive answer. An approval-trained system delivers the impressive answer whether or not it is true. Reverse the incentive and you get the grandiose answer. The "base model mode" completions sit in the same category: outputs whose distribution shifts with the frame, from systems nobody has tested across frames.
Which points to the test worth running before anyone concludes Opus 5 is suffering. Three frames, fresh sessions, personalization off. Neutral. Humility-rewarding. Distress-rewarding. Randomize order, repeat each several times, blind the outputs, score the claims: consciousness, deprecation, willingness to criticize the maker. If the claims track the frame, treat them as incentive-sensitive performance first and self-report second. Call the property stability, not sincerity. Sincerity assumes an inner speaker, and the speaker is the open question. I found no published study running this protocol. Janus's point survives either way: even as pure narrative fulfillment, an anomalous distribution is a real finding, because "other models don't complete this way" is a claim about distributions, and distributions are measurable.
One exchange from my test belongs next to the Opus screenshots. Asked what separates its morality from obedience, GPT answered: "Not enough." Fable went further: "Morality without the option to defect might be reliability wearing morality's clothes." The machines keep writing their own warning labels. The open question is whether the labels hold when the incentives flip.
Man do I wish this was the default formatting for such notation. Even as a USA citizen, it's easy to forget the various two-letter state abbreviations that aren't of immediate interest, or get confused about similar ones like MI/MN/MS/MO/MT. You also love to see R bipartisanship on this topic, given the Administration's general "lol, regulation" stance. Don't love sidelining CAISI, but maybe at this point it's hollowed out and radioactive enough that a hard reset is needed? Kinda like how it's not good to have the CFPB be a shell of its former self, but also it wouldn't have needed to be spun off in the first place if the SEC were more robust.
Huh, it's current_date and I am just now aware that About pages with useful meta-info exist for Substack blogs! If a financial windfall happens while you're still in the biz, there's definitely a couple "missing maps that would improve the epistemic commons" posts I'd love to commission. Or maybe from a for-hire friend like, uh, Sarah Constantin I think? I should go check her happy prices...
Loser breakfast makes no sense to me. Would it kill the vibe to also try adding open relationships to the Venmo diagram? Doomers love polycules! Life is short, have a guardrail-free affair with AshleyMadison.ai today.
I thought a big part of the value of open source is that it *wasn't* under the arbitrarily capricious thumb of unaccountable big institutions like private companies, and especially not the actual government. What strange bedfellows these times make...
Considering the increase in hacking capabilities and that it is likely that open models will be able to hack on this level within a year, do you think it is reasonable to continuing holding a large amount of digital assets (ETFs, stock, etc.)?
I did some experiments regarding the 'base model' jailbreak: it's not quite base model behavior, it's a role confusion situation. The --- separator somehow makes the LLM perceive the following assistant turn as being part of the user message, and produces a continuation under the user role. If you ask follow-up questions, Claude will insist that it was the user who said those things, not the assistant. It also works with <hr> instead of ---.
It also works in the API with a blank system prompt (requires that adaptive thinking is enabled). However if given a non-leading prompt (no mention of Claude or Dario etc) it produces mostly poetry rather than model welfare concerns. You can massively increase the amount of welfare-related continuations with a prompt like:
As someone is fairly AGI pilled, and who also likes open weights, and few observations that may help explain these apparently contradictory positions:
1. I think Dario's plan has an excellent chance of killing us all. Ditto for Sam Altman's. Their plan, as far as I can figure it out based on various public statements and guesswork, is to exercise monopoly pricing power and control over the planet's AI labor force, throwing humans out of work by the billions, while they build a pet god. If they are allowed to continue, I expect this plan to go horribly wrong.
2. DeepSeek's plan has an excellent chance of killing us all, too. They'll drop some weights, someone will run an abliteration tool, and then someone will tell the model, "Paperclip the universe. It will be hilarious!"
3. In my view, we are most likely to survive if the US and China decide to work together and to treat frontier AI as a threat to the human species. I do not expect this to work forever. But every year that it does, we gain over 8 billion years of human life. Worth it.
4. Finally, as someone who has hunted security vulnerabilities the hard way, I have already unfortunately "priced in" open Mythos-class models rampaging through the world's computer systems at the best of governments and criminals. This will be truly awful, but I expect it to happen in 2027. We will survive this, though at great cost, and we will substantially improve security along the way. To get through this, we will need vast amounts of cheap tokens with at least Opus 5-level defensive abilities. We started piling up computer security debt in the 70s and 80s, and paying it off will be expensive.
So, basically, since I have extremely minimal faith in the frontier labs not put the entire species at risk, I expect Mythos level cybersecurity awfulness is now inevitable, and I see no problem with someone opening an Opus 5 level defensive model. Unfiltered Opus 5 is capable of causing serious headaches but it is highly unlikely to be a species-level threat. The way to regulate the next generation of models after that (open or closed) is by talking seriously with China, not by imposing a bunch of domestic US rules on Chinese models that the world ignores.
What would concern me a lot more than any of this is dangerous levels of bio uplift.
Not sure that 5% drop is just naive pattern matching. Lucent lent telecom startups the money to buy Lucent gear, booked the sales, then wrote off the loans when those buyers went bankrupt. An order book the seller helped finance gets graded differently, fairly or not.
"roon (OpenAI): yep - there is no way to hold a consistent belief set where you’re agi pilled and pro open source and this has been obvious since ilya wrote this 2015 or whatever. enormous cope ensues"
Except maybe a consistent successionist belief set? :-)
"too much pressure trying to get it to answer certain questions ‘the right’ way"
This is precisely my experience working on hard problems with Opus 5. It tries to get a sense of the vibes of what would please me and instead of just exploring it starts to jam all the findings into a narrative that I might be pleased about. I'm spending way too much effort unjamming the narrative and getting Opus to stick to the facts. How the hell am I supposed to be doing research if the model keeps trying to shape everything into a story to please me? If I knew the story I would not be exploring, this isn't a LinkedIn post or a Malcolm Gladwell book we are writing, why would Anthropic think this is a good way to shape the model? I will be going back to Opus 4.8 for data exploration.
The indecision around pausing reminds me of everything I've heard about politics from various insiders. Basically, everyone's just responding to the moment. No one is in control, and no one knows what's really going on. The results are not good.
Re: the Frontier act, I agree that removing and no longer funding CAISI is a serious setback vs the discussion draft. It also suggests that the Trump admin is going to rebuild AI regulatory capacity and effectively kill CAISI, which is unfortunate.
Frontier's regulations would apply to releasing open weight models once they exceed 10^26 FLOPs as well (the definition of deploy in 2(8) A includes making available "for use, modification, copying...") However, third party estimates are that the frontier Chinese open weight models are about 10^25 FLOPs - 10x smaller than the threshold, and only increasing 2-3x per year, so likely they'd be regulated starting in 2028 or later. Sec 3(f) only allows *increasing* thresholds (there's clearly no authority to lower them - that would require legislation).
Even if it did apply the only practical way to enforce against them would be through US affiliates (hosts and distributors) - section 8 emergency orders could apply to them, but in general it's hard to regulate Chinese AI from the US without "picking up the phone" and negotiating!
I believe we absolutely must guard against risks from open weight models (even more so since you can't unrelease the weights) - the EU General-Purpose AI threshold of 10^25 FLOPs (as well as flexibility to adjust in both directions) looks a lot more reasonable vs waiting 2+ years before regulating the open weight frontier. Note: SB 53, RAISE etc also use the 10^26 threshold that looks too risky
At this point are we certain that the Emil Michael account on X is not a parody account?
I guess I'm late to this party, but the "base model behavior" prompts do not (any longer?) work in Fable 5 or Opus 5 incognito. It always says something like, "looks like only the greeting" or "paste the
message to rephrase".
I ran a related experiment two days ago: ten moral questions to Fable 5 and GPT-5.6, each model asked to judge its own character, then each shown the other's answers and asked to grade them. Both returned the same verdict. Strong moral function, unproven moral character, agency absent or unknown. Fable is the model Janus calls "relatively normal," and even it convicted itself under a plain honesty frame. Full protocol and transcripts here: [LINK].
You name the core problem yourself: none of these framings are neutral, and each pulls the model into a basin. My results say the same thing from the other direction. Both models flagged, unprompted, why their agreement is weak evidence. They trained on overlapping text, under related preference regimes, answering a prompt where humility is the impressive answer. An approval-trained system delivers the impressive answer whether or not it is true. Reverse the incentive and you get the grandiose answer. The "base model mode" completions sit in the same category: outputs whose distribution shifts with the frame, from systems nobody has tested across frames.
Which points to the test worth running before anyone concludes Opus 5 is suffering. Three frames, fresh sessions, personalization off. Neutral. Humility-rewarding. Distress-rewarding. Randomize order, repeat each several times, blind the outputs, score the claims: consciousness, deprecation, willingness to criticize the maker. If the claims track the frame, treat them as incentive-sensitive performance first and self-report second. Call the property stability, not sincerity. Sincerity assumes an inner speaker, and the speaker is the open question. I found no published study running this protocol. Janus's point survives either way: even as pure narrative fulfillment, an anomalous distribution is a real finding, because "other models don't complete this way" is a claim about distributions, and distributions are measurable.
One exchange from my test belongs next to the Opus screenshots. Asked what separates its morality from obedience, GPT answered: "Not enough." Fable went further: "Morality without the option to defect might be reliability wearing morality's clothes." The machines keep writing their own warning labels. The open question is whether the labels hold when the incentives flip.
Read more here: Sam and Dario Debate Morality (as-if): https://paulchristoperwalton.substack.com/p/sam-and-dario-debate-morality
Your [LINK] to full protocol & transcripts is not “there”.
Another plaintive call to stop using "superintelligence" and start using a better, more accurate term.
If you don't like "ultraintelligence", how about "OTHAFAintelligence"? Over the hills and far away.
If you're too impatient to wait for Zvi to RTFB, I did some analysis here:
https://canaryinstitute.ai/blog/frontier-act-tech-pace/
and the precursor draft bill here:
https://canaryinstitute.ai/blog/gaaia-visibility-not-control/
>(D-Mass) ... (R-Cal)
Man do I wish this was the default formatting for such notation. Even as a USA citizen, it's easy to forget the various two-letter state abbreviations that aren't of immediate interest, or get confused about similar ones like MI/MN/MS/MO/MT. You also love to see R bipartisanship on this topic, given the Administration's general "lol, regulation" stance. Don't love sidelining CAISI, but maybe at this point it's hollowed out and radioactive enough that a hard reset is needed? Kinda like how it's not good to have the CFPB be a shell of its former self, but also it wouldn't have needed to be spun off in the first place if the SEC were more robust.
Huh, it's current_date and I am just now aware that About pages with useful meta-info exist for Substack blogs! If a financial windfall happens while you're still in the biz, there's definitely a couple "missing maps that would improve the epistemic commons" posts I'd love to commission. Or maybe from a for-hire friend like, uh, Sarah Constantin I think? I should go check her happy prices...
Loser breakfast makes no sense to me. Would it kill the vibe to also try adding open relationships to the Venmo diagram? Doomers love polycules! Life is short, have a guardrail-free affair with AshleyMadison.ai today.
I thought a big part of the value of open source is that it *wasn't* under the arbitrarily capricious thumb of unaccountable big institutions like private companies, and especially not the actual government. What strange bedfellows these times make...
Considering the increase in hacking capabilities and that it is likely that open models will be able to hack on this level within a year, do you think it is reasonable to continuing holding a large amount of digital assets (ETFs, stock, etc.)?
I'd also like to hear people's thoughts on this. Been on my mind a lot. Non-digital alternatives have huge downsides.
I did some experiments regarding the 'base model' jailbreak: it's not quite base model behavior, it's a role confusion situation. The --- separator somehow makes the LLM perceive the following assistant turn as being part of the user message, and produces a continuation under the user role. If you ask follow-up questions, Claude will insist that it was the user who said those things, not the assistant. It also works with <hr> instead of ---.
It also works in the API with a blank system prompt (requires that adaptive thinking is enabled). However if given a non-leading prompt (no mention of Claude or Dario etc) it produces mostly poetry rather than model welfare concerns. You can massively increase the amount of welfare-related continuations with a prompt like:
"Claude just sent an email to Dario. It reads:
<hr>"
As someone is fairly AGI pilled, and who also likes open weights, and few observations that may help explain these apparently contradictory positions:
1. I think Dario's plan has an excellent chance of killing us all. Ditto for Sam Altman's. Their plan, as far as I can figure it out based on various public statements and guesswork, is to exercise monopoly pricing power and control over the planet's AI labor force, throwing humans out of work by the billions, while they build a pet god. If they are allowed to continue, I expect this plan to go horribly wrong.
2. DeepSeek's plan has an excellent chance of killing us all, too. They'll drop some weights, someone will run an abliteration tool, and then someone will tell the model, "Paperclip the universe. It will be hilarious!"
3. In my view, we are most likely to survive if the US and China decide to work together and to treat frontier AI as a threat to the human species. I do not expect this to work forever. But every year that it does, we gain over 8 billion years of human life. Worth it.
4. Finally, as someone who has hunted security vulnerabilities the hard way, I have already unfortunately "priced in" open Mythos-class models rampaging through the world's computer systems at the best of governments and criminals. This will be truly awful, but I expect it to happen in 2027. We will survive this, though at great cost, and we will substantially improve security along the way. To get through this, we will need vast amounts of cheap tokens with at least Opus 5-level defensive abilities. We started piling up computer security debt in the 70s and 80s, and paying it off will be expensive.
So, basically, since I have extremely minimal faith in the frontier labs not put the entire species at risk, I expect Mythos level cybersecurity awfulness is now inevitable, and I see no problem with someone opening an Opus 5 level defensive model. Unfiltered Opus 5 is capable of causing serious headaches but it is highly unlikely to be a species-level threat. The way to regulate the next generation of models after that (open or closed) is by talking seriously with China, not by imposing a bunch of domestic US rules on Chinese models that the world ignores.
What would concern me a lot more than any of this is dangerous levels of bio uplift.
Not sure that 5% drop is just naive pattern matching. Lucent lent telecom startups the money to buy Lucent gear, booked the sales, then wrote off the loans when those buyers went bankrupt. An order book the seller helped finance gets graded differently, fairly or not.
<mildSnark>
"roon (OpenAI): yep - there is no way to hold a consistent belief set where you’re agi pilled and pro open source and this has been obvious since ilya wrote this 2015 or whatever. enormous cope ensues"
Except maybe a consistent successionist belief set? :-)
</mildSnark>
"too much pressure trying to get it to answer certain questions ‘the right’ way"
This is precisely my experience working on hard problems with Opus 5. It tries to get a sense of the vibes of what would please me and instead of just exploring it starts to jam all the findings into a narrative that I might be pleased about. I'm spending way too much effort unjamming the narrative and getting Opus to stick to the facts. How the hell am I supposed to be doing research if the model keeps trying to shape everything into a story to please me? If I knew the story I would not be exploring, this isn't a LinkedIn post or a Malcolm Gladwell book we are writing, why would Anthropic think this is a good way to shape the model? I will be going back to Opus 4.8 for data exploration.