Discussion about this post

User's avatar
Olivia Horbowy's avatar

Regarding the extensive risk factors: the safe harbour that protects forward-looking statements from securities suits doesn’t apply to an IPO. Section 11 makes the issuer strictly liable for anything misleading in the registration statement. There’s no need to prove intent. So every risk that could materialise has to be in there. Legally, over-warning costs nothing, and under-warning can be costly. So statements like “concerns have been raised” disclose the risk without stating what the company believes, or with what probability.

As a lawyer, I find the governance risk more worrying. At OpenAI, we saw employees informally override the board in 2023. At Anthropic, that right is formally in the charter: the founders plus the employees can remove the LTBT. Anthropic is incorporated in Delaware. Under Delaware law, only stockholders holding 2% of the shares (or $2m for a listed company) can sue to enforce the public benefit. Society is the beneficiary but has no standing. By contrast, large US banks must have a risk committee with a risk-management expert. But banking regulation is extensive because actual crises drove it. Meanwhile, a company whose own S-1 mentions “existential risks” has no such requirement for its Safety and Risk Committee, mostly because the regulations haven’t been adopted yet. And now we’re seeing Anthropic produce its own draft of federal regulation. I’m not saying that’s bad, but it’s obviously biased towards the sponsor who paid for it.

“Our civilization really does not want to live.” I think we’re simply too slow. You wrote about being AGI-pilled rather than ASI-pilled, and I don’t think there’s enough public awareness of the top layer yet, not even among the people whose job is to write the laws. Without regulation, labs agreeing to slow down look like competitors colluding in breach of antitrust law. Without a safe harbour, compliance and safety pull in opposite directions. And none of that addresses the race with China. That’s why I think pacing needs a body above the nations, to which the management of the major risks is delegated.

Isha Yiras Hashem's avatar

"There are four broad categories of risk: misuse, loss of control, authoritarian abuse and economic disruption."

This reminded me of how Harvard Magazine proposes (apparently seriously) a canine agentic framework of AI, also in 4 suspiciously neat categories, though theirs has an aura of historical respectability as well.

The "canine agentic framework" is derived from medieval English animal law which had four categories:

1. domesticated dogs

These are low risk AI systems, with clear owners, like the NYC one. How domesticated they are is how corrigible the model is to the person or company using it.

They didn't use the word corrigible, but IYH knows what they meant.

2. dangerous owned animals

These are higher risk systems that remain under control, they're kind of like pit bulls

3. wild foxes

Systems that are low risk and unknown are kind of like foxes we can tolerate them and they'll be difficult to control

4. wolves

High risk AI systems operating without identifiable responsibility or human oversight this reminded me of hugging face and in fact that's exactly the example they gave who would be responsible for the loss incurred by such an attack.

They do not think that dangerous AI need to be eliminated, like wolves, but they do need to be regulated by the government, perhaps with dog tags. Digital ones.

22 more comments...

Ready for more?