OpenAI #16: A History and a Proposal
The real news today is that Anthropic has partnered with the top companies in cybersecurity to try and patch everyone’s systems to fix all the thousands of zero-day exploits found by their new model Claude Mythos.
I’ll be sorting through that over the coming days. For now, we instead have stories from OpenAI.
In particular there are three stories.
There’s a massive 18,000 word article in The New Yorker about Sam Altman and the history of OpenAI as it relates to his trustworthiness. No trust.
There’s also OpenAI’s proposal for a ‘new deal’ of sorts. No deal.
Then there is an actual deal, where they bought TBPN. RIP.
Table of Contents
Part 1: OpenAI: The Histories
If you ask a question in a headline, the answer is almost always no.
This week’s example, from Ronan Farrow and Andrew Marantz at The New Yorker is ‘Sam Altman May Control Our Future—Can He Be Trusted?’
To which in earlier days Altman himself replied no, and this week he replied with a PR workshopped answer that was still ultimately no, and also very obviously the answer to this question is no.
The article is less about the question in the headline - we all know the answer to that one - and more about creating a profile of Altman, laying out the history of various incidents in Altman’s and OpenAI’s past, and litigating basically everything.
Mostly I found the article fair, and definitely fully within the rules of Bounded Distrust, but it is certainly in the ‘make it look suspicious’ school of journalistic description. I think they were trying to be fair, and mostly succeeded, but there are a decent number of things that could have been cut, or where the framing is more of a gotcha than I would like.
It is 18k words long and covers a lot of stuff we’ve covered extensively before, so I’ll skip over a lot. It is well-written, in case you want to consider reading the whole thing.
I will instead attempt to hit the highlights.
The Battle of the Board
We start out with the Battle of the Board, as the central event in OpenAI’s history.
All the reporting here was consistent with my past reporting. There were a handful of new meaningful details or confirmations.
One ongoing question is how Altman reacted to the initial firing. Altman claims he was going to accept the firing and then others fought back, eventually getting him to get his head back in the game. That’s not the picture here.
Ronan Farrow and Andrew Marantz: The day that Altman was fired, he flew back to his twenty-seven-million-dollar mansion in San Francisco, which has panoramic views of the bay and once featured a cantilevered infinity pool, and set up what he called a “sort of government-in-exile.”
… Altman interrupted his “war room” at six o’clock each evening with a round of Negronis. “You need to chill,” he recalls saying. “Whatever’s gonna happen is gonna happen.” But, he added, his phone records show that he was on calls for more than twelve hours a day.
Thanks For The Memos
We get confirmation that then-board-member Ilya Sutskever assembled seventy pages of Slack messages to show why Altman and Brockman shouldn’t be running OpenAI, which out of fear of Altman he alas sent as disappearing messages, and then the board attempting to fire Altman. It is a shame that we cannot see it directly.
I Am What I Am
This is quite the quote, which Altman doesn’t recall but doesn’t deny.
Ronan Farrow and Andrew Marantz: In a tense call after Altman’s firing, the board pressed him to acknowledge a pattern of deception. “This is just so fucked up,” he said repeatedly, according to people on the call. “I can’t change my personality.”
No, I suppose you can’t, sir. And yes, I agree, that was all pretty fucked up.
Altman says he is under tons of stress and suffering from decision fatigue. On that point I fully believe him, and I sympathize.
That’s Not What I Said
Paul Graham disputes the account given here of how Altman left YC, but his claim that the post’s account is false seems to be an inaccurate statement about what the post’s account actually said, and fails to contradict it.
There Will Be No Investigation
In a later section they outline how Altman got Summers and Taylor to not do their promised investigation of Altman’s previous actions, focusing only on narrow claims about specific potential criminality and looking for a way to ‘acquit’ Altman and get things back to normal. No criminal violations? Then get back to work. The investigation was mostly fake and failed to even release a report.
Many former and current OpenAI employees told us that they were shocked by the lack of disclosure. Altman said he believed that all the board members who joined in the aftermath of his reinstatement received the oral briefings. “That’s an absolute, outright lie,” a person with direct knowledge of the situation said.
Some board members told us that ongoing questions about the integrity of the report could prompt, as one put it, “a need for another investigation.”
Look at those nerds, acting as if words have meaning and truth matters. Fools.
Musk Versus Altman
The story of the falling out with Musk seems consistent with Altman’s version of events, and with my previous understanding, and being inconsistent with the story Musk tells. For all the places I think Altman is inconsistently candid, I think his telling of these events is the more accurate one.
The thing that stuck out most was getting this in detail:
But, unbeknownst to them, he also struck a secret handshake deal with Brockman and Sutskever: Altman would get the C.E.O. title; in exchange, he agreed to resign if the other two deemed it necessary. (He disputed this characterization, saying he took the C.E.O. role only because he was asked to. All three men confirmed that the pact existed, though Brockman said that it was informal. “He unilaterally told us that he’d step down if we ever both asked him to,” he told us. “We objected to this idea, but he said it was important to him. It was purely altruistic.”) Later, the board was alarmed to learn that its C.E.O. had essentially appointed his own shadow board.
By this point, we all know what an informal agreement like that is worth.
Amodei Versus Altman
This detail seems important in the story of how and why Dario Amodei left OpenAI.
Amodei presented Altman with a ranked list of safety demands, placing the preservation of the merge-and-assist clause at the very top. Altman agreed to that demand, but in June, as the deal was closing, Amodei discovered that a provision granting Microsoft the power to block OpenAI from any mergers had been added. “Eighty per cent of the charter was just betrayed,” Amodei recalled.
He confronted Altman, who denied that the provision existed. Amodei read it aloud, pointing to the text, and ultimately forced another colleague to confirm its existence to Altman directly. (Altman doesn’t remember this.)
Correction: Altman claims he does not remember this.
After that, one can see how things might escalate quickly.
Ronan Farrow and Andrew Marantz: Amodei’s notes describe escalating tense encounters, including one, months later, in which Altman summoned him and his sister, Daniela, who worked in safety and policy at the company, to tell them that he had it on “good authority” from a senior executive that they had been plotting a coup. Daniela, the notes continue, “lost it,” and brought in that executive, who denied having said anything.
As one person briefed on the exchange recalled, Altman then denied having made the claim. “I didn’t even say that,” he said. “You just said that,” Daniela replied. (Altman said that this was not quite his recollection, and that he had accused the Amodeis only of “political behavior.”) In 2020, Amodei, Daniela, and other colleagues left to found Anthropic, which is now one of OpenAI’s chief rivals.
Sydney Versus Altman
Here’s one confirmation, of the anecdote that when someone at Microsoft tested an early version of GPT-4 in India without safety board approval, in ways that ended up having permanent ramifications via embedding the concept of Sydney into the training data, he neglected to even inform the board that this had happened.
As McCauley, the board member and entrepreneur, left the meeting, an employee pulled her aside and asked if she knew about “the breach” in India. Altman, during many hours of briefing with the board, had neglected to mention that Microsoft had released an early version of ChatGPT in India without completing a required safety review. “It just was kind of completely ignored,” Jacob Hilton, an OpenAI researcher at the time, said.
Various examples are cited of people raising concerns about OpenAI and Altman’s lack of actual commitment to safety, and prioritization of profits over safety.
Highest Bidder Versus Altman
They lay out the whole ‘let’s offer it to Putin and auction AI to the highest bidder.’
[Hedley] was aghast: “The premise, which they didn’t dispute, was ‘We’re talking about potentially the most destructive technology ever invented—what if we sold it to Putin?’
… Brainstorming sessions often produce outlandish ideas. Hedley hoped that this one, which came to be known internally as the “countries plan,” would be dropped. Instead, according to several people involved and to contemporaneous documents, OpenAI executives seemed to grow only more excited about it. Brockman’s goal, according to Jack Clark, OpenAI’s policy director at the time, was to “set up, basically, a prisoner’s dilemma, where all of the nations need to give us funding,” and that “implicitly makes not giving us funding kind of dangerous.” A junior researcher recalled thinking, as the plan was detailed at a company meeting, “This is completely fucking insane.”
Executives discussed the approach with at least one potential donor. But later that month, after several employees talked about quitting, the plan was abandoned. Altman “would lose staff,” Hedley said. “I feel like that was always something that had more weight in Sam’s calculations than ‘This is not a good plan because it might cause a war between great powers.’ ”
Brockman claims he was not serious, but this seems like well-documented reporting that it was actually rather serious. I think you should be rather alarmed that this got taken seriously as a proposal.
I won’t spoil tales of various other attempts to raise money. They’re not pretty.
I will note his randomly announcing, on Twitter, his intent to raise trillions.
In February, 2024, the Wall Street Journal published a description of Altman’s vision for ChipCo. He conceived of it as a joint entity funded by an investment of five to seven trillion dollars. (“fk it why not 8,” he tweeted.) This was how many employees learned about the plan. “Everyone was, like, ‘Wait, what?’ ” Leike recalled. Altman insisted at an internal meeting that safety teams had been “looped in.” Leike sent a message urging him not to falsely suggest that the effort had been approved.
Risky Business
One common throughline of the post is Altman’s evolving positions and communications concerning AI risks, including AI existential risks, as he moved from being a relatively good voice to a relatively poor voice.
When it was useful to him to speak about risks, he played them up. When it was useful for him to downplay them, as he increasingly finds it now, he downplays them, often using similar language such as emphasizing different aspects of a ‘Manhattan Project.’
I do think Altman was being more candid for longer than was efficient for the business, but this is not obvious. It’s possible he followed incentives the whole way, if he had to focus a lot on employee retention. OpenAI was relying, from the start, on employees thinking it was an especially noble place to work.
Altman is quoted warning about AI dangers before it was cool, and back when such statements did not conflict with his business. According to this account, it was Altman who convinced Musk to contribute, on the basis that Google should not be allowed to be the one to do AI.
One thing emphasized is that ‘we are the good guys doing the right thing’ was absolutely a key selling point, even the key selling point, to getting OpenAI off the ground and recruiting key initial people, including Ilya Sutskever. This greatly reduced cost of capital.
This was centered around making it clear that AI going wrong carried big risks, and in particular that the AIs getting out of our control could be an existential risk to humanity, but also concentration of power and other risks.
He wrote on his blog in 2015 that superhuman machine intelligence “does not have to be the inherently evil sci-fi version to kill us all. A more probable scenario is that it simply doesn’t care about us much either way, but in an effort to accomplish some other goal . . . wipes us out.”
OpenAI’s founders vowed not to privilege speed over safety, and the organization’s articles of incorporation made benefitting humanity a legally binding duty. If A.I. was going to be the most powerful technology in history, it followed that any individual with sole control over it stood to become uniquely powerful—a scenario that the founders referred to as an “AGI dictatorship.”
Altman told early recruits that OpenAI would remain a pure nonprofit, and programmers took significant pay cuts to work there.
Altman continued to use OpenAI’s commitment to safety, and promises of grand investments and gestured, as a recruiting tool. The article cites a clear example from late 2022, where he talks of things he did not do.
Superalignment Was Always Fake
We then get to the grand ‘superalignment team.’
But, in the course of several meetings in the spring of 2023, Altman seemed to waver. He stopped talking about endowing a prize. Instead, he advocated for establishing an in-house “superalignment team.”
An official announcement, referring to the company’s reserves of computing power, pledged that the team would get “20% of the compute we’ve secured to date”—a resource potentially worth more than a billion dollars. The effort was necessary, according to the announcement, because, if alignment remained unsolved, A.G.I. might “lead to the disempowerment of humanity or even human extinction.”
Jan Leike, who was appointed to lead the team with Sutskever, told us, “It was a pretty effective retention tool.”
The twenty-per-cent commitment evaporated, however. Four people who worked on or closely with the team said that the actual resources were between one and two per cent of the company’s compute. Furthermore, a researcher on the team said, “most of the superalignment compute was actually on the oldest cluster with the worst chips.”
The researchers believed that superior hardware was being reserved for profit-generating activities. (OpenAI disputes this.) Leike complained to Murati, then the company’s chief technology officer, but she told him to stop pressing the point—the commitment had never been realistic.
… But the superalignment team was dissolved the following year, without completing its mission.
OpenAI and Altman continue to attempt to memory hole that this happened, and that this promise was both valuable and load bearing and then massively broken.
I am always especially enraged when:
People make you a promise of [X], which was their own idea.
You act on the basis of [X].
Nothing changes.
They don’t give you [X].
When asked why, they say [X] was never realistic.
This happened to me once, in a deeply similar way, where someone offered to fund a potential new company in a particular way, then after efforts were made told me (paraphrased) that it wasn’t realistic, and offered a dramatically worse deal, right after I had subsequently invested time and blown up a bunch of things including a key friendship that took me years to repair, and other opportunities had been lost. Should I have predicted it was reasonably likely to go down that way? Oh, absolutely, but seriously, not okay.
This Is Fine
This is a good summary of the switch that happened over time:
Last June, on his personal blog, Altman wrote, referring to artificial superintelligence, “We are past the event horizon; the takeoff has started.” This was, according to the charter, arguably the moment when OpenAI might stop competing with other companies and start working with them.
But in that post, called “The Gentle Singularity,” he adopted a new tone, replacing existential terror with ebullient optimism. “We’ll all get better stuff,” he wrote. “We will build ever-more-wonderful things for each other.” He acknowledged that the alignment problem remained unsolved, but he redefined it—rather than being a deadly threat, it was an inconvenience, like the algorithms that tempt us to waste time scrolling on Instagram.
Liar Liar Master Persuader
Another common theme is that yes, Altman lies quite a lot. This is not news.
The problem was that these lies, in isolation, are not smoking guns. Altman is smart. But if you add up the pattern, the result is clear.
Neither collection of documents contains a smoking gun. Rather, they recount an accumulation of alleged deceptions and manipulations, each of which might, in isolation, be greeted with a shrug: Altman purportedly offers the same job to two people, tells contradictory stories about who should appear on a live stream, dissembles about safety requirements. But Sutskever concluded that this kind of behavior “does not create an environment conducive to the creation of a safe AGI.” Amodei and Sutskever were never close friends, but they reached similar conclusions. Amodei wrote, “The problem with OpenAI is Sam himself.”
… most of the people we spoke to shared the judgment of Sutskever and Amodei: Altman has a relentless will to power that, even among industrialists who put their names on spaceships, sets him apart.
“He’s unconstrained by truth,” the board member told us. “He has two traits that are almost never seen in the same person. The first is a strong desire to please people, to be liked in any given interaction. The second is almost a sociopathic lack of concern for the consequences that may come from deceiving someone.”
The board member was not the only person who, unprompted, used the word “sociopathic.”
As others have pointed out, contra the board member, those two traits are seen in the same person reasonably often. There is no contradiction between maximizing likeability in each interaction and engaging in reflexive deception. Indeed, one could even call (a lesser version of) this a common cultural attitude in various parts, especially in California.
The goal is to instill good vibes and keep everyone thinking and doing whatever you need them to be thinking and doing, you are mostly indifferent to whether the statements required for this are true and you don’t worry so much about the reputational consequences or what happens if your contradictory statements are pointed out.
Altman simply extends this common pattern to an extreme extent.
“He’s unbelievably persuasive. Like, Jedi mind tricks,” a tech executive who has worked with Altman said. “He’s just next level.” A classic hypothetical scenario in alignment research involves a contest of wills between a human and a high-powered A.I. In such a contest, researchers usually argue, the A.I. would surely win, much the way a grandmaster will beat a child at chess. Watching Altman outmaneuver the people around him during the Blip, the executive continued, had been like watching “an A.G.I. breaking out of the box.”
If you think Altman can be ‘unbelievably persuasive’ then yes perhaps consider what a sufficiently advanced intelligence could do, or assist a human in doing, on this front.
Here’s a fun one I hadn’t heard before.
Even former colleagues can be affected. Murati left OpenAI in 2024 and began building her own A.I. startup. Josh Kushner, the close Altman ally, called her. He praised her leadership, then made what seemed to be a veiled threat, noting that he was “concerned about” her “reputation” and that former colleagues now viewed her as an “enemy.” (Kushner, through a representative, said that this account did not “convey full context”; Altman said that he was unaware of the call.)
As every good leader and follower knows, the leader wants to be unaware of that call.
This In Particular Is Securities Fraud
Here’s a rather serious allegation, which OpenAI denies, to further the story of how the second largest theft in human history was pulled off, here’s how it started:
At the beginning of his tenure as C.E.O., Altman had announced that OpenAI would create a “capped profit” company, which would be owned by the nonprofit. This byzantine corporate structure apparently did not exist until Altman devised it. In the midst of the conversion, a board member named Holden Karnofsky objected to it, arguing that the nonprofit was being severely undervalued. “I can’t do that in good faith,” Karnofsky, who is Amodei’s brother-in-law, said. According to contemporaneous notes, he voted against it. However, after an attorney for the board said that his dissent “might be a flag to investigate further” the legitimacy of the new structure, his vote was recorded as an abstention, apparently without his consent—a potential falsification of business records. (OpenAI told us that several employees recall Karnofsky abstaining, and provided the minutes from the meeting recording his vote as an abstention.)
Last October, OpenAI “recapitalized” as a for-profit entity. The firm touts its associated nonprofit, now called the OpenAI Foundation, as one of the “best resourced” in history. But it is now a twenty-six-per-cent stakeholder of the company, and its board members are also, with one exception, members of the for-profit board.
Given that the capped profit structure was unsustainable in practice, and investors would be handed all their uncapped profits in exchange for nothing, what to think?
Regulation Two Step
As you would expect, they chronicle some of OpenAI’s history where Altman will publicly call for regulation, while behind the scenes OpenAI quietly lobbies against it, including bills whose provisions largely match things Altman publicly called for. This includes the attempts to subpoena nonprofits, and the funding of Leading the Future.
Offered without comment, other than to contrast it with more recent choices, as the article later notes:
Altman has long supported Democrats. “I’m very suspicious of powerful autocrats telling a story of fear to gang up on the weak,” he told us. “That’s a Jewish thing, not a gay thing.”
OpenAI has been trying to extract government money under false pretenses, Altman says he ‘does not recall describing Beijing’s efforts in exactly this way’ but who are we kidding and it was false at the time no matter how he worded it:
In a meeting with U.S. intelligence officials in the summer of 2017, he claimed that China had launched an “A.G.I. Manhattan Project,” and that OpenAI needed billions of dollars of government funding to keep pace. When pressed for evidence, Altman said, “I’ve heard things.” It was the first of several meetings in which he made the claim. After one of them, he told an intelligence official that he would follow up with evidence. He never did.
Easy Mode
Quoting troubling things about Altman does feel like shooting fish in a barrel at this point, and in this case I get to edit out all the shots that failed to hit a fish, or only hit a relatively unimpressive or known dead fish.
I mean, yeah, this is a fine paragraph, but it’s easy mode:
But others in Silicon Valley think that Altman’s behavior has created unacceptable managerial dysfunction. “It’s more about a practical inability to govern the company,” the board member said. And some still believe that the architects of A.I. should be evaluated more stringently than executives in other industries.
The vast majority of people we spoke to agreed that the standards by which Altman now asks to be judged are not those he initially proposed. During one conversation, we asked Altman whether running an A.I. company came with “an elevated requirement of integrity.”
This was supposed to be an easy question. Until recently, when asked a version of it, his answer was a clear, unqualified yes.
Now he added, “I think there’s, like, a lot of businesses that have potential huge impact, good and bad, on society.” (Later, he sent an additional statement: “Yes, it demands a heightened level of integrity, and I feel the weight of the responsibility every day.”)
Yes, yes, OpenAI has dissolved its superalignment team after starving it of promised compute, then dissolved the AGI-readiness team, and now on its IRS disclosure form no longer includes the concept of safety under its ‘most significant activities.’ More fish, more barrels. But man, do they make it so easy:
“My vibes don’t match a lot of the traditional A.I.-safety stuff,” Altman said. He insisted that he continued to prioritize these matters, but when pressed for specifics he was vague: “We still will run safety projects, or at least safety-adjacent projects.”
When we asked to interview researchers at the company who were working on existential safety—the kinds of issues that could mean, as Altman once put it, “lights-out for all of us”—an OpenAI representative seemed confused. “What do you mean by ‘existential safety’?” he replied. “That’s not, like, a thing.”
I’m somewhat sympathetic there, since ‘existential safety’ is not a standard term, but you should damn well know what that means, and also know not go give a reporter that kind of a pull quote. What are they going to do, not use it?
Because yes, existential safety is totally a thing.
The Right Amount of Alignment Research Is Not Zero
For avoidance of doubt, OpenAI does do a bunch of alignment work, and indeed is a rather important division for many obvious reasons.
roon (OpenAI): the alignment team continues to exist and is one of the largest and most compute rich research programs at OpenAI (i am on it, I should know). specific teams dissolving usually has more to do with people than functions
relatively new blog [here].Austin Wallace: Just for an avoidance of doubt, as I’m pretty sure the answer is yes, are there people on the team who have as a large part of their job being to explicitly (if indirectly) work on what most would agree counts as “existential safety” even if by a different term?
roon (OpenAI): no more or less than the original superalignment team ie looking for deceptive behaviors, confessions, cot supervision. if you don’t buy the premise of any of these work streams then you wouldn’t have bought what superalignment was originally doing.
To me, all of this is mostly mundane alignment, as in helping improve the performance of existing or near future mundane AI systems, with the hope that improving in mundane alignment will scale or otherwise assist you later. The work is certainly good and worth doing, but this was never to me a viable path to something worthy of the name ‘superalignment.’ If it was always fake in that sense? Okay, sure.
From what I have seen, OpenAI is not seriously pursuing a viable strategy for aligning future sufficiently advanced (superintelligent) AI systems, nor is it spending as much overall on alignment as it would be wise to do even on a pure commercial basis. And I do not believe OpenAI has honored its commitments in this area, or those surrounding the foundation.
But that’s very different from saying they’re doing nothing. They’re doing some things.
This then fits into what we will see in Part 2, where I dissect the latest policy proposal from OpenAI, and find it not to be a serious document.
OpenAI Proposes Policy
OpenAI in particular proposes Industrial Policy for the Intelligence Age: Ideas To Keep People First.
OpenAI: Now, we’re beginning a transition toward superintelligence: AI systems capable of outperforming the smartest humans even when they are assisted by AI.
There is very much a written-by-committee-for-committee feel to the text, of a third-tier politician giving a profoundly milquetoast non-rousing drinking-game-level speech about maximizing AI gains while minimizing AI risks.
It is a PR stunt. It is an attempt to frame, once again, the upcoming problems of superintelligence as focused on redistribution and jobs and ordinary mitigations of ordinary harms we can muddle through, when what we are actually talking about is creating a new class of minds that are smarter and more capable than ourselves.
There is nothing in this document that is downstream of the fact that we are talking about creating sufficiently advanced AI that have minds that are smarter, faster, more competitive and more capable than us. This is the kind of document you produce when you’re introducing, say, the car, or the internet… and you want a PR statement.
Up front the goals are ‘share prosperity broadly,’ ‘mitigate risks’ and ‘democratize access and agency.’
Mitigate risks. The transition toward superintelligence will come with serious risks—from economic disruption, to misuse in areas like cybersecurity and biology, to the loss of alignment or control over increasingly powerful systems. Without effective mitigation, people will be harmed.
People will be harmed? Oh no. Sounds like a problem. This is not how people actually serious about those risks would be talking. Meanwhile the other two points are both distributional. What to do?
A new industrial policy agenda should use government’s existing toolbox for aligning public and private activities: research funding, workforce development, market-shaping tools, and targeted regulation.
Wading through a lot of drivel, what are they actually proposing?
The same grab bag of good sounding gestures as always, basically. The thing that matters is intent to do redistribution as needed. The rest is window dressing or worse.
‘Give workers a voice’ to prioritize deployments and set limits that could ‘erode job quality’ or undermine fair scheduling and pay.
Mandating this seems hard? How would you do it without paralyzing the economy and killing companies?
This is by default a horribly inefficient way to guard worker welfare.
If you want to do redistribution, then do it directly. Minimal deadweight loss.
Financing for AI-first entrepreneurs.
Give us money. Classic.
Right to AI. Education plus massive subsidies at every stage.
Give us all the money. Super classic.
This is distinct from a 1A free speech style ‘right to AI.’
Modernize the tax base. Raise capital gains and corporate taxes and pair this with ‘wage-linked incentives.’
It’s tough for me to take ‘wage-linked incentives’ seriously in a world with such high taxes on wages. You want to do this? Cut income taxes.
Rapid growth from AI/AGI/ASI allows you to cut taxes.
Capital gains and corporate taxes are generally anti-growth, especially if they are general and not concentrated in AI, but way better than wealth taxes.
Why not use consumption taxes?
The core efficiency problem is that human labor is heavily taxed and non-human labor that competes with it will go untaxed. If you want to solve that, you’ll want to tackle it head-on, and taxes on profits won’t solve this.
Public wealth fund.
Look, no. You don’t create a public wealth fund with a massive debt.
You have the power to tax, can we all not be idiots please?
Well, you probably lose control and then probably die, but if you lose the power to tax you also lose the power to use the public wealth fund.
This feels like an attempt to cozy up and excuse the policy of taking shares in various companies including Intel. It was bad then, it is bad now.
Accelerate grid expansion.
Give us money.
But also, yeah, this needs to happen. Focus should be on regulatory barriers.
Efficiency dividends. Convert gains into shorter worker hours, better benefits.
The government does not get to do this.
Or, when they do get to do this, it doesn’t turn out well. This is EU stuff.
If you want to do redistribution, do it directly. Minimal deadweight loss.
I’m sure the ‘beat China’ people will condemn these horrible ideas. Right?
Adaptive safety nets that work for everyone. Measure impact to be prepared.
Yes, this is The Way, in terms of dealing with distributional issues.
Except the call is for a ‘temporary’ package. Why would you expect to be able to end that later? Won’t the situation only escalate?
Portable benefits for those who switch jobs.
You don’t want to be mandating such things, only enabling.
Enabling is a great idea but for reasons that have nothing to do with AI.
Pathways into human-centered work. Healthcare, education, community services, elder care as places for displaced workers to go.
There is limited flexibility in how many real jobs we can put in such areas.
We do have a bunch of slack, especially with currently training-and-license limited jobs like doctors and nurses. More training would be great.
Accelerate scientific discovery ‘and scale the benefits.’
I mean, sure, yes, please.
Trump’s budget wants to cut science funding dramatically. Don’t let him.
Part two is about building a resilient society, because you see some things might go wrong after deployment of these superintelligent systems. So what do we do?
Here we have some things that are good on the margin, but nothing that takes the existential risks and loss of control risks and other major concerns seriously.
Safety systems for emerging risks. AI threat modeling, countermeasures for specific threats, ‘catalyze competitive safety markets.’
So basically muddle through like always.
AI trust stack. Let people trust and verify AI systems and content.
This is helpful on the margin but doesn’t address core issues.
Auditing regimes.
I mean I would certainly hope so. But again, insufficient.
Model-containment playbooks. Plan to ‘contain dangerous AI systems once they have been released into the world.’ Learn from cybersecurity and public health.
Okay, now we’re starting to have a conversation. What makes you think that you can do that? How are you going to do that?
This does not read like they understand how hard that would be.
Mission-aligned corporate governance. Use PBCs, explicit commitments. Harden systems against corporate or insider capture.
That sounds like a great idea. How’s it working out for you?
Guardrails for government use. Policymakers need to lay out restrictions.
Or else what? OpenAI seems content to provide the models anyway.
But yes, agreed, we need this.
Mechanisms for public input.
Public has plenty of ways to input, they just get ignored.
The public does not actually have the ability to provide meaningful or wise input on many of these topics and we shouldn’t pretend that they do.
Incident reporting.
Okay, sure.
International information-sharing around AI capabilities, risks and mitigations.
Yes, please. Good idea.
Nate Soares tries to find the most generous interpretation of all this.
Nate Soares (MIRI): Glad OpenAI’s clear about risking “loss of alignment or control over increasingly powerful systems”, which is presumably part euphemism for “we might make a superintelligence that kills everyone.” But they’re gonna need a better response than “we’ll try our best to contain it.”
A fellowship to investigate public wealth funds and four-day workweeks as well as “containment playbooks” does not quite discharge your duties to the civilization you’re gambling with, sorry.
Nor does assuring reporters that you’re very concerned about the extinction threat and that you’re vaguely working on it (even as employees insist “That’s not, like, a thing”).
Maybe try coming right out and saying “this is horribly dangerous and we would prefer the global pace be much slower.” It’s not 2023 any more. You don’t have to softpedal.
On the contrary, in 2023 OpenAI was willing to softpedal far less. We did not know how good we had it.
Charles and FleetingBits more or less call it a PR exercise, which is correct.
Anton Leicht points out that something like this would be a good idea for part of accelerationist policy (he even says ‘the best direction’), but that this looks like the bad version of such an approach, where you put out a PR policy doc and then use that to oppose anyone else’s attempts to do anything and spend a lot of money (as OpenAI is going by nominal proxy) to defeat anyone who tries to regulate you in any way.
Nathan Calvin: Currently the correct lens of viewing this document is as a cynical comms document that doesn’t represent OpenAI’s actual influence on policy/politics. I agree with Anton that if it wasn’t a cynical comms doc then that would be good. OAI - take costly actions to prove me wrong!
Daniel Eth: Nathan is 100% correct here. Put yourself in the shoes of a politician - OpenAI leadership is spending tens of millions of dollars in super PAC money to block AI regulations, while simultaneously putting out rhetoric supporting regulatory guardrails
Adrien Ecoffet (OpenAI): Totally reasonable to be skeptical. For what it’s worth this was my first involvement in a policy project and my role was to lead a group of researchers who suggested many of these proposals and gave extensive feedback on all of them. I realize that at this stage these are just words, but we have to start with something and I am with you on the fact that we will need to do much more for this to really pay off for the world, as I think are my researcher friends.
If we see OpenAI spending big amounts of political and actual capital to push for these kinds of changes, to the point of risking actively antagonizing people like the Abundance Institute or David Sacks, as opposed to spending that capital on things like Leading the Future? Then okay, I will believe they mean it. Until then, no.
RIP TBPN
OpenAI has bought TBPN. Sam Altman calls it his favorite tech show. Everyone vows they will stay the same, and Altman ‘doesn’t expect them to go any easier on us.’ That is, unfortunately, not the way any of this works, even if believe Altman wants it to be.
The contract says the right things.
rat king: interesting... as part of negotiations with OpenAI, TBPN has a “Commitment to Editorial Independence” written into the contract, which states the following:
• TBPN retains full control over its daily programming, editorial decisions, guest selection, and production schedule.
• TBPN will continue to host a broad range of voices and perspectives.
• TBPN independently determines its external appearances and commentary.
• OpenAI will not control TBPN’s planning materials or working documents.
• OpenAI will not provide direction on TBPN’s editorial calendar.
• OpenAI will not influence who TBPN books or what topics it covers.
• OpenAI will not materially alter, discontinue, or rebrand TBPN.
• OpenAI does not have rights to TBPN host likeness.
Matthew Yglesias: If you can’t trust Sam Altman to scrupulously follow the original terms of an arrangement come what may, who can you trust?
Yeah, I don’t buy it. The revealed preference is already crystal clear.
If you have your media acquisition reporting to Chris Lehane it is already dead.
If it is also within your ‘strategy’ org? Even more so.
Peter Wildeford: Buying a media property and having it report to “master of the dark arts” Chris Lehane does have a certain vibe to it... not that TBPN was ever particularly critical in the past though.
Shakeel: Sorry, how do you plan to retain “full editorial control” while reporting to the company’s chief lobbyist?
Alex Bores: If they want to prove their editorial independence from Chris Lehane, they could always invite me on 🤷
Entertaining and well-executed state media can be useful, but it is what it is.
The tragic thing about all this is, what were they hoping for?
Anton Leicht: even if you take OpenAI’s ‘we want to support their broader tech/AGI comms’ seriously, I don’t get how buying TBPN helps with that?
if there’s a successful, independent-ish media outlet doing things you like, the last thing you want to do is buy them and hurt their credibility.A similar logic applies to the Anthropic Institute btw.
The Anthropic Institute makes sense in that it would not exist ‘but for’ Anthropic, so you’re trading off the conflict of interest to get the thing to happen, and given its mission there’s less danger from this, but yeah. If you’re being funded by the labs, or a particular lab, you can’t be fully objective even if everyone is trying hard.
jasmine sun: there’s a reason I’ve elected to write on substack and in legacy journalistic outlets over working for a tech company in this moment
frontier lab brain drain is real, and there are very few thinkers who get AI and can be truly editorially independent
I actively turned down investing in Anthropic, even though it was an obviously fantastic investment, in order to avoid both the appearance and risk of bias. Doing a proper journalism is not easy. Keeping to your principles is not easy, nor cheap.
That’s the way it goes.
Podcasts aren't my thing, so maybe it's obvious in-context to fans of that media, but it felt weird to have to look up "TBPN" because I had no idea what it was. Some sort of media property, but what format? I don't think it's ever been linked here before either, to the best recollection of dozens of This Week In Audio. If the news had been "sama buys Odd Lots" or "OAI bankrolls Dwarkesh" or something, then I'd know the appropriate amount of freakouttery to have.
The structural lesson from the board saga isn't really about Altman's character — it's that governance mechanisms have an implied threat model, and he consistently operates outside it. A board designed to constrain a master persuader is a lock designed by the lockpick. The 20% compute commitment to superalignment lasted roughly 12 months before Leike quit publicly citing resource starvation and the team dissolved. The nonprofit shell lasted until the valuation made it load-bearing. Each constraint works exactly until the moment it would actually bind.