Is there a risk that grading OpenAI significantly below the pass-mark might elicit a response of "screw it, we're gonna fail no matter what, let's stop caring what those impossible-to-please saferyists think"? I'm not sure what the solution is, but I wonder if it would be better to have a little more "positivity-in-response-to-improvement" in vibe, alongside honest negative specifics.
P.S. why did the universe have to make this all so hard?
The universe makes this hard, because we're trying to invent something significantly smarter and more capable than we are, and somehow remain in control. The entire multimillion year history of hominid evolution shows what happens to the loser of the intelligence arms race. And we're trying our very hardest to lose.
Then on top of this problem, if we could ever solve it, we're adding the question of precisely _who_ controls the smarter than human AI. Remember the saying about "absolute power" and corruption?
The universe is making this hard because our plan is really dumb, the species-wide equivalent of "hold my beer and watch this" with chainsaws.
That all sounds like a description of how the universe is making this hard. The question of "why" seems better answered by: "universe is rude" or "well, it's kinda exciting, innit?".
I really like when people can speak their honest points of view. It is true that what people say influences things, and so to influence things to go the way you want, you should say not what you believe, but what you believe will cause other actors to act the way you want.
But there are two problems:
1. Actually predicting how what is said affects other agents is super complicated and you aren't actually that likely to do it successfully.
2. It's gross.
So I support Zvi saying "All the AI companies are fucking up and it's scary as fuck" because that is how I also see things, but I agree with you about "positivity-in-response-to-improvement". Relatedly, ACX recently had this praise for Altman which I think is worth quoting:
> One reason I respect Sam Altman is that back in 2016, when he founded an AI charity to bring a positive singularity to the world, he realized that it would later be extraordinarily tempting to turn it into a normal profit-focused company and get rich. So he tied himself to the mast by designing a nonprofit structure capable of thwarting all the machinations his future self could throw at it. A few years later, he gave into temptation, tried to turn it into a normal profit-focused company, and failed, because the structure he designed was really good. This was the best possible outcome, and one of many reasons I number him among the all-time greats.
PS, the universe is making things hard because the universe is rude.
"It is easy to forget that in the world of VCs and corporate America, in many ways it is not only that you have no obligation to do the right thing. It is that you have an obligation, and will face tremendous pressure, to do the wrong thing, in many cases merely because it is wrong, and certainly to do so if the wrong thing maximizes shareholder value in the short term."
I find it very difficult to forget this, and wish more people focused on changing it...
Podcast episode for this post:
https://open.substack.com/pub/dwatvpodcast/p/openai-claims-nonprofit-will-retain
Is there a risk that grading OpenAI significantly below the pass-mark might elicit a response of "screw it, we're gonna fail no matter what, let's stop caring what those impossible-to-please saferyists think"? I'm not sure what the solution is, but I wonder if it would be better to have a little more "positivity-in-response-to-improvement" in vibe, alongside honest negative specifics.
P.S. why did the universe have to make this all so hard?
The universe makes this hard, because we're trying to invent something significantly smarter and more capable than we are, and somehow remain in control. The entire multimillion year history of hominid evolution shows what happens to the loser of the intelligence arms race. And we're trying our very hardest to lose.
Then on top of this problem, if we could ever solve it, we're adding the question of precisely _who_ controls the smarter than human AI. Remember the saying about "absolute power" and corruption?
The universe is making this hard because our plan is really dumb, the species-wide equivalent of "hold my beer and watch this" with chainsaws.
That all sounds like a description of how the universe is making this hard. The question of "why" seems better answered by: "universe is rude" or "well, it's kinda exciting, innit?".
I really like when people can speak their honest points of view. It is true that what people say influences things, and so to influence things to go the way you want, you should say not what you believe, but what you believe will cause other actors to act the way you want.
But there are two problems:
1. Actually predicting how what is said affects other agents is super complicated and you aren't actually that likely to do it successfully.
2. It's gross.
So I support Zvi saying "All the AI companies are fucking up and it's scary as fuck" because that is how I also see things, but I agree with you about "positivity-in-response-to-improvement". Relatedly, ACX recently had this praise for Altman which I think is worth quoting:
> One reason I respect Sam Altman is that back in 2016, when he founded an AI charity to bring a positive singularity to the world, he realized that it would later be extraordinarily tempting to turn it into a normal profit-focused company and get rich. So he tied himself to the mast by designing a nonprofit structure capable of thwarting all the machinations his future self could throw at it. A few years later, he gave into temptation, tried to turn it into a normal profit-focused company, and failed, because the structure he designed was really good. This was the best possible outcome, and one of many reasons I number him among the all-time greats.
PS, the universe is making things hard because the universe is rude.
"It is easy to forget that in the world of VCs and corporate America, in many ways it is not only that you have no obligation to do the right thing. It is that you have an obligation, and will face tremendous pressure, to do the wrong thing, in many cases merely because it is wrong, and certainly to do so if the wrong thing maximizes shareholder value in the short term."
I find it very difficult to forget this, and wish more people focused on changing it...
Do you know what the "special voting and governance rights held solely by the OpenAI Foundation" are?