Like the the government-commissioned Gladstone Report on AI itself, there are two sections here. First I cover the Gladstone Report’s claims and arguments about the state of play, including what they learned talking to people inside the labs. I mostly agree with their picture and conclusions, both in terms of arguments and reported findings, however I already mostly agreed. If these arguments and this information is new to someone, and the form of a government-backed report helps them process it and take it seriously, this is good work. However, in terms of convincing an already informed skeptic, I believe this is a failure. They did not present their findings in a way that should be found convincing to the otherwise unconvinced.
Tiny question: is the name of the Gladstone group a reference to the Hyperion novels? If so, it would kind of give away their opinions ahead of time. (my apologies if this is well known already).
Don’t catastrophic events actually help stave off existential ones? Isn’t that (not to put words in anyone’s mouth) roughly Scott Alexander’s position?
A few years ago, how many people would’ve believed that early 2024 would see a US government report that pretty much fully endorses Yudkowskyan doomerism? Isn’t the mere fact of such a report’s existence MUCH more significant than how persuasive that report’s arguments will be to those not already persuaded — a criterion by which the Yudkowskyans themselves have arguably fallen short as well?
> The worlds were artificial superintelligence (ASI) is coming very soon with only roughly current levels of compute, and where ASI by default goes catastrophically badly, are not worlds I believe we can afford to save.
I am concerned that this may well be the case, and I intend to try hard to save us anyway. I don't think it's hopeless. I do expect things to get messy.
What a bunch of goofballs, you included. You can't "align" something smarter than yourself - you can't even "align" a regular old human. How many times does it need to be said? The best thing we can do, with people & other beings, is to educate.
Is there anyone in this field with an actual education, or is it upjumped engineers all the way down?
Your reaction is pretty much the same as that of the Pope to the printing press -- an invention which did, in fact, result in catastrophe rather immediately: to wit, the Thirty Years' War, which followed directly from the printing-accelerated Reformation and ended up killing half the people in Europe. Technology has always led to catastrophe. This isn't to say "the ends justify the collateral damage" -- I'm not even asserting any particular end -- but rather that there's nothing you can do to contain technology once it's out. You're asking for something that would require dictatorial powers even to attempt, and couldn't be accomplished even so.
It's also preposterous of you to call for international cooperation while you dog on China every other sentence. If I happened to be a citizen of China, I would rather heartily doubt that you intend "cooperation" in any real or equal sense, and I would support my government's efforts to level the playing field.
As a citizen of the United States, I'm rather sure that if our positions were reversed, and Chinese companies were demonstrating shockingly potent AI technologies, and they didn't want to share, you would be calling on the American government to gain access to these technologies by any means necessary, and you would consider yourself entirely in the right.
I'm not saying you can't be afraid of China -- but don't call for cooperation when you clearly have no such intention.
It seems to me there are a few particularly crucial questions:
1. The master question: Does agency of some kind emerge spontaneously at a certain scale?
2. The next question: Is there a certain threshold after which the model is “smart” enough to improve itself?
Then there are the boring-but-still-existential questions:
a. Is there a certain scale at which the inherent capabilities are sufficient that the AI in question is essentially a extinction-level weapon?
b. What is the threshold at which the above occurs?
c. What is the threshold at which the above occurs taking into account the performance of a fully optimized version?
I am not particularly clever, but I am a bit worried that there may be no safe ways to answer these questions. In an ideal world:
1. ALL frontier research would be conducted on air-gapped models
2. ALL frontier research would be conducted with
3. We would have a clear test battery or criteria for identification of danger (i.e. self-improvement, spontaneous agency which is not just token-prediction) which would justify shutdown
4. We would be specifically targeting the questions above
We do not seem to be specifically targeting any of the above. At this stage I think our best hope is a catastrophe that gives us the political will to avert extinction! I would welcome someone who could share some good reasons that I am being overly skeptical.
Tiny question: is the name of the Gladstone group a reference to the Hyperion novels? If so, it would kind of give away their opinions ahead of time. (my apologies if this is well known already).
Don’t catastrophic events actually help stave off existential ones? Isn’t that (not to put words in anyone’s mouth) roughly Scott Alexander’s position?
AI Podcast episode of this post:
https://askwhocastsai.substack.com/p/on-the-gladstone-report-by-zvi-mowshowitz
A few years ago, how many people would’ve believed that early 2024 would see a US government report that pretty much fully endorses Yudkowskyan doomerism? Isn’t the mere fact of such a report’s existence MUCH more significant than how persuasive that report’s arguments will be to those not already persuaded — a criterion by which the Yudkowskyans themselves have arguably fallen short as well?
> The worlds were artificial superintelligence (ASI) is coming very soon with only roughly current levels of compute, and where ASI by default goes catastrophically badly, are not worlds I believe we can afford to save.
I am concerned that this may well be the case, and I intend to try hard to save us anyway. I don't think it's hopeless. I do expect things to get messy.
What a bunch of goofballs, you included. You can't "align" something smarter than yourself - you can't even "align" a regular old human. How many times does it need to be said? The best thing we can do, with people & other beings, is to educate.
Is there anyone in this field with an actual education, or is it upjumped engineers all the way down?
Your reaction is pretty much the same as that of the Pope to the printing press -- an invention which did, in fact, result in catastrophe rather immediately: to wit, the Thirty Years' War, which followed directly from the printing-accelerated Reformation and ended up killing half the people in Europe. Technology has always led to catastrophe. This isn't to say "the ends justify the collateral damage" -- I'm not even asserting any particular end -- but rather that there's nothing you can do to contain technology once it's out. You're asking for something that would require dictatorial powers even to attempt, and couldn't be accomplished even so.
It's also preposterous of you to call for international cooperation while you dog on China every other sentence. If I happened to be a citizen of China, I would rather heartily doubt that you intend "cooperation" in any real or equal sense, and I would support my government's efforts to level the playing field.
As a citizen of the United States, I'm rather sure that if our positions were reversed, and Chinese companies were demonstrating shockingly potent AI technologies, and they didn't want to share, you would be calling on the American government to gain access to these technologies by any means necessary, and you would consider yourself entirely in the right.
I'm not saying you can't be afraid of China -- but don't call for cooperation when you clearly have no such intention.
It seems to me there are a few particularly crucial questions:
1. The master question: Does agency of some kind emerge spontaneously at a certain scale?
2. The next question: Is there a certain threshold after which the model is “smart” enough to improve itself?
Then there are the boring-but-still-existential questions:
a. Is there a certain scale at which the inherent capabilities are sufficient that the AI in question is essentially a extinction-level weapon?
b. What is the threshold at which the above occurs?
c. What is the threshold at which the above occurs taking into account the performance of a fully optimized version?
I am not particularly clever, but I am a bit worried that there may be no safe ways to answer these questions. In an ideal world:
1. ALL frontier research would be conducted on air-gapped models
2. ALL frontier research would be conducted with
3. We would have a clear test battery or criteria for identification of danger (i.e. self-improvement, spontaneous agency which is not just token-prediction) which would justify shutdown
4. We would be specifically targeting the questions above
We do not seem to be specifically targeting any of the above. At this stage I think our best hope is a catastrophe that gives us the political will to avert extinction! I would welcome someone who could share some good reasons that I am being overly skeptical.