In your commentary on Peter Wildeford’s summary, after number 10, you wrote “In which case, can we stop waving them, such as with H20 sales to China?” - should be “waiving” not “waving”
I don’t really understand the obsession Zvi and others have with beating China. If it’s true that TSMC is manufacturing constrained then selling them chips slows down AI progress for everyone which is good.
Further as we get into the future, assuming we don’t have extremely short timelines, the H20s will be obsolete (this contradicts my first point ik). If compute really is all you need (as xAI demonstrated) then I’m not sure why allowing the Chinese to train better models now would have any effect on their plans to train human level intelligence.
Also them using the CUDA stack for longer prevents them from properly focusing on their own Chinese versions of the whole thing (not very certain on this point).
I also don’t care that much about China “winning”. Maybe if it was like the North Koreans I’d be concerned but Chinese hegemony would be a billion times better than us all dropping dead.
And finally I’m not so convinced the Chinese will be less safe than we are.
Obviously I’m quite uncertain on this everyone else seems to be convinced I’m wrong.
I think the point Zvi is failing to communicate is that China is positioning itself as an advocate for open models to promote multilateralism, which would effectively strangely any capacity to regulate AI development.
I find that quite unlikely. If China had a lead in models I doubt they would just hand it over to us. I’m quite confident the best models will remain private
That's fair. I'm viewing that part of Zvi's posts as more of a: "Look, administration, if you _are_ trying to win this race, could you at least be sane in your tactics? Maximizing nVidia's market share is very much NOT the same thing as have a US company get to AGI first." (And the latter, not the former, is the relevant "win" here, if there is one.)
Re:
>I also don’t care that much about China “winning”. Maybe if it was like the North Koreans I’d be concerned but Chinese hegemony would be a billion times better than us all dropping dead.
Well... Sort-of mostly agreed, but
a) The CCP is the lineal descendant of Mao's CCP, with his Great Leap Corpseward (arguable 45 million dead...)
b) (writing from the USA) It also depends on how bloodily Chinese hegemony is achieved. "killed by the PRC" and "killed by an ASI" can look pretty similar...
That said, a worst-case MAGA or Woke surveillance state and a median CCP surveillance state might not be that different...
>Zvi has made it pretty clear he genuinely cares about the US getting to superintelligence before China.
You are probably right. I'm somewhat unsure. Zvi is often in the unenviable position of having to say that e.g. one of the labs' policies will probably kill us, but is less bad than it could have been.
>And yeah I would probably slightly prefer the US to dictate the future but the CCP isn’t *that* bad.
Partly agreed. After all, almost all of the people _in_ China can live with it. Still, a lot depends on the guy at the top. If Xi's successor is as bad as Mao, it could quickly become a whole lot worse...
For most weaponry these are very good arguments, but I'd argue less so for AI. The main way AI in the hands of the CCP is dangerous for us is essentially the same way it is dangerous for *them*: An unaligned AI that kills or subjugates all, or gets into terrorist hands or whatever. Mao might not have cared about the lives he caused to end, but if he had a magic button to kill 50 million people he would not have pressed it.
Many Thanks! I'm a bit unsure which part(s) of my comment you disagree with.
"The main way AI in the hands of the CCP is dangerous for us is essentially the same way it is dangerous for *them*: An unaligned AI that kills or subjugates all, or gets into terrorist hands or whatever."
I mostly, but not entirely, agree with you. I agree that that is a large fraction of the hazard to us of an AI in CCP hands. I don't think we can tell if the fraction is 25% or 75%, so I;m not sure if "main" (presumably > 50%) is correct.
In the case where AI is successfully aligned by the PRC, "winning" the race, then the question for our fate becomes: "How do they achieve hegemony (bloodily or less so)? and What is their hegemony like (liveable or not)?" - but these are only relevant in the part of the probability mass where the AI(s) is/are following CCP orders, where they have been successfully aligned.
1) It is better if only one country races for AI. Since the US has the possibility of preventing China from racing to AI, it is better that they attempt to do so. China has no realistic possibility of preventing the US racing to AI. (that's a simplification, but a valid strategy could be hoping to have a year lead time or something)
2)There is also some political reasons. It would be outside the overton window to suggest to (current) US policymakers that it would not be bad if China won.
Having said that, I agree with you. People in China, on average, seem to live fairly satisfactory lives, and it seems crazy to prefer to irresponsibly race with China, given that an aligned AI would usher in an age of abundance for the world, such that as an European I don't care whether I get the American or Chinese scraps.
>I will keep repeating, if necessary until I am blue in the face, that this is effectively a call (the motivations for which I do not care to speculate) for sacrificing the future and get us all killed in order to maximize Nvidia’s market share.
Well yeah, when the kids talk about "late-stage" capitalism did you think it was a joke?
Does Zuckerberg think he is Harry in HPMOR when the others ban traitors and he's like mwhahaha, I'll still do what I want? He's truly an agent of chaos trying to win at any cost.
> You can run any most any AI model on most any advanced chip, and switch on an hour’s notice.
This is directionally correct but heavily overstated: fused kernels are not hot-swappable like that, and at the moment there's not even any alternative to swap to. Yes, CUDA can and will be replaced if the money is right. No, it is not something you can do in a matter of hours, or days, or even weeks. You could *maybe* replace the parts required for an LLM (which are a relative small portion of the overall software package) in a matter of months if you knew *exactly* what you were doing and had unlimited money to burn on poaching people. Realistically, it's a multi-year project at best.
Am I wrong in saying this is an argument that selling them H20s would slow down their development? Assuming sometime in the future we tighten export controls (pretty likely) then the Chinese would have been less likely to switch from the CUDA stack.
"I will keep repeating, if necessary until I am blue in the face, that this is effectively a call (the motivations for which I do not care to speculate) for sacrificing the future and get us all killed in order to maximize Nvidia’s market share. "
Any possibility of getting Nvidia to STFU about its market share if, e.g. its sales to US companies were lightly subsidized?
"The good news is that there are some steps in the actual concrete plan to start preparing for those problems, even if they are insufficient and it can’t be explained, but it’s a rough path trying to sustain even that level of responsibility under this kind of rhetorical oppression. "
<mildSnark>
The first rule of ASI risk mitigation is that we do not talk about ASI risk mitigation. :-)
</mildSnark>
Re the RAND paper:
"We focus on verification options that oversee large-scale AI compute use, and which have the potential to be confidential and effective internationally. Large-scale AI compute use, in this report, refers to the use of thousands of AI chips for multiple months (Section 2.2). These can be overseen to set rules on frontier AI development and associated mass deployment, without intruding on smallerscale activities. Confidentiality refers to protecting sensitive information—especially models, data, and code—from unauthorized access, especially theft. Finally, international effectiveness means effectiveness even against states determined to conceal non-compliance (Section 1)."
Remember the jump to reasoning models - modest fine training budgets, in a phase AFTER the "thousands of AI chips for multiple months". Ok, verification mechanisms focussed on the massively compute intensive phases of AI research do SOMETHING, but I really, really, really doubt that they will catch the pivotal moments of the next few years.
Re the privacy provisions in the PRC plan (and the EU AI Code of Practice): Who are they trying to kid? Does anyone honestly expect either government(s) to refrain from using AI to penetrate individual privacy? Really???
Re item 22 in the administration's US AI Action plan: "Continued congressional action is needed for appropriations to fund critical AI-related R&D across agencies like the National Science Foundation." a line from Chrysanthemum Tea comes to mind: "And it wouldn't be so tangled if you hadn't had him strangled."
I think chatting with 4o is leading your reasoning astray
Podcast episode for this post:
https://open.substack.com/pub/dwatvpodcast/p/the-week-in-ai-governance?r=67y1h&utm_campaign=post&utm_medium=web&showWelcomeOnShare=true
In your commentary on Peter Wildeford’s summary, after number 10, you wrote “In which case, can we stop waving them, such as with H20 sales to China?” - should be “waiving” not “waving”
I don’t really understand the obsession Zvi and others have with beating China. If it’s true that TSMC is manufacturing constrained then selling them chips slows down AI progress for everyone which is good.
Further as we get into the future, assuming we don’t have extremely short timelines, the H20s will be obsolete (this contradicts my first point ik). If compute really is all you need (as xAI demonstrated) then I’m not sure why allowing the Chinese to train better models now would have any effect on their plans to train human level intelligence.
Also them using the CUDA stack for longer prevents them from properly focusing on their own Chinese versions of the whole thing (not very certain on this point).
I also don’t care that much about China “winning”. Maybe if it was like the North Koreans I’d be concerned but Chinese hegemony would be a billion times better than us all dropping dead.
And finally I’m not so convinced the Chinese will be less safe than we are.
Obviously I’m quite uncertain on this everyone else seems to be convinced I’m wrong.
I think the point Zvi is failing to communicate is that China is positioning itself as an advocate for open models to promote multilateralism, which would effectively strangely any capacity to regulate AI development.
I find that quite unlikely. If China had a lead in models I doubt they would just hand it over to us. I’m quite confident the best models will remain private
That's fair. I'm viewing that part of Zvi's posts as more of a: "Look, administration, if you _are_ trying to win this race, could you at least be sane in your tactics? Maximizing nVidia's market share is very much NOT the same thing as have a US company get to AGI first." (And the latter, not the former, is the relevant "win" here, if there is one.)
Re:
>I also don’t care that much about China “winning”. Maybe if it was like the North Koreans I’d be concerned but Chinese hegemony would be a billion times better than us all dropping dead.
Well... Sort-of mostly agreed, but
a) The CCP is the lineal descendant of Mao's CCP, with his Great Leap Corpseward (arguable 45 million dead...)
b) (writing from the USA) It also depends on how bloodily Chinese hegemony is achieved. "killed by the PRC" and "killed by an ASI" can look pretty similar...
That said, a worst-case MAGA or Woke surveillance state and a median CCP surveillance state might not be that different...
Zvi has made it pretty clear he genuinely cares about the US getting to superintelligence before China.
And yeah I would probably slightly prefer the US to dictate the future but the CCP isn’t *that* bad.
Many Thanks!
>Zvi has made it pretty clear he genuinely cares about the US getting to superintelligence before China.
You are probably right. I'm somewhat unsure. Zvi is often in the unenviable position of having to say that e.g. one of the labs' policies will probably kill us, but is less bad than it could have been.
>And yeah I would probably slightly prefer the US to dictate the future but the CCP isn’t *that* bad.
Partly agreed. After all, almost all of the people _in_ China can live with it. Still, a lot depends on the guy at the top. If Xi's successor is as bad as Mao, it could quickly become a whole lot worse...
For most weaponry these are very good arguments, but I'd argue less so for AI. The main way AI in the hands of the CCP is dangerous for us is essentially the same way it is dangerous for *them*: An unaligned AI that kills or subjugates all, or gets into terrorist hands or whatever. Mao might not have cared about the lives he caused to end, but if he had a magic button to kill 50 million people he would not have pressed it.
Many Thanks! I'm a bit unsure which part(s) of my comment you disagree with.
"The main way AI in the hands of the CCP is dangerous for us is essentially the same way it is dangerous for *them*: An unaligned AI that kills or subjugates all, or gets into terrorist hands or whatever."
I mostly, but not entirely, agree with you. I agree that that is a large fraction of the hazard to us of an AI in CCP hands. I don't think we can tell if the fraction is 25% or 75%, so I;m not sure if "main" (presumably > 50%) is correct.
In the case where AI is successfully aligned by the PRC, "winning" the race, then the question for our fate becomes: "How do they achieve hegemony (bloodily or less so)? and What is their hegemony like (liveable or not)?" - but these are only relevant in the part of the probability mass where the AI(s) is/are following CCP orders, where they have been successfully aligned.
1) It is better if only one country races for AI. Since the US has the possibility of preventing China from racing to AI, it is better that they attempt to do so. China has no realistic possibility of preventing the US racing to AI. (that's a simplification, but a valid strategy could be hoping to have a year lead time or something)
2)There is also some political reasons. It would be outside the overton window to suggest to (current) US policymakers that it would not be bad if China won.
Having said that, I agree with you. People in China, on average, seem to live fairly satisfactory lives, and it seems crazy to prefer to irresponsibly race with China, given that an aligned AI would usher in an age of abundance for the world, such that as an European I don't care whether I get the American or Chinese scraps.
>I will keep repeating, if necessary until I am blue in the face, that this is effectively a call (the motivations for which I do not care to speculate) for sacrificing the future and get us all killed in order to maximize Nvidia’s market share.
Well yeah, when the kids talk about "late-stage" capitalism did you think it was a joke?
Ok, but this predates _late_ stage capitalism:
“The Capitalists will sell us the rope with which we will hang them.” Vladimir Ilich Lenin
Does Zuckerberg think he is Harry in HPMOR when the others ban traitors and he's like mwhahaha, I'll still do what I want? He's truly an agent of chaos trying to win at any cost.
> You can run any most any AI model on most any advanced chip, and switch on an hour’s notice.
This is directionally correct but heavily overstated: fused kernels are not hot-swappable like that, and at the moment there's not even any alternative to swap to. Yes, CUDA can and will be replaced if the money is right. No, it is not something you can do in a matter of hours, or days, or even weeks. You could *maybe* replace the parts required for an LLM (which are a relative small portion of the overall software package) in a matter of months if you knew *exactly* what you were doing and had unlimited money to burn on poaching people. Realistically, it's a multi-year project at best.
Am I wrong in saying this is an argument that selling them H20s would slow down their development? Assuming sometime in the future we tighten export controls (pretty likely) then the Chinese would have been less likely to switch from the CUDA stack.
I think that's almost certainly wrong, yes. Developing a software stack is hard. Developing a hardware stack and a software stack is much harder.
"I will keep repeating, if necessary until I am blue in the face, that this is effectively a call (the motivations for which I do not care to speculate) for sacrificing the future and get us all killed in order to maximize Nvidia’s market share. "
Any possibility of getting Nvidia to STFU about its market share if, e.g. its sales to US companies were lightly subsidized?
"The good news is that there are some steps in the actual concrete plan to start preparing for those problems, even if they are insufficient and it can’t be explained, but it’s a rough path trying to sustain even that level of responsibility under this kind of rhetorical oppression. "
<mildSnark>
The first rule of ASI risk mitigation is that we do not talk about ASI risk mitigation. :-)
</mildSnark>
Re the RAND paper:
"We focus on verification options that oversee large-scale AI compute use, and which have the potential to be confidential and effective internationally. Large-scale AI compute use, in this report, refers to the use of thousands of AI chips for multiple months (Section 2.2). These can be overseen to set rules on frontier AI development and associated mass deployment, without intruding on smallerscale activities. Confidentiality refers to protecting sensitive information—especially models, data, and code—from unauthorized access, especially theft. Finally, international effectiveness means effectiveness even against states determined to conceal non-compliance (Section 1)."
Remember the jump to reasoning models - modest fine training budgets, in a phase AFTER the "thousands of AI chips for multiple months". Ok, verification mechanisms focussed on the massively compute intensive phases of AI research do SOMETHING, but I really, really, really doubt that they will catch the pivotal moments of the next few years.
Re the privacy provisions in the PRC plan (and the EU AI Code of Practice): Who are they trying to kid? Does anyone honestly expect either government(s) to refrain from using AI to penetrate individual privacy? Really???
Re item 22 in the administration's US AI Action plan: "Continued congressional action is needed for appropriations to fund critical AI-related R&D across agencies like the National Science Foundation." a line from Chrysanthemum Tea comes to mind: "And it wouldn't be so tangled if you hadn't had him strangled."