"The risk is that this report greatly expands and other requirements are added over time, in a way that do introduce an undue burden, and that this also applies to a wider range of systems, and also that none of that was actually necessary for safety. That is a possible world. I don’t expect it, but it makes sense to keep an eye."
Assuming the "death by a thousand cuts" scenario is plausible, isn't the best time for pushback _right now_ rather than in a few years when the requirements get ratcheted up? Yes, the requirements _today_ are to send out a few reports, but the optimal strategy for fighting this is to ignore the "it's just reports" part and blow it up into a huge political fight.
This is a good parallel to the fight against extinction-level AGI: safety people aren't exactly convinced by the idea that we shouldn't pushback against AGI development today because AI capabilities are not that significant just yet. Instead, safety people want to start the fight _now_ and not wait for these capabilities to actually happen.
Yep, I agree that if your only goal is to not be cut a thousand times, you should try and stop people from buying knives, if you think you can be effective doing that.
But also if you go full blast when someone calls for writing reports, the risk is that then people tune you out, because your objections are not data on whether something is reasonable.
I think you can just admit that it is a slippery slope instead of telling critics that their concerns are just about a report that ChatGPT can develop for them. It’s a snarky response and I think you can do better.
But more importantly why start requiring reporting now? Your article indicates that no one is going to train at that level until 2025. Give the field some time to develop. We can regulate later. I take your concerns about the existential threat seriously and think the best course of action is to recruit talent and put funding into R&D on this issue but let’s not start regulatory regimes that begin metastasizing before we even have a good agreed upon definition of AI.
They are requiring reporting for when compute used rises to the level that there could be substantial existential risk from training such a model (some say that level starts sooner, technically yes but not enough that I'd want to fight that battle). So whenever it happens, you want to know about it, and that's all this is saying.
I'm affirming the slope arguments rather explicitly. But this is exactly what 'regulate later' looks like - you get reports done on what options you have, and then decide later. There's a conflation here, I think, between 'any step towards regulation, including exploring options, is part of a slope towards inevitable bad regulations, so we should never even ask the question of how one might regulate' which is... a position, and 'it is too early to do concrete rules that would lock us in' and it seems highly legit to say 'well then good, this doesn't do that.'
I get your point but way leads on to way as Frost said. This isn’t a question about how to regulate. It’s a nascent regulatory framework. I’d much rather an explicit statement that regulation is premature at this point or if you can’t say that then that an EO is not the medium for this kind of regulation (ie telling agencies how to use AI is fine but setting up regulatory regimes is not). Say that AI and its definition need to be further refined. Fully admit to the existential issues. Fund people and research into those issues. Why not just a report on what regulatory options there are and their pros and cons? That would be consistent with the report nature of the EO.
One other issue is that by doing this as an EO instead of a legislative process, you’ve open the door to politicizing the issue.
Come on, Zvi. This is the exact story of every regulation: “The risk is that this report greatly expands and other requirements are added over time, in a way that do introduce an undue burden, and that this also applies to a wider range of systems, and also that none of that was actually necessary for safety. That is a possible world. I don’t expect it, but it makes sense to keep an eye.”
You greatly underestimate this possibility. It’s a guarantee.
Thanks for your two posts. FYI, in Varun's take they mention "10^26 floating-point operations (or 0.1 exaflops)" which is a completely wrong conversion, off by a billion. The exa prefix means 1e18. Instead it should be one of the new prefixes, specifically 0.1 ronnaflops (or 100 yottaflops). The mistake is repeated a few lines below.
"Matthew Yglesias: My thought is that it’s probably not a good idea to cover the administration’s AI safety efforts much because if this becomes a high-profile topic it will induce Republicans to oppose it."
That is not the way this would go. Instead, Republicans would criticize it as not going far enough. The issue is not yet clearly polarized, but when/if it does, it's likely that the right will be the more pro-safety side, unless something changes a lot. (Polls: "The uncontrollable effects of artificial intelligence (AI) could be so harmful that it may risk the future of humankind", Trump voters - 70% agree, Biden voters - 60%, Ipsos poll; "How worried are you that machines with artificial intelligence could eventually pose a threat to the existence of the human race – very, somewhat, not too, or not at all worried?", Republicans - 31% "very worried", Democrats - 21%, 31% each for "somewhat worried", Monmouth. Among politicians, it's less obvious a skew, but Sunak, von der Leyen, and Netanyahu are all right-wing within their systems.) This will likely end up being a problem, because academia, big tech, and the media are all held by the left.
I love the symmetry between the opposing sides in this debate. For some, AI unaligned with the interests of powerful corporations and governments seems like the overwhelming existential risk to humanity. For others, it’s AI aligned with those interests that seems most risky.
Note that I reject the notion that AI could be aligned with humanity as such. Clearly, nobody serious wants AI aligned with North Korea, religious extremists, anti-vaxxers, or other assorted “bad actors”.
I think AI *truly* aligned with NK, religious extremists and anti-vaxxers would probably be at least almost as good as AI aligned with myself.
Everybody's lives in NK are almost certainly enormously improved by a Friendly Singularity, including the ruling family's and the military's. Antivaxxers have certain beliefs that I think are wrong; but if antivaxxers are right, I would want AI aligned with me to act like it was aligned with them; conversely, if antivaxxers were wrong, they would certainly not want the superintelligence to pretend otherwise. Again, their lives are massively improved by any Friendly Singularity worth having. Religious extremists are the biggest stretch, because they're the ones who have moral claims on other people. However, religions are already selected pretty heavily for coexistence, so I suspect it would still work out.
As I observe the world, some of the most salient “human values” include the destruction of one’s enemies.
People don’t just want wealth, they want others to be poor.
People don’t just want attention, they want others to be ignored.
People don’t just want happiness, they want others to be miserable.
Anti-vaxxers, for instance, don’t just want “the truth” to win, they want pharmaceutical industry leaders, public health officials, and pro-vax philanthropists to be stripped of status, humiliated, and prosecuted.
So, should AI actually become aligned with “human values”, existential risk doesn’t get eliminated. Alignment with human values just means that AI will likely have some of the same “dark tetrad” traits that humans do.
Narcissism, psychopathy, Machiavellianism, and sadism are not merely moral flaws, these traits have evolutionary fitness because they facilitate dominant behaviors — disproportionate command of human and material capital — which increase the survival of one’s offspring.
Obama actually drove the effort and communicated/coordinated on Zoom with the various tech bros..
Excellent links, thanks! The EO doesn’t sound too diabolical, but (as with all EOs) there is usually an activist/donor-led agenda not always visible in the text. The good news is that it may be a year or two away from mattering.
IMO "tens of B" means between 10.1B and 19.9B. 'Tens', in this case, being used instead of the common usage 'teens.' And we all know what teens means.
Sooner or later we may find out the actual threshold, but if I read you right, you have concerns about power in the mid-teens. Would be funny if a bunch of developers wasted their time on 19B models, based on that interpretation, as a means to dodge reporting. Do your paperwork!
"The risk is that this report greatly expands and other requirements are added over time, in a way that do introduce an undue burden, and that this also applies to a wider range of systems, and also that none of that was actually necessary for safety. That is a possible world. I don’t expect it, but it makes sense to keep an eye."
Assuming the "death by a thousand cuts" scenario is plausible, isn't the best time for pushback _right now_ rather than in a few years when the requirements get ratcheted up? Yes, the requirements _today_ are to send out a few reports, but the optimal strategy for fighting this is to ignore the "it's just reports" part and blow it up into a huge political fight.
This is a good parallel to the fight against extinction-level AGI: safety people aren't exactly convinced by the idea that we shouldn't pushback against AGI development today because AI capabilities are not that significant just yet. Instead, safety people want to start the fight _now_ and not wait for these capabilities to actually happen.
Yep, I agree that if your only goal is to not be cut a thousand times, you should try and stop people from buying knives, if you think you can be effective doing that.
But also if you go full blast when someone calls for writing reports, the risk is that then people tune you out, because your objections are not data on whether something is reasonable.
I think you can just admit that it is a slippery slope instead of telling critics that their concerns are just about a report that ChatGPT can develop for them. It’s a snarky response and I think you can do better.
But more importantly why start requiring reporting now? Your article indicates that no one is going to train at that level until 2025. Give the field some time to develop. We can regulate later. I take your concerns about the existential threat seriously and think the best course of action is to recruit talent and put funding into R&D on this issue but let’s not start regulatory regimes that begin metastasizing before we even have a good agreed upon definition of AI.
They are requiring reporting for when compute used rises to the level that there could be substantial existential risk from training such a model (some say that level starts sooner, technically yes but not enough that I'd want to fight that battle). So whenever it happens, you want to know about it, and that's all this is saying.
I'm affirming the slope arguments rather explicitly. But this is exactly what 'regulate later' looks like - you get reports done on what options you have, and then decide later. There's a conflation here, I think, between 'any step towards regulation, including exploring options, is part of a slope towards inevitable bad regulations, so we should never even ask the question of how one might regulate' which is... a position, and 'it is too early to do concrete rules that would lock us in' and it seems highly legit to say 'well then good, this doesn't do that.'
I get your point but way leads on to way as Frost said. This isn’t a question about how to regulate. It’s a nascent regulatory framework. I’d much rather an explicit statement that regulation is premature at this point or if you can’t say that then that an EO is not the medium for this kind of regulation (ie telling agencies how to use AI is fine but setting up regulatory regimes is not). Say that AI and its definition need to be further refined. Fully admit to the existential issues. Fund people and research into those issues. Why not just a report on what regulatory options there are and their pros and cons? That would be consistent with the report nature of the EO.
One other issue is that by doing this as an EO instead of a legislative process, you’ve open the door to politicizing the issue.
Come on, Zvi. This is the exact story of every regulation: “The risk is that this report greatly expands and other requirements are added over time, in a way that do introduce an undue burden, and that this also applies to a wider range of systems, and also that none of that was actually necessary for safety. That is a possible world. I don’t expect it, but it makes sense to keep an eye.”
You greatly underestimate this possibility. It’s a guarantee.
Are there any signs that the EO was written at all by an LLM because that would be pretty funny.
Thanks for your two posts. FYI, in Varun's take they mention "10^26 floating-point operations (or 0.1 exaflops)" which is a completely wrong conversion, off by a billion. The exa prefix means 1e18. Instead it should be one of the new prefixes, specifically 0.1 ronnaflops (or 100 yottaflops). The mistake is repeated a few lines below.
"Matthew Yglesias: My thought is that it’s probably not a good idea to cover the administration’s AI safety efforts much because if this becomes a high-profile topic it will induce Republicans to oppose it."
That is not the way this would go. Instead, Republicans would criticize it as not going far enough. The issue is not yet clearly polarized, but when/if it does, it's likely that the right will be the more pro-safety side, unless something changes a lot. (Polls: "The uncontrollable effects of artificial intelligence (AI) could be so harmful that it may risk the future of humankind", Trump voters - 70% agree, Biden voters - 60%, Ipsos poll; "How worried are you that machines with artificial intelligence could eventually pose a threat to the existence of the human race – very, somewhat, not too, or not at all worried?", Republicans - 31% "very worried", Democrats - 21%, 31% each for "somewhat worried", Monmouth. Among politicians, it's less obvious a skew, but Sunak, von der Leyen, and Netanyahu are all right-wing within their systems.) This will likely end up being a problem, because academia, big tech, and the media are all held by the left.
Except... you know... Trump actually promised repealing the Order?
https://www.nextgov.com/artificial-intelligence/2024/11/trump-promised-repeal-bidens-ai-executive-order-heres-what-expect-next/400934/
Fun note: the Libertarian position seems to be that "generating a bunch of text" will lead to "bad things happening."
Applying this logic to other domains is left as an exercise for the reader.
I love the symmetry between the opposing sides in this debate. For some, AI unaligned with the interests of powerful corporations and governments seems like the overwhelming existential risk to humanity. For others, it’s AI aligned with those interests that seems most risky.
Note that I reject the notion that AI could be aligned with humanity as such. Clearly, nobody serious wants AI aligned with North Korea, religious extremists, anti-vaxxers, or other assorted “bad actors”.
I think AI *truly* aligned with NK, religious extremists and anti-vaxxers would probably be at least almost as good as AI aligned with myself.
Everybody's lives in NK are almost certainly enormously improved by a Friendly Singularity, including the ruling family's and the military's. Antivaxxers have certain beliefs that I think are wrong; but if antivaxxers are right, I would want AI aligned with me to act like it was aligned with them; conversely, if antivaxxers were wrong, they would certainly not want the superintelligence to pretend otherwise. Again, their lives are massively improved by any Friendly Singularity worth having. Religious extremists are the biggest stretch, because they're the ones who have moral claims on other people. However, religions are already selected pretty heavily for coexistence, so I suspect it would still work out.
Would you consider yourself a “mistake theorist”?
As I observe the world, some of the most salient “human values” include the destruction of one’s enemies.
People don’t just want wealth, they want others to be poor.
People don’t just want attention, they want others to be ignored.
People don’t just want happiness, they want others to be miserable.
Anti-vaxxers, for instance, don’t just want “the truth” to win, they want pharmaceutical industry leaders, public health officials, and pro-vax philanthropists to be stripped of status, humiliated, and prosecuted.
So, should AI actually become aligned with “human values”, existential risk doesn’t get eliminated. Alignment with human values just means that AI will likely have some of the same “dark tetrad” traits that humans do.
Narcissism, psychopathy, Machiavellianism, and sadism are not merely moral flaws, these traits have evolutionary fitness because they facilitate dominant behaviors — disproportionate command of human and material capital — which increase the survival of one’s offspring.
Obama actually drove the effort and communicated/coordinated on Zoom with the various tech bros..
Excellent links, thanks! The EO doesn’t sound too diabolical, but (as with all EOs) there is usually an activist/donor-led agenda not always visible in the text. The good news is that it may be a year or two away from mattering.
IMO "tens of B" means between 10.1B and 19.9B. 'Tens', in this case, being used instead of the common usage 'teens.' And we all know what teens means.
Sooner or later we may find out the actual threshold, but if I read you right, you have concerns about power in the mid-teens. Would be funny if a bunch of developers wasted their time on 19B models, based on that interpretation, as a means to dodge reporting. Do your paperwork!