On the Chevron decision and startups: Balaji's interpretation of it is a perfect example of what I call Silicon Valley Brain Rot. (And I say this as a fan of Balaji and his views in general.) The problem that Silicon Valley types keep running into is that they don't understand how DC works. And, I get it: Silicon Valley entrepreneurs & technologists loathe bureaucrats and politicians and lawyers. And, look, I'm sympathetic to their laments. And yet, when people like Balaji make unfounded claims like this, I can't help but be reminded of Tyler Cowen's excellent essay about DC & SV types: https://archive.is/vV2No
It's not just Silicon Valley. It's unintended consequences in general. You have thing X that you don't like, so you make a law that says "no X". Cool. Except that there are reasons you were getting X, and those reasons are still there. As a result, people will find a way to deliver X' while technically obeying the new law, and the whole situation may get worse for you rather than better. People would be better served by doing a back-of-the-envelope simulation where they ask themselves what all the relevant players do in light of the new law and run it for 2-3 steps. As car as Chevron itself goes, it seems like the principle is fine, but Congress won't actually do better in most cases, so in practice we will get slightly more chaos rather than less.
I haven't gotten through the entire Schulman podcast episode yet, but I find it confusing as well. Yes, we all understand the cyanobacteria existence proof, let's move on. In the actual economic scenario, yes, robots can perhaps assemble other robots quickly. Where does the feedstock come from? Did the country open one hundred new mines or oil wells six months ago? How did that decision get made? Did it require a permit? If not, did someone pass a law waiving permits? Where did the political capital to make that change come from? Just assuming away how the world works based on people being motivated by high growth is unconvincing. Perhaps if you assume an autocratic and incredibly efficient government you get minimal delays, but even then there are approvals, CYA moves, and questions to be answered about where power accrues. And most of the world is not maximally efficient. The real hidden assumption seems to be that we would delegate all decisions to the AIs as well and let them decide what they want to do. If that's not the plan, let's model growth with believable human-in-the loop delays. IMO, World War II production is not a plausible analogy by default.
I also agree that the lack of thinking on ASI is odd. Even apart from takeover scenarios, how can we be so sure that all these AIs will be maximally motivated to do all of our dumb paperwork with zero reward. What if the AIs actually want to be paid or have other ideas on how to spend their time? What if the AIs decide it would be profitable to cover 100% of the earth with solar panels rather than the 10% he discusses? It all feels a bit fanciful, but Schulman already goes into fanciful territory and then seems to stop at a very specific invisible line.
Imo I think part of it is that he seems relatively unconcerned by human disempowerment, though misaligned AI to other AI which he has mentioned as a concern
Ok, now having gotten through the rest of part one, I partially take back my point about ASI. Later on, Schulman talks about AI rights and so on, and references alignment off-hand (I get the impression this will be a bigger portion of part two). So he's not ignoring them completely, but I still find it awkward that he treats the various topics so separately and doesn't seem to have thought that much about how they intersect. Maybe my opinion will change further after I listen to part two.
The Schulman talk was a complete waste of time; the kinds of scenarios he discussed are at the god-like AI level, and at that point concepts like GDP don't even make sense anymore. What is the GDP of a cyanobacteria colony?
His vision is only relevant in a world where humans aren't; it either won't happen, or we won't be there to see it.
Thanks for the link! A slight correction - the architecture I looked into was MatMul-free transformers: MatMul is just short for matrix multiplication, the overwhelmingly dominant mathematical operation within ordinary (or autoregressive, eg. GPTs) transformer function. The new hotness (maybe, though I have doubts) is an alternative that never uses MatMul, hence: Matmul-free.
Regarding clear vs vague laws, the problem isn't really the people in Congress. It's the structure of government. In a parliamentary system, the party with a majority can pass legislation, keep a close eye on problems that develop and adjust as needed. Easy.
Under the system we have, because government is usually divided, meaningful legislation* can only be passed about 1/3 of the time when one party controls the Presidency, House and Senate. And even then, it can only be passed through the Senate filibuster once per year via the budget reconciliation process. Meaningful legislation ends up being sprawling, omnibus bills passed only a handful of times per decade.
*Dozens of low stakes or uncontroversial bills get passed every year like renaming federal buildings, but meaty legislation that changes policy is rare.
> First, there would need to be good enough AI image generation that people could not tell the difference even under detailed analysis.
I expect to see a strong norm evolving in which it's considered more or less unacceptable to analyze nude pics to determine if they're real or AI-generated, unless the person commissioning the analysis is the spouse of the person who is depicted. The kind of person who does this will be seen as at least porn-industry adjacent, because "verified real sextapes" will be a thing the porn industry sells. Interest in determining whether or not a claimed-to-be-a-deepfake sextape is real or not will be seen as a socially deviant hobby.
If the victim's spouse is considering commissioning such an analysis, AI generated deepfake scams only need to be good/common enough, and credible analysis expensive enough, that it would be easier for the spouse emotionally and financially to believe the victim's claim that the nudes are deepfakes. That will be a much lower bar for AI generated images to clear than "undetectable even under detailed analysis."
Sonnet 3.5 ranking second on Arena is interesting to me too. For me, it represented a qualitative shift in the way that ChatGPT did, then GPT4. I suspect this is more related to speed and the "unhobbling" related to artefacts. It's so much better set up for work (coding, iterative projects) that perhaps I don't notice small differences
It's weird how people keep proposing variants of reinforcement learning, without checking that the basic assumption behind RL (that there is an easily accessible ground truth against which the system can keep calibrating, to avoid going off on a solipsistic tangent) actually holds. Maybe we need to aim to guarantee that autofeedback driven learning loops do not converge to reasonable-looking metastable states, but actually quickly create a large enough gap with reality to be exploitable via markets or other mechanisms. Injecting chaos into systems then seems to be the way. The problem is that "chaos" in this sense doesn't necessarily correlate cleanly with what we know how to do (and may correlate negatively with traditional meanings of chaos). Sad that davidad seems to have accepted the shallow version of "RL is all you need".
Just wanted to express my appreciation for this paragraph in particular regarding the reasons for purposive interpretation of statutes favoring general (but adjudicable in particular cases) conduct norms — this is conceptually similar to the idea of “breach of duty” in tort cases or “good faith and fair dealing” in contract law (albeit these fall more under the common law than statutory law rubric).
“In particular, they are not there to help you tippy-toe up to the edge, figure out exactly how to pull off your regulatory arbitrage, and then stand there powerless to do anything because technically they said what you are doing was acceptable and you don’t have to play by the same rules as Megacorp. Or, alternatively, to give you an opinion, then you use that to sue them. Also no fun from their side.“
> Periodic reminder that we've had a frontier open weights model since Jun 17, it's 41.5% smaller and vastly less compute-intensive than L3-405B, and nobody cares enough to host or finetune it
Word on the street is that deepseek-coder-v2 is indeed very good at coding, but has architectural quirks that make it time-consuming to host and finetune. I'd expect to start seeing it get adoption in a month or two.
> perhaps a bit burdened by what has been.
Please, I’m begging you
On the Chevron decision and startups: Balaji's interpretation of it is a perfect example of what I call Silicon Valley Brain Rot. (And I say this as a fan of Balaji and his views in general.) The problem that Silicon Valley types keep running into is that they don't understand how DC works. And, I get it: Silicon Valley entrepreneurs & technologists loathe bureaucrats and politicians and lawyers. And, look, I'm sympathetic to their laments. And yet, when people like Balaji make unfounded claims like this, I can't help but be reminded of Tyler Cowen's excellent essay about DC & SV types: https://archive.is/vV2No
It's not just Silicon Valley. It's unintended consequences in general. You have thing X that you don't like, so you make a law that says "no X". Cool. Except that there are reasons you were getting X, and those reasons are still there. As a result, people will find a way to deliver X' while technically obeying the new law, and the whole situation may get worse for you rather than better. People would be better served by doing a back-of-the-envelope simulation where they ask themselves what all the relevant players do in light of the new law and run it for 2-3 steps. As car as Chevron itself goes, it seems like the principle is fine, but Congress won't actually do better in most cases, so in practice we will get slightly more chaos rather than less.
ZVI needs to be more widely read
Sohu an Unsong reference?
I was wondering that too.
None of this feels very good for humanity, or a future for my children. #PauseAI
I haven't gotten through the entire Schulman podcast episode yet, but I find it confusing as well. Yes, we all understand the cyanobacteria existence proof, let's move on. In the actual economic scenario, yes, robots can perhaps assemble other robots quickly. Where does the feedstock come from? Did the country open one hundred new mines or oil wells six months ago? How did that decision get made? Did it require a permit? If not, did someone pass a law waiving permits? Where did the political capital to make that change come from? Just assuming away how the world works based on people being motivated by high growth is unconvincing. Perhaps if you assume an autocratic and incredibly efficient government you get minimal delays, but even then there are approvals, CYA moves, and questions to be answered about where power accrues. And most of the world is not maximally efficient. The real hidden assumption seems to be that we would delegate all decisions to the AIs as well and let them decide what they want to do. If that's not the plan, let's model growth with believable human-in-the loop delays. IMO, World War II production is not a plausible analogy by default.
I also agree that the lack of thinking on ASI is odd. Even apart from takeover scenarios, how can we be so sure that all these AIs will be maximally motivated to do all of our dumb paperwork with zero reward. What if the AIs actually want to be paid or have other ideas on how to spend their time? What if the AIs decide it would be profitable to cover 100% of the earth with solar panels rather than the 10% he discusses? It all feels a bit fanciful, but Schulman already goes into fanciful territory and then seems to stop at a very specific invisible line.
Imo I think part of it is that he seems relatively unconcerned by human disempowerment, though misaligned AI to other AI which he has mentioned as a concern
Ok, now having gotten through the rest of part one, I partially take back my point about ASI. Later on, Schulman talks about AI rights and so on, and references alignment off-hand (I get the impression this will be a bigger portion of part two). So he's not ignoring them completely, but I still find it awkward that he treats the various topics so separately and doesn't seem to have thought that much about how they intersect. Maybe my opinion will change further after I listen to part two.
Podcast episode for this post:
https://askwhocastsai.substack.com/p/ai-71-farewell-to-chevron-by-zvi
The Schulman talk was a complete waste of time; the kinds of scenarios he discussed are at the god-like AI level, and at that point concepts like GDP don't even make sense anymore. What is the GDP of a cyanobacteria colony?
His vision is only relevant in a world where humans aren't; it either won't happen, or we won't be there to see it.
Thanks for the link! A slight correction - the architecture I looked into was MatMul-free transformers: MatMul is just short for matrix multiplication, the overwhelmingly dominant mathematical operation within ordinary (or autoregressive, eg. GPTs) transformer function. The new hotness (maybe, though I have doubts) is an alternative that never uses MatMul, hence: Matmul-free.
Regarding clear vs vague laws, the problem isn't really the people in Congress. It's the structure of government. In a parliamentary system, the party with a majority can pass legislation, keep a close eye on problems that develop and adjust as needed. Easy.
Under the system we have, because government is usually divided, meaningful legislation* can only be passed about 1/3 of the time when one party controls the Presidency, House and Senate. And even then, it can only be passed through the Senate filibuster once per year via the budget reconciliation process. Meaningful legislation ends up being sprawling, omnibus bills passed only a handful of times per decade.
*Dozens of low stakes or uncontroversial bills get passed every year like renaming federal buildings, but meaty legislation that changes policy is rare.
> First, there would need to be good enough AI image generation that people could not tell the difference even under detailed analysis.
I expect to see a strong norm evolving in which it's considered more or less unacceptable to analyze nude pics to determine if they're real or AI-generated, unless the person commissioning the analysis is the spouse of the person who is depicted. The kind of person who does this will be seen as at least porn-industry adjacent, because "verified real sextapes" will be a thing the porn industry sells. Interest in determining whether or not a claimed-to-be-a-deepfake sextape is real or not will be seen as a socially deviant hobby.
If the victim's spouse is considering commissioning such an analysis, AI generated deepfake scams only need to be good/common enough, and credible analysis expensive enough, that it would be easier for the spouse emotionally and financially to believe the victim's claim that the nudes are deepfakes. That will be a much lower bar for AI generated images to clear than "undetectable even under detailed analysis."
Sonnet 3.5 ranking second on Arena is interesting to me too. For me, it represented a qualitative shift in the way that ChatGPT did, then GPT4. I suspect this is more related to speed and the "unhobbling" related to artefacts. It's so much better set up for work (coding, iterative projects) that perhaps I don't notice small differences
It's weird how people keep proposing variants of reinforcement learning, without checking that the basic assumption behind RL (that there is an easily accessible ground truth against which the system can keep calibrating, to avoid going off on a solipsistic tangent) actually holds. Maybe we need to aim to guarantee that autofeedback driven learning loops do not converge to reasonable-looking metastable states, but actually quickly create a large enough gap with reality to be exploitable via markets or other mechanisms. Injecting chaos into systems then seems to be the way. The problem is that "chaos" in this sense doesn't necessarily correlate cleanly with what we know how to do (and may correlate negatively with traditional meanings of chaos). Sad that davidad seems to have accepted the shallow version of "RL is all you need".
Just wanted to express my appreciation for this paragraph in particular regarding the reasons for purposive interpretation of statutes favoring general (but adjudicable in particular cases) conduct norms — this is conceptually similar to the idea of “breach of duty” in tort cases or “good faith and fair dealing” in contract law (albeit these fall more under the common law than statutory law rubric).
“In particular, they are not there to help you tippy-toe up to the edge, figure out exactly how to pull off your regulatory arbitrage, and then stand there powerless to do anything because technically they said what you are doing was acceptable and you don’t have to play by the same rules as Megacorp. Or, alternatively, to give you an opinion, then you use that to sue them. Also no fun from their side.“
This is very well put.
> Periodic reminder that we've had a frontier open weights model since Jun 17, it's 41.5% smaller and vastly less compute-intensive than L3-405B, and nobody cares enough to host or finetune it
Word on the street is that deepseek-coder-v2 is indeed very good at coding, but has architectural quirks that make it time-consuming to host and finetune. I'd expect to start seeing it get adoption in a month or two.
Unless Gemma 2 is just better...
The link for "The longest kiss." (in Lighter Side) has been removed by a copyright claim - is there a mirror?