Welcome, new readers! This is my weekly AI post, where I cover everything that is happening in the world of AI, from what it can do for you today (‘mundane utility’) to what it can promise to do for us tomorrow, and the potentially existential dangers future AI might pose for humanity, along with covering the discourse on what we should do about all of that.
"We must ensure that we are shaping and commanding the more capable A.I.s of the coming years, rather than letting them shape and command us. The critical first step in making that possible is to enact legislation requiring visibility into the training of any new A.I. model that potentially approaches or exceeds the state of the art. Mandatory oversight of cutting-edge models will not solve the underlying problem, but it will be necessary in order to make finding a future solution possible."
Bravo, well done getting this to be the take-away point of your NYT op-ed.
"Technically I would say: Powerful entities generally caring about X tends not to be a stable equilibrium, even if it is stable ‘on reflection’ within a given entity. It will only hold if caring more about X provides a competitive advantage against other similarly powerful entities, or if there can never be a variation in X-caring levels between such entities that arises other than through reflection, and also reflection never causes reductions in X-caring despite this being competitively advantageous."
I will note one place I see an avenue to victory (though likely still a worse approach): If we successfully get the first system capable of an intelligence explosion to care enough to prioritize continuing to care as it self-improves, the head start may well let it assume full control of the future even if later competitors' approaches would've been more efficient.
(Not sure if I should comment here or at your Wordpress site, so I did both)
I disagree with Megan McCardle who said: "Our ability to evaluate student learning through out-of-class writing projects is coming to an end."
I've posted this before, but it's very easy to prevent students from using ChatGpt to write their papers. All the teacher has to do is force students to use Microsoft Word with parameters set such that Word both autosaves the paper every minute and also keeps a large version history. Then, the teacher requires the student to not only turn in the final document, but also the entire version history thus providing documentation of the writing process. This solves the cheating problem because when writing their own paper, students will add, delete and move sections around as the document grows in an organic way. When using ChatGPT, the paper simply appears out of whole cloth in a finished form. Even if the student takes the trouble to type in the document, the version history will not look anything like it would have had the student written the document themselves. There is no easy way at this time to use ChatGPT simulate the writing process that a human would use.
The teacher would not even have to look at the version history unless they suspected cheating. And just the threat of being able to be caught would deter most students.
> Tyler Cowen reviews AI music generator Suno. It is technically impressive. That does not mean one would want to listen to the results.
I've found it to be pretty good, but probably a good chunk of that is me disliking the (what feels like) 90% of music being about love/sadness. Getting Suno to generate music about transhumanism, space operas, or even medieval inquisitors is nice. There simply isn't much music like that, especially in genres other than filk. (Like the released Secular Solstice songs are good, but I like harder music)
Definitely not as good overall, as you say voices should be better, but fitting my tastes better makes up for that decently well. Is not more than 10% of my music listening however.
Do you know of any guides written to extract mundane utility from the current crop of LLMs? It seems most knowledge (like all things) is left implicit in Tweets and conversations in comment sections.
> You can of course Read the Whole Thing, and I encourage that if you have the time and interest, but these posts are long, so they also designed to also let you pick the sections that you find most interesting. Each week, I pick the sections I feel are the most important, and put them in bold in the table of contents.
> Yes, there are several things here of potential interest if they are thoughtful. But, I mean, ow, my eyes. I would like to think we could all agree that human extinction is bad, that increasing the probability of it is bad, and that lowering that probability or delaying when it happens is good. And yet, here we are?
It seems like you think the Pettigrew paper argues for human extinction? It actually argues against longtermism, by arguing that longtermism + risk aversion => we should hasten human extinction. But of course another response is to reject (the paper's version of) risk aversion. Cf. https://rychappell.substack.com/p/risk-aversion-as-a-humility-heuristic
On the whole Noah Smith discourse, if humans could have comparative advantage in the world of AGI, it begs the question of why horses don’t currently have a comparative advantage in the age of the car.
It may be because horses are already maximally employed in a niche that does not directly compete with a car, but I suspect it also has to do with the fact that there is an inherit friction associated with deploying an inefficient slow cog in a machine that’s used to operating at 10000x speed.
Gemini's cybersecurity test suite is already better than OpenAI's preparedness framework. Turning popular CTFs into evals and publishing your results seems like the least and most obvious thing you could do if you just wanted to try at all.
Is that enough?
Camera pans to the appropriately named "DarkGemini ... a powerful new GenAI chatbot now being sold on the dark web for a $45 monthly subscription. It can generate a reverse shell, build malware, or even locate people based on an image."
Good News Everyone! I'm lowering my odds of ASI taking over the internet. Centaurs will assuredly melt the internet into unusable slag well before that's of any concern!
I find this mega thread impossible to consume due to its sheer content volume. Does anyone have a recommendation on how to read something like this productively without spending 4 hours going line by line?
>> All we need is ONE regulation – a limit on the fraction of energy that can go to data centers.
> That still requires driving up the cost of any compute useful for inference by orders of magnitude from where it is today, and keeping it there by global fiat.
I'm not understanding how energy restrictions became compute restrictions here. What am I missing? As I see it ASI is using every regulated joule to successfully discover new compute strategies as well as new energy sources, while leaving an increasingly shrinking percentage of that energy budget to humans. Energy and compute still gets cheaper, and comparative advantage is still in play, humans doing everything less important than designing new compute architectures and fusion power plants.
“If your rules technically say that Killmonger gets to be in charge, and you know he is going to throw out all the rules and become a bloodthirsty warmongering dictator the second he gains power, what do you do?”
Or, to quote ultimate Killmonger Anton Chigurh: “If the rule you followed brought you to this, of what use was the rule?”
“It’s not that scenario number two makes zero sense. I presume the argument is ‘well, if we can’t understand how things work, the AI won’t understand how anything works either?’ So… that makes everything fine, somehow? What a dim hope.”
Surely the actual argument is more, “If we can’t understand how things work, we won’t be able to replicate the full range of things undergirding human intelligence, so maybe our creations will be not so omnipotent after all”?
"We must ensure that we are shaping and commanding the more capable A.I.s of the coming years, rather than letting them shape and command us. The critical first step in making that possible is to enact legislation requiring visibility into the training of any new A.I. model that potentially approaches or exceeds the state of the art. Mandatory oversight of cutting-edge models will not solve the underlying problem, but it will be necessary in order to make finding a future solution possible."
Bravo, well done getting this to be the take-away point of your NYT op-ed.
What are "privacy breeches"?
"Technically I would say: Powerful entities generally caring about X tends not to be a stable equilibrium, even if it is stable ‘on reflection’ within a given entity. It will only hold if caring more about X provides a competitive advantage against other similarly powerful entities, or if there can never be a variation in X-caring levels between such entities that arises other than through reflection, and also reflection never causes reductions in X-caring despite this being competitively advantageous."
I will note one place I see an avenue to victory (though likely still a worse approach): If we successfully get the first system capable of an intelligence explosion to care enough to prioritize continuing to care as it self-improves, the head start may well let it assume full control of the future even if later competitors' approaches would've been more efficient.
(Not sure if I should comment here or at your Wordpress site, so I did both)
I disagree with Megan McCardle who said: "Our ability to evaluate student learning through out-of-class writing projects is coming to an end."
I've posted this before, but it's very easy to prevent students from using ChatGpt to write their papers. All the teacher has to do is force students to use Microsoft Word with parameters set such that Word both autosaves the paper every minute and also keeps a large version history. Then, the teacher requires the student to not only turn in the final document, but also the entire version history thus providing documentation of the writing process. This solves the cheating problem because when writing their own paper, students will add, delete and move sections around as the document grows in an organic way. When using ChatGPT, the paper simply appears out of whole cloth in a finished form. Even if the student takes the trouble to type in the document, the version history will not look anything like it would have had the student written the document themselves. There is no easy way at this time to use ChatGPT simulate the writing process that a human would use.
The teacher would not even have to look at the version history unless they suspected cheating. And just the threat of being able to be caught would deter most students.
> Tyler Cowen reviews AI music generator Suno. It is technically impressive. That does not mean one would want to listen to the results.
I've found it to be pretty good, but probably a good chunk of that is me disliking the (what feels like) 90% of music being about love/sadness. Getting Suno to generate music about transhumanism, space operas, or even medieval inquisitors is nice. There simply isn't much music like that, especially in genres other than filk. (Like the released Secular Solstice songs are good, but I like harder music)
Definitely not as good overall, as you say voices should be better, but fitting my tastes better makes up for that decently well. Is not more than 10% of my music listening however.
Do you know of any guides written to extract mundane utility from the current crop of LLMs? It seems most knowledge (like all things) is left implicit in Tweets and conversations in comment sections.
> You can of course Read the Whole Thing, and I encourage that if you have the time and interest, but these posts are long, so they also designed to also let you pick the sections that you find most interesting. Each week, I pick the sections I feel are the most important, and put them in bold in the table of contents.
"so they also designed"
they're*
What a slow week.
by the way, peek AI -> peak AI
> Yes, there are several things here of potential interest if they are thoughtful. But, I mean, ow, my eyes. I would like to think we could all agree that human extinction is bad, that increasing the probability of it is bad, and that lowering that probability or delaying when it happens is good. And yet, here we are?
It seems like you think the Pettigrew paper argues for human extinction? It actually argues against longtermism, by arguing that longtermism + risk aversion => we should hasten human extinction. But of course another response is to reject (the paper's version of) risk aversion. Cf. https://rychappell.substack.com/p/risk-aversion-as-a-humility-heuristic
On the whole Noah Smith discourse, if humans could have comparative advantage in the world of AGI, it begs the question of why horses don’t currently have a comparative advantage in the age of the car.
It may be because horses are already maximally employed in a niche that does not directly compete with a car, but I suspect it also has to do with the fact that there is an inherit friction associated with deploying an inefficient slow cog in a machine that’s used to operating at 10000x speed.
Gemini's cybersecurity test suite is already better than OpenAI's preparedness framework. Turning popular CTFs into evals and publishing your results seems like the least and most obvious thing you could do if you just wanted to try at all.
Is that enough?
Camera pans to the appropriately named "DarkGemini ... a powerful new GenAI chatbot now being sold on the dark web for a $45 monthly subscription. It can generate a reverse shell, build malware, or even locate people based on an image."
https://twitter.com/ItakGol/status/1772212211370406214
Good News Everyone! I'm lowering my odds of ASI taking over the internet. Centaurs will assuredly melt the internet into unusable slag well before that's of any concern!
I find this mega thread impossible to consume due to its sheer content volume. Does anyone have a recommendation on how to read something like this productively without spending 4 hours going line by line?
>> All we need is ONE regulation – a limit on the fraction of energy that can go to data centers.
> That still requires driving up the cost of any compute useful for inference by orders of magnitude from where it is today, and keeping it there by global fiat.
I'm not understanding how energy restrictions became compute restrictions here. What am I missing? As I see it ASI is using every regulated joule to successfully discover new compute strategies as well as new energy sources, while leaving an increasingly shrinking percentage of that energy budget to humans. Energy and compute still gets cheaper, and comparative advantage is still in play, humans doing everything less important than designing new compute architectures and fusion power plants.
“If your rules technically say that Killmonger gets to be in charge, and you know he is going to throw out all the rules and become a bloodthirsty warmongering dictator the second he gains power, what do you do?”
Or, to quote ultimate Killmonger Anton Chigurh: “If the rule you followed brought you to this, of what use was the rule?”
https://youtu.be/p93w7MpbZRw?si=s1uvHlHOX9rWNvVl
“It’s not that scenario number two makes zero sense. I presume the argument is ‘well, if we can’t understand how things work, the AI won’t understand how anything works either?’ So… that makes everything fine, somehow? What a dim hope.”
Surely the actual argument is more, “If we can’t understand how things work, we won’t be able to replicate the full range of things undergirding human intelligence, so maybe our creations will be not so omnipotent after all”?