"We must ensure that we are shaping and commanding the more capable A.I.s of the coming years, rather than letting them shape and command us. The critical first step in making that possible is to enact legislation requiring visibility into the training of any new A.I. model that potentially approaches or exceeds the state of the art. Mandatory oversight of cutting-edge models will not solve the underlying problem, but it will be necessary in order to make finding a future solution possible."
Bravo, well done getting this to be the take-away point of your NYT op-ed.
You did well, and without it sounding like you were trying to shoehorn alignment / not-kill-everyone into a different discussion. Good natural common ground.
"Technically I would say: Powerful entities generally caring about X tends not to be a stable equilibrium, even if it is stable ‘on reflection’ within a given entity. It will only hold if caring more about X provides a competitive advantage against other similarly powerful entities, or if there can never be a variation in X-caring levels between such entities that arises other than through reflection, and also reflection never causes reductions in X-caring despite this being competitively advantageous."
I will note one place I see an avenue to victory (though likely still a worse approach): If we successfully get the first system capable of an intelligence explosion to care enough to prioritize continuing to care as it self-improves, the head start may well let it assume full control of the future even if later competitors' approaches would've been more efficient.
(Not sure if I should comment here or at your Wordpress site, so I did both)
I disagree with Megan McCardle who said: "Our ability to evaluate student learning through out-of-class writing projects is coming to an end."
I've posted this before, but it's very easy to prevent students from using ChatGpt to write their papers. All the teacher has to do is force students to use Microsoft Word with parameters set such that Word both autosaves the paper every minute and also keeps a large version history. Then, the teacher requires the student to not only turn in the final document, but also the entire version history thus providing documentation of the writing process. This solves the cheating problem because when writing their own paper, students will add, delete and move sections around as the document grows in an organic way. When using ChatGPT, the paper simply appears out of whole cloth in a finished form. Even if the student takes the trouble to type in the document, the version history will not look anything like it would have had the student written the document themselves. There is no easy way at this time to use ChatGPT simulate the writing process that a human would use.
The teacher would not even have to look at the version history unless they suspected cheating. And just the threat of being able to be caught would deter most students.
Think about how you write a paper. It doesn't come out perfect on the first go around. You add sentences, delete them, and move them around. You do the same thing with paragraphs. You also backspace a lot to correct typos, etc. I challenge anyone to start from nothing, and a generate a bunch of realistic edits every minute to wind up with the final ChatGPT result.
I guarantee that ChatGPT (or some future LLM) could be instructed to make incremental edits that you copy and paste back and forth with Word to game the system you’re describing. Not that we shouldn’t try (I think your idea is really smart), but the effort will likely continue to be sisyphean.
At some point in the future you are probably right so this might just be a stopgap solution. But is should at least buy the teachers at least a few years since what you suggest sounds much harder for the LLM to do than simply writing a paper.
> Tyler Cowen reviews AI music generator Suno. It is technically impressive. That does not mean one would want to listen to the results.
I've found it to be pretty good, but probably a good chunk of that is me disliking the (what feels like) 90% of music being about love/sadness. Getting Suno to generate music about transhumanism, space operas, or even medieval inquisitors is nice. There simply isn't much music like that, especially in genres other than filk. (Like the released Secular Solstice songs are good, but I like harder music)
Definitely not as good overall, as you say voices should be better, but fitting my tastes better makes up for that decently well. Is not more than 10% of my music listening however.
I've used it more for instrumentals. I've found the opposite of Tyler, the melodies are often catchy, to the point I'll sometimes have them running through my head the next day. The composition is part where it can't draw fingers. It will go into this sweeping transition... then just forget why and return to the main section one measure later.
It needs some audio version of inpainting but of course as noted this is the worst they will be.
Also it is really bad at distinguishing genres that changed over time. Like if you ask for first wave ska it gets confused and you get distinctive later big horns fast tempo stuff. Any request for any jazz might come out as big band. As if it is staring at you blankly with "you said jazz and this is jazz why are you upset." That is just a risk you must accept for now.
Do you know of any guides written to extract mundane utility from the current crop of LLMs? It seems most knowledge (like all things) is left implicit in Tweets and conversations in comment sections.
There are a bunch of people who write posts like that, or generally provide advice, but no general definitive guide I am aware of. I think a lot of that is because it is different for each person based on their skills and needs.
Is double-checking the output from LLMs something that you (or your son) find yourself frequently doing when trying to learn something in a completely new area, or have you only used it for domains where you already have some background knowledge?
> You can of course Read the Whole Thing, and I encourage that if you have the time and interest, but these posts are long, so they also designed to also let you pick the sections that you find most interesting. Each week, I pick the sections I feel are the most important, and put them in bold in the table of contents.
> Yes, there are several things here of potential interest if they are thoughtful. But, I mean, ow, my eyes. I would like to think we could all agree that human extinction is bad, that increasing the probability of it is bad, and that lowering that probability or delaying when it happens is good. And yet, here we are?
It seems like you think the Pettigrew paper argues for human extinction? It actually argues against longtermism, by arguing that longtermism + risk aversion => we should hasten human extinction. But of course another response is to reject (the paper's version of) risk aversion. Cf. https://rychappell.substack.com/p/risk-aversion-as-a-humility-heuristic
On the whole Noah Smith discourse, if humans could have comparative advantage in the world of AGI, it begs the question of why horses don’t currently have a comparative advantage in the age of the car.
It may be because horses are already maximally employed in a niche that does not directly compete with a car, but I suspect it also has to do with the fact that there is an inherit friction associated with deploying an inefficient slow cog in a machine that’s used to operating at 10000x speed.
Noah Smith discusses the horse example in his post on the subject. His argument is that horses competed with cars for scarce resources like land, and time and effort to care for them. The scarce resources were soon better spent on cars than horses. With AI and humans, the relevant shared scarce resource is energy. https://www.noahpinion.blog/p/plentiful-high-paying-jobs-in-the
Gemini's cybersecurity test suite is already better than OpenAI's preparedness framework. Turning popular CTFs into evals and publishing your results seems like the least and most obvious thing you could do if you just wanted to try at all.
Is that enough?
Camera pans to the appropriately named "DarkGemini ... a powerful new GenAI chatbot now being sold on the dark web for a $45 monthly subscription. It can generate a reverse shell, build malware, or even locate people based on an image."
Good News Everyone! I'm lowering my odds of ASI taking over the internet. Centaurs will assuredly melt the internet into unusable slag well before that's of any concern!
I find this mega thread impossible to consume due to its sheer content volume. Does anyone have a recommendation on how to read something like this productively without spending 4 hours going line by line?
>> All we need is ONE regulation – a limit on the fraction of energy that can go to data centers.
> That still requires driving up the cost of any compute useful for inference by orders of magnitude from where it is today, and keeping it there by global fiat.
I'm not understanding how energy restrictions became compute restrictions here. What am I missing? As I see it ASI is using every regulated joule to successfully discover new compute strategies as well as new energy sources, while leaving an increasingly shrinking percentage of that energy budget to humans. Energy and compute still gets cheaper, and comparative advantage is still in play, humans doing everything less important than designing new compute architectures and fusion power plants.
“If your rules technically say that Killmonger gets to be in charge, and you know he is going to throw out all the rules and become a bloodthirsty warmongering dictator the second he gains power, what do you do?”
Or, to quote ultimate Killmonger Anton Chigurh: “If the rule you followed brought you to this, of what use was the rule?”
“It’s not that scenario number two makes zero sense. I presume the argument is ‘well, if we can’t understand how things work, the AI won’t understand how anything works either?’ So… that makes everything fine, somehow? What a dim hope.”
Surely the actual argument is more, “If we can’t understand how things work, we won’t be able to replicate the full range of things undergirding human intelligence, so maybe our creations will be not so omnipotent after all”?
"We must ensure that we are shaping and commanding the more capable A.I.s of the coming years, rather than letting them shape and command us. The critical first step in making that possible is to enact legislation requiring visibility into the training of any new A.I. model that potentially approaches or exceeds the state of the art. Mandatory oversight of cutting-edge models will not solve the underlying problem, but it will be necessary in order to make finding a future solution possible."
Bravo, well done getting this to be the take-away point of your NYT op-ed.
Thanks! I did fight for that (and to get even more, but I think this was the right compromise).
You did well, and without it sounding like you were trying to shoehorn alignment / not-kill-everyone into a different discussion. Good natural common ground.
What are "privacy breeches"?
What you wear when you want to go to the bathroom without anyone else watching.
Almost any pair of pants that cover your unmentionables.
"Technically I would say: Powerful entities generally caring about X tends not to be a stable equilibrium, even if it is stable ‘on reflection’ within a given entity. It will only hold if caring more about X provides a competitive advantage against other similarly powerful entities, or if there can never be a variation in X-caring levels between such entities that arises other than through reflection, and also reflection never causes reductions in X-caring despite this being competitively advantageous."
I will note one place I see an avenue to victory (though likely still a worse approach): If we successfully get the first system capable of an intelligence explosion to care enough to prioritize continuing to care as it self-improves, the head start may well let it assume full control of the future even if later competitors' approaches would've been more efficient.
(Not sure if I should comment here or at your Wordpress site, so I did both)
I disagree with Megan McCardle who said: "Our ability to evaluate student learning through out-of-class writing projects is coming to an end."
I've posted this before, but it's very easy to prevent students from using ChatGpt to write their papers. All the teacher has to do is force students to use Microsoft Word with parameters set such that Word both autosaves the paper every minute and also keeps a large version history. Then, the teacher requires the student to not only turn in the final document, but also the entire version history thus providing documentation of the writing process. This solves the cheating problem because when writing their own paper, students will add, delete and move sections around as the document grows in an organic way. When using ChatGPT, the paper simply appears out of whole cloth in a finished form. Even if the student takes the trouble to type in the document, the version history will not look anything like it would have had the student written the document themselves. There is no easy way at this time to use ChatGPT simulate the writing process that a human would use.
The teacher would not even have to look at the version history unless they suspected cheating. And just the threat of being able to be caught would deter most students.
Couldn’t the student just copy and paste between word and ChatGPT a whole bunch of times?
Think about how you write a paper. It doesn't come out perfect on the first go around. You add sentences, delete them, and move them around. You do the same thing with paragraphs. You also backspace a lot to correct typos, etc. I challenge anyone to start from nothing, and a generate a bunch of realistic edits every minute to wind up with the final ChatGPT result.
I guarantee that ChatGPT (or some future LLM) could be instructed to make incremental edits that you copy and paste back and forth with Word to game the system you’re describing. Not that we shouldn’t try (I think your idea is really smart), but the effort will likely continue to be sisyphean.
At some point in the future you are probably right so this might just be a stopgap solution. But is should at least buy the teachers at least a few years since what you suggest sounds much harder for the LLM to do than simply writing a paper.
That sounds almost as hard as writing a paper.
Comments are welcome at either but this is the primary forum.
> Tyler Cowen reviews AI music generator Suno. It is technically impressive. That does not mean one would want to listen to the results.
I've found it to be pretty good, but probably a good chunk of that is me disliking the (what feels like) 90% of music being about love/sadness. Getting Suno to generate music about transhumanism, space operas, or even medieval inquisitors is nice. There simply isn't much music like that, especially in genres other than filk. (Like the released Secular Solstice songs are good, but I like harder music)
Definitely not as good overall, as you say voices should be better, but fitting my tastes better makes up for that decently well. Is not more than 10% of my music listening however.
I've used it more for instrumentals. I've found the opposite of Tyler, the melodies are often catchy, to the point I'll sometimes have them running through my head the next day. The composition is part where it can't draw fingers. It will go into this sweeping transition... then just forget why and return to the main section one measure later.
It needs some audio version of inpainting but of course as noted this is the worst they will be.
Also it is really bad at distinguishing genres that changed over time. Like if you ask for first wave ska it gets confused and you get distinctive later big horns fast tempo stuff. Any request for any jazz might come out as big band. As if it is staring at you blankly with "you said jazz and this is jazz why are you upset." That is just a risk you must accept for now.
Do you know of any guides written to extract mundane utility from the current crop of LLMs? It seems most knowledge (like all things) is left implicit in Tweets and conversations in comment sections.
There are a bunch of people who write posts like that, or generally provide advice, but no general definitive guide I am aware of. I think a lot of that is because it is different for each person based on their skills and needs.
Unfortunate, but understandable.
Is double-checking the output from LLMs something that you (or your son) find yourself frequently doing when trying to learn something in a completely new area, or have you only used it for domains where you already have some background knowledge?
Ethan Mollick's substack and book are pretty detailed and have a lot of great examples of using them in different ways that drive utility.
Thank you! I may report back on my thoughts.
> You can of course Read the Whole Thing, and I encourage that if you have the time and interest, but these posts are long, so they also designed to also let you pick the sections that you find most interesting. Each week, I pick the sections I feel are the most important, and put them in bold in the table of contents.
"so they also designed"
they're*
What a slow week.
by the way, peek AI -> peak AI
> Yes, there are several things here of potential interest if they are thoughtful. But, I mean, ow, my eyes. I would like to think we could all agree that human extinction is bad, that increasing the probability of it is bad, and that lowering that probability or delaying when it happens is good. And yet, here we are?
It seems like you think the Pettigrew paper argues for human extinction? It actually argues against longtermism, by arguing that longtermism + risk aversion => we should hasten human extinction. But of course another response is to reject (the paper's version of) risk aversion. Cf. https://rychappell.substack.com/p/risk-aversion-as-a-humility-heuristic
On the whole Noah Smith discourse, if humans could have comparative advantage in the world of AGI, it begs the question of why horses don’t currently have a comparative advantage in the age of the car.
It may be because horses are already maximally employed in a niche that does not directly compete with a car, but I suspect it also has to do with the fact that there is an inherit friction associated with deploying an inefficient slow cog in a machine that’s used to operating at 10000x speed.
Noah Smith discusses the horse example in his post on the subject. His argument is that horses competed with cars for scarce resources like land, and time and effort to care for them. The scarce resources were soon better spent on cars than horses. With AI and humans, the relevant shared scarce resource is energy. https://www.noahpinion.blog/p/plentiful-high-paying-jobs-in-the
Gemini's cybersecurity test suite is already better than OpenAI's preparedness framework. Turning popular CTFs into evals and publishing your results seems like the least and most obvious thing you could do if you just wanted to try at all.
Is that enough?
Camera pans to the appropriately named "DarkGemini ... a powerful new GenAI chatbot now being sold on the dark web for a $45 monthly subscription. It can generate a reverse shell, build malware, or even locate people based on an image."
https://twitter.com/ItakGol/status/1772212211370406214
Good News Everyone! I'm lowering my odds of ASI taking over the internet. Centaurs will assuredly melt the internet into unusable slag well before that's of any concern!
I find this mega thread impossible to consume due to its sheer content volume. Does anyone have a recommendation on how to read something like this productively without spending 4 hours going line by line?
>> All we need is ONE regulation – a limit on the fraction of energy that can go to data centers.
> That still requires driving up the cost of any compute useful for inference by orders of magnitude from where it is today, and keeping it there by global fiat.
I'm not understanding how energy restrictions became compute restrictions here. What am I missing? As I see it ASI is using every regulated joule to successfully discover new compute strategies as well as new energy sources, while leaving an increasingly shrinking percentage of that energy budget to humans. Energy and compute still gets cheaper, and comparative advantage is still in play, humans doing everything less important than designing new compute architectures and fusion power plants.
“If your rules technically say that Killmonger gets to be in charge, and you know he is going to throw out all the rules and become a bloodthirsty warmongering dictator the second he gains power, what do you do?”
Or, to quote ultimate Killmonger Anton Chigurh: “If the rule you followed brought you to this, of what use was the rule?”
https://youtu.be/p93w7MpbZRw?si=s1uvHlHOX9rWNvVl
“It’s not that scenario number two makes zero sense. I presume the argument is ‘well, if we can’t understand how things work, the AI won’t understand how anything works either?’ So… that makes everything fine, somehow? What a dim hope.”
Surely the actual argument is more, “If we can’t understand how things work, we won’t be able to replicate the full range of things undergirding human intelligence, so maybe our creations will be not so omnipotent after all”?