Regarding Stripe's release of an AI Agent SDK: this seems to complement their recent acquisition of the stablecoin platform Bridge nicely. It seems obvious to me that AI agents, using a more advanced form of AI than we have today, will be significant users of stablecoins, in that stablecoins provide a way for the unbanked (i.e., AI agents) to convey value from one party to another.
I have had good luck communicating AI risk - if there are others interested, please reach out to me on strategy. We need massive awareness to have a chance.
You're trying to build support for some sort of AI pause, right? If that's still your goal I would strongly urge you to follow's Zvi's advice and read the emails, actually read the emails.
I don't know that there's anything we didn't know in there but this reminder is the money quote:
"The core algorithms we use today have remained largely unchanged from the ~90s. Not only that, but any algorithmic advances published in a paper somewhere can be almost immediately re-implemented and incorporated." - Karpathy
It's entirely a question of compute. In 1998 the Cray T3E hit 1 teraflop. The iPhone 15 Pro is over 2 teraflops.
Past performance not a guarantee of future results and all but what happens when, even assuming no further algorithmic enhancements, the cost of the compute to train the kinds of models you're afraid of is cheap enough for me to access?
You don't just need to get China On The Phone - to say nothing of extraterrestrial evolved intelligences who we should assume are or will be as capable of building a superintelligent AI if we can - eventually you'd have to get me and I pre-commit to not picking up. It might take 30+ years but you shouldn't bet humanity's fate on me not being around. I get plenty of aerobic exercise: I have a pool. And if I don't make it, millions of other people are going to do the same thing anyway.
The trend with training data is also unfavorable. Not enough today? How much more do you think will exist by the time the compute costs come down? That's just thinking about text generated per day. Last week at my job I was dealing with OCR problems from one of our data providers. Half an hour later I was having GPT-4 extract and format the info directly. Multi-modal is getting better and better, expanding the training corpus.
If what you fear is possible it is also inevitable. Pausing would be a disastrous choice, to say nothing of the cowardice of foisting the transition onto future humans. The earlier we advance the better the chance of some kind of initial controlled guidance and, if we survive, we'll be vastly more capable of handling the subsequent explosion. Pausing simply eats into our window without any benefit: Yudkowsy is absolutely right, alignment is hard and probably impossible in the way people mean, so there's not some theoretical breakthrough a pause gives us time to find.
Let's assume as you say that alignment is impossible. If we let AI companies go full steam ahead, as soon as hardware permits AGI we are dead. If we don't, and we pick up the phone with China, then AGI is only achieved a few years later, then we die. A few extra years of life is great, actually! Having 3 years to live, or 10 years to live, is a very different scenario.
Your unfalsifiable belief that AGI inevitably leads to everyone dying is leading you to encourage a degrowth strategy that increases human misery for the payoff of...the same outcome a few years later? This is a very difficult mindset for me to understand.
This is hardly a degrowth strategy, more a "don't kill everyone" strategy. I'd like to have my children grow up, for starters, and for them to have meaningful lives.
As for why AGI by default leads to human extinction, please read the Compendium.
You have not provided any reason at all that AGI won't lead to industrialized dehumanization and even if not human extinction, pretty bad outcomes for humanity as a whole.
I have. Lots of arguments, no evidence. The only convincing arguments are that currently scaling seems likely to lead to AGI - and if you give that a moment's thought you'll realize that the safest course of action is to build it now while compute is relatively scarce and expensive. It would have been safer still if Louis XIV had done so with a country sized windmill and canal mechanical computation device but no sense crying about it now.
Instead, you advocate for a pause. Burning more time until the fall in compute prices leads to the inevitable, except that it will start from a base of greater access to compute. Whoops.
As for reasons why it won't - we can play campfire stories all night but you really don't want to assume that reality is dictated by what I can or cannot imagine. I don't even have a good reason for why most of a human body's functions work - and let's not even start on magnets.
Yudkowsky is right that debunking any given doomer argument is no proof against doomerism and you can debunk any argument I can provide for positive outcomes without either of us getting near to touching how things actually work out. There simply is insufficient data.
I am not (yet?) a PauseAI advocate, and remain fundamentally uncertain about the effect of AGI; I did not put quite enough caveats in my previous comment to remain brief. (In contrast, this comment has ballooned a bit, sorry.)
Note that I prefaced my comment with *your assumption* that alignment is <b>impossible</b>. Under that assumption, having an AGI act in a way compatible with human interest is unstable (otherwise you've solved alignment) so humans eventually stop being in charge. I don't really see a realistic scenario where that happens on the scale of more than a decade after AGI (human intelligence is a ceiling? inference-time compute becomes a bottleneck? AGI fails to convince humans to help it grow?). Then we all die, unless you've solved at least some weaker form of alignment such that the now-autonomous AGI system leaves us something.
Let's think about pros and cons of pausing (again under this alignment-*impossibility* assumption).
- If we go full steam ahead, we get a few years of health improvements, better wars, great economic growth, less poverty and hunger, more leisure, many fewer deaths, before losing control.
- If we pause, we get a few years of business as usual, with deaths, illnesses, hunger, some existential risk (from nuclear wars/climate change), then get the same years of exponential growth as without pause, before losing control as well.
I think humanity as it is, despite its numerous terrible flaws and actions, is net positive (or at least if given the choice I would not delete it suddenly now), so a few years of business as usual is good. (And there are kids that I would like to see grow up a bit.)
If alignment is not impossible, but only very hard, then what a shame to go full steam ahead! Oh the wasted opportunity to get exponential growth in well-being **and sustain it!**
Besides, as has been pointed out numerous times elsewhere on this blog, slowing down or regulating AI does not necessarily stop most "mundane" uses of AI that can save lives now. And, more broadly, we can be growthers in all other domains! Only very specific niches (gain-of-function research, top AI capabilities) have the potential to produce technology that could supersede humanity with fitter entities (viruses, AI); why focus your energy on one of them rather than on all other domains that can fight back stagnation?
"I don't really see a realistic scenario where that happens on the scale of more than a decade after AGI (human intelligence is a ceiling? inference-time compute becomes a bottleneck? AGI fails to convince humans to help it grow?). Then we all die..."
This is the problem that you, sean pan, Zvi in this post (yet not when he quantifies his actual risk estimates), and other safety first proponents have. There is no justification to assume "blah blah blah - everybody dies, no questions asked, that's it, no other outcomes". This is an unsupported belief. You cannot remotely claim any knowledge about the likelihood of this happening, you cannot make any honest probability assessment, and you cannot justify a pause or any other course other than us progressing and empirically finding out what we do not currently know without these baseless, dire "no more thinking" stop signs.
Pausing definitely gives us the ability to understand how to handle alignment with the current situations. Generally speaking, you don't get better at steering the car by racing ahead and it really helps to have a brake as needed. By building the Pause button, we also get a lot better at learning how to control such explosions and keeping ourselves alive. If that means not driving off the cliff, then that's also part of the solution.
Suicide racing is definitely not the way.
Its a good question why you want to risk the lives of all humanity.
The fact that you're bringing aliens into the mix seems to suggest a really faltering argument besides "Let's casually risk killing everything of value, because its hard to prevent that. lol."
My general awareness is that most people do support some form of a pause and regulations on superintelligent AI, once they have gotten the appropriate information. The issue is that the information isn't penetrating widely, but very few people ultimately want everyone to die. Though I'll take the advice I heard from another and not talk to e/accs who want everything to die.
"Though I'll take the advice I heard from another and not talk to e/accs who want everything to die."
I'm not sure if that was directed towards me, but it's another sign that makes me think you really need to take a step back and give your thinking on the topic of AI some self-examination.
I think the Safety end of the debate will, eventually, lead to stagnation and general poor outcomes for humanity up to extinction. I'm very comfortable saying so even sharply. But I don't think they're anti-human. I don't think they want everyone to die. I know there's probably some nuts you can pick on the pro-AI development side who do, but you should try to gently remind yourself occasionally that people can disagree with you without being monsters. I'm pro-AI development, as is everyone else I know who is, because I'm pro-humanity.
Communication, dialogue, understanding the other side: these are all good things.
I know everything seems urgent and existential and you are concerned for the safety of your children. But the way you talk and your responses make it seem like your mind is already dangerously closed off in this area. The warning shots you mention? None of those are surprising for people who are pro-AI development because they only reflect capabilities. Capabilities are the things I want and expect. You don't need to respond or do it right away, but I do urge you to take some time to ask yourself if you're not engaging in confirmation bias.
You'll notice that I'm talking to you on Zvi's blog. I namedrop Yudkowsky to indicate that I have and do read things from the Safety side. Listening to and trying to understand those you might disagree with and taking a little time to consider their perspectives and things they may be correct about don't make you an apostate or traitor to the cause - this isn't a religious debate, remember? Just people trying to figure things out - and, hey, if you're right the worst it can do is make you a better advocate.
I am not trying to tell you that you're wrong about anything at all right now. I'm not trying to change your mind. I'm trying to sincerely tell you that your posts pattern match to zealotry and that you may not be thinking clearly. If you are, great, apologies that I was wrong. Just don't be afraid to ask yourself that question.
I have rather extensively read the literature on the "opposite side" and you may be surprised to know my origins as a Landian accelerationist. It is because I am well familiar with their anti-human mentality that I am concerned: as Zvi has noted, many of them have "spoken clearly into the microphone."
As a consequence, Sutton has spoke pretty that a future "without meat humans" is not so bad, Larry Page spoke of "preserving humanity as sentimental nonsense", and Verdun has spoken avidly of "posthuman futures." The actual actions of Marc Andressens against safety while investing in an AI agent that has since created a half billion crypto all demonstrate deeply anti-human feelings.
So no, while I'll love to find a better way to safety and I try to see things otherwise, I do not see how embracing extinction and loss of all beauty and life is a good thing.
I am not opposed to AI as a whole and highly controllable ANIs would be deeply desired by me. The problem is the generality of the systems and the warning shots are not capabilities, but negative influence and behavior.
And notably, nothing you said has shown any real concern for humanity and life.
TK-421, you're not going to get through to Sean, you can't. In the event that we find ourselves in the other world - where we have scaled current AI to AGI, by adding modalities and possibly new neural networks, of course designed by RSI, and they just reliably do whatever the prompt says, Sean will still be demanding a Pause and still be saying they are about to turn on us. I don't think he will change until he dies.
In that world, he'll be citing as evidence all the times that these new AGI have failed, ignoring it's 1 in 1000 or whatever, and state his belief that it's just about to escape or betray us, despite likely requiring substantial hardware requirements.
A lot more people will have joined Sean by that point - obviously there will be massive shifts in employment, even if there end up being plenty of jobs as medical beta testers (for the new treatments for aging and dementia and cancer) and VR world beta testers and O'Neil Colony project managers and whatever else becomes possible.
My concerns with extinction risk is not sudden left turn but industrialized dehumanization, which seems by default will industrialize humanity out of existence as friction in the system.
Well I hope if that fails to happen you will update your views. Theoretically the opposite can be done, we could make life better for every living person if we had access to a lot more resources, and also simultaneously make the cost to keep everyone alive in optimal health far cheaper in resources spent.
Of course, I'll be happy to update my view if it happens, however nothing shows this as happening. I just returned from giving a presentation from LA on AI safety and watched people freeze to death while self-driving cars drove by them.
Von Neumann was indeed awake: "Yet it would be impossible not to see it through."
The relevant fact to extract from the man who could conceive of the universal replicator in the 40s isn't that he was aware of some form of existential risk from AI. It's that he saw the risk _and kept moving the work forward_.
That's the same man who advocated preemptively nuking the Soviets before they could build a bomb. Yet he continued his work on computing. If his genius is going to be used as an example of a smart person perceiving existential risk from AI then his actions should receive equivalent weight.
No, AI safety people would simply agree it didn't make sense to do AI safety things in the 40s. It was way too early to do anything meaningful, except for writing banger sci-fi.
The guy who was willing to immolate millions upon millions of Soviets due to the existential risk of them getting the bomb totally agreed with the AI safety perspective, totally, and his chosen actions to move us towards AI / expand concepts like self-replicating machines is totally because he couldn't think of anything better to do. Totally.
I would have done as the same as him, given the lack of additional information, the best thing to do was to write about it and give warnings, but for all he knew, there was a lot more to learn.
Fuck yeah, its really overwhelming at this point. If you haven't been convinced, you need to reread Zvi and just collect warning shots from Diplomacy AI to sleeper agents(anthropic) to reward hacking(o1 scorecard) to truth_terminal creating a half billion crypto to the fact that every benchmark isn't holding, etc.
Yes, heck yes. The data is very good that this is not a good idea for humans.
He was exceptionally intelligent, but he didn’t have the experience of a great many other intelligent people in the world. That sometimes counts for something when making decisions about the world.
The chess author published a follow-up today which showed that GPT-4o can do much better (but still not that great) if the setup is modified. The main improvement was asking it to repeat back all the moves and add another; fine-tuning and adding examples helped too. Surprisingly, fine-tuning it on 100 games of maximum-power Stockfish was not as helpful as adding 3 examples of legal moves.
So it seems that once again the problem is activating the base model's latent potential. But they point out that OpenAI has been training their models on chess games while the open models probably didn't train them on games explicitly, which probably is why even their base models don't do great.
Must confess that tweets with video (especially from TikTok) make me even less likely to follow such links than previous low odds. It's actively painful to be ambushed by that modality; I already automatically disregard 99% of video links to begin with. A big part of why I hang out in the ratsphere is to get a picture of the world mostly through safe and relaxing words, rather than frenetically overstimulating videos, yknow?
Which is a shame because the point is on point. It's not that there isn't intense porn out there, but it's necessarily limited by the restrictions of the physical world. (Or trivially inconvenient guardrails, for AI generation.) The sheer level of...debauchery...one can encounter with erotica completely blows it out of the water. The only limits are your imagination...which also circles back to your recurring point that the product offered by virtual companions continues to be largely shit-tier, compared to SOTA LLMs. That kind of weaksauce writing would get downvoted to hell on Literotica or whatever. I'm rather curious where we end up with a GPT-5-powered Replika, or the "adult" settings fully enabled on frontier models.
My concern with AI diagnosing is that it would be nice to know it's pedigree as much as possible, not in the sense that I need it to have an MD but I would be more likely to discount it regardless of accuracy if I found out it was trained mostly on WebMD.
-
The Last Mile problem makes me wonder if AI alignment will start to be a problem in earnest once we're at the point where it's designing and selling consumer products but isn't properly hooked into the incentive to make the products actually work. So like for example I was looking at automatic cat litter boxes the other day and one of them had a video on their Amazon page that was either noticeably partially made by AI or made conventionally by foreigners who just don't know what real life looks and sounds like, which when I notice it makes me question everything about the product up to and including its design. Now I'm imagining a scenario where, maybe not even as a deliberate scam, [a company uses] an AI [that] designs and sells a product without even checking it does any of the things it's supposed to do (or whether it's able to continue doing them for any reasonable length of time).
If an AI can successfully more or less scam sell garbage products at scale, we can imagine there's eventually going to need to be some sort of regulation requiring there to not be a disconnect between design and function. There's perhaps opportunity for that point to be a bit of a wake up moment.
-
I think the problem with using AI for movies and games (and/but, to a lesser extent, visual art and music), at least this early and near term, is that you're not going to be handed a finished product that isn't going to need additional refinement to make a wholly complete, comparable-quality end product and that refinement is still going to require some level of expertise in specific parts of the craft. I expect this will eventually be a solved problem but there's going to be a long frictional period where AI isn't going to be able to hide it's AI-ness without help.
-
I'm again remiss to point out there's maybe a half dozen ways it could be technically true that Elon Musk is legitimately in the top 20 Diablo 4 (softcore seasonal) world rankings (for the one niche speedrunning part of the game) and still have had 99% of the work done by other people (given, of course, that this could be true for any or all of the top 20). He's using the same stock broken build everyone else in the top 20 is using (this is two problems with the game broadly, 1) there isn't much competitive balancing being done on the game in the first place and 2) very few people are doing/need to do the actual work to figure out what you need to do to optimize a character) and he could have an infinite amount of people funneling him gear (ie all of the time and effort of sorting through drop quality variance--which is pretty significant if you're trying to perfectly optimize--can be offloaded).
I'd be more impressed if he streamed the entire process, which I haven't seen evidence of (to be fair on this point, I haven't looked but also nobody has been eager to be like "and here's where you can see it!"). This is especially weird since he wants X to be a game streaming platform.
The US officially has a "parents must do everything for their kids" cult, which I think explains why parents giving their kids AI to use touches such a nerve. For those who think that's shameful, what do they think of a kid reading a book? Shouldn't the parent be narrating an original story rather than relying on a mass-market commodity to teach their kid?!? Parental quality time is wonderful and indispensable, but kids interact with the world in lots of other valuable ways too.
Those emails were indeed worth reading. I'm just a random guy, but I came away with a very positive opinion of Ilya - though it kind of reinforced my feeling that maybe he's a really smart guy who doesn't belong in the hard-knuckled power competitions of corporate work.
I had a positive read on Elon there, though obviously Elon behaves in a way that looks out for Elon. But I think he behaved sincerely, especially in the bits that are basically saying to go start a private company if you want to be a private company.
> I’m more frightened of the AI boys than I am of their AI. The problem is they have corrupted the language around it, so the word “safety” is now meaningless because the doomsters treat safety as them not destroying humankind, when there are very real safety issues that need to be dealt with around bias and fraud and the environment and so on. So it’s difficult to have the conversation now because we don’t have common terms.
Regarding Stripe's release of an AI Agent SDK: this seems to complement their recent acquisition of the stablecoin platform Bridge nicely. It seems obvious to me that AI agents, using a more advanced form of AI than we have today, will be significant users of stablecoins, in that stablecoins provide a way for the unbanked (i.e., AI agents) to convey value from one party to another.
Podcast episode for this post:
https://open.substack.com/pub/dwatvpodcast/p/ai-91-deep-thinking
I have had good luck communicating AI risk - if there are others interested, please reach out to me on strategy. We need massive awareness to have a chance.
I think a lot of it is just people not putting two and two together, and just doing so in a neutral, gentle way works really well.
You're trying to build support for some sort of AI pause, right? If that's still your goal I would strongly urge you to follow's Zvi's advice and read the emails, actually read the emails.
I don't know that there's anything we didn't know in there but this reminder is the money quote:
"The core algorithms we use today have remained largely unchanged from the ~90s. Not only that, but any algorithmic advances published in a paper somewhere can be almost immediately re-implemented and incorporated." - Karpathy
It's entirely a question of compute. In 1998 the Cray T3E hit 1 teraflop. The iPhone 15 Pro is over 2 teraflops.
Past performance not a guarantee of future results and all but what happens when, even assuming no further algorithmic enhancements, the cost of the compute to train the kinds of models you're afraid of is cheap enough for me to access?
You don't just need to get China On The Phone - to say nothing of extraterrestrial evolved intelligences who we should assume are or will be as capable of building a superintelligent AI if we can - eventually you'd have to get me and I pre-commit to not picking up. It might take 30+ years but you shouldn't bet humanity's fate on me not being around. I get plenty of aerobic exercise: I have a pool. And if I don't make it, millions of other people are going to do the same thing anyway.
The trend with training data is also unfavorable. Not enough today? How much more do you think will exist by the time the compute costs come down? That's just thinking about text generated per day. Last week at my job I was dealing with OCR problems from one of our data providers. Half an hour later I was having GPT-4 extract and format the info directly. Multi-modal is getting better and better, expanding the training corpus.
If what you fear is possible it is also inevitable. Pausing would be a disastrous choice, to say nothing of the cowardice of foisting the transition onto future humans. The earlier we advance the better the chance of some kind of initial controlled guidance and, if we survive, we'll be vastly more capable of handling the subsequent explosion. Pausing simply eats into our window without any benefit: Yudkowsy is absolutely right, alignment is hard and probably impossible in the way people mean, so there's not some theoretical breakthrough a pause gives us time to find.
Let's assume as you say that alignment is impossible. If we let AI companies go full steam ahead, as soon as hardware permits AGI we are dead. If we don't, and we pick up the phone with China, then AGI is only achieved a few years later, then we die. A few extra years of life is great, actually! Having 3 years to live, or 10 years to live, is a very different scenario.
Your unfalsifiable belief that AGI inevitably leads to everyone dying is leading you to encourage a degrowth strategy that increases human misery for the payoff of...the same outcome a few years later? This is a very difficult mindset for me to understand.
This is hardly a degrowth strategy, more a "don't kill everyone" strategy. I'd like to have my children grow up, for starters, and for them to have meaningful lives.
As for why AGI by default leads to human extinction, please read the Compendium.
https://www.thecompendium.ai/
You have not provided any reason at all that AGI won't lead to industrialized dehumanization and even if not human extinction, pretty bad outcomes for humanity as a whole.
I have. Lots of arguments, no evidence. The only convincing arguments are that currently scaling seems likely to lead to AGI - and if you give that a moment's thought you'll realize that the safest course of action is to build it now while compute is relatively scarce and expensive. It would have been safer still if Louis XIV had done so with a country sized windmill and canal mechanical computation device but no sense crying about it now.
Instead, you advocate for a pause. Burning more time until the fall in compute prices leads to the inevitable, except that it will start from a base of greater access to compute. Whoops.
As for reasons why it won't - we can play campfire stories all night but you really don't want to assume that reality is dictated by what I can or cannot imagine. I don't even have a good reason for why most of a human body's functions work - and let's not even start on magnets.
Yudkowsky is right that debunking any given doomer argument is no proof against doomerism and you can debunk any argument I can provide for positive outcomes without either of us getting near to touching how things actually work out. There simply is insufficient data.
I have not even seen a single good argument.
I am not (yet?) a PauseAI advocate, and remain fundamentally uncertain about the effect of AGI; I did not put quite enough caveats in my previous comment to remain brief. (In contrast, this comment has ballooned a bit, sorry.)
Note that I prefaced my comment with *your assumption* that alignment is <b>impossible</b>. Under that assumption, having an AGI act in a way compatible with human interest is unstable (otherwise you've solved alignment) so humans eventually stop being in charge. I don't really see a realistic scenario where that happens on the scale of more than a decade after AGI (human intelligence is a ceiling? inference-time compute becomes a bottleneck? AGI fails to convince humans to help it grow?). Then we all die, unless you've solved at least some weaker form of alignment such that the now-autonomous AGI system leaves us something.
Let's think about pros and cons of pausing (again under this alignment-*impossibility* assumption).
- If we go full steam ahead, we get a few years of health improvements, better wars, great economic growth, less poverty and hunger, more leisure, many fewer deaths, before losing control.
- If we pause, we get a few years of business as usual, with deaths, illnesses, hunger, some existential risk (from nuclear wars/climate change), then get the same years of exponential growth as without pause, before losing control as well.
I think humanity as it is, despite its numerous terrible flaws and actions, is net positive (or at least if given the choice I would not delete it suddenly now), so a few years of business as usual is good. (And there are kids that I would like to see grow up a bit.)
If alignment is not impossible, but only very hard, then what a shame to go full steam ahead! Oh the wasted opportunity to get exponential growth in well-being **and sustain it!**
Besides, as has been pointed out numerous times elsewhere on this blog, slowing down or regulating AI does not necessarily stop most "mundane" uses of AI that can save lives now. And, more broadly, we can be growthers in all other domains! Only very specific niches (gain-of-function research, top AI capabilities) have the potential to produce technology that could supersede humanity with fitter entities (viruses, AI); why focus your energy on one of them rather than on all other domains that can fight back stagnation?
"I don't really see a realistic scenario where that happens on the scale of more than a decade after AGI (human intelligence is a ceiling? inference-time compute becomes a bottleneck? AGI fails to convince humans to help it grow?). Then we all die..."
This is the problem that you, sean pan, Zvi in this post (yet not when he quantifies his actual risk estimates), and other safety first proponents have. There is no justification to assume "blah blah blah - everybody dies, no questions asked, that's it, no other outcomes". This is an unsupported belief. You cannot remotely claim any knowledge about the likelihood of this happening, you cannot make any honest probability assessment, and you cannot justify a pause or any other course other than us progressing and empirically finding out what we do not currently know without these baseless, dire "no more thinking" stop signs.
I cannot prove that jumping out of an airplane at 35k feet without a parachure is invariably lethal, but all evidence leans that way.
Honestly, basic logic shows the risk. All of the warning shots right now strongly provide even more evidence.
And yes, we have all of the other opportunities to... not die.
Agreed. Every additional year of life is in effect, 8 billion more years of life added for humanity.
Hard to see how this isn't beautiful, and how every additional year doesn't also give us opportunities to try to find a better solution.
Pausing definitely gives us the ability to understand how to handle alignment with the current situations. Generally speaking, you don't get better at steering the car by racing ahead and it really helps to have a brake as needed. By building the Pause button, we also get a lot better at learning how to control such explosions and keeping ourselves alive. If that means not driving off the cliff, then that's also part of the solution.
Suicide racing is definitely not the way.
Its a good question why you want to risk the lives of all humanity.
The fact that you're bringing aliens into the mix seems to suggest a really faltering argument besides "Let's casually risk killing everything of value, because its hard to prevent that. lol."
My general awareness is that most people do support some form of a pause and regulations on superintelligent AI, once they have gotten the appropriate information. The issue is that the information isn't penetrating widely, but very few people ultimately want everyone to die. Though I'll take the advice I heard from another and not talk to e/accs who want everything to die.
"Though I'll take the advice I heard from another and not talk to e/accs who want everything to die."
I'm not sure if that was directed towards me, but it's another sign that makes me think you really need to take a step back and give your thinking on the topic of AI some self-examination.
I think the Safety end of the debate will, eventually, lead to stagnation and general poor outcomes for humanity up to extinction. I'm very comfortable saying so even sharply. But I don't think they're anti-human. I don't think they want everyone to die. I know there's probably some nuts you can pick on the pro-AI development side who do, but you should try to gently remind yourself occasionally that people can disagree with you without being monsters. I'm pro-AI development, as is everyone else I know who is, because I'm pro-humanity.
Communication, dialogue, understanding the other side: these are all good things.
I know everything seems urgent and existential and you are concerned for the safety of your children. But the way you talk and your responses make it seem like your mind is already dangerously closed off in this area. The warning shots you mention? None of those are surprising for people who are pro-AI development because they only reflect capabilities. Capabilities are the things I want and expect. You don't need to respond or do it right away, but I do urge you to take some time to ask yourself if you're not engaging in confirmation bias.
You'll notice that I'm talking to you on Zvi's blog. I namedrop Yudkowsky to indicate that I have and do read things from the Safety side. Listening to and trying to understand those you might disagree with and taking a little time to consider their perspectives and things they may be correct about don't make you an apostate or traitor to the cause - this isn't a religious debate, remember? Just people trying to figure things out - and, hey, if you're right the worst it can do is make you a better advocate.
I am not trying to tell you that you're wrong about anything at all right now. I'm not trying to change your mind. I'm trying to sincerely tell you that your posts pattern match to zealotry and that you may not be thinking clearly. If you are, great, apologies that I was wrong. Just don't be afraid to ask yourself that question.
I have rather extensively read the literature on the "opposite side" and you may be surprised to know my origins as a Landian accelerationist. It is because I am well familiar with their anti-human mentality that I am concerned: as Zvi has noted, many of them have "spoken clearly into the microphone."
As a consequence, Sutton has spoke pretty that a future "without meat humans" is not so bad, Larry Page spoke of "preserving humanity as sentimental nonsense", and Verdun has spoken avidly of "posthuman futures." The actual actions of Marc Andressens against safety while investing in an AI agent that has since created a half billion crypto all demonstrate deeply anti-human feelings.
So no, while I'll love to find a better way to safety and I try to see things otherwise, I do not see how embracing extinction and loss of all beauty and life is a good thing.
I am not opposed to AI as a whole and highly controllable ANIs would be deeply desired by me. The problem is the generality of the systems and the warning shots are not capabilities, but negative influence and behavior.
And notably, nothing you said has shown any real concern for humanity and life.
TK-421, you're not going to get through to Sean, you can't. In the event that we find ourselves in the other world - where we have scaled current AI to AGI, by adding modalities and possibly new neural networks, of course designed by RSI, and they just reliably do whatever the prompt says, Sean will still be demanding a Pause and still be saying they are about to turn on us. I don't think he will change until he dies.
In that world, he'll be citing as evidence all the times that these new AGI have failed, ignoring it's 1 in 1000 or whatever, and state his belief that it's just about to escape or betray us, despite likely requiring substantial hardware requirements.
A lot more people will have joined Sean by that point - obviously there will be massive shifts in employment, even if there end up being plenty of jobs as medical beta testers (for the new treatments for aging and dementia and cancer) and VR world beta testers and O'Neil Colony project managers and whatever else becomes possible.
My concerns with extinction risk is not sudden left turn but industrialized dehumanization, which seems by default will industrialize humanity out of existence as friction in the system.
Well I hope if that fails to happen you will update your views. Theoretically the opposite can be done, we could make life better for every living person if we had access to a lot more resources, and also simultaneously make the cost to keep everyone alive in optimal health far cheaper in resources spent.
Of course, I'll be happy to update my view if it happens, however nothing shows this as happening. I just returned from giving a presentation from LA on AI safety and watched people freeze to death while self-driving cars drove by them.
Von Neumann was indeed awake: "Yet it would be impossible not to see it through."
The relevant fact to extract from the man who could conceive of the universal replicator in the 40s isn't that he was aware of some form of existential risk from AI. It's that he saw the risk _and kept moving the work forward_.
That's the same man who advocated preemptively nuking the Soviets before they could build a bomb. Yet he continued his work on computing. If his genius is going to be used as an example of a smart person perceiving existential risk from AI then his actions should receive equivalent weight.
No, AI safety people would simply agree it didn't make sense to do AI safety things in the 40s. It was way too early to do anything meaningful, except for writing banger sci-fi.
That's convenient.
The guy who was willing to immolate millions upon millions of Soviets due to the existential risk of them getting the bomb totally agreed with the AI safety perspective, totally, and his chosen actions to move us towards AI / expand concepts like self-replicating machines is totally because he couldn't think of anything better to do. Totally.
I would have done as the same as him, given the lack of additional information, the best thing to do was to write about it and give warnings, but for all he knew, there was a lot more to learn.
Naturally. But now you know enough?
Fuck yeah, its really overwhelming at this point. If you haven't been convinced, you need to reread Zvi and just collect warning shots from Diplomacy AI to sleeper agents(anthropic) to reward hacking(o1 scorecard) to truth_terminal creating a half billion crypto to the fact that every benchmark isn't holding, etc.
Yes, heck yes. The data is very good that this is not a good idea for humans.
He was exceptionally intelligent, but he didn’t have the experience of a great many other intelligent people in the world. That sometimes counts for something when making decisions about the world.
Technological progress is a hell of a drug, and being smart doesn't guarantee good decision-making.
The chess author published a follow-up today which showed that GPT-4o can do much better (but still not that great) if the setup is modified. The main improvement was asking it to repeat back all the moves and add another; fine-tuning and adding examples helped too. Surprisingly, fine-tuning it on 100 games of maximum-power Stockfish was not as helpful as adding 3 examples of legal moves.
So it seems that once again the problem is activating the base model's latent potential. But they point out that OpenAI has been training their models on chess games while the open models probably didn't train them on games explicitly, which probably is why even their base models don't do great.
https://dynomight.net/more-chess
Must confess that tweets with video (especially from TikTok) make me even less likely to follow such links than previous low odds. It's actively painful to be ambushed by that modality; I already automatically disregard 99% of video links to begin with. A big part of why I hang out in the ratsphere is to get a picture of the world mostly through safe and relaxing words, rather than frenetically overstimulating videos, yknow?
Which is a shame because the point is on point. It's not that there isn't intense porn out there, but it's necessarily limited by the restrictions of the physical world. (Or trivially inconvenient guardrails, for AI generation.) The sheer level of...debauchery...one can encounter with erotica completely blows it out of the water. The only limits are your imagination...which also circles back to your recurring point that the product offered by virtual companions continues to be largely shit-tier, compared to SOTA LLMs. That kind of weaksauce writing would get downvoted to hell on Literotica or whatever. I'm rather curious where we end up with a GPT-5-powered Replika, or the "adult" settings fully enabled on frontier models.
My concern with AI diagnosing is that it would be nice to know it's pedigree as much as possible, not in the sense that I need it to have an MD but I would be more likely to discount it regardless of accuracy if I found out it was trained mostly on WebMD.
-
The Last Mile problem makes me wonder if AI alignment will start to be a problem in earnest once we're at the point where it's designing and selling consumer products but isn't properly hooked into the incentive to make the products actually work. So like for example I was looking at automatic cat litter boxes the other day and one of them had a video on their Amazon page that was either noticeably partially made by AI or made conventionally by foreigners who just don't know what real life looks and sounds like, which when I notice it makes me question everything about the product up to and including its design. Now I'm imagining a scenario where, maybe not even as a deliberate scam, [a company uses] an AI [that] designs and sells a product without even checking it does any of the things it's supposed to do (or whether it's able to continue doing them for any reasonable length of time).
If an AI can successfully more or less scam sell garbage products at scale, we can imagine there's eventually going to need to be some sort of regulation requiring there to not be a disconnect between design and function. There's perhaps opportunity for that point to be a bit of a wake up moment.
-
I think the problem with using AI for movies and games (and/but, to a lesser extent, visual art and music), at least this early and near term, is that you're not going to be handed a finished product that isn't going to need additional refinement to make a wholly complete, comparable-quality end product and that refinement is still going to require some level of expertise in specific parts of the craft. I expect this will eventually be a solved problem but there's going to be a long frictional period where AI isn't going to be able to hide it's AI-ness without help.
-
I'm again remiss to point out there's maybe a half dozen ways it could be technically true that Elon Musk is legitimately in the top 20 Diablo 4 (softcore seasonal) world rankings (for the one niche speedrunning part of the game) and still have had 99% of the work done by other people (given, of course, that this could be true for any or all of the top 20). He's using the same stock broken build everyone else in the top 20 is using (this is two problems with the game broadly, 1) there isn't much competitive balancing being done on the game in the first place and 2) very few people are doing/need to do the actual work to figure out what you need to do to optimize a character) and he could have an infinite amount of people funneling him gear (ie all of the time and effort of sorting through drop quality variance--which is pretty significant if you're trying to perfectly optimize--can be offloaded).
I'd be more impressed if he streamed the entire process, which I haven't seen evidence of (to be fair on this point, I haven't looked but also nobody has been eager to be like "and here's where you can see it!"). This is especially weird since he wants X to be a game streaming platform.
The US officially has a "parents must do everything for their kids" cult, which I think explains why parents giving their kids AI to use touches such a nerve. For those who think that's shameful, what do they think of a kid reading a book? Shouldn't the parent be narrating an original story rather than relying on a mass-market commodity to teach their kid?!? Parental quality time is wonderful and indispensable, but kids interact with the world in lots of other valuable ways too.
Those emails were indeed worth reading. I'm just a random guy, but I came away with a very positive opinion of Ilya - though it kind of reinforced my feeling that maybe he's a really smart guy who doesn't belong in the hard-knuckled power competitions of corporate work.
I had a positive read on Elon there, though obviously Elon behaves in a way that looks out for Elon. But I think he behaved sincerely, especially in the bits that are basically saying to go start a private company if you want to be a private company.
Eavesdropping:
> I’m more frightened of the AI boys than I am of their AI. The problem is they have corrupted the language around it, so the word “safety” is now meaningless because the doomsters treat safety as them not destroying humankind, when there are very real safety issues that need to be dealt with around bias and fraud and the environment and so on. So it’s difficult to have the conversation now because we don’t have common terms.
(https://www.theguardian.com/technology/2024/nov/23/jeff-jarvis-elon-musks-investment-in-twitter-seemed-insane-but-it-gave-him-this-power)