Discussion about this post

User's avatar
Rapa-Nui's avatar

"Seems important to think ahead here - is this a good fire alarm? Would anyone be willing to say in advance ‘if this happens, with the following qualifications, then that is scary and I will then start to worry about existential risks from such agents in the future when they get more capable?’"

Yes, that's a good fire alarm, but the details matter. If it's just really good at noticing free dollar bills lying around on the street (arbitrage), then it's only a little alarming. But if it develops capabilities to somehow change/trigger human behaviors in a way that allows it to profit, that is a HUGE RED WAILING fire alarm to me. We've seen hints of this in the Diplomacy bot and poker bots, but nothing like "creating a false narrative that gets reported by the WSJ which causes a trading cascade where it ends up making 100 million dollars over 72 hours by taking it from actual humans and human-run institutions".

Expand full comment
Ed Stackler's avatar

Thank you for this overview and your thoughts. Brilliantly well-covered. And so brief. : )

Expand full comment
16 more comments...

No posts