I have a handful of thoughts (not terribly well organized, sorry) in response to these kinds of speculations. At least in part, I'm going to question the premises. (1) presuming an agent "can copy and instantiate itself" implies that an ASI can solve the alignment problem. Thus, the premise includes that it's a solvable problem and one t…
I have a handful of thoughts (not terribly well organized, sorry) in response to these kinds of speculations. At least in part, I'm going to question the premises. (1) presuming an agent "can copy and instantiate itself" implies that an ASI can solve the alignment problem. Thus, the premise includes that it's a solvable problem and one that an ASI solves before humans do. (2) Lots of very smart humans exist and have not accumulated nigh-unlimited and unaccountable resources just by being smart. Large parts of the financial and governmental infrastructure of the world is specifically dedicated to ensuring that unaccountable entities do not accumulate resources. It's not clearly how merely superhuman intelligence solves that problem practically. (3) Compute capacity is not unlimited, perfectly elastic, perfectly fungible, nor unobserved. (4) Plans that depend on gullible or malicious humans aiding the AI in manipulating the physical world will operate at human speed and thus be vulnerable to human-speed counters. (5) it will be hard, even for an ASI, to solve all these problems (and more I haven't thought of) **correctly in the first attempt** and any detection likely decreases the odds of the next attempt succeeding. So, overall, I think the premises as stated seem flawed. Either there needs to be a very fast takeoff such that "magic ASI" solves all these problems too fast for a counter -- which isn't any more compelling that the nanotech/etc magic story. Or there needs to be an extremely slow takeoff such that the conditions for your premises are established without detection until it's too late. Or there needs to be another premise for how we get to that point, e.g. terrorists/nihilists/authoritarians/TonyStark build proto-ASI systems then push gain-of-function until they lose control. Put differently, I accept doom given your premise. But the premise is that "ASI" already exists, and the crux for me is a set of credible, inescapable real-world paths that take us from "AGI" to "ASI" given the real-world, human interactions that would require.
I have a handful of thoughts (not terribly well organized, sorry) in response to these kinds of speculations. At least in part, I'm going to question the premises. (1) presuming an agent "can copy and instantiate itself" implies that an ASI can solve the alignment problem. Thus, the premise includes that it's a solvable problem and one that an ASI solves before humans do. (2) Lots of very smart humans exist and have not accumulated nigh-unlimited and unaccountable resources just by being smart. Large parts of the financial and governmental infrastructure of the world is specifically dedicated to ensuring that unaccountable entities do not accumulate resources. It's not clearly how merely superhuman intelligence solves that problem practically. (3) Compute capacity is not unlimited, perfectly elastic, perfectly fungible, nor unobserved. (4) Plans that depend on gullible or malicious humans aiding the AI in manipulating the physical world will operate at human speed and thus be vulnerable to human-speed counters. (5) it will be hard, even for an ASI, to solve all these problems (and more I haven't thought of) **correctly in the first attempt** and any detection likely decreases the odds of the next attempt succeeding. So, overall, I think the premises as stated seem flawed. Either there needs to be a very fast takeoff such that "magic ASI" solves all these problems too fast for a counter -- which isn't any more compelling that the nanotech/etc magic story. Or there needs to be an extremely slow takeoff such that the conditions for your premises are established without detection until it's too late. Or there needs to be another premise for how we get to that point, e.g. terrorists/nihilists/authoritarians/TonyStark build proto-ASI systems then push gain-of-function until they lose control. Put differently, I accept doom given your premise. But the premise is that "ASI" already exists, and the crux for me is a set of credible, inescapable real-world paths that take us from "AGI" to "ASI" given the real-world, human interactions that would require.