Discussion about this post

User's avatar
Nikita Sokolsky's avatar

Which of the following outcomes as of 2200 would you find acceptable?

1) AGI is banned, only humans exist, civilization chugs along as usual

2) AGI is allowed, is helpful to humans thanks to fully solved alignment, civilization grows at amazing pace

3) AGI exists, humans exist, and "virtual humans" exist. All co-exist peacefully.

4) No AGI, no humans, only "virtual humans". But these "virtual humans" are ~99% equivalent to normal humans, they just happen to live "in the cloud".

5) Only AGI, no humans, no "virtual humans". But AGI has its own civilization, develops new technologies, has some sort of an economy, explores the stars, etc.

6) Only AGI, no humans. The AGI is just making paper clips.

Clearly most people think that #6 is bad. But what about the others? Are we happy to let AGI takeover if it promises to keep on building space ships, exploring other planets, researching quantum mechanics, etc? Are we happy to let "AGI" takeover if it exists in the form of "virtual humans"? What is it _exactly_ that we find so offensive about an AGI takeover?

Expand full comment
Dmitrii Zelenskii's avatar

"Does it value any amount of property above any risk to a human? If so, how does it drive a car at all? If not, how dare it?" - I feel like "if so" and "if not" are reversed here?

Expand full comment
16 more comments...

No posts