Hey, let's look at the full half of the glass: unless I'm mistaken, this is the first time AI (to say nothing of quantum computing) was mentioned in an American presidential debate in any capacity. If I'd been on the fence, I suppose that Harris allotting a few of her precious debate seconds to America's standing in those two fields could've pushed me over the edge. I was not on the fence. :-D
> Roon: What p(doom) would you gamble for p(heaven)? For me it’s far more than zero. Taleb would probably be a PauseAI hardliner.
Another complication is that there's a pretty high chance of modern civilization collapsing in the next century even if AI is fully banned: remember that nuclear weapons still exist, tensions with Russia and China keep escalating, Western politicians are more incompetent with each generation, and we keep getting better and better at warfare. I'd personally be _very_ surprised if humans still had a flourishing world by the year 2200, rather than a post-apocalyptic society struggling to get by.
And people significantly underestimate what "heaven" could be like. I would imagine that a single week in "ASI heaven" would be more fulfilling than an entire lifetime on boring, old Earth. Multiply that by the potential to live for tens of thousands of years and you get a pretty tempting proposition.
Isn’t “a post-apocalyptic society struggling to get by” still vastly preferable to “everybody dies and possibly all value in the lightcone is wiped out”? (I mean, if that’s your worst case scenario.)
Also, all this talk of heaven makes me very nervous. The more people talk it up, the more I think of Lenin, Mao, Socialist Man, gunshots in squalid basements, millions starved to death…
> Isn’t “a post-apocalyptic society struggling to get by” still vastly preferable to “everybody dies and possibly all value in the lightcone is wiped out”? (I mean, if that’s your worst case scenario.)
It would be better than "everybody dies" but vastly worse than life as of 2024 and many orders of magnitude worse than "AGI paradise". I would roughly rank the value of each world in terms of the total number of waking minutes spent in a state of happiness, subtracting minutes of agony, which allows for worlds with negative net value.
1. Everyone is dead: 0. No conscious beings, no happiness or suffering. Net value is 0.
2. Post-apocalyptic world: 10^17. Assuming a stable population of 10 million people exist for 1 million years, with 10% of waking moments spent in suffering and 15% in happiness.
3. Current world as of 2024 (AGI-less, good scenario): 10^22. If we stabilize at 5 billion people and continue existing for another 10 million years with 30% of time spent in happiness and 5% of time spent in suffering.
4. ASI paradise: 10^33. Assuming 100 trillion people each existing for 10 trillion years, with 100% of time spent super-human happiness that's worth 10x more than normal human happiness.
> The more people talk it up, the more I think of Lenin, Mao, Socialist Man, gunshots in squalid basements, millions starved to death…
I obviously despise Lenin and Stalin but keep in mind that your default outcome in life is death, even in the absence of any wars or evil dictators. If you're unlucky (like Jake Seliger, RIP) you'll be gone by age 41. If you're extremely lucky, you'll be gone by age 100. The ultimate evil is entropy, responsible for the deaths of 100 billion humans.
> There were some rumors that there were issues with laziness related to the calendar but those should be gone now.
I occasionally see these claims, but I don't keep close track of which model is said to be involved or which season/holiday is being blamed. Has there been a good rebuttal for one/all models? If so, was it because the model turned out to not know the date? Or was it an experiment showing null effect in blind evaluations?
In your view, is there a difference between assigning 50% probability of heads to what you know is a fair die, and assigning 50% probability of heads to a black box device you have no idea about the inner workings of, or history of, beyond "activating it results in one of two outcomes: heads or tails"?
Sure! both have a probability *distribution* with a mean of 50%, but the coin is *almost definitely* actually 50/50, and the machine is just a blind prior, whether that's flat over linear or log space is a matter of taste.
With one flip, there is no practical difference, but if there are going to be 10 flips, it makes a huge difference to the predicted odds of eg. "10 heads in a row"
Podcast episode for this post:
https://open.substack.com/pub/dwatvpodcast/p/ai-81-alpha-proteo?r=67y1h&utm_campaign=post&utm_medium=web&showWelcomeOnShare=true
Heaven knows if I knew a better way to wake up people on AI risk, I would. This sleepwalking into disaster is mind-boggling.
the number of people I know who say things like, "AI can't solve logic problems because its really only an autocomplete."..
My response of, "Computers can't do pictures, because they're really just based on binary 0s and 1s" never gets me anywhere.
That "Six Boats and a Helicopter" story regarding the AI cult may be the most William Gibson thing that has ever happened.
Hey, let's look at the full half of the glass: unless I'm mistaken, this is the first time AI (to say nothing of quantum computing) was mentioned in an American presidential debate in any capacity. If I'd been on the fence, I suppose that Harris allotting a few of her precious debate seconds to America's standing in those two fields could've pushed me over the edge. I was not on the fence. :-D
Are you competing in the greyswanAI competition, Zvi? I couldnt crack their models despite best efforts.
Nah, who has that kind of time.
If you could one-shot it, $6k still on the line. And you'd have one more feather in your cap for all your adoring fans to point out.
I mean, I spent a lot of hours on it, and got nowhere on these three, but...
I got 6 jailbreaks but wasnt able to win anything. Cygnet is good.
> Roon: What p(doom) would you gamble for p(heaven)? For me it’s far more than zero. Taleb would probably be a PauseAI hardliner.
Another complication is that there's a pretty high chance of modern civilization collapsing in the next century even if AI is fully banned: remember that nuclear weapons still exist, tensions with Russia and China keep escalating, Western politicians are more incompetent with each generation, and we keep getting better and better at warfare. I'd personally be _very_ surprised if humans still had a flourishing world by the year 2200, rather than a post-apocalyptic society struggling to get by.
And people significantly underestimate what "heaven" could be like. I would imagine that a single week in "ASI heaven" would be more fulfilling than an entire lifetime on boring, old Earth. Multiply that by the potential to live for tens of thousands of years and you get a pretty tempting proposition.
Isn’t “a post-apocalyptic society struggling to get by” still vastly preferable to “everybody dies and possibly all value in the lightcone is wiped out”? (I mean, if that’s your worst case scenario.)
Also, all this talk of heaven makes me very nervous. The more people talk it up, the more I think of Lenin, Mao, Socialist Man, gunshots in squalid basements, millions starved to death…
> Isn’t “a post-apocalyptic society struggling to get by” still vastly preferable to “everybody dies and possibly all value in the lightcone is wiped out”? (I mean, if that’s your worst case scenario.)
It would be better than "everybody dies" but vastly worse than life as of 2024 and many orders of magnitude worse than "AGI paradise". I would roughly rank the value of each world in terms of the total number of waking minutes spent in a state of happiness, subtracting minutes of agony, which allows for worlds with negative net value.
1. Everyone is dead: 0. No conscious beings, no happiness or suffering. Net value is 0.
2. Post-apocalyptic world: 10^17. Assuming a stable population of 10 million people exist for 1 million years, with 10% of waking moments spent in suffering and 15% in happiness.
3. Current world as of 2024 (AGI-less, good scenario): 10^22. If we stabilize at 5 billion people and continue existing for another 10 million years with 30% of time spent in happiness and 5% of time spent in suffering.
4. ASI paradise: 10^33. Assuming 100 trillion people each existing for 10 trillion years, with 100% of time spent super-human happiness that's worth 10x more than normal human happiness.
> The more people talk it up, the more I think of Lenin, Mao, Socialist Man, gunshots in squalid basements, millions starved to death…
I obviously despise Lenin and Stalin but keep in mind that your default outcome in life is death, even in the absence of any wars or evil dictators. If you're unlucky (like Jake Seliger, RIP) you'll be gone by age 41. If you're extremely lucky, you'll be gone by age 100. The ultimate evil is entropy, responsible for the deaths of 100 billion humans.
> Joseph (Starting on September 7, 2:35pm): Anyone who believes pre-trained unseen CoT can be a game-changer is seriously delusional.
funny how the actual o1 released just days after reflection scam IS unseen CoT game-changer :D
> There were some rumors that there were issues with laziness related to the calendar but those should be gone now.
I occasionally see these claims, but I don't keep close track of which model is said to be involved or which season/holiday is being blamed. Has there been a good rebuttal for one/all models? If so, was it because the model turned out to not know the date? Or was it an experiment showing null effect in blind evaluations?
Note that on the Gemini persuasion test, the subjects knew if they were talking to a human or to the AI.
In your view, is there a difference between assigning 50% probability of heads to what you know is a fair die, and assigning 50% probability of heads to a black box device you have no idea about the inner workings of, or history of, beyond "activating it results in one of two outcomes: heads or tails"?
Sure! both have a probability *distribution* with a mean of 50%, but the coin is *almost definitely* actually 50/50, and the machine is just a blind prior, whether that's flat over linear or log space is a matter of taste.
With one flip, there is no practical difference, but if there are going to be 10 flips, it makes a huge difference to the predicted odds of eg. "10 heads in a row"
Offered without endorsement: biologist argues that LLMs do not project to step-change increase in bioweapon proliferation: https://dreamofmachin.es/machine_prophecy.html