As the Trump transition continues and we try to steer and anticipate its decisions on AI as best we can, there was continued discussion about one of the AI debate’s favorite questions: Are we making huge progress real soon now, or is deep learning hitting a wall?
"The cancer case is likely similar to the asthma case, where slow developing cancers lead to more other health care,"
I'm a great example of that: Prostate cancer saved my life, because the preoperative physical for the surgery discovered my lymphoma while it was still stage 1. And as lymphoma only really has a good cure rate when it's accidentally discovered early, most lymphoma survivors have similar stories.
> Terrance Tao: These are extremely challenging. I think they will resist AIs for several years at least.
This is a pretty substantial claim. Tao is both a brilliant mathematician, and someone who has put in real effort to engage with generative AI and evaluate its capabilities for research-level math. If AIs start doing well on this bench mark before "several years", that would be worth a significant update in AGI timelines.
I think Tao has reserved his right to change his mind if the time comes, but he also has the privilege of being high in the mountains as AI makes its ascent, which gives him particular insight as to the particular hurdles it has to overcome.
On the other hand, AI meeting Tao's expectation in the next several years also means that AI will be expect to start performing in the top 1%, 0.1%, 0.01%, etc. in the next few years. Even being on pace would be significant progress.
I'm a retired Computer Systems Engineer. Garbage in garbage out has always been true. Any AI model build with input from "X" or a myriad of other misinformation is going to be susceptible to hallucinations and worse. Is there any at this time that have been carefully fed only the truth? Probably not. This is not a good bedrock to be building on.
Hallucinations aren't necessarily a product of misinformation being available. Sometimes there'll be no information and the LLM will hallucinate to fill in gaps. Even models like Claude who are fed curated datasets are prone to a decent amount of hallucinations. Luckily there are many ways to circumvent these issues by providing additional instructions.
Garbage in garbage out is true indeed, but it's not all garbage in and there are steps you can take to filter the trash from the treasure.
I'm contemplating writing an article called, "Hallucinations Will Kill The Dream Of AGI!" It's a deliberately provocative title, but the more I work with LLMs, the more amazed I am at how useless so many tasks are for even the most advanced models because of hallucinations. The form that the hallucinations take is stunningly broad. From research articles to case law, historical weather reports to simple baseball statistics, LLMs will confidently provide the wrong answers. Even so-called specialized models and wrappers for research will return results (i.e., supposedly relevant articles) that are completely fabricated. Most incredibly, even when the information is uploaded into the context window (which dramatically helps but renders models much less *generally* useful, as the user has to basically do his or her own initial research), models will still hallucinate the wrong answer even when the correct information is "in context" - i.e., immediately available to the model. As you say, additional instructions (prompt engineering) and chain of thought protocols can avoid some of the most egregious answers, but that makes the models less useful and, more important, do not necessarily work. From what I can tell, more compute and more data will not solve the hallucinations problem, yet those topics get much more attention and coverage than the fact that LLMs will simply make up "facts" because they seem plausible without even alerting the user to the errors. Most important work in the world requires factual accuracy. Put another way, AI that cannot provide and process factually accurate information even when the information is perfectly clear, readily available, and uncontroversial (I'm not talking about "facts" that reasonable humans might debate) will never be AGI.
One consideration - don't humans sometimes "hallucinate" facts in the same way? After being called out, they might say, "I thought I remembered X, but I must have been mistaken," but they don't realize they were wrong until the evidence is shown to them. Humans regularly fail at reading comprehension, also.
Obviously AI hallucinations are a problem, but even if the issue can't be eliminated, that doesn't seem like it precludes the existence AGI.
> Wolfram goes full ‘your preferences are invalid and human extinction is good because what matters is computation?’
No, he didn’t. He asked for a clarification, in the lines of (not exact quotes) "when you say that humans being replaced by a successor, unaligned, artificial species is a bad thing, do you mean bad in a parochial sense of bad from the narrow point of view of humans, or do you have a more objective notion of bad", then Yudkowsky said it was the first case, and yes Wolfram said something close to "this looks like more spiritual and unscientific" but also completed with "but I agree that as an human I too would very much prefer if we could make it".
Yes, there was a lot of unfortunate philosophical not-very-productive-for-the-question-at-hand side debates (I feel the moderator should have intervened quite a few times, but both Yudkowsky and Wolfram made very clear to the moderator that he was out of his depth on his first intervention). Like "you say AGI can make better technology than us but do we have a sufficiently good definition of technology that we can say for sure that a technology is better than another one" like c'mon man.
I *think* I can pinpoint where Wolfram and Yudkowsky talked past each other (I’m pretty familiar with both of those, by having avidly read all the sequences from the first and religiously followed most of his live streams when he was working on his Physics Project for the second). Both agree than under current paradigm, inner alignment is hard, in the sense that the outer objective is reached by completely alien and non-understandable sub-goals/proxies/heuristics. But for Wolfram those subgoals are like his simple cellular automata, while they can exhibit very complex behaviors, there is no reason to think that they have to have some kind of goal-directness, which is the dangerous part. Yudkowsky did not respond to that because he did not get the point (to be fair, Wolfram did not made that point very clear), and that very important part of the debate was rushed towards the last minutes of the long debate, because so much damn time had been lost to philosophical rabbit holes.
I didn't listen myself so - you're saying Wolfram didn't actively assert an alternative notion of preference or value, simply he pulled a rabbit hole where he didn't accept the premise that human extinction was bad, without actively disagreeing with it?
Wofram: "I viscerally agree with you. Scientifically I have a bit of a hard time in a sense that feels like a very kind of spiritual statement, which is not necessarily bad, but it’s just worth understanding what kind of a thing it is. It is saying that there is something very kind of sacred about these attributes of humans and that we have perhaps even a higher purpose to which we don’t really know where it comes from. [...] As humans who like doing what we’re doing, it would be nice if we could go on doing that without all being killed by AIs"
Wolfram : "One question is what’s the right thing to have happen ? I don’t think there’s any abstract way to answer that. I think it’s a question of how we humans feel about it. And I think you and I seem to feel — I know I feel that — that preserving the kind of things human do is a good thing"
My overall understanding is that it’s essentially "yes we would prefer existing, but the AI would probably prefer existing too, and there’s no objective way to decide who’s right, it all boils down to subjective preference, and I agree that subjectively as a humans I prefer for humans to exist".
"The cancer case is likely similar to the asthma case, where slow developing cancers lead to more other health care,"
I'm a great example of that: Prostate cancer saved my life, because the preoperative physical for the surgery discovered my lymphoma while it was still stage 1. And as lymphoma only really has a good cure rate when it's accidentally discovered early, most lymphoma survivors have similar stories.
> Terrance Tao: These are extremely challenging. I think they will resist AIs for several years at least.
This is a pretty substantial claim. Tao is both a brilliant mathematician, and someone who has put in real effort to engage with generative AI and evaluate its capabilities for research-level math. If AIs start doing well on this bench mark before "several years", that would be worth a significant update in AGI timelines.
I think Tao has reserved his right to change his mind if the time comes, but he also has the privilege of being high in the mountains as AI makes its ascent, which gives him particular insight as to the particular hurdles it has to overcome.
On the other hand, AI meeting Tao's expectation in the next several years also means that AI will be expect to start performing in the top 1%, 0.1%, 0.01%, etc. in the next few years. Even being on pace would be significant progress.
Podcast episode for this post:
https://open.substack.com/pub/dwatvpodcast/p/ai-90-the-wall
> Exceeding human-level reasoning will require training methods beyond next-token prediction, such as reinforcement learning and self-play
So AGI will be brought on by...mental masturbation?
I'm a retired Computer Systems Engineer. Garbage in garbage out has always been true. Any AI model build with input from "X" or a myriad of other misinformation is going to be susceptible to hallucinations and worse. Is there any at this time that have been carefully fed only the truth? Probably not. This is not a good bedrock to be building on.
Hallucinations aren't necessarily a product of misinformation being available. Sometimes there'll be no information and the LLM will hallucinate to fill in gaps. Even models like Claude who are fed curated datasets are prone to a decent amount of hallucinations. Luckily there are many ways to circumvent these issues by providing additional instructions.
Garbage in garbage out is true indeed, but it's not all garbage in and there are steps you can take to filter the trash from the treasure.
I'm contemplating writing an article called, "Hallucinations Will Kill The Dream Of AGI!" It's a deliberately provocative title, but the more I work with LLMs, the more amazed I am at how useless so many tasks are for even the most advanced models because of hallucinations. The form that the hallucinations take is stunningly broad. From research articles to case law, historical weather reports to simple baseball statistics, LLMs will confidently provide the wrong answers. Even so-called specialized models and wrappers for research will return results (i.e., supposedly relevant articles) that are completely fabricated. Most incredibly, even when the information is uploaded into the context window (which dramatically helps but renders models much less *generally* useful, as the user has to basically do his or her own initial research), models will still hallucinate the wrong answer even when the correct information is "in context" - i.e., immediately available to the model. As you say, additional instructions (prompt engineering) and chain of thought protocols can avoid some of the most egregious answers, but that makes the models less useful and, more important, do not necessarily work. From what I can tell, more compute and more data will not solve the hallucinations problem, yet those topics get much more attention and coverage than the fact that LLMs will simply make up "facts" because they seem plausible without even alerting the user to the errors. Most important work in the world requires factual accuracy. Put another way, AI that cannot provide and process factually accurate information even when the information is perfectly clear, readily available, and uncontroversial (I'm not talking about "facts" that reasonable humans might debate) will never be AGI.
One consideration - don't humans sometimes "hallucinate" facts in the same way? After being called out, they might say, "I thought I remembered X, but I must have been mistaken," but they don't realize they were wrong until the evidence is shown to them. Humans regularly fail at reading comprehension, also.
Obviously AI hallucinations are a problem, but even if the issue can't be eliminated, that doesn't seem like it precludes the existence AGI.
I don't want AI bird watching binoculars for bird watching, I want AI kudzu killing binoculars for kudzu killing.
https://preview.redd.it/dcfnmv0jz84d1.jpeg?auto=webp&s=4fac6de72e4cf9fec2c8b1145c182f1a57c7960f
> Wolfram goes full ‘your preferences are invalid and human extinction is good because what matters is computation?’
No, he didn’t. He asked for a clarification, in the lines of (not exact quotes) "when you say that humans being replaced by a successor, unaligned, artificial species is a bad thing, do you mean bad in a parochial sense of bad from the narrow point of view of humans, or do you have a more objective notion of bad", then Yudkowsky said it was the first case, and yes Wolfram said something close to "this looks like more spiritual and unscientific" but also completed with "but I agree that as an human I too would very much prefer if we could make it".
Yes, there was a lot of unfortunate philosophical not-very-productive-for-the-question-at-hand side debates (I feel the moderator should have intervened quite a few times, but both Yudkowsky and Wolfram made very clear to the moderator that he was out of his depth on his first intervention). Like "you say AGI can make better technology than us but do we have a sufficiently good definition of technology that we can say for sure that a technology is better than another one" like c'mon man.
I *think* I can pinpoint where Wolfram and Yudkowsky talked past each other (I’m pretty familiar with both of those, by having avidly read all the sequences from the first and religiously followed most of his live streams when he was working on his Physics Project for the second). Both agree than under current paradigm, inner alignment is hard, in the sense that the outer objective is reached by completely alien and non-understandable sub-goals/proxies/heuristics. But for Wolfram those subgoals are like his simple cellular automata, while they can exhibit very complex behaviors, there is no reason to think that they have to have some kind of goal-directness, which is the dangerous part. Yudkowsky did not respond to that because he did not get the point (to be fair, Wolfram did not made that point very clear), and that very important part of the debate was rushed towards the last minutes of the long debate, because so much damn time had been lost to philosophical rabbit holes.
I didn't listen myself so - you're saying Wolfram didn't actively assert an alternative notion of preference or value, simply he pulled a rabbit hole where he didn't accept the premise that human extinction was bad, without actively disagreeing with it?
Trying to capture the essence of the discussion on this specific point with a few excerpts :
https://www.youtube.com/watch?v=xjH2B_sE_RQ&t=1345s
Yudkowsky: "It’s not clear to me you can wipe out humanity and replace it with arbitrary stuff, and everything just gets better as a result"
Wolfram: "I don’t know what better means. Better is a very human concept"
(segues into debating about consciousness and ethics)
https://www.youtube.com/watch?v=xjH2B_sE_RQ&t=35m10s
Wofram: "I viscerally agree with you. Scientifically I have a bit of a hard time in a sense that feels like a very kind of spiritual statement, which is not necessarily bad, but it’s just worth understanding what kind of a thing it is. It is saying that there is something very kind of sacred about these attributes of humans and that we have perhaps even a higher purpose to which we don’t really know where it comes from. [...] As humans who like doing what we’re doing, it would be nice if we could go on doing that without all being killed by AIs"
https://www.youtube.com/watch?v=xjH2B_sE_RQ&t=37m15s
Wolfram : "One question is what’s the right thing to have happen ? I don’t think there’s any abstract way to answer that. I think it’s a question of how we humans feel about it. And I think you and I seem to feel — I know I feel that — that preserving the kind of things human do is a good thing"
My overall understanding is that it’s essentially "yes we would prefer existing, but the AI would probably prefer existing too, and there’s no objective way to decide who’s right, it all boils down to subjective preference, and I agree that subjectively as a humans I prefer for humans to exist".
I saw Pliny got one of his agents to successfully sign up for a Google account so that's probably going to continue to get weird.