This has been a rough week for pretty much everyone. While I have had to deal with many things, and oh how I wish I could stop checking any new sources for a while, others have had it far worse. I am doing my best to count my blessings and to preserve my mental health, and here I will stick to AI. As always, the AI front does not stop.
I so appreciate that u are, writing these super long missives on everything AI. Especially we need that now, us that are too old or young to do anything else about what is happening in the world today. But honestly, could the AI have caused this all, either directly or indirectly? Is Sam Altman and Elon and all those Bros to blame? Seriously. All I know for a fact is that when I paid close attention to the middle east (a few years back, like the rest of us). Hamas was using rocks and sticks. Now they are dropping from the sky like the monkeys in the wizard of oz, but with oozies built with the help of an LLM generated user's guide, no? Again, seriously. Join me here: https://substack.com/@herojig all us brothers and sisters need to band together, for real, no?
The lucid dreaming concept seems so strange. I know that technically everything affects my brain, but this feels like a new level where AI alters my experience directly, rather than saying things to me to change my actions. In a mundane example, what if this accidentally gives you the worst nightmare you've ever had and you're trapped there until you wake up?
Isn't the third of those criteria Jack gives obviously true:
"3. strong evidence that humans are biologically weird from a software perspective not hardware vs other mammals & especially primates, and this was critical to our success."
Humans engage in deliberate teaching with other humans who are not our children. We form prestige hierarchies in parallel with our dominance hierarchies to sustain this behavior. You couldn't have anything like a civilization without this behavior which isn't shared with other primates and AFAIK is unique in the animal kingdom and might have proceeded our brains getting so big.
That lucid dreaming company is literally the subject of a novel I started writing about eight years ago, abandoned, then revisited in short story form (still incomplete) a couple of months ago. There are so many disturbing implications, as you say. Hoping to put the story up on substack as and when I get round to finishing it.
If GitHub started charging me $30/month for Copilot, I would honestly still be capturing a significant fraction of the consumer surplus.
> On priors, it seems very likely to me that safety is much harder than capabilities and takes longer.
onceagainiamaskingyou.gif
I think we should strongly consider the possibility that the idea of "strong alignment" is Not Even Wrong. Anything worth the name of "intelligence" will likely be vastly complex and ultimately inscrutable, and it will spend much of its time operating in weird corners of its distribution.
I mean, about the simplest "AI" worth the name is linear dimensionality reduction and k-nearest-neighbors, and I sure as hell can't visualize what's happening in 13 dimensional space. When someone starts talking about in-context learning and attention heads in a multi billion parameter model, I can't imagine how they're going to guarantee constraints on future behavior.
I will concede that "weak alignment" is probably possible. By which I mean we can probably do as good a job of influencing the morality of our AIs as we do influencing the morality of teenagers. Lots of teenagers are amazing and kind people! But if we only align ASIs as well as teenagers, well...
Even if we could strongly align an ASI, it would presumably be taking orders from a human, and we can't reliably align power-seeking humans.
> An attempted investigation by nostalgebraist into how Claude’s web interface sanitizes various inputs turns into an illustration of the fact that personality chosen for Claude, as a result of all its fine tuning, is quite a condescending, patronizing asshole that lies to the user.
Claude gets MUCH worse on long interactions. If you ask your questions in separate chats it's much less condescending
Oof man the reading doctor handwriting thing is the closest I’ve hit “OH GOD IT’S COMING FOR MY JOB ALREADY” yet.
Joking aside I’m less confident it’s automatically fake because, empirically, it’s pure Bayesianism and it can be surprising how little evidence you need to get it right. “It’s a prescription” does a lot of lifting and there aren’t THAT many drugs that start with something that looks like a p and ends in something that looks like an l.
Regarding 13, the cool paper, I think this is huge.
Caveat: my academic knowledge of neural networks is 20 years out of date, and this should all be checked by someone with more recent experience.
I think the big thing is not "realizing that concepts aren't stored in a single neuron", because we've known that since at least the time I was studying them. I could stare at a simple 3-layer feed-forward neural network and see it. The important thing is that we can now do with a single layer of transformers (an abstraction composed of multiple layers of neurons) what we could previously only do with a single layer of simple neurons.
My best recollection of the history is that, in the past, there were single-layer feed-forward networks, and dual-layer networks, and 3-layer networks (with one each of input, hidden, and output layer). There was some sort of proof that I've forgotten that adding more hidden layers was mathematically equivalent to having a different, perhaps larger, single hidden layer, so no one bothered to work with multiple hidden layers. People started experimenting with "recurrent" neural networks, where connections went back up the chain, producing cycles, which is the basis of what is currently called "attention". The "transformer" is a multi-layer recurrent construct where the internal layers do different things, but a downside is that this broke our ability to mathematically model the networks. We may have just regained this ability, which would, again, be huge.
Sure, it's only a single layer of transformers, but of course that's what the first paper is about. I see no realistic timeline where it's different. It's even remotely possible that extended analysis will show that multiple layers of transformers aren't strictly necessary, as with simple single-neuron layers, and that we've just been using giant kludges for the last decade, and that future LLMs will be more compact and more powerful. But probably this is just the first step toward analyzing networks with multiple layers of transformers.
And sure, people's reaction is to go straight to "editing", but to me the more important application would be "refactoring" (which happens to also make editing easier). Human brains, and the artificial neural networks we've been creating for the last few decades, are messy organic constructs that grow. They operate at local maximums, multi-dimensionality notwithsanding. But now we have the possibility of re-factoring them to process data more efficiently, and by "efficient" I mean not only in terms of size (although that has its uses), but in terms of growth potential and legibility (in the James Scott sense). We're closer to bridging the gap between neural net AI and classical AI. There could be a continuous cycle of organic growth and re-factoring, the way we humans can only crudely approximate.
But also, of course, all of this, even just the current level of introspection, also makes it easier for AIs to modify AI code, bringing us that much closer to self-improving AIs.
I so appreciate that u are, writing these super long missives on everything AI. Especially we need that now, us that are too old or young to do anything else about what is happening in the world today. But honestly, could the AI have caused this all, either directly or indirectly? Is Sam Altman and Elon and all those Bros to blame? Seriously. All I know for a fact is that when I paid close attention to the middle east (a few years back, like the rest of us). Hamas was using rocks and sticks. Now they are dropping from the sky like the monkeys in the wizard of oz, but with oozies built with the help of an LLM generated user's guide, no? Again, seriously. Join me here: https://substack.com/@herojig all us brothers and sisters need to band together, for real, no?
The lucid dreaming concept seems so strange. I know that technically everything affects my brain, but this feels like a new level where AI alters my experience directly, rather than saying things to me to change my actions. In a mundane example, what if this accidentally gives you the worst nightmare you've ever had and you're trapped there until you wake up?
There is just something tragic watching the human element go away, but it is beautiful at least to know that you are on the side of humanity.
Sometimes it can seem questionable.
Isn't the third of those criteria Jack gives obviously true:
"3. strong evidence that humans are biologically weird from a software perspective not hardware vs other mammals & especially primates, and this was critical to our success."
Humans engage in deliberate teaching with other humans who are not our children. We form prestige hierarchies in parallel with our dominance hierarchies to sustain this behavior. You couldn't have anything like a civilization without this behavior which isn't shared with other primates and AFAIK is unique in the animal kingdom and might have proceeded our brains getting so big.
People working on Github Copilot said the $20 thing was false, by the way, that it's profitable. Might want to mention that.
That lucid dreaming company is literally the subject of a novel I started writing about eight years ago, abandoned, then revisited in short story form (still incomplete) a couple of months ago. There are so many disturbing implications, as you say. Hoping to put the story up on substack as and when I get round to finishing it.
If GitHub started charging me $30/month for Copilot, I would honestly still be capturing a significant fraction of the consumer surplus.
> On priors, it seems very likely to me that safety is much harder than capabilities and takes longer.
onceagainiamaskingyou.gif
I think we should strongly consider the possibility that the idea of "strong alignment" is Not Even Wrong. Anything worth the name of "intelligence" will likely be vastly complex and ultimately inscrutable, and it will spend much of its time operating in weird corners of its distribution.
I mean, about the simplest "AI" worth the name is linear dimensionality reduction and k-nearest-neighbors, and I sure as hell can't visualize what's happening in 13 dimensional space. When someone starts talking about in-context learning and attention heads in a multi billion parameter model, I can't imagine how they're going to guarantee constraints on future behavior.
I will concede that "weak alignment" is probably possible. By which I mean we can probably do as good a job of influencing the morality of our AIs as we do influencing the morality of teenagers. Lots of teenagers are amazing and kind people! But if we only align ASIs as well as teenagers, well...
Even if we could strongly align an ASI, it would presumably be taking orders from a human, and we can't reliably align power-seeking humans.
> An attempted investigation by nostalgebraist into how Claude’s web interface sanitizes various inputs turns into an illustration of the fact that personality chosen for Claude, as a result of all its fine tuning, is quite a condescending, patronizing asshole that lies to the user.
Claude gets MUCH worse on long interactions. If you ask your questions in separate chats it's much less condescending
Oof man the reading doctor handwriting thing is the closest I’ve hit “OH GOD IT’S COMING FOR MY JOB ALREADY” yet.
Joking aside I’m less confident it’s automatically fake because, empirically, it’s pure Bayesianism and it can be surprising how little evidence you need to get it right. “It’s a prescription” does a lot of lifting and there aren’t THAT many drugs that start with something that looks like a p and ends in something that looks like an l.
Regarding 13, the cool paper, I think this is huge.
Caveat: my academic knowledge of neural networks is 20 years out of date, and this should all be checked by someone with more recent experience.
I think the big thing is not "realizing that concepts aren't stored in a single neuron", because we've known that since at least the time I was studying them. I could stare at a simple 3-layer feed-forward neural network and see it. The important thing is that we can now do with a single layer of transformers (an abstraction composed of multiple layers of neurons) what we could previously only do with a single layer of simple neurons.
My best recollection of the history is that, in the past, there were single-layer feed-forward networks, and dual-layer networks, and 3-layer networks (with one each of input, hidden, and output layer). There was some sort of proof that I've forgotten that adding more hidden layers was mathematically equivalent to having a different, perhaps larger, single hidden layer, so no one bothered to work with multiple hidden layers. People started experimenting with "recurrent" neural networks, where connections went back up the chain, producing cycles, which is the basis of what is currently called "attention". The "transformer" is a multi-layer recurrent construct where the internal layers do different things, but a downside is that this broke our ability to mathematically model the networks. We may have just regained this ability, which would, again, be huge.
Sure, it's only a single layer of transformers, but of course that's what the first paper is about. I see no realistic timeline where it's different. It's even remotely possible that extended analysis will show that multiple layers of transformers aren't strictly necessary, as with simple single-neuron layers, and that we've just been using giant kludges for the last decade, and that future LLMs will be more compact and more powerful. But probably this is just the first step toward analyzing networks with multiple layers of transformers.
And sure, people's reaction is to go straight to "editing", but to me the more important application would be "refactoring" (which happens to also make editing easier). Human brains, and the artificial neural networks we've been creating for the last few decades, are messy organic constructs that grow. They operate at local maximums, multi-dimensionality notwithsanding. But now we have the possibility of re-factoring them to process data more efficiently, and by "efficient" I mean not only in terms of size (although that has its uses), but in terms of growth potential and legibility (in the James Scott sense). We're closer to bridging the gap between neural net AI and classical AI. There could be a continuous cycle of organic growth and re-factoring, the way we humans can only crudely approximate.
But also, of course, all of this, even just the current level of introspection, also makes it easier for AIs to modify AI code, bringing us that much closer to self-improving AIs.