If you didn’t have any future shock over the past two months, either you weren’t paying attention to AI developments or I am very curious how you managed that. I would not exactly call this week’s pace of events slow. It was still distinctly slower than that which we have seen in the previous six weeks of updates. I don’t feel
Italy's privacy authority decision is not one of the many entirely stupid things that our government is currently trying to do (fines on foreign words, the "Italian restaurant" badge, banning synthetic meat) - it is in my opinion more interesting than that.
1) Massive collection of personal data and potentially using it for training.
2) Inaccurate information made available by ChatGPT could amount to "inaccurate personal data processing".
3) Lack of age verification mechanisms.
(1) and (3) seem straightforward to address, but (2) is a radical problem: the Italian Garante says that ChatGPT "hallucinating" information about people that is not true could configure "incorrect processing of personal data". I kind of see their point, in a way it's similar to false information being put on a website or newspaper... but how could that possibly be fixed, in the current model?
Perhaps they'll let them get away with some kind of banner "The stories and information posted here are artistic works of fiction and falsehood. Only a fool would take anything posted here as fact", but perhaps not.
Another interesting fact is that, as I understand it, anyone who uses OpenAI's API becomes responsible for personal data processing (including third parties to which the data is sent), and again they could probably address (1) and (3) fairly easily, but not (2). Really curious to see how this evolves - my prediction is an "Accept" banner similar to cookies.
One thing that definitely occurs to me is that the laws on the books as written don't actually mix with what LLM chatbots do - if you enforced the rules, especially in Europe, the chatbots couldn't exist, so the question is whether they will get enforced that way...
Ok, but the GDPR and similar "laws on the books" demonstrably do not do what they are purported to do (by design?). European data regulation, currently, 1. smacks the US around on purported surveillance issues* like Privacy Shield and 2. enables some countries that care more to show that they Care More, while "accidentally" shielding companies with clever lawyers from impact by enabling them to keep their European operations in countries that Care Less as a matter of public policy. So shouldn't we expect the same to apply going forwards?
*Of course, I'm sure that Europeans never surveil American networks. Gentlemen don't read each other's mail.
I haven't gotten through all of this yet but just wanted to thank Zvi for organizing, curating, and writing up all of this info (both for AI, covid, and all the other stuff). The time and effort put into this couldn't have been small and if my substack comment of gratitude helps offset that in some way, that would be great.
I don’t know if you saw this but the decompression of the lambda calculus graf isn’t just inexact, it’s importantly wrong. It exactly reversed the original true fact (STLC well-typed terms terminate) to the opposite (are not proven to terminate.). This is meaningful, wrong, and a theorem certainly in the training set, so…it’s still pretty cool to see the compression happen, but Shoggothtongue isn’t magic.
(The second one with the secret message seems to have worked better, so there’s probably some variability here.)
Here’s my biggest problem with the pause/halt arguments: “we have to call it AI-notkilleveryoneism now.”
Our governments lack operational competence, can’t build shit, don’t understand AI at all, and are generally…bad, but they’re staffed entirely by world-class political operatives and killers. Timnit Gebru is not the political A-team, or anywhere close to it…yet she and her clique ate Eliezer alive so bad that the normal term we started with, “AI safety”, doesn’t actually refer to anything but their culture war shit anymore. _Why the hell do you want to draw the attention of the political A-team_? They have no interest in your goals, you are made of political power they can use, and they are so much better at this than you that you can’t understand what they do. If Eliezer can’t beat Timnit Gebru at this, I don’t believe he can beat the White House.
Making a fuss about pausing/halting training is not going to result in the halt or pause you want, but something vaguely shaped like it that is good for the major party political operatives. I judge that as likely to be worse than the status quo. Don’t you? The best thing to do is to make sure they don’t notice or care about us (though it’s probably too late.)
I'm a little skeptical on the proposition in "More Than Meets the Eye". Confidentiality aside, surely any insight that can be neatly packaged and communicated would already have been, but isn't there space for tacit knowledge that cannot? I have a better understanding / intuition / forecasting capabilities on a lot of systems I worked on across the board, but I don't know if I could pass the "here is a secret insight gated by experience" test in any.
Well, yes, if you can communicate it then that by construction is something you can grok without getting hands-on experience.
The question is: Can we gesture at a concrete instance of tacit knowledge that you can't get another way? Can we get SOME idea what the hell this is all about, in some way?
If the answer is no, I'd ask: Can someone give me some non-AI example of a practice where (1) you need hands-on experience with doing some X to develop tacit knowledge (2) where you cannot gesture at what that tacit knowledge is? But where (3) there is reason for me to trust you that such tacit knowledge still exists and is important?
There are plenty of places where such tacit knowledge is important and hard to otherwise communicate, but in those places I feel I could gesture at the thing (even if the gesture does not substitute for the hands-on experience)?
Mmh.. I'd have to think about it, not sure I can come up with a very convincing parallel.
I have toy examples like MTG in mind, I'd buy that (3) I'd trust someone that a play is better than mine even if (2) they can't really explain why or when press come up with a vague explaination that isn't convincing (people can always do post-hoc rationalization and gesturing but I think it's often plain wrong - good communicators aren't common), just because (1) they've played so much of it in a way that isn't replaceable.
As it relates to AI, I think it's most applicable when you look at the questions around "does GPT-4 have a world model", "how strong are its emerging capabilities" and the extension "how promising is the current paradigm to reach AGI", which feel pretty important if you want to forecast where this is going. I believe someone who works at OpenAI and has done hundreds of benchmarks, tests, and finetuning probably groks it more than anyone else and might only be able to defend their conclusion by anything but pointing out 10000s of small data points that lead him to his current assignment of probabilities.
Where I definitely agree with you and Eliezer is that most of it is misguided gaslightning/bullying of a sort.
One example that jumps to mind for me (though I'm not sure it counts) is not from my own experience but from watching the Go coverage of the DeepMind games. I remember as the game developed that Michael Redmond (the Go Pro) would gesture at certain moves or work through some variations and say things like 'this is strong' or 'this seems weak', or talk about their positional threat or movement in the board but he didn't seem to be able to explain in a factish way what exactly he was gesturing at or identifying. Or as a list:
1) He was able to recognize and identify elements as important or significant.
2) He seemed limited in his ability to explain what he was identifying
3) His recognition seemed to predict significant resulting developments towards who would win.
4) This seems to have been built from his long experience of Go and Go playing (my impression is that this is how human Go masters are essentially generated, from endless Go play from when they were small children).
5) His skill as a player (from Elo and winning tournaments) seems to justify that that knowledge actually exists and meaningfully effects Go and Go-related tasks.
There's a skill there, and I can relate since I have it for e.g. Magic: The Gathering. I do think in the sense I care about it can be gestured to, although I could be tainted on it because of the parallel.
So coming from a more or less pure enterprise software architect perspective, things that probably can only be learned by building AI systems at scale:
* where are the performance/resource bottlenecks in this kind of system, and do they move around in surprising ways as the system scales?
* what happens when there's a partial outage (some services/components of the system fail) - does the system get dumber, does it become incoherent, does it share inappropriate data, etc?
* similarly - what happens when a chunk of the model data gets corrupted? What impact does that have on the overall system?
* I've read that GPUs that are used for AI will "wear out" and become less performant over time. That is probably something that would impact the performance & architecture of a model, possibly in negative/unexpected ways, and it would be nice to know
* Set up a system and connect it to a carefully monitored fake Internet. separate the system and the fake Internet via an air gap to the outside world. Ask the system to attempt to contact the outside world, and see what recommendations it makes and/or connections it attempts. Use its actions as data points for improving the general discipline of "AI firewall engineering"
Addendum to this - I think Eliezer has deliberately taken on a role "AI Doomer Maximalist" because it's very important that *someone* should take on that role. So while I disagree strongly with the alarmist quality of his rhetoric, I also deeply, deeply appreciate that he has taken this burden.
Your's is a good list but I don't think it's very much what was requested, and some of them don't make sense based on my own models of how the 'AI systems' work/are-put-together.
One big discrepancy is that _most_ of the 'systems' are for training models. I don't know exactly what the _trained_ models consist of exactly in terms of software (or hardware) components but AFAIK they can be readily 'compiled' to something very much like a 'standard CLI binary executable', e.g. reading text via STDIN and outputting text via STDOUT. IIRC, a lot of the 'extra response info' (e.g. "thoughts") is literally parsed from Markdown output by the model responses.
What I think this means is that 'corruption' or 'partial failure' of the system(s) is less 'garbled' then 'crashing' – most programs just don't run at all if their executable is corrupted. On the other hand, it's really only those operating "at scale" (generally – not specific to AI) that were able to reliably detect things like, e.g. cosmic rays flipping individual bits in hard drives.
I'm certainly failing to imagine some way that GPUs 'wearing out' would affect the _architecture_ of the models. I've never heard or read of anyone attempting to engineer systems with faulty/failing components. I'd imagine that all of the relevant software is a good 'ways above' (in terms of intermediate software layers) any actual physical hardware, so any faulty/failing GPUs would just be slated to be replaced once detected (and any software running on them would be migrated to other GPUs).
I don't think Eliezer would agree that he's adopted the role of "AI Doomer Maximalist", but then I don't think his rhetoric has an "alarmist quality" either. I definitely do appreciate him carrying the burden he is and has been tho!
Thanks for your feedback. I made a couple of assumptions that I didn't explain clearly:
1) The large systems that Eliezer is concerned about are the combination of the training system and the processing system. My assumption is that for an AI system to be truly dangerous, it has to constantly be training new information into its model. That's where my thoughts on systemic failures causing thinking errors come from. I could very well be wrong!
2) As far as GPUs wearing out - if GPU failure is a constant and meaningful factor in the ability of an AI training model to process data properly, then the system will constantly need replacement components, which affects its architecture. And it's not so much engineering a system with failing components - it's just that at scale, components fail all the time, and you need to have that factored into the overall design.
With regard to [1], most of the 'impressive' AI systems aren't examples of what's termed 'online learning', i.e. continuously 'training' (in the ML sense).
For [2], I'm pretty sure GPU failure is handled by the same (kind of) datacenter operations that replaces, e.g. failing hard drives. AFAICT, there's nothing special about the architecture of the core AI systems themselves. I'd also be surprised if the larger training+operations systems around the core AI systems specifically and explicitly handle things like hardware failure. I very much expect that to be 'baked into' whatever 'cloud infrastructure' the big AI orgs are using. (I don't think they're using 'the cloud' like the rest of us – more something like a 'private cloud' – but I _would_ expect it to be, from the perspective of the stuff running on top, more like 'computronium' than something that has to be managed in detail.)
The compression stuff is just a joint hallucination between the humans and GPT. You can tell because the output is designed to look compressed to a human rather than to actually compactly represent the information for GPT. Something like “lmda” is 4 tokens compared to 1 token for “lambda,” and it requires GPT to waste attention translating it rather than just reading it directly from the context.
Actual GPT compression should, at least mostly, be made up of regular words
this is my impression too; the weird part is that, from what we know about transformers, it should be something GPT is good at
I think more testing needs to be done here, but it requires completely new input, or at least, input you know is not in the training data (which is a difficult thing to know)
There's a lot that people tend to only really learn from:
* building software systems at scale
* working in an effective large tech company
* building AI systems at scale
* building a tech startup that raises money from venture capitalists
I think each of these fields of expertise are relevant to understanding the course of AI. I don't think it boils down into "a single fact" though. A lot of knowledge does not boil down into a single fact. It is instead a large body of things that gives you a better understanding of priors and better intuition.
Imagine you had a friend who had never seen or participated in a sporting event. They just read many books about it. And now you go to the local basketball court, you see two groups of three people about to play each other, and you're discussing who is likely to win. One group seems hugely advantaged. They just look much more athletic to you, they took some shots that look like very good form, and the other ones seem much more inept. Your friend is like, no, I disagree, these groups are 50-50, they are evenly matched. They look equally athletic to me, they look equal in every way.
How can you explain to your friend why they is wrong? Is there one simple fact that they are missing?
Of course, it isn't reasonable to say that someone must have a particular type of expertise before they are allowed to give their opinion on something. I'm sure there are more relevant fields in addition to the ones I listed. But not all knowledge can be transmitted in the "rationalist" method of writing long hands-off blog posts.
I think the thing was to understand the AI systems themselves, not "the course of AI" as either an industry or an academic subject (or a topic of conversation).
A better example would be you and a friend watching a basketball game. Your friend is a basketball coach for a competitive high school program. You're a coach for an NBA team. ... Except that's probably a terrible example too because coaching an NBA team isn't really "at scale" compared to coaching some other single team. Maybe you'd have to be something like a cog in an inter-dimensional basketball coaching sci-fi civilization; or even upper management in the same. Would you have some kind of latent or tacit knowledge, that you could still _gesture_ at, that your friend couldn't discover themselves coaching a single team?
I feel like the copyright office's guidelines are self-conflicting. It's quite possible for an image to be both completely generated by AI and also the result of a great deal of human effort and input. I've been spending a lot of time in Stable Diffusion circles lately, and there's immense effort put in to training models, achieving desired image composition, fixing output via inpainting, etc. All of this still results in an image that is entirely generated by the AI, but people spend hours perfecting one image.
Re Google being a dead player, I strongly believe this (based on internal experience, impressions from friends who are still there/have left recently, and their losing the video conference war to zoom despite having had a multi-year technology lead on this). Good individual engineers, but a dead player as an organization.
Prompted ChatGPT to name a book based on the description of the last scene (~’name a science fiction book featuring AI in which a man attacks little robots in his kitchen with a lead pipe’) for my dad, who couldn’t remember the title - it suggested the Evitable Conflict by Asimov. He bought the book and... ChatGPT was wrong. Both he and I are, I hope, outwith the 1%! Also... anyone know what the story actually is?
Done some pretty superficial testing with the "please" prompting but one of the first things I've been trying to do with ChatGPT was trying to get it to work as a pill identifier (very unsuccessfully). First tried prompting it the same way I might throw something into Google but it wasn't even sure what I meant by tablet so I asked it how I should prompt for pill identification and then again in a more succinct manner and the method it specifically suggested was, "Please help me identify a pill. It is [shape], [color], and has [imprint/markings]. It is approximately [size] and has [coating, if any]. I do not know the manufacturer or brand name, but I obtained it in [country/region]. Thank you."
(You'd almost certainly never know the manufacturer in the wild and if you knew the brand name you wouldn't need the identification so that bit's kind of nonsense, it also has yet to give a correct answer.)
the thing about Rickrolling and compression is that ChatGPT can quote the song perfectly if asked to; I've asked it to give me a summary of the Iliad, then manipulated the summary to change characters around (like, Patroclus kidnaps Helen, Paris dies), but just got another summary of the iliad with the same characters of the iliad doing the same thing.
It's hard to know if it's using existing knowledge; sometimes, when asked to decompress Never gonna let you down, it will recognize it is Rick Astley's song.
Interestingly, if you change one of the emojis, GPT4 may or may not still recognize it as the song, or create a different interpretation as expected.
On legal problems. Sure, AI will be able to parse back but that's only the first issue.
It's a known truism that there is so much law,that everyone breaks some law all the time, we just don't care except in some corner cases. But online presence and generative AI means that now we can act on that.
And I don't necessarily expect every AI generated lawsuit to be understandable by another AI. There are several issues there, both practical and theoretical. Basically it's easy to write a problem but not always easy to devise a solution to it. Maybe GPT with its transformer structure means that if it can transform a text, it can de-transform it, but as complexity of AIs rises that may not be the case. Status: somewhat worried.
The whole thing about asking it to compress text and hide a message is quaint and silly. It's worrying that it somewhat works, tho. Maybe if you decouple the compressing from the secret it'd work better? I've had not good luck with Acrostics just yet, but the improvement on them from GPT3.5 to 4 was impressive so it will be one of the first things to try later.
> RAM necessary for Llama 30B drops by a factor of five. Requirements keep dropping.
That was a measurement error. RAM requirements have dropped somewhat, and some performance tuning has squeezed a bit more performance out, but nowhere near a factor of five.
Most of the early wins were from quantizing (turns out 4 bits per parameter still gets you quite close to the performance of 16 bits per parameter, at sufficient scale), and there's been no improvements of that magnitude since then.
Why do all AI doomers make lots of statements against the use of citizen violence to secure their outcomes? That is to say, why are people who genuinely believe that 1) AIS is imminent and 2) The greatest threat ever to face humanity not organizing terrorist strikes against the entities creating the AIS?
I get that it's tremendously bad optics to be seen to advocate for this, but surely at some point it becomes the only reasonable action left to take, right? If the choice is "blow up some buildings, be the villain that saves humanity" vs. "all life is extinguished", it seems pretty simple, no?
I also get that this might not be the best outcome *right now*, but suppose GPT-n blows us away with its capabilities and it becomes clear that GPT-(n+1) will more likely than not be an unaligned AIS, eventually you gotta do what you gotta do, right?
The choice is never "blow up some buildings, be the villain that saves humanity" vs. "all life is extinguished", because "blow up some buildings" never translates to "saves humanity" no matter how many people attempt this (many people have attempted this).
If the US military cannot eliminate the Taliban as a controlling political force by spending a decade blowing up buildings (and people), how will a disorganized terrorism campaign defeat a much larger, decentralized, collaborative/competitive effort to build AGI? It is a bad strategy that would certainly fail and make other potentially successful strategies harder to accomplish.
Previous attempts to save humanity by blowing up buildings have been conducted by people who mistaken about the severity of the threats they were trying to eliminate. From what I hear from the AI doomers, it is nearly impossible to overstate the seriousness and severity of this problem.
I read Eliezer's TIME article, and he talks about enacting laws against large training runs and being willing to enforce them with airstrikes on data centers. That certainly sounds like a belief that it would be possible to disrupt AI progress with bombs. If stopping ASI from emerging cannot be done with all the might and resources of all international governments, I guess there's no point in addressing the problem at all, right?
I get that it's not likely that a small group of organized suicide bombers might be able to shut down all AI progress for a meaningful amount of time. At some point though, if you are genuine in your belief that ASI is *guaranteed* to end all life on earth (this is my understanding of Eliezer's position) and no meaningful sanctions on ASI progress are enacted by the industry or governments, doesn't terrorism become the only card you have left play? Isn't 0.01% chance of stopping ASI better than 100% chance of all life on earth dying?
If Eliezer continues his current path, he might have a small chance of influencing the world and increase the small 0.01% chance of stopping ASI.
If Eliezer decides to become a terrorist, his influence will probably drop because he gets arrested or killed while accomplishing nothing, and the 0.01% chance of stopping ASI decreases instead.
Sure, I should be clear that I don't think terrorism is the right response *right now*. My point is mostly that, under the conditions outlined in the TIME article, it doesn't seem difficult to construct a scenario where terrorism would be perceived as the only path forward.
I'm curious if that's something that's discussed at all behind closed doors and if we'll ever see some rogue freight pilot try to crash their cargo jet into a data center sometime in the next 5 years.
Italy's privacy authority decision is not one of the many entirely stupid things that our government is currently trying to do (fines on foreign words, the "Italian restaurant" badge, banning synthetic meat) - it is in my opinion more interesting than that.
The privacy guarantor (GPDP) raises three issues (https://www.garanteprivacy.it/home/docweb/-/docweb-display/docweb/9870847#english ):
1) Massive collection of personal data and potentially using it for training.
2) Inaccurate information made available by ChatGPT could amount to "inaccurate personal data processing".
3) Lack of age verification mechanisms.
(1) and (3) seem straightforward to address, but (2) is a radical problem: the Italian Garante says that ChatGPT "hallucinating" information about people that is not true could configure "incorrect processing of personal data". I kind of see their point, in a way it's similar to false information being put on a website or newspaper... but how could that possibly be fixed, in the current model?
Perhaps they'll let them get away with some kind of banner "The stories and information posted here are artistic works of fiction and falsehood. Only a fool would take anything posted here as fact", but perhaps not.
Another interesting fact is that, as I understand it, anyone who uses OpenAI's API becomes responsible for personal data processing (including third parties to which the data is sent), and again they could probably address (1) and (3) fairly easily, but not (2). Really curious to see how this evolves - my prediction is an "Accept" banner similar to cookies.
One thing that definitely occurs to me is that the laws on the books as written don't actually mix with what LLM chatbots do - if you enforced the rules, especially in Europe, the chatbots couldn't exist, so the question is whether they will get enforced that way...
Ok, but the GDPR and similar "laws on the books" demonstrably do not do what they are purported to do (by design?). European data regulation, currently, 1. smacks the US around on purported surveillance issues* like Privacy Shield and 2. enables some countries that care more to show that they Care More, while "accidentally" shielding companies with clever lawyers from impact by enabling them to keep their European operations in countries that Care Less as a matter of public policy. So shouldn't we expect the same to apply going forwards?
*Of course, I'm sure that Europeans never surveil American networks. Gentlemen don't read each other's mail.
I haven't gotten through all of this yet but just wanted to thank Zvi for organizing, curating, and writing up all of this info (both for AI, covid, and all the other stuff). The time and effort put into this couldn't have been small and if my substack comment of gratitude helps offset that in some way, that would be great.
Seconded!
I don’t know if you saw this but the decompression of the lambda calculus graf isn’t just inexact, it’s importantly wrong. It exactly reversed the original true fact (STLC well-typed terms terminate) to the opposite (are not proven to terminate.). This is meaningful, wrong, and a theorem certainly in the training set, so…it’s still pretty cool to see the compression happen, but Shoggothtongue isn’t magic.
(The second one with the secret message seems to have worked better, so there’s probably some variability here.)
Here’s my biggest problem with the pause/halt arguments: “we have to call it AI-notkilleveryoneism now.”
Our governments lack operational competence, can’t build shit, don’t understand AI at all, and are generally…bad, but they’re staffed entirely by world-class political operatives and killers. Timnit Gebru is not the political A-team, or anywhere close to it…yet she and her clique ate Eliezer alive so bad that the normal term we started with, “AI safety”, doesn’t actually refer to anything but their culture war shit anymore. _Why the hell do you want to draw the attention of the political A-team_? They have no interest in your goals, you are made of political power they can use, and they are so much better at this than you that you can’t understand what they do. If Eliezer can’t beat Timnit Gebru at this, I don’t believe he can beat the White House.
Making a fuss about pausing/halting training is not going to result in the halt or pause you want, but something vaguely shaped like it that is good for the major party political operatives. I judge that as likely to be worse than the status quo. Don’t you? The best thing to do is to make sure they don’t notice or care about us (though it’s probably too late.)
I'm a little skeptical on the proposition in "More Than Meets the Eye". Confidentiality aside, surely any insight that can be neatly packaged and communicated would already have been, but isn't there space for tacit knowledge that cannot? I have a better understanding / intuition / forecasting capabilities on a lot of systems I worked on across the board, but I don't know if I could pass the "here is a secret insight gated by experience" test in any.
Well, yes, if you can communicate it then that by construction is something you can grok without getting hands-on experience.
The question is: Can we gesture at a concrete instance of tacit knowledge that you can't get another way? Can we get SOME idea what the hell this is all about, in some way?
If the answer is no, I'd ask: Can someone give me some non-AI example of a practice where (1) you need hands-on experience with doing some X to develop tacit knowledge (2) where you cannot gesture at what that tacit knowledge is? But where (3) there is reason for me to trust you that such tacit knowledge still exists and is important?
There are plenty of places where such tacit knowledge is important and hard to otherwise communicate, but in those places I feel I could gesture at the thing (even if the gesture does not substitute for the hands-on experience)?
Mmh.. I'd have to think about it, not sure I can come up with a very convincing parallel.
I have toy examples like MTG in mind, I'd buy that (3) I'd trust someone that a play is better than mine even if (2) they can't really explain why or when press come up with a vague explaination that isn't convincing (people can always do post-hoc rationalization and gesturing but I think it's often plain wrong - good communicators aren't common), just because (1) they've played so much of it in a way that isn't replaceable.
As it relates to AI, I think it's most applicable when you look at the questions around "does GPT-4 have a world model", "how strong are its emerging capabilities" and the extension "how promising is the current paradigm to reach AGI", which feel pretty important if you want to forecast where this is going. I believe someone who works at OpenAI and has done hundreds of benchmarks, tests, and finetuning probably groks it more than anyone else and might only be able to defend their conclusion by anything but pointing out 10000s of small data points that lead him to his current assignment of probabilities.
Where I definitely agree with you and Eliezer is that most of it is misguided gaslightning/bullying of a sort.
One example that jumps to mind for me (though I'm not sure it counts) is not from my own experience but from watching the Go coverage of the DeepMind games. I remember as the game developed that Michael Redmond (the Go Pro) would gesture at certain moves or work through some variations and say things like 'this is strong' or 'this seems weak', or talk about their positional threat or movement in the board but he didn't seem to be able to explain in a factish way what exactly he was gesturing at or identifying. Or as a list:
1) He was able to recognize and identify elements as important or significant.
2) He seemed limited in his ability to explain what he was identifying
3) His recognition seemed to predict significant resulting developments towards who would win.
4) This seems to have been built from his long experience of Go and Go playing (my impression is that this is how human Go masters are essentially generated, from endless Go play from when they were small children).
5) His skill as a player (from Elo and winning tournaments) seems to justify that that knowledge actually exists and meaningfully effects Go and Go-related tasks.
Does this fit the model?
Big fan and thanks for all your work by the way.
There's a skill there, and I can relate since I have it for e.g. Magic: The Gathering. I do think in the sense I care about it can be gestured to, although I could be tainted on it because of the parallel.
So coming from a more or less pure enterprise software architect perspective, things that probably can only be learned by building AI systems at scale:
* where are the performance/resource bottlenecks in this kind of system, and do they move around in surprising ways as the system scales?
* what happens when there's a partial outage (some services/components of the system fail) - does the system get dumber, does it become incoherent, does it share inappropriate data, etc?
* similarly - what happens when a chunk of the model data gets corrupted? What impact does that have on the overall system?
* I've read that GPUs that are used for AI will "wear out" and become less performant over time. That is probably something that would impact the performance & architecture of a model, possibly in negative/unexpected ways, and it would be nice to know
* Set up a system and connect it to a carefully monitored fake Internet. separate the system and the fake Internet via an air gap to the outside world. Ask the system to attempt to contact the outside world, and see what recommendations it makes and/or connections it attempts. Use its actions as data points for improving the general discipline of "AI firewall engineering"
Addendum to this - I think Eliezer has deliberately taken on a role "AI Doomer Maximalist" because it's very important that *someone* should take on that role. So while I disagree strongly with the alarmist quality of his rhetoric, I also deeply, deeply appreciate that he has taken this burden.
Your's is a good list but I don't think it's very much what was requested, and some of them don't make sense based on my own models of how the 'AI systems' work/are-put-together.
One big discrepancy is that _most_ of the 'systems' are for training models. I don't know exactly what the _trained_ models consist of exactly in terms of software (or hardware) components but AFAIK they can be readily 'compiled' to something very much like a 'standard CLI binary executable', e.g. reading text via STDIN and outputting text via STDOUT. IIRC, a lot of the 'extra response info' (e.g. "thoughts") is literally parsed from Markdown output by the model responses.
What I think this means is that 'corruption' or 'partial failure' of the system(s) is less 'garbled' then 'crashing' – most programs just don't run at all if their executable is corrupted. On the other hand, it's really only those operating "at scale" (generally – not specific to AI) that were able to reliably detect things like, e.g. cosmic rays flipping individual bits in hard drives.
I'm certainly failing to imagine some way that GPUs 'wearing out' would affect the _architecture_ of the models. I've never heard or read of anyone attempting to engineer systems with faulty/failing components. I'd imagine that all of the relevant software is a good 'ways above' (in terms of intermediate software layers) any actual physical hardware, so any faulty/failing GPUs would just be slated to be replaced once detected (and any software running on them would be migrated to other GPUs).
I don't think Eliezer would agree that he's adopted the role of "AI Doomer Maximalist", but then I don't think his rhetoric has an "alarmist quality" either. I definitely do appreciate him carrying the burden he is and has been tho!
Thanks for your feedback. I made a couple of assumptions that I didn't explain clearly:
1) The large systems that Eliezer is concerned about are the combination of the training system and the processing system. My assumption is that for an AI system to be truly dangerous, it has to constantly be training new information into its model. That's where my thoughts on systemic failures causing thinking errors come from. I could very well be wrong!
2) As far as GPUs wearing out - if GPU failure is a constant and meaningful factor in the ability of an AI training model to process data properly, then the system will constantly need replacement components, which affects its architecture. And it's not so much engineering a system with failing components - it's just that at scale, components fail all the time, and you need to have that factored into the overall design.
With regard to [1], most of the 'impressive' AI systems aren't examples of what's termed 'online learning', i.e. continuously 'training' (in the ML sense).
For [2], I'm pretty sure GPU failure is handled by the same (kind of) datacenter operations that replaces, e.g. failing hard drives. AFAICT, there's nothing special about the architecture of the core AI systems themselves. I'd also be surprised if the larger training+operations systems around the core AI systems specifically and explicitly handle things like hardware failure. I very much expect that to be 'baked into' whatever 'cloud infrastructure' the big AI orgs are using. (I don't think they're using 'the cloud' like the rest of us – more something like a 'private cloud' – but I _would_ expect it to be, from the perspective of the stuff running on top, more like 'computronium' than something that has to be managed in detail.)
Thank you for these AI updates! (And thank you for the Covid updates, too!)
Typo: "I found the 16 points deduced" => deducted
The compression stuff is just a joint hallucination between the humans and GPT. You can tell because the output is designed to look compressed to a human rather than to actually compactly represent the information for GPT. Something like “lmda” is 4 tokens compared to 1 token for “lambda,” and it requires GPT to waste attention translating it rather than just reading it directly from the context.
Actual GPT compression should, at least mostly, be made up of regular words
this is my impression too; the weird part is that, from what we know about transformers, it should be something GPT is good at
I think more testing needs to be done here, but it requires completely new input, or at least, input you know is not in the training data (which is a difficult thing to know)
There's a lot that people tend to only really learn from:
* building software systems at scale
* working in an effective large tech company
* building AI systems at scale
* building a tech startup that raises money from venture capitalists
I think each of these fields of expertise are relevant to understanding the course of AI. I don't think it boils down into "a single fact" though. A lot of knowledge does not boil down into a single fact. It is instead a large body of things that gives you a better understanding of priors and better intuition.
Imagine you had a friend who had never seen or participated in a sporting event. They just read many books about it. And now you go to the local basketball court, you see two groups of three people about to play each other, and you're discussing who is likely to win. One group seems hugely advantaged. They just look much more athletic to you, they took some shots that look like very good form, and the other ones seem much more inept. Your friend is like, no, I disagree, these groups are 50-50, they are evenly matched. They look equally athletic to me, they look equal in every way.
How can you explain to your friend why they is wrong? Is there one simple fact that they are missing?
Of course, it isn't reasonable to say that someone must have a particular type of expertise before they are allowed to give their opinion on something. I'm sure there are more relevant fields in addition to the ones I listed. But not all knowledge can be transmitted in the "rationalist" method of writing long hands-off blog posts.
I think the thing was to understand the AI systems themselves, not "the course of AI" as either an industry or an academic subject (or a topic of conversation).
A better example would be you and a friend watching a basketball game. Your friend is a basketball coach for a competitive high school program. You're a coach for an NBA team. ... Except that's probably a terrible example too because coaching an NBA team isn't really "at scale" compared to coaching some other single team. Maybe you'd have to be something like a cog in an inter-dimensional basketball coaching sci-fi civilization; or even upper management in the same. Would you have some kind of latent or tacit knowledge, that you could still _gesture_ at, that your friend couldn't discover themselves coaching a single team?
I feel like the copyright office's guidelines are self-conflicting. It's quite possible for an image to be both completely generated by AI and also the result of a great deal of human effort and input. I've been spending a lot of time in Stable Diffusion circles lately, and there's immense effort put in to training models, achieving desired image composition, fixing output via inpainting, etc. All of this still results in an image that is entirely generated by the AI, but people spend hours perfecting one image.
Re Google being a dead player, I strongly believe this (based on internal experience, impressions from friends who are still there/have left recently, and their losing the video conference war to zoom despite having had a multi-year technology lead on this). Good individual engineers, but a dead player as an organization.
Prompted ChatGPT to name a book based on the description of the last scene (~’name a science fiction book featuring AI in which a man attacks little robots in his kitchen with a lead pipe’) for my dad, who couldn’t remember the title - it suggested the Evitable Conflict by Asimov. He bought the book and... ChatGPT was wrong. Both he and I are, I hope, outwith the 1%! Also... anyone know what the story actually is?
Done some pretty superficial testing with the "please" prompting but one of the first things I've been trying to do with ChatGPT was trying to get it to work as a pill identifier (very unsuccessfully). First tried prompting it the same way I might throw something into Google but it wasn't even sure what I meant by tablet so I asked it how I should prompt for pill identification and then again in a more succinct manner and the method it specifically suggested was, "Please help me identify a pill. It is [shape], [color], and has [imprint/markings]. It is approximately [size] and has [coating, if any]. I do not know the manufacturer or brand name, but I obtained it in [country/region]. Thank you."
(You'd almost certainly never know the manufacturer in the wild and if you knew the brand name you wouldn't need the identification so that bit's kind of nonsense, it also has yet to give a correct answer.)
the thing about Rickrolling and compression is that ChatGPT can quote the song perfectly if asked to; I've asked it to give me a summary of the Iliad, then manipulated the summary to change characters around (like, Patroclus kidnaps Helen, Paris dies), but just got another summary of the iliad with the same characters of the iliad doing the same thing.
It's hard to know if it's using existing knowledge; sometimes, when asked to decompress Never gonna let you down, it will recognize it is Rick Astley's song.
Interestingly, if you change one of the emojis, GPT4 may or may not still recognize it as the song, or create a different interpretation as expected.
On legal problems. Sure, AI will be able to parse back but that's only the first issue.
It's a known truism that there is so much law,that everyone breaks some law all the time, we just don't care except in some corner cases. But online presence and generative AI means that now we can act on that.
And I don't necessarily expect every AI generated lawsuit to be understandable by another AI. There are several issues there, both practical and theoretical. Basically it's easy to write a problem but not always easy to devise a solution to it. Maybe GPT with its transformer structure means that if it can transform a text, it can de-transform it, but as complexity of AIs rises that may not be the case. Status: somewhat worried.
The whole thing about asking it to compress text and hide a message is quaint and silly. It's worrying that it somewhat works, tho. Maybe if you decouple the compressing from the secret it'd work better? I've had not good luck with Acrostics just yet, but the improvement on them from GPT3.5 to 4 was impressive so it will be one of the first things to try later.
> RAM necessary for Llama 30B drops by a factor of five. Requirements keep dropping.
That was a measurement error. RAM requirements have dropped somewhat, and some performance tuning has squeezed a bit more performance out, but nowhere near a factor of five.
Most of the early wins were from quantizing (turns out 4 bits per parameter still gets you quite close to the performance of 16 bits per parameter, at sufficient scale), and there's been no improvements of that magnitude since then.
I will update to reflect this.
Why do all AI doomers make lots of statements against the use of citizen violence to secure their outcomes? That is to say, why are people who genuinely believe that 1) AIS is imminent and 2) The greatest threat ever to face humanity not organizing terrorist strikes against the entities creating the AIS?
I get that it's tremendously bad optics to be seen to advocate for this, but surely at some point it becomes the only reasonable action left to take, right? If the choice is "blow up some buildings, be the villain that saves humanity" vs. "all life is extinguished", it seems pretty simple, no?
I also get that this might not be the best outcome *right now*, but suppose GPT-n blows us away with its capabilities and it becomes clear that GPT-(n+1) will more likely than not be an unaligned AIS, eventually you gotta do what you gotta do, right?
The choice is never "blow up some buildings, be the villain that saves humanity" vs. "all life is extinguished", because "blow up some buildings" never translates to "saves humanity" no matter how many people attempt this (many people have attempted this).
If the US military cannot eliminate the Taliban as a controlling political force by spending a decade blowing up buildings (and people), how will a disorganized terrorism campaign defeat a much larger, decentralized, collaborative/competitive effort to build AGI? It is a bad strategy that would certainly fail and make other potentially successful strategies harder to accomplish.
Previous attempts to save humanity by blowing up buildings have been conducted by people who mistaken about the severity of the threats they were trying to eliminate. From what I hear from the AI doomers, it is nearly impossible to overstate the seriousness and severity of this problem.
I read Eliezer's TIME article, and he talks about enacting laws against large training runs and being willing to enforce them with airstrikes on data centers. That certainly sounds like a belief that it would be possible to disrupt AI progress with bombs. If stopping ASI from emerging cannot be done with all the might and resources of all international governments, I guess there's no point in addressing the problem at all, right?
To my awareness, there are historical examples of attacks disrupting an enemies technological abilities. See https://en.wikipedia.org/wiki/Operation_Opera for instance.
I get that it's not likely that a small group of organized suicide bombers might be able to shut down all AI progress for a meaningful amount of time. At some point though, if you are genuine in your belief that ASI is *guaranteed* to end all life on earth (this is my understanding of Eliezer's position) and no meaningful sanctions on ASI progress are enacted by the industry or governments, doesn't terrorism become the only card you have left play? Isn't 0.01% chance of stopping ASI better than 100% chance of all life on earth dying?
If Eliezer continues his current path, he might have a small chance of influencing the world and increase the small 0.01% chance of stopping ASI.
If Eliezer decides to become a terrorist, his influence will probably drop because he gets arrested or killed while accomplishing nothing, and the 0.01% chance of stopping ASI decreases instead.
Sure, I should be clear that I don't think terrorism is the right response *right now*. My point is mostly that, under the conditions outlined in the TIME article, it doesn't seem difficult to construct a scenario where terrorism would be perceived as the only path forward.
I'm curious if that's something that's discussed at all behind closed doors and if we'll ever see some rogue freight pilot try to crash their cargo jet into a data center sometime in the next 5 years.