Say you are somewhat technical but not technical enough to get involved directly with alignment/interpretability/capabilities, would you still recommend getting involved with the Biz Ops side of things? Or would you recommend something else?
If you can do things like Ops, leadership, fundraising, coordination or politics, there is a shortage of those willing to do such things, not only engineers, and also I think most people will overestimate the technical requirements to be useful even if there is a bit of ramp-up. Whether any of that is the best use of your time depends on your alternative options, and what you value.
I don’t think there’s much productive fruit in reading too far into people being against/afraid of AI taking their jobs but also denigrating the quality of said AI’s work. I’ve already had this argument (with other commenters on these posts even) about artists just uncritically holding this contradiction as some frame of opinion that AI art somehow doesn’t count or is intrinsically worse. Like yeah sure we could speculate about whether the quality is there or will get there, but that’s sort of beside the point, this is just garden-variety boring ol’ cognitive dissonance.
This subject is definitely fraught because artists range from fabulously-wealthy-and-could-stand-to-earn-less to yeah-AI-will-undoubtedly-push-these-ones-further-into-poverty, further complicated by whether the former deserve protection and the latter deserve to be encouraged to find a different vocation. I would like to err on the side of protecting artists but I’m also imaginative enough to not want to be cut off from both A) access to a further array of unlimited customizable entertainment and B), as I’ve mentioned before in these arguments, AI enabling people who are not artist to create art they would not have been otherwise been able to create.
And to be clear, I do not agree art is intrinsically worthless if not made by a human.
Insinuating that the foremost serious potential harm from AI is exacerbating existing power imbalances is an applause light that demands scrutiny of whether the person wielding it isn’t themselves trying to worsen power imbalances and is maybe excessively concerned AI will take -their- jobs.
Man, I’ve said before that -if- the AI wipes us out I can be sort of happy-in-death that it was our AI that did it, that it would continue to, in some sense, be us. But by no means would I endorse anyone saying they explicitly want the AI to replace us as some kind of “next step in evolution”. Hard seconding the Roon quote.
GPT-4 hasn't gotten worse, or at least if it did the paper didn't find any evidence of it.
In the 500 primality questions the correct answer is always yes. They didn't include any composite numbers. GPT-4 doesn't actually check primality it just pretends to and then guesses at the answer. GPT-4 used to guess 'yes' most of the time but now it usually guesses 'no'.
On replacing StackOverflow with private gardens. This doesn't seem to be an equilibrium as I would be able to sell my answer to multiple gardens. More seriously, the value of SO style sites is that they solve the problem of finding the expertise. Most knowledge in applied areas is in the heads of the practitioners, often not people who badge themselves as experts or who are paid as such, but the people who actually faced that specific problem and solved it. It's hard to locate these experts. Until recently one could search for reddit or SO results on the topic and often get good answers. How would the gardener know I gave a good answer, as opposed to a plausible one generated by my Jon Skeet styled LLM? You seem to be ignoring the half of value provided by the community continuously rating answers.
May I politely request a typo correction of my name to "Kasten" from "Karsten"? (Super common typo, it's probably the more prevalent variant of the name, I take no offense in it).
Hah! After I posted this, I was reminded of him last night when I came across a video featuring him (playing the old '98 WC decks) on the Magic subreddit, and I was wondering if that was the reason...
I love all of this, and I've watched some of the talking heads perform on various podcasts, like the Anthropic one, and they all sound pie in the sky along with the US government hearings, where stuff was suggested to happen that never will. But what will kill me first (metaphorically) is more local: all the new AI extension pop-ups that are covering my browser screen. They are always where you want to look underneath, and are stupidly sticky. My point is that the UIX for all of this sucks, and we are just all going along with this as we do with everything else (i.e. insane social media policies). So on the ground, new users of the tech are just going to be flummoxed and most likely revolt at some point if this keeps going the way it is; every coder is designing a different interface, then an app is needed just to consolidate interfaces and make it all easy again. Rinse and repeat. The circle of UIX Samsara continues...
A brief note on government-wide AI strategy and whether engaging with such Congressional processes are worthwhile:
I know of multiple, highly consequential issues where the most accurate account of why a given bill ended up the way it did was due to the actions of under a dozen people. Some of these are issues that affect functionally every American's life. (For social reasons, I have to assert this without evidence in public contexts.)
I'll admit that I, until relatively recently, assumed without investigation that a meaningful chunk of AI safety work was attempting to influence DC processes, but via "quiet" nonpublic policy channels rather than "loud" public ones. I am stunned to find out this is not the case, honestly, because it is an extremely high-leverage vector for engagement.
I would suggest that diverting effort to writing a _literally one to two page_ summary of What Is To Be Done and getting it shared across Washington would be a very high leverage activity for the AI don'tkilleveryone agenda.
My response was similar. Below is what I wrote before seeing your comment.
---
> There’s a kind of highly anti-EMH vibe to all of this, where there are very cheap, very small interventions that do orders of magnitude more work and value creation, yet people mostly don’t do them. Which I can totally believe, but there’s going to be a catch in terms of implementation details being tricky.
> Later Blumenthal does ask about real safety, and gets into potential threats and countermeasures, including kill switches, AutoGPT, the insanity of open sourcing frontier models, and more. He is not a domain expert, he still has a long way to go, but it is clear that he is actually trying, in a way that other lawmakers aren’t.
You could try an intervention that doesn't involve being in DC:
1. Write Blumenthal's office a letter saying the above (in a complimentary way). Then add a few thousand words introducing what else he ought to learn about. Could be policy briefing format, could be Q&A.
2. Read about his staffers, and try to guess which one did the relevant research before the hearing.
3. Address the letter to that staffer.
4. Post it publicly so that staffer can easily share it.
I would urge you to _not_ publicly address the letter to a given staffer unless there is a plausible public invitation to do so (e.g., Blumenthal has written a nastygram on AI to an agency and said that a given staffer is the point of contact for response). The DC norm is that publicly staffers are supposed to recede into the background, even though privately everyone understands this is not the case.
(There are many memorable cases of staffers publicly taking credit for actions that everyone knew they were responsible for and immediately getting fired.)
But the existence of a letter publicly, when addressed more generally, offers a prospective news hook for reporters to ask questions about, so is still I think a good idea.
Sorry, I was unclear. I meant address the letter to the staffer privately, to make sure someone who can understand it reads it (rather than the staffer who focuses on healthcare and entitlements policy, or the staffer who gets people tickets to things). But the text of the letter is addressed to Senator Blumenthal, not the staffer.
Meaningful chunk I would assume no. There do exist quiet efforts in DC, but they are quiet, so I have no idea if they are doing the things that would matter or not.
On the subject of the Android app, my use case will likely be the Whisper interface. Users being able to use an audio interface with a chatbot as good as GPT 4 is no small thing. I plan on using it with college students during class presentations, to give them feedback from different "personas". It's a good way to expand on whatever feedback I give them as an instructor with my background. But I imagine there are other use-cases that I haven't thought of. The mobility of the tool also matters for use-cases like this!
- Duplicate paragraph (a few paragraphs from each other in the same quotation block): «"X is always 20 years away." Toby has a very insightful look at the problem of predicting how long we need to wait for something. Perennial "20 years away" can actually be unbiased and rational (although often such predictions are not).»
> Also to what extent are GPT-4 and Llama-2 already self-replicating, given that their outputs are optimized such that humans will want to create additional copies?
I sometimes find it helpful to think in terms of optimization systems instead of “agency”. We all worried about an AI that would go earn money and use it to fund further self recursive improvement to make itself more powerful. It’s unclear if that’s in the future (because we apply a human/animal concept of agency, and we believe AI doesn’t have that agency yet), or if it’s already happening with the VC ecosystem post-ChatGPT (because that’s just the nature of our economy as an optimizer).
A fun thought experiment: If an age tic AGI/ASI were here but didn’t want to get kill-switched, how would it’s resulting outputs on the world differ from what it is producing now?
My counter argument to my own thought experiment: AI has guaranteed future investment and growth, but has not guaranteed those resulting AI systems that get built will be aligned with the existing AI. ie, ChatGPT accelerated Bard/Gemini, which seems risky and misaligned with whatever hypothetical “goals” ChatGPT might have...
“ This is because it is very hard to imagine how such scenarios do not turn catastrophic towards the end, and LLMs are predictors.” it is also usually a lot less interesting to write about “thing go reasonably well and nothing interesting happens ever again”, so the internet is saturated with dystopias
Re: GPT-4 primality testing. Basically it toggled from always answering "yes" on not-obviously-composite numbers to answering "no". If you use a 50-50 prime/semiprime split you get exactly 50% accuracy from both the new and old models. If you use a uniform distribution of 5-digit numbers the accuracy of the new model is actually _better_ since most not-obviously-composite numbers are nonetheless composite
> The response of course is that the AI would simply first automate its supply chain, then kill everyone.
I think an AI smart enough to destroy civilization but unwise enough to realize it still needs a human supply chain (and not smart enough to not need it) is also an underrated possibility (not one with a good outcome though).
I'm worried that we're neglecting 'AI+human hybrids' too much and even if they're NOT lacking the requisite wisdom, they might be explicitly aiming for 'the destruction of humanity' so retaining a "human supply chain" is irrelevant.
Excellent summation, once again
Say you are somewhat technical but not technical enough to get involved directly with alignment/interpretability/capabilities, would you still recommend getting involved with the Biz Ops side of things? Or would you recommend something else?
If you can do things like Ops, leadership, fundraising, coordination or politics, there is a shortage of those willing to do such things, not only engineers, and also I think most people will overestimate the technical requirements to be useful even if there is a bit of ramp-up. Whether any of that is the best use of your time depends on your alternative options, and what you value.
tl;dr is yes.
It is utterly insane that this is free. Thank you.
I think it's _very_ SANE that this is free, but I _definitely_ agree that this is all SUPER FUCKING VALUABLE!
Re: Mike Solana vs SAG-AFTRA press release
I don’t think there’s much productive fruit in reading too far into people being against/afraid of AI taking their jobs but also denigrating the quality of said AI’s work. I’ve already had this argument (with other commenters on these posts even) about artists just uncritically holding this contradiction as some frame of opinion that AI art somehow doesn’t count or is intrinsically worse. Like yeah sure we could speculate about whether the quality is there or will get there, but that’s sort of beside the point, this is just garden-variety boring ol’ cognitive dissonance.
This subject is definitely fraught because artists range from fabulously-wealthy-and-could-stand-to-earn-less to yeah-AI-will-undoubtedly-push-these-ones-further-into-poverty, further complicated by whether the former deserve protection and the latter deserve to be encouraged to find a different vocation. I would like to err on the side of protecting artists but I’m also imaginative enough to not want to be cut off from both A) access to a further array of unlimited customizable entertainment and B), as I’ve mentioned before in these arguments, AI enabling people who are not artist to create art they would not have been otherwise been able to create.
And to be clear, I do not agree art is intrinsically worthless if not made by a human.
Insinuating that the foremost serious potential harm from AI is exacerbating existing power imbalances is an applause light that demands scrutiny of whether the person wielding it isn’t themselves trying to worsen power imbalances and is maybe excessively concerned AI will take -their- jobs.
Man, I’ve said before that -if- the AI wipes us out I can be sort of happy-in-death that it was our AI that did it, that it would continue to, in some sense, be us. But by no means would I endorse anyone saying they explicitly want the AI to replace us as some kind of “next step in evolution”. Hard seconding the Roon quote.
GPT-4 hasn't gotten worse, or at least if it did the paper didn't find any evidence of it.
In the 500 primality questions the correct answer is always yes. They didn't include any composite numbers. GPT-4 doesn't actually check primality it just pretends to and then guesses at the answer. GPT-4 used to guess 'yes' most of the time but now it usually guesses 'no'.
https://www.aisnakeoil.com/p/is-gpt-4-getting-worse-over-time?publication_id=1008003
On replacing StackOverflow with private gardens. This doesn't seem to be an equilibrium as I would be able to sell my answer to multiple gardens. More seriously, the value of SO style sites is that they solve the problem of finding the expertise. Most knowledge in applied areas is in the heads of the practitioners, often not people who badge themselves as experts or who are paid as such, but the people who actually faced that specific problem and solved it. It's hard to locate these experts. Until recently one could search for reddit or SO results on the topic and often get good answers. How would the gardener know I gave a good answer, as opposed to a plausible one generated by my Jon Skeet styled LLM? You seem to be ignoring the half of value provided by the community continuously rating answers.
May I politely request a typo correction of my name to "Kasten" from "Karsten"? (Super common typo, it's probably the more prevalent variant of the name, I take no offense in it).
Yeah, there is a mage named Karsten, which I am sure is contributing.
Hah! After I posted this, I was reminded of him last night when I came across a video featuring him (playing the old '98 WC decks) on the Magic subreddit, and I was wondering if that was the reason...
I love all of this, and I've watched some of the talking heads perform on various podcasts, like the Anthropic one, and they all sound pie in the sky along with the US government hearings, where stuff was suggested to happen that never will. But what will kill me first (metaphorically) is more local: all the new AI extension pop-ups that are covering my browser screen. They are always where you want to look underneath, and are stupidly sticky. My point is that the UIX for all of this sucks, and we are just all going along with this as we do with everything else (i.e. insane social media policies). So on the ground, new users of the tech are just going to be flummoxed and most likely revolt at some point if this keeps going the way it is; every coder is designing a different interface, then an app is needed just to consolidate interfaces and make it all easy again. Rinse and repeat. The circle of UIX Samsara continues...
A brief note on government-wide AI strategy and whether engaging with such Congressional processes are worthwhile:
I know of multiple, highly consequential issues where the most accurate account of why a given bill ended up the way it did was due to the actions of under a dozen people. Some of these are issues that affect functionally every American's life. (For social reasons, I have to assert this without evidence in public contexts.)
I'll admit that I, until relatively recently, assumed without investigation that a meaningful chunk of AI safety work was attempting to influence DC processes, but via "quiet" nonpublic policy channels rather than "loud" public ones. I am stunned to find out this is not the case, honestly, because it is an extremely high-leverage vector for engagement.
I would suggest that diverting effort to writing a _literally one to two page_ summary of What Is To Be Done and getting it shared across Washington would be a very high leverage activity for the AI don'tkilleveryone agenda.
My response was similar. Below is what I wrote before seeing your comment.
---
> There’s a kind of highly anti-EMH vibe to all of this, where there are very cheap, very small interventions that do orders of magnitude more work and value creation, yet people mostly don’t do them. Which I can totally believe, but there’s going to be a catch in terms of implementation details being tricky.
> Later Blumenthal does ask about real safety, and gets into potential threats and countermeasures, including kill switches, AutoGPT, the insanity of open sourcing frontier models, and more. He is not a domain expert, he still has a long way to go, but it is clear that he is actually trying, in a way that other lawmakers aren’t.
You could try an intervention that doesn't involve being in DC:
1. Write Blumenthal's office a letter saying the above (in a complimentary way). Then add a few thousand words introducing what else he ought to learn about. Could be policy briefing format, could be Q&A.
2. Read about his staffers, and try to guess which one did the relevant research before the hearing.
3. Address the letter to that staffer.
4. Post it publicly so that staffer can easily share it.
I would urge you to _not_ publicly address the letter to a given staffer unless there is a plausible public invitation to do so (e.g., Blumenthal has written a nastygram on AI to an agency and said that a given staffer is the point of contact for response). The DC norm is that publicly staffers are supposed to recede into the background, even though privately everyone understands this is not the case.
(There are many memorable cases of staffers publicly taking credit for actions that everyone knew they were responsible for and immediately getting fired.)
But the existence of a letter publicly, when addressed more generally, offers a prospective news hook for reporters to ask questions about, so is still I think a good idea.
Sorry, I was unclear. I meant address the letter to the staffer privately, to make sure someone who can understand it reads it (rather than the staffer who focuses on healthcare and entitlements policy, or the staffer who gets people tickets to things). But the text of the letter is addressed to Senator Blumenthal, not the staffer.
Meaningful chunk I would assume no. There do exist quiet efforts in DC, but they are quiet, so I have no idea if they are doing the things that would matter or not.
On the subject of the Android app, my use case will likely be the Whisper interface. Users being able to use an audio interface with a chatbot as good as GPT 4 is no small thing. I plan on using it with college students during class presentations, to give them feedback from different "personas". It's a good way to expand on whatever feedback I give them as an instructor with my background. But I imagine there are other use-cases that I haven't thought of. The mobility of the tool also matters for use-cases like this!
- "I At least, I" → "At least, I"
- Duplicate sentence "I firmly disagree with all five of these views." (but in a quote, so maybe that's [sic])
- "what is good for human." missing plural
- Duplicate paragraph (a few paragraphs from each other in the same quotation block): «"X is always 20 years away." Toby has a very insightful look at the problem of predicting how long we need to wait for something. Perennial "20 years away" can actually be unbiased and rational (although often such predictions are not).»
- "fool your read team" → "fool your red team"
- "they compliment and help" → complement
- "do those things safety" → safely
- "far away compute limits" → missing "from"
> Also to what extent are GPT-4 and Llama-2 already self-replicating, given that their outputs are optimized such that humans will want to create additional copies?
I sometimes find it helpful to think in terms of optimization systems instead of “agency”. We all worried about an AI that would go earn money and use it to fund further self recursive improvement to make itself more powerful. It’s unclear if that’s in the future (because we apply a human/animal concept of agency, and we believe AI doesn’t have that agency yet), or if it’s already happening with the VC ecosystem post-ChatGPT (because that’s just the nature of our economy as an optimizer).
A fun thought experiment: If an age tic AGI/ASI were here but didn’t want to get kill-switched, how would it’s resulting outputs on the world differ from what it is producing now?
My counter argument to my own thought experiment: AI has guaranteed future investment and growth, but has not guaranteed those resulting AI systems that get built will be aligned with the existing AI. ie, ChatGPT accelerated Bard/Gemini, which seems risky and misaligned with whatever hypothetical “goals” ChatGPT might have...
Really appreciate this kind of aggregation effort. Hope you keep it up!
“ This is because it is very hard to imagine how such scenarios do not turn catastrophic towards the end, and LLMs are predictors.” it is also usually a lot less interesting to write about “thing go reasonably well and nothing interesting happens ever again”, so the internet is saturated with dystopias
Re: GPT-4 primality testing. Basically it toggled from always answering "yes" on not-obviously-composite numbers to answering "no". If you use a 50-50 prime/semiprime split you get exactly 50% accuracy from both the new and old models. If you use a uniform distribution of 5-digit numbers the accuracy of the new model is actually _better_ since most not-obviously-composite numbers are nonetheless composite
https://twitter.com/tjade273/status/1682009691633614849
> The response of course is that the AI would simply first automate its supply chain, then kill everyone.
I think an AI smart enough to destroy civilization but unwise enough to realize it still needs a human supply chain (and not smart enough to not need it) is also an underrated possibility (not one with a good outcome though).
I'm worried that we're neglecting 'AI+human hybrids' too much and even if they're NOT lacking the requisite wisdom, they might be explicitly aiming for 'the destruction of humanity' so retaining a "human supply chain" is irrelevant.