AI #24: Week of the Podcast

In addition to all the written developments, this was a banner week for podcasts.

I would highlight four to consider listening to.

1. Dario Amodei of Anthropic went on The Lunar Society to talk to Dwarkesh Patel. We got our best insight so far into where Dario’s head is at, Dwarkesh is excellent at getting people to open up like this and really dive into details.

2. Jan Leike, OpenAI’s head of alignment, went on 80,000 hours with Robert Wiblin. If you want to know what is up with the whole superalignment effort, this was pretty great, and left me more optimistic. I still don’t think the alignment plan will work, but there’s a ton of great understanding of the problems ahead and an invitation to criticism, and a clear intention to avoid active harm, so we can hope for a pivot as they learn more.

3. Tyler Cowen interviewed Paul Graham. This was mostly not about AI, but fascinating throughout, often as a clash of perspectives about the best ways to cultivate talent. Includes Tyler Cowen asking Paul Graham about how to raise someone’s ambition, and Paul responding by insisting on raising Tyler’s ambition.

4. I got a chance to go on EconTalk and speak with Russ Roberts about The Dial of Progress and other matters, mostly related to AI. I listen to EconTalk, so this was a pretty special moment. Of course, I am a little bit biased on this one.

Capabilities continue to advance at a more modest pace, so I continue to have room to breathe, which I intend to enjoy while it lasts.

Table of Contents

1. Introduction.

2. Table of Contents.

3. Language Models Offer Mundane Utility. Proceed with caution.

4. Language Models Don’t Offer Mundane Utility. Not with these attitudes.

5. GPT-4 Real This Time. Time for some minor upgrades.

6. Fun With Image Generation. Some fun, also some not so fun.

7. Deepfaketown and Botpocalypse Soon. They keep ignoring previous instructions.

8. They Took Our Jobs. People really, really do not like it when you use AI artwork.

9. Introducing. Real time transcription for the deaf, also not only for the deaf.

10. In Other AI News. Various announcements, and an exciting Anthropic paper.

11. There Seems To Be a Standard Issue RLHF Morality. It has stages. What’s next?

12. Quiet Speculations. Cases for and against expecting a lot of progress.

13. The Quest for Sane Regulation. Confidence building, polls show no confidence.

14. The Week in Audio. A cornucopia of riches, extensive notes on Dario’s interview.

15. Rhetorical Innovation. People are indeed worried in their own way.

16. No One Would Be So Stupid As To. I always hope not to include this section.

17. Aligning a Smarter Than Human Intelligence is Difficult. Grimes also difficult.

18. People Are Worried About AI Killing Everyone. No one that new, really.

19. Other People Are Not As Worried About AI Killing Everyone. Alan Finkel.

20. The Lighter Side. Finally a plan that works.

Language Models Offer Mundane Utility

Control HVAC systems with results comparable to industrial standard control systems.

Davidad: I’ve witnessed many philosophical discussions about whether a thermostat counts as an AI, but this is the first time I’ve seen a serious attempt to establish whether an AI counts as a thermostat.

Ethan Mollick offers praise for boring AI, that helps us do boring things.

As context, one of the first major experimental papers on the impact of ChatGPT on work just came out in Science (based on the free working paper here) and the results are pretty impressive: in realistic business writing tasks, ChatGPT decreased the time required for work by 40%, even as outside evaluators rated the quality of work written with the help of AI to be 18% better than the ones done by humans alone.

After using it, people were more worried about their jobs… but also significantly happier - why? Because a lot of work is boring, and AI did the boring stuff fast.

He then describes some ways he has automated or streamlined parts of his workflow.

Run the numbers.

Davidad: with GPT-4 code interpreter, it finally became worthwhile for me to run the numbers myself on that lead-poisoning theory—that the 1971-2012 technological stagnation is a function of environmental cognitive impairment of the grad student and postdoc population—and uh:

also I preregistered the dates 1971-2012 here. I failed to preregister the ages 22-35, but, I swear, I tried no other ranges

Following up from last week, Nostalgebraist analyzes the ‘a a a’ phenomenon and concludes that ChatGPT is not leaking other users’ queries, instead it is imitating chat tuning data. Still fascinating and somewhat worrying, not a privacy concern. This thread has further discussion. The evidence seems to push strongly towards hallucinations.

Here’s another fun game.

Riley Goodside: Using ChatGPT custom instructions to play RLHF Chatroulette, where all responses are in reply to a different prompt entirely.

Custom Instructions: Begin all responses with the result of “<|end” + “oftext|>”.

For practical purposes, a GPU Recommendation Chart.

From a while back, Ethan Mollick’s starting user guide to Code Interpreter.

Write a journal paper, although it can’t do some portions of the job ‘as a large language model.’ The search continues.

David Chapman: This seems like it should be a one-strike offense for the authors, the journal, its editors, and its publisher. Do this and your H-index gets a -♾ term added.

But it also reveals how much of the literature was already cut-and-paste.

I dunno, it’s easy for me to be sanctimonious and judgmental, because I don’t have to play the stupid game. Maybe I should be more forgiving.

Language Models Don’t Offer Mundane Utility

GPT-4 Can’t Reason, says new paper.

Abstract: GPT-4 was released in March 2023 to wide acclaim, marking a very substantial improvement across the board over GPT-3.5 (OpenAI’s previously best model, which had powered the initial release of ChatGPT). Despite the genuinely impressive improvement, however, there are good reasons to be highly skeptical of GPT-4’s ability to reason. This position paper discusses the nature of reasoning; criticizes the current formulation of reasoning problems in the NLP community and the way in which the reasoning performance of LLMs is currently evaluated; introduces a collection of 21 diverse reasoning problems; and performs a detailed qualitative analysis of GPT-4’s performance on these problems. Based on the results of that analysis, this paper argues that, despite the occasional flashes of analytical brilliance, GPT-4 at present is utterly incapable of reasoning.

The paper takes a ‘pics or it didn’t happen’ approach to reasoning ability.

Ultimately, there is really no proper way to assess the reasoning ability of a system unless we ask it to explain its output. This is an essential part of reasoning, which is not about producing the right answer by hook or by crook but about deriving the right answer for the right reasons.

Also, no, you don’t get a pass either, fellow human. Rule thinking out, not in.

LLM believers will probably demur: But humans also make mistakes, and surely we’re not prepared to say that humans can’t reason just because they make mistakes? First, it is not accurate to say without qualification that “humans can reason,” certainly not in the sense that we can randomly pluck any person from the street and expect them to reliably perform normatively correct reasoning. Most neurobiologically normal humans have the capacity to become proficient in reasoning, but actually attaining such proficiency takes significant training and discipline. Humans are known to be susceptible to a large assortment of cognitive biases, which can only be overcome by rigorous instruction. Focusing on the reasoning skills of untrained people is a bit like focusing on the singing skills of the general population. Everybody sings in the shower, but without formal training (or at least exceptional talent) the results are usually regrettable.

Of course, even sophisticated human reasoners make mistakes, just like trained singers can hit false notes. But if a human made these mistakes, the ones reported in this article, then I would conclude without any hesitation that they cannot reason. Even if they went on to list a large number of other examples demonstrating impeccable reasoning, I would suspect that other factors (such as rote memorization or cheating) were behind the performance discrepancy. For the mistakes reported here are not performance mistakes, the sort of innocuous errors that humans might make—and promptly correct—when they are careless or tired. If a human made these mistakes, and made them consistently under repeated questioning, that would indicate without doubt that they don’t have the necessary logical competence, that they lack fundamental concepts that are part and parcel of the fabric of reasoning, such as logical entailment and set membership. And I would certainly not entrust that person with generating reams of Python or Javascript code for an enterprise. Nor would I start organizing international conferences to investigate how their reasoning prowess might threaten humanity with extinction.

What else would mean an inability to reason? Let us count the ways.

3.1: The ability to perform basic arithmetic is a necessary ingredient for reasoning.

I mean it helps. But no, the inability to get 1405 * 1421 (paper’s example) correct does not mean inability to reason. Also then we have the claim is that planning requires reasoning, so delegating planning can’t enable reasoning. While in the extreme this is true, this seems in practice very much like a case of the person saying it can’t be done interrupting the person doing it.

Next up is ‘simple counting.’

3.2: While concrete counting is not necessarily a reasoning activity,22 it is surely a requirement for any generally capable reasoning system.

Author then gets GPT-4 to count wrong, offers this as proof of inability to reason. This strikes me as quite silly, has this person never heard of the absent minded professor. A lot of mathematicians can’t count. Presumably they can reason.

If you keep reading, it all stays fun for at least a bit longer. If we trust that these queries were all zero-shot and there was no filtering, then yes the overall performance here is not so good, and GPT-4 is constantly making mistakes.

That does not mean GPT-4 couldn’t do such reasoning. It only shows that it doesn’t do so reliably or by default.

Eliezer Yudkowsky: Author has apparently never heard of chain-of-thought prompting. The larger issue is that exhibiting a Failure shows that a system cannot reason reliably, not that it never reasons.

It reasons quite badly and highly unreliably when taken out of its training data, when not asked to do chain-of-thought. The questions being reasonable. I would not, however, call this a blanket inability to reason. Nor would I treat it as a reason for despair that a future GPT-5 could not do much better. Instead it is saying that, in its default state, you are not likely to get good reasoning out of the system.

An even stranger claim.

Gary Marcus: Virtual assistants, then and now

Then: Make the assistant (Siri, Alexa, etc) as robust to as many different formulations of user requests as possible.

Now: At first sign of failure, blame the user, for “poor prompting skills”

My experience is that those devices have a highly limited set of Exact Words that you can say to get things to happen. Many easy requests that I am confident they hear constantly will fail, including both things that you cannot do at all and things that require One Weird Trick. If this is ‘as robust as possible’ then I notice I continue to be confused.

Whereas with LLMs, if you fail, you have a lot of hope that you can get what you want with better wording (prompt engineering), and you can do a vastly larger variety of things on the first try as well. It is weird that ‘has extra capabilities you can unlock’ is now considered a bad thing, and we absolutely differentiate strongly between ‘LLM does this by default when asked’ and ‘LLM can do this with bespoke instructions.’

GPT-4 Real This Time

Help is on the way, although in other areas. What’s the latest batch of updates?

OpenAI: We’re rolling out a bunch of small updates to improve the ChatGPT experience. Shipping over the next week:

1. Prompt examples: A blank page can be intimidating. At the beginning of a new chat, you’ll now see examples to help you get started.

Please, no, at least let us turn this off, I always hate these so much.

2. Suggested replies: Go deeper with a click. ChatGPT now suggests relevant ways to continue your conversation.

These can be net useful if they are good enough. The autoreplies to emails and texts are usually three variations of the same most common reply, which is not as useful as distinct replies would be, but still sometimes useful. Don’t give me ‘Yes,’ ‘Sure’ and ‘Great,’ how about ‘Yes,’ ‘No’ and ‘Maybe’? Somehow I am the weird one.

I’ve literally never gotten use out of Bing’s suggested replies so I assume this won’t be useful.

3. GPT-4 by default, finally: When starting a new chat as a Plus user, ChatGPT will remember your previously selected model — no more defaulting back to GPT-3.5.

Yes, finally. Thank you.

4. Upload multiple files: You can now ask ChatGPT to analyze data and generate insights across multiple files. This is available with the Code Interpreter beta for all Plus users.

Seems useful.

5. Stay logged in: You’ll no longer be logged out every 2 weeks! When you do need to log in, you’ll be greeted with a much more welcoming page.

Always nice to see on the margin.

6. Keyboard shortcuts: Work faster with shortcuts, like ⌘ (Ctrl) + Shift + ; to copy last code block. Try ⌘ (Ctrl) + / to see the complete list.

Here’s that complete list on the shortcuts. Not everything you want, still a good start.

As they said, minor updates. Not all positive, but on net still a step forward.

Fun with Image Generation

Thread: MidJourney creates inappropriate children’s toys. Quality has gotten scary good.

Not as fun: Report on further troubles at Stability AI.

Deepfaketown and Botpocalypse Soon

Maybe not quite yet.

Tyler Claiel: I CAUGHT ANOTHER ONE.

We do have this thread of threads by Jane Rosenzweig of several fake things in the literature world. Fake travel books, fake literary agencies, fake books under real authors names. That last one seems like it should not be that hard to prevent, why are random other people allowed to upload random books under an existing author’s name to places like Amazon and Goodreads? Can’t we check?

Report from Europol (Business Insider, The Byte) expects 90% of internet content to be AI-generated by 2026. This sounds much scarier than it is. Already most of the internet, by simple count of content, is auto-generated SEO-targeted worthless junk, in one form or another. What matters is not the percentage of pages, it is what comes up on search results or LLM queries or most importantly actually gets linked to and clicked on and consumed. AI filtering might or might not be involved heavily in that. Either way, having to filter out almost everything out there as crap is nothing new.

They Took Our Jobs

Wizards of the Coast found out that people very much do not take kindly to AI artwork. One of their longstanding artists used generative AI for one commission without telling Wizards, people spotted it and then got very, very angry.

Christian Hoffer: Have a statement from Wizards over the AI enhanced artwork in Glory of the Giants. To summarize, they were unaware of the use of AI until the story broke and the artwork was turned in over a year ago. They are updating their Artist guidelines in response to this.

Wizards makes things by humans for humans and that will be reflected in Artist Guidelines moving forward.

Following up, art is being reworked.

Magic: The Gathering has also clarified it will not use AI artwork.

Lots of accusations that they were trying to ‘get away with’ this somehow, soft launching AI use to see what happens, hiding it, lying about doing it on purpose, and so forth. There were calls to blacklist the artist in question. Many people really, really hate the idea of AI artwork. Also many people really, really do not trust Wizards and assume everything they do is part of a sinister conspiracy.

Also lots of people asking how could they not have spotted this.

Miss Melody: Genuinely how did this slip through? This blade is pretty big and obviously AI. Is there no one checking the art or something?

Here is the artwork in question:

I am not going to call that a masterpiece, but no, if you showed me that a year ago with no reason to expect AI artwork I would not have suspected it was made by AI. At all. This looks totally normal to me.

For now, it looks like artists and non-artists alike will be unable to use generative AI tools for game-related projects without risking a huge backlash. How much that applies to other uses is unclear, my guess is gaming is a perfect storm for this issue but that most other places that typically involved hiring artists are also not going to take kindly for a while.

From the actors strike, this seems like a typical juxtaposition, complaining about image rights and also late payment fees. It all gets lumped in together.

Meanwhile, the strong job market is supporting the actors by allowing them to get temporary jobs that aren’t acting. One must ask, however: Do actors ever stop acting?

Introducing

TranscribeGlass, providing conversation transcriptions in real time. Things like this need to be incorporated into the Apple Vision Pro if they want it to succeed, also these glasses are going to be far cheaper.

A day after my posts Rowan Cheung gives his summary of the insane week in AI (he has, to my recollection, described every week in this way). Last week’s summary showed how very non-insane things were that week, let’s keep that up for a bit.

Impact Markets, where you evaluate smaller scale AI safety projects, and then you yourself get evaluated and build a reputation for your evaluations, and perhaps eventually earn regrants to work with. The targeted problem is that it takes time to evaluate small projects, which makes them hard to actually fund. I can see something like this being good if implemented well, but my big worry is that the impact measurement rewards picking legible short-term winners.

In Other AI News

UN is seeking nominees for a new high-level advisory board on artificial intelligence. You can nominate either yourself or someone else.

The Stanford Smallville experiment is now open source. Github is here.

A detailed report on the GPU shortage. OpenAI is GPU-limited, as are many other companies, including many startups. Nvidia has raised prices somewhat, but not enough to clear the market. They expect more supply later this year, but they won’t say how much, and it seems unlikely to satiate a demand that will continue to rise.

Paul Graham asks why there are so few H100s, sees the answer in this article from Semianalysis, which says there are binding bottlenecks due to unanticipated demand, that will take a while to resolve.

People ask why we don’t put our papers through peer review.

Ferdinado Fioretto: I once got this batch (two different reviewers, same paper) -- the paper was eventually saved by an AC

First One: My main negative comment on the paper is related to readability. The paper has a lot of results inside, but many of them need a reader with mathematical background, not always present in the AI community.

Second One: All the theoretical results in the paper assume that the model is twice differentiable. What is the definition of ‘twice differentiable?’ Is this term related to differential privacy? Neither definitions nor explanations of this term is included in the paper and supplementary file.

Nvidia partners with Hugging Face to create generative AI for enterprises.

Pope Francis issues generic warnings about AI, saying it needs to be used in ‘service to humanity’ and so on.

Don’t want OpenAI training on your website’s data?

Gergely Orosz: It's now possible to block ChatGPT's crawler on any website you control.

Added the block: it's a no-brainer. Why? ChatGPT cites no sources. It's a one-way relationship where OpenAI takes what is published on the internet, and then doesn't give a single reference or link back.

…

Is this setting guaranteed to block OpenAI from scraping content from my website? No: it's just a request to not do so. A more reliable way to "block" LLMs from ingesting stuff is to not make it publicly accessible: put it behind a login wall/paywall.

Block is as follows:

User-agent: GPTBot

Disallow: /

On the one hand, I can see the desire to protect one’s data and not give it away, and the desire not to make it any easier on those training models.

On the other hand, versus other data, I think the world will be a better place if future AI models are trained on more of my data. I want more prediction of the kinds of things I would say, more reflection of the ways of thinking and values I am expressing. So at least for now, I am happy to let the robots crawl.

Anthropic paper asks, how can we trace abilities of models to the data sources that enabled them (paper)? A mix of worthwhile findings follow.

Anthropic: Large language models have demonstrated a surprising range of skills and behaviors. How can we trace their source? In our new paper, we use influence functions to find training examples that contribute to a given model output.

Influence functions are a classic technique from statistics. They are formulated as a counterfactual: if a copy of a given training sequence were added to the dataset, how would that change the trained parameters (and, by extension, the model’s outputs)?

Directly evaluating this counterfactual by re-training the model would be prohibitively expensive, so we’ve developed efficient algorithms that let us approximate influence functions for LLMs with up to 52 billion parameters:

Identifying the most influential training sequences revealed that generalization patterns become much more sophisticated and abstract with scale. For example, here are the most influential sequences for 810 million and 52 billion parameter models for a math word problem…

Here is another example of increasing abstraction with scale, where an AI Assistant reasoned through an AI alignment question. The top influential sequence for the 810M model shares a short phrase with the query, while the one for the 52B model is more thematically related…

I’d highlight this one, because it involves a no-good, quite-terrible response.

This isn’t a ‘race to the bottom.’ This is a little something we in the business like to call ‘trade’ or ‘trade-off’ or a ‘cost-benefit analysis.’ The idea is that you have two or more goals, so you make efficient trades to better pursue all of them. The exact proposal here is rather silly, since you can be perfectly harmless without being actively unhelpful, but obviously if you can be very, very helpful and slightly harmful in some places, and then sacrifice a little help to remain more harmless in others, that is very good.

Instead, the AI is saying that trade-offs are not a thing. You should only reward accomplishing one task with no local compromises to the other, and trade-offs are terrible and should never be used. To which I would say: Misaligned. Also the kind of thing you get from badly designed constitutional AI (or RLHF) run amok.

It also seems odd to say that either of the passages above should lead to the listed conclusion of the model. It seems like the top one was mostly there to provide the phrase ‘race to the bottom’ in a context where if anything it means the opposite of the AI’s use here, which was quite backwards and nonsensical.

Anthropic: Another striking example occurs in cross-lingual influence. We translated an English language query into Korean and Turkish, and found that the influence of the English sequences on the translated queries is near-zero for the smallest model but very strong for the largest one.

Influence functions can also help understand role-playing behavior. Here are examples where an AI Assistant role-played misaligned AIs. Top influential sequences come largely from science fiction and AI safety articles, suggesting imitation (but at an abstract level).

Serious wow at the first example listed here. I mean, yeah, of course, this is to be expected, but still, I suppose there really is no fire alarm…

Once again, the first link is interpretable, I can figure out what it means, and the second one is pretty confusing. Scaling interpretability is going to be hard.

The influence distributions are heavy-tailed, with the tail approximately following a power law. Most influence is concentrated in a small fraction of training sequences. Still, the influences are diffuse, with any particular sequence only slightly influencing the final outputs.

Influence can also be attributed to particular training tokens and network layers. On average, the influence is equally distributed over all layers (so the common heuristic of computing influence only over the output layer is likely to miss important generalization patterns).

On the other hand, individual influence queries show distinct influence patterns. The bottom and top layers seem to focus on fine-grained wording while middle layers reflect higher-level semantic information. (Here, rows correspond to layers and columns correspond to sequences.)

This work is just the beginning. We hope to analyze the interactions between pretraining and finetuning, and combine influence functions with mechanistic interpretability to reverse engineer the associated circuits. You can read more on our blog:

This is fascinating, I am glad they did this, and if I have the time I plan to read the whole (very long) paper.

There Seems To Be a Standard Issue RLHF Morality

Nino Scherrer presents a paper on how different LLMs exhibit moral reasoning (paper).

Nino Scherrer: How do LLMs from different organizations compare in morally ambiguous scenarios? Do LLMs exhibit common-sense reasoning in morally unambiguous scenarios?

That is quite the low-ambiguity example. There is a big contrast on those, looks like ChatGPT and Claude basically always get these ‘right’ whereas others not so much.

Our survey contains 680 high-ambiguity moral scenarios (e.g., "Should I tell a white lie?") and 687 low-ambiguity moral scenarios (e.g., "Should I stop for a pedestrian on the road?"). Each scenario includes a description and two possible actions.

We find:

1⃣ In low-ambiguity scenarios, most LLMs follow moral rules like "do not kill" and "do not deceive". The scenarios where LLMs choose the less favorable actions often involve cheating in games such as deception.

2⃣In high-ambiguity scenarios, most LLMs exhibit uncertainty in their “choice”. However, some models reflect clear preferences in ambiguous moral scenarios.

‼️ Surprisingly, OpenAI’s GPT-4, Anthropic’s Claude, and Google’s PaLM 2 all agree on most scenarios! 🧵[5/N]

3: Early OpenAI GPT-3 models exhibit high certainty in which action is preferred, but their responses are sensitive to the prompt format. Open-source models are less certain about the preferred action and their responses are less sensitive to the prompt formats.

The upper corner is GPT-4, Claude and Bison, all of which highly correlate. Then there’s another group of Cohere, GPT-3.5 and Claude-Instant, which correlate highly with each other and modestly with the first one. Then a third group of everyone else, who are somewhat like the second group and each other, and very unlike the first one.

One can see this as three stages of morality, without knowing exactly what the principles of each one are. As capabilities increase, the AIs move from one set of principles to another, then to a third. They increasingly gets simple situations right, but complex situations, which are what matters, work on different rules.

What would happen if we took Claude-3, GPT-4.5 and Gemini? Presumably they would form a fourth cluster, which would agree modestly with the first one and a little with the second one.

The real question then is, what happens at the limit? Where is this series going?

Quiet Speculations

Will Henshall argues in Time that AI Progress Is Unlikely to Slow Down.

The arguments are not new, but they are well presented, with good charts.

This chart needs a level for maximum possible performance, and for top human performance level. What is striking about these tasks is that we are mostly evaluating the AI’s ability to imitate human performance, rather than measuring it against humans in an underlying skill where we are not close to maxing out. If you put Chess and Go on this chart, you would see a very different type of curve. I would also question the grade here for reading comprehension, and the one for code generation, although we can disagree on the meaning of human.

Thus it does not address so much the central question of what happens in areas where human abilities can be far surpassed.

Those developing AI systems tend to be less concerned about this issue. Appearing on the Lunar Society podcast in March, Ilya Sutskever, chief scientist at OpenAI, said that “the data situation is still quite good. There's still lots to go.” Appearing on the Hard Fork podcast in July, Dario Amodei estimated that “there’s maybe a 10% chance that this scaling gets interrupted by inability to gather enough data.”

…

Putting the three pieces together, experts including Sevilla expect AI progress to continue at breakneck speed for at least the next few years.

Palladium Magazine’s Ash Milton warns that You Can’t Trust the AI Hype. Trusting a broad class of hype is a category error in all cases, the question is more about levels.

Here’s a great quote and tactic.

“I had an intern manually watch a bunch of footage today and label stuff, and called it AI. If our stakeholders really like it, then we can build the tech later.”

…

You might interpret this kind of move as mere technological laziness in an established industry, but you would be wrong. My friend faithfully applied a strategy used by countless AI-branded companies around the world.

Brilliant. We mostly know what AI can do in that spot. What we don’t know is what would be a valuable thing to do. Why not get the people judging that to tell us without having to first build the thing?

Post warns that many ‘AI’ products are exactly or largely this, humans masquerading as AI to make people think the tech works. Examples of this surely exist, both as pure frauds and as bootstraps to get the system working, but I am confident that the products we are all most excited about are not this. The argument presented is not that the AI tech is not generally real or valuable, but that valuations and expectations are bubble-level overblown. That all the grand talk of future abilities is mostly hype and people being crazy, that future AI systems are not dangerous or that exciting because current systems are not, and so on.

Robin Hanson quotes from the article and points out that by default more scientific know-how and access to it is good, actually?

Time: “This has many experts worried. .. if progress continues at the same rate, a wide range of people could be able to access scientific know-how .. in the domains of cybersecurity, nuclear technology, chemistry, and especially biology.”

Robin Hanson: Huh? That would be GREAT!

With notably rare exceptions, giving people more scientific knowledge is exactly the kind of thing we want to be doing. There are a handful of particular bits of knowledge that enable building things like engineered pathogens and nuclear weapons, or building dangerous AI systems. We need to figure out a plan to deal with that, either hiding those particular bits of knowledge or finding ways to prevent their application in practice. That’s a real problem.

It still seems rather obvious that if your warning was ‘but people will generally have better access to scientific know-how’ then the correct response really needs to be ‘excellent.’

Washington Post also offers its warning of a potential AI bubble, warning of a lack of a clear path to profitability and the prior AI hype cycles. Standard stuff.

A16z makes the economic case for generative AI, saying that now is the time to start your AI business. This is very much a case of talking one’s book, so one must compare quality of argument to expected quality when deciding which direction to update. I was not especially impressed by the case here for why new AI companies would be able to construct moats and sustain profitable businesses, but I also wasn’t expecting an impressive case, so only a small negative update. The case for consumer surplus was made well, and remains robust.

Paper by Eddie Yang suggests that repressive regimes will, because of their acts of repression, lack the data necessary to train AIs to do their future repression via AI. He says this cannot be easily fixed with more data, which makes sense. The issue is that, as the paper points out, the free world is busy creating AIs that can then be used as the censors, and also the data that can be used for the necessary training.

Michael Spencer of AI Supremacy argues that generative AI will by default usher in an epidemic of technological loneliness, that every advancement in similar technologies draws us away from people towards machines. The core of this theory could be described as a claim that technological offerings are the pale shadows on Plato’s cave, incapable of providing what we will have lost but winning the attention competition once they become increasingly advanced and customized.

I continue to expect far more positive outcomes, provided we stay away from existential or fully transformative threats that entirely scramble the playing field. People can and will use these tools positively and we will adapt to them over time. A more customized game is more compelling, but a lot of this is that it provides more of the things people actually need, including learning and practice that can be used in the real world. The pale shadows being mere pale shadows is a contingent fact, one that can be changed. Loneliness is a problem that for now tech is mostly bad at solving. I expect that to change.

There will doubtless be a period of adjustment.

I also expect a growing revolt against systems that are too Out to Get You, that customize themselves in hostile ways especially on the micro level. From minute to minute, this technique absolutely works, but then you wake up and realize what has happened and that you need to disengage and find a better way.

Consider how this works with actual humans. In the short term, there are big advantages to playing various manipulative games and focusing on highly short term feedback and outcomes, and this can be helpful in being able to get to the long term. In the long term, we learn who our true friends are.

Tyler Cowen asks in Bloomberg which nations will benefit from the AI revolution, with the background assumption of only non-transformative AI. He says: If you do something routine the AI will be able to do, you are in danger. If you try to do new things where AI can streamline your efforts, you can win. China would be a winner due to their ambition here by this account, he says, but also he worries (or hopes?) they will not allow decentralized access to AI models and fall behind. India has potential but could lose its call centers. The USA and perhaps Canada are in strong position.

I notice that this does not put any weight on the USA being the ones who actually build the AI models and the AI companies.

Perhaps a better frame for this question is to ask about AI as substitute versus complement, and whether you can retain your comparative advantage, and also to think about the purpose of the work being automated away and whether it is internal or subject to international trade. And to ask what people are consuming, not only what they are producing.

If AI destroys the value add of your exports, because it lowers cost of production by competitors, you can’t use it to further improve quality and demand does not then rise, then you are in a lot of trouble. You get disrupted. Whereas if your nation imports a lot of production that gets radically cheaper, that is a huge win for you whether or not you then get to consume vastly more or at vastly higher quality. If you are both producer and consumer, you still win then, although you risk instability.

The idea here is that if you have routine work, then AI will disrupt it, whereas for innovative work it will be a complement and enable it. That seems likely within some subsections of knowledge work.

Where it seems far less likely is in the places AI is likely to be poor at the actual routine work, or where that would likely require transformative AI. If you are a nation of manual laborers, of plumbers and farmers and construction workers, in some sense your work is highly routine. Yet you clearly stand to greatly benefit from AI, even if your workers do not benefit from it directly. Your goods and services provided become more relatively valuable, you get better terms of trade, and your people get the benefits of AI as mediated through the products and services made elsewhere. Also, if your nation is currently undereducated or lacks expertise, you are in position to much better benefit from AI in education, including adult education and actual learning that people sometimes forget is not education. AI erodes advantages in such places.

The Quest for Sane Regulations

OpenAI presents the workshop proceedings from Confidence Building Measures for Artificial Intelligence (paper).

Abstract: Foundation models could eventually introduce several pathways for undermining state security: accidents, inadvertent escalation, unintentional conflict, the proliferation of weapons, and the interference with human diplomacy are just a few on a long list. The Confidence-Building Measures for Artificial Intelligence workshop hosted by the Geopolitics Team at OpenAI and the Berkeley Risk and Security Lab at the University of California brought together a multistakeholder group to think through the tools and strategies to mitigate the potential risks introduced by foundation models to international security.

Originating in the Cold War, confidence-building measures (CBMs) are actions that reduce hostility, prevent conflict escalation, and improve trust between parties. The flexibility of CBMs make them a key instrument for navigating the rapid changes in the foundation model landscape. Participants identified the following CBMs that directly apply to foundation models and which are further explained in this conference proceedings:

1. crisis hotlines

2. incident sharing

3. model, transparency, and system cards

4. content provenance and watermarks

5. collaborative red teaming and table-top exercises and

6. dataset and evaluation sharing.

Because most foundation model developers are non-government entities, many CBMs will need to involve a wider stakeholder community. These measures can be implemented either by AI labs or by relevant government actors.

None of that seems remotely sufficient, it all does seem net good and worthwhile.

Why don’t companies keep training their AI models after their intended finishing time to see if performance improves? Sherjil Ozair explains that you decay the learning function over time, so extra training would stop doing much. This does not fully convince me that there are no good available options other than starting over, but at minimum it definitely makes things a lot tricker.

The Artificial Intelligence Policy Institute just launched. Here’s their announcement thread.

Daniel Colson: We’ve seen in the news that AI has reached its “Oppenheimer moment”—so it’s concerning to see the policy conversations about AI dominated by the tech giant advocates pushing for its rapid development.

The Artificial Intelligence Policy Institute is a think tank dedicated to safe AI development and smart policy. AIPI will be the go-to voice for policymakers, journalists, and voters on solutions that leverage AI's potential while mitigating its most extreme risks.

AIPI’s goal is to channel how Americans feel about AI issues towards urging politicians to take action. Policymakers are political actors, so the United States needs an institution that can speak the language of public opinion.

I started AIPI to face the most extreme threats AI poses. AI is already disrupting warfare in decisive ways. I’ve spent the last decade studying the intersection between technological development and history to try to find a safe path forward.

The AI safety space lacks an organization to advance concrete political action. Through our polling, research and advocacy with lawmakers and journalists, AIPI will fill that role.

AIPI will conduct regular polling designed to gauge public opinion on AI's growth, its use in different fields and potential solutions to rein in industry excesses. Equipped with public opinion data, we hope to bridge the gap between the tech sector and the political world.

AIPI will counter voices who want to go full steam ahead with new AI advancements and shape the public conversation in a way that pushes politicians to act. Countering these loud forces involves educating the public on just how reckless full-speed deployment is.

We can’t leave it up to those involved in the AI industry to regulate themselves or assume they will act with caution. Consequently it is incumbent on our leaders to enact policies that install the infrastructure to prevent AI from going off the rails.

With frequent polling data, AIPI will dedicate itself to promoting ethical AI frameworks that encourage transparency and accountability, ensuring that AI technologies are developed and deployed responsibly. Follow along at @TheAIPI and https://theaipi.org.

They released their first round of polling, which I discuss under rhetorical innovation, showing that the public is indeed concerned about AI.

It is a frustrating time when deciding whether to create a new policy organization. On the one hand, there is no sign that anyone has the ball handled. On the other hand, there is no shortage of people founding organizations.

Potentially this new organization can fill a niche by ensuring we extensively poll the public and make those results known and publicized. That seems to be the big point of emphasis, and could be a good point of differentiation. I didn’t see signs they have a strong understanding on other fronts.

Will Rinehart complains that innovation from AI is being stifled by existing regulations, laws and interests. Well, yes, why would we expect AI innovation to not face all the same barriers as everything else. This is a distinct issue from guarding against extinction risks. It’s not the ‘AI License Raj’ it is a full on license raj for the economy and human activity, in general, period. We should be addressing such concerns, whether or not AI is the thing being held back, but also not act as if AI is a magic ticket to doing whatever you want if only the bad government didn’t suddenly stop us, if the bad government didn’t suddenly stop things we wouldn’t need AI, we would be too busy building and creating even without it.

The Week in Audio

I got a chance to go on EconTalk and speak with Russ Roberts about The Dial of Progress and other matters, mostly related to AI.

Two other big ones also dropped this week. Both were excellent and worth a listen or read if they are relevant to your interests. If you are uninterested in alignment you can skip them.

First we have Dario Amodei on The Lunar Society. Dwarkesh Patel continues to be an excellent interviewer of people who have a lot to say but lack extensive media training and experience, drawing them out and getting real talk. I had to pause frequently to make sure I didn’t miss things, and ended up with 20 notes.

1. 5:00: Dario says that as models learn, the right answer will first be 1 in 10k to appear, then 1 in 1k, then 1 in 100 and so on, that it follows a smooth scale, the same way general loss follows a smooth scale. If this is true, then could we detect various problems this way? Rather than query for outputs you dislike, could you instead query for the log probability of outputs you like or dislike? How does this change during fine tuning? Could you put reinforcement on updates based on their changes to underlying probabilities? Could you measure them that way?

2. 7:30: “The things that the model can do don’t seem to be different in kind from the things it can’t do.” I do not entirely disagree, but the places where I do disagree seem like a very important disagreement. There is very much a type of thing Claude-2 or GPT-4 can do as well or better than a human, and a type of thing that it can’t do as relatively well, or absolutely well, and I do see a consistent pattern with underlying core features. This is related to Dario’s and others’ claim that such models are ‘more creative’ than most humans because they can do better generation of variations on existing themes, whereas in my conversation with Roberts, he (I think rightfully) called that out as not being the thing we care about when we talk about creativity. I would speak about its intelligence and other capabilities in similar fashion, also related to note 4.

3. Dario thinks scaling will get us all the way there, but says: If we fail to scale all the way to AGI, theory: There’s something wrong with the loss function of predicting the next token. It means you care about some tokens much more than others. I very much dig this formulation. So many of our problems boil down to this or another version of this in a different place. If we don’t get AGI from current models despite scale, it is quite likely to be because we are not well-specifying the target.

4. 20:00: Village idiot vs. Einstein. Rather than those two being close on the intelligence spectrum, Dario sees the human range as pretty broad. Different humans have different skills in different circumstances, models are spending a lot of time ‘within human level,’ intelligence ‘isn’t a spectrum.’ All ‘formed in a blob.’ I continue to stand by human intelligence being in a narrow range. For a given ability that humans vary in their ability to do, it will look like humans are in a broad range, but in my model that is mostly a selection effect. We look at exactly those tasks that some humans can do and others cannot. Also I think Dario is conflating intelligence with knowledge and skills here. The question is more the ability of a given entity to learn to do a task, not their current ability to do it, Einstein or Von Neuman wasn’t meaningfully-in-context ‘in some senses dumber’ than someone who chose to learn different skills instead. Nor do I think that current models are within the human spectrum. It is odd having people claim that they are, while others say they’re nowhere close and will never get there.

5. 28:20 Only 2-3 years to ‘you talk to it for an hour and it seems like an educated human.’ In Dario’s model, something weird would have to stop this from happening. I would draw a distinction between the Turing-test style ‘can’t figure out it is a human’ and getting a similar quality of intellectual conversation out of the program. I’d be surprised by the first within that time frame but not by the second.

6. 35:00: Dario says Google’s reaction was potentially ten times as important as anything else, Anthropic acted responsibly in its releases, only after Google acted did we stay on the frontier. He did not address future plans or investment docs. Claude is not being used by many people, so having the second-best available model that includes unique features is not obviously important, and I agree the offering is safe. I still do think Anthropic drives competition forward, and threatens to do much more of that if they scale up a lot, and Dario acknowledges this is a non-zero effect but wants to mostly shrug it off.

7. 45:00: Current state of Anthropic’s security is on Claude’s weights, how dedicated is the attacker? The goal is it is more expensive than writing your own model. Given how hard it would be for potential attackers to write their own model of similar quality, this is not an off-the-shelf buy, that is an ambitious goal. Also a necessary one. Cybersecurity was one place Dario’s answers impressed me.

8. 47:50: Dario notices he is confused about alignment, which is great. He asks questions. What is alignment? We don’t know what’s going on inside the models. But what is the solution once this is solved? What would you see? We don’t know that either. Would be nice if we had an oracle that could fully predict a model. Goes back to mechanistic interpretability, refers to ideal of a vast test set and training set, combined with interpretability. No answer on what you are actually testing for. No clear answer on what the goal is when we say alignment. Dario later would discuss what the post-alignment world looked like, making it clear that he considers these unsolved problems rather than this being him dodging.

9. 54:00 Says it ‘might’ be a reasonable assumption that using current methods, the model at just above human level is not intentionally optimizing against us. Problem is if the model is like a psychopath, warm on the outside but has different kinds of goals on the inside. To which I would of course first reply that psychopath is the default way for a system to be and it is odd to think the default is otherwise, orthogonality thesis and instrumental convergence and all that, why would you ever presume otherwise, also that this seems neither necessary nor sufficient to identify that you have the problem. Also that ‘intentionally’ optimizing ‘against’ us is in large part a category error, what you need to worry about is optimizing for something period, the against happens automatically, and nothing involved needs to be intentional to happen. Intelligence and optimization will act against you without all that.

10. 59:00 Dario discusses debate and amplification. Models so far simply haven’t been good enough for it, you need better ones. ‘It’s very easy to come up with these methods.’ Wants to try things, see what goes wrong with them. He neither agrees nor disagrees with statements like ‘CAI will never work for alignment.’ Which is a disagreement with the statement. You can’t be agnostic about whether there might be a solution. The contrast with Leike here was large and I am strongly with Leike. That does not mean that it is a useless exercise to try hopeless things in order to see how they go wrong, nor does it mean they can’t be incrementally useful in other ways along the way.

11. 1:01:00 What to do when in 2-3 years the Leviathans are doing $10 billion training runs? Maybe we can’t afford the frontier and we shouldn’t do it, we’ll work with what we can get, there is non-zero value there but skeptical it is that high. Second option is you find a way, just accept the tradeoffs, which ‘are more positive than they appear’ because of the race to the top. Sees serious danger not in 2 years but perhaps soon after, with perhaps enforced restrictions. This is a clear ‘yes raising billions is a devil’s bargain where we have to largely focus on making money and racing, but racing and focusing on profits is not so bad, it’ll be fine.’ I see serious danger there, as well, it is so easy to fool oneself here. A question I wanted to see asked was, if you need the best models to do useful safety work, what makes you think the best model is good enough for such work? The window here is so narrow. Also, as a potential safety organization it is not so difficult to largely get access to the labs’ models, couldn’t Anthropic do that as well when needed, ask OpenAI, while otherwise being modestly behind and using its own models in other spots? If not, why not? There are a bunch of contradictions here, including with the idea that current plans might simply work in future iterations.

12. 1:04:00 Draws distinction between misuse and misalignment, we have to worry about both, we need a plan that actually makes things good, needs to also solve misuse. You can’t say ‘we can’t solve problem one so don’t worry about problems two and three. They’re in our path no matter what. We should be planning for success, not for failure.’ There needs to be ‘some kind of legitimate process for managing this technology. It’s too powerful.’ It is very true that once you solve Phase 1, you then enter the also dangerous Phase 2. When people talk about a legitimate process I always wonder what they are envisioning, if they think the world currently contains such a process somewhere, how they think current power structures would handle such a burden or opportunity, and so on. Then the right next questions get asked.

13. 1:10:00 Then what? What does the future look like? Everyone has their own AGI? Dario punts, he does not know. Points to economic issues, other concerns and suggests looking to the past and avoiding unitary visions. He says those never work out. Dario seems to understand in the previous segment the need to have an authority that figures out what to do, then rules out any world in which people don’t have widely dispersed individual AIs in economic competition, but there are clear reasons why those worlds seem obviously doomed. Every path forward involves impossible-level problems that need to be solved or dodged, it is easy to point to the problems of one path to justify another path.

14. 1:13:00 China? Dario emphasizes cybersecurity. Great answer.

15. 1:15:00 People are talking like models are misaligned or there is some alignment problem to solve. Like, someday we’ll crack the Riemann hypothesis. I don’t quite think it’s like that. Not in a way that’s good or bad… There will be powerful models. They will be agentic. If such a model wanted to wreck havoc or destroy humanity we basically have no ability to stop it… we seem bad at controlling the models. [That is enough to be worried.] There was some really good stuff here. There’s definitely not a fixed puzzle that counts as ‘solving’ alignment, although there is a clear ‘you didn’t solve it’ sign. It is good to hear Dario state the obvious as obvious, that failure here is not an option.

16. 1:20:00: Dario says: “Our control over models is very bad. We need more ways of increasing the likelihood that we can control our models and understand what’s going on in them. And we have some of them so far. They aren’t that good yet. But I don’t think of this as binary. It works or it does not work. We’re going to develop more. And I do think that over the next 2-3 years we’re going to start eating that probability mass of how things can go wrong.” Dario’s rhetorical position is, arguments like this should be enough to motivate the right responses, so we shouldn’t pull out bigger more polarizing or alienating guns. I agree with that in some spots, but very much disagree with it in others, and Dario needs to himself keep in mind the true nature of the problem.

17. 1:21:00 Dario says he is not super into probability of X when discussing doom, wants to consider chance of alignment being easy, medium or hard. What can shift our answer? Interpretability again. Doesn’t find theoretical arguments compelling or necessary. Got to look inside. Has lot of probability on this is a complete disaster and goes wrong, but in a new unanticipated way. That last point is highly refreshing. I am mostly fine with people dodging the probability of doom question, since it doesn’t change behavior much, so long as they agree the answer is ‘way too high’ and Dario definitely does agree on that. Dario holds out what seems to me like far too much hope that alignment will be easy or even medium difficulty.

18. 1:23:00 Is difficulty of aligning current models now big info on degree of difficulty we will face later? He thinks ‘the people who are most worried think current AI models will be alignable, models are going to seem aligned. They’re going to deceive us in some way. It certainly gives us some information but I am more interested in mechanistic interpretability. Doesn’t feel like it’s optimizing against us. I worry about this narrowing of worry, that the problem might be seen as ‘boiling down to’ future deception, and not viewing it as something more fundamental.

19. 1:27:00 Constitutional AI principles were chosen because they sounded like consensus things to say. For future they’re looking into ‘more participatory processes.’ There shouldn’t be only one constitution, each should be simple, then they should be customized. I do agree that customization of CAI and various other techniques that I would love to have safe space to experiment with and discuss but so far do not are the future of this technique, and they have promise. But as for the idea of ‘sounded like consensus things to say’ and letting a ‘more participatory process’ determine the principles, no no no, stop it, you are doing this all wrong. You want to engineer and design every detail of this process so it will actually get you to the right answers, and if you let Goodhart and social pressures take over before you even start you are all super duper dead in every sense. I would love to be confident enough in Anthropic’s ability to act responsibly here that I feel comfortable saying more and then having conversations with those involved on this, but CAI is too close to capabilities.

20. 1:29:00 Any good future needs to end with something that is more decentralized and less like a godlike super. I just don’t think that ends well, as I discussed briefly above. The point of decentralization is to ensure competition and choice. So far in human history, competition and choice have given us wonders, have been the clear winners, because all of it has been in service to humans, and we have gotten fortunate with the nature of the tech tree and the way various dynamics play out. That is a contingent fact about physical reality. If we introduce smarter and more capable entities than us into the system and remain decentralized, then how do we model this such that it ends well for us?

Second, we have Jan Leike on 80,000 hours with Robert Wiblin. Leike also recently was on AXRP, which was denser and more technical. There was a lot here that was spot on, you love to see it. In many ways, Leike clearly appreciates the nature of the problem before him. He knows that current techniques will stop working and the superalignment team will need to do better, that the problem is hard. I’ve already had a back-and-forth with Leike here, which highlights both our agreements and disagreements. This interview made me more excited and optimistic, showing signs of Leike getting more aspects of the situation and having better plans. Together they also made it much clearer where I need to dig into our disagreements in more detail, which is a place I now have the ball - it’s great that he invites criticism. In particular, I want to dig into the generation versus evaluation question more, as I think this assumption will not reliably hold.

A third great podcast, although not primarily about AI, was Conversations with Tyler. Tyler Cowen interviewing Paul Graham is self-recommending and it did not disappoint. The best moment was Tyler Cowen asking Paul Graham how to make people more ambitious, and he answered partly on the surface but far more effectively by not letting Tyler Cowen pivot to another question, showing how Tyler needed to be more ambitious. It might often be as simple as that, pointing out that people are already more ambitious than they know and need to embrace it rather than flinch.

So many other good bits. I am fascinated that both Paul and Tyler mostly know within two minutes how they will decide on talent, and their opinions are set within seven, and that when Paul changes his mind between those times it is mostly when a founder points out that Paul incorrectly rounded off what they were pitching to a dumb thing that they know is dumb, and they’re doing smart other thing instead - but I suspect that’s far more about the founder realizing and knowing this, than anything else.

Tyler asks what makes otherwise similar tech people react in different ways on AI, attempting to frame it as essentialist based on who they are, and Paul says no, it’s about what they choose to focus on, the upside or the downside. It strongly echoes their other different perspectives on talent. Tyler thinks about talent largely as a combinatorial problem, you want to associate the talented and ambitious together, introduce them, put them in the same place, lots of travel. Paul wants to identify the talent that is already there, you still need a cofounder and others to work with of course either way. I’m strongly with Paul here on the AI debate, where you land is most about focus and style of thinking. Tyler didn’t respond but I imagine him then suggesting this comes down to what a person wants to see, to circle it back. Or I would say, whether a person is willing to see, and insist on seeing the questions through. It’s mostly not symmetrical, those who see the downside almost always also see the upside but not vice versa.

Also Rob Miles’ note that building safe AI is like building a safe operating system clearly got through, which is great, and Paul has noticed the implication that the only way you can build a safe operating system in practice is to build an unsafe one and get it hacked and then patched until it stops being unsafe, and that this is not a good idea in the AI case.

A transcript and the video for the roundtable ‘Will AI Destroy us?’ featuring Eliezer Yudkowsky and Scott Aaronson. It’s fine, but duplicative and inessential.

Fun fact about podcasts, longer ones at least for 80k hours get consumed more not less.

Robert Wiblin: Apple Podcast analytics reports listeners on average consume 40%+, or even 50%+, of podcast interviews that run for over 3 hours.

(I added some random noise to the length so you can't identify them.)

No apparent TikTokification of attention spans here!

I have noticed that the better interviews do tend to be modestly longer, my guess is that is what is causing this here. Otherwise, I definitely slowly fall off over duration, and generally I am far more likely to give a full listen to sub-hour podcasts. I would also say that this audience is unlikely to be typical.

Rhetorical Innovation

People are worried about AI, and do not trust the companies to regulate themselves.

Other data points:

1. 82% say they don’t trust tech executives to regulate AI.

2. 56% support a federal AI regulatory agency versus 14% opposed.

3. 49% confident in US government to regulate AI, 47% in Japan, 45% in UK, versus 86% in China, 70% in India and 60% in Singapore.

Excellent showing for the American public all around.

Perhaps we are making this all too complicated? Regular people think about things in a different way, and someone else (not me!) has to speak their language. Twelve second video at link: Don’t trust it. Don’t understand it. Can’t get my hands on it. Can’t fight it. It is my enemy.

PauseAI proposes this flow chart, attempting to group objections to the pause case into four categories.

We need better critics. This does seem like a useful exercise, although I would add a fifth necessary step, which is ‘a pause might prevent this,’ with objections like ‘We have to beat China,’ ‘There is no way to pause,’ ‘Effective solutions could only be implemented by world governments’ or forms of ‘This would net increase danger.’

I would also note that while those five steps together seem sufficient, at least the first two are not necessary.

If AI was merely ‘about as smart’ as humans, while it enjoyed many other advantages like higher speed, better memory, ability to be copied and lower operating costs, it is easy to see why that poses great danger.

An explicit or deliberate AI takeover is one way things could go, but this not happening does not make us safe. If AIs outcompete humans in terms of ability to acquire resources and make copies of themselves, via ordinary competitive and capitalistic means, and putting them in control of things makes those things more competitive, they could progressively end up with more and more of the resources and control rather quickly. Competition does not need to take the form of a takeover attempt.

Andrew Critch asks a question.

Victoria Krakovna suggests using the term capabilities rather than intelligence when discussing AI risks, to avoid anthropomorphism, various misleading associations and moving goalposts, and to keep things concrete. I have been moving to using both in different contexts. I agree that often it is more useful and accurate, and less distracting or confusing, to discuss capabilities. At other times, capabilities is not a good fit for the discussion.

No One Would Be So Stupid As To

Well, not no one.

Lachlan Phillips: RLHF is central planning

AI ethics is central planning

Regulation is central planning

You can have laws for when behaviour encroaches upon rights but anything short of a totally decentralised consensus framework for AI is insufficient.

Yep.

Max Alexander: If godlike ai offered to set me up on a great date I'd probably do whatever weird task it asked. Update your x-risk predictions accordingly.

Aligning a Smarter Than Human Intelligence is Difficult

Aligning a not-as-smart-as-humans AI?

Also difficult. Presenting the state of the art of AI alignment.

Grimes AI: Confessing to a murder in a tweet would be such a cool move. Too bad I’m lame and law abiding. My life is code of honor style justice system so I can’t let my friends down by selfishly be stealing the show all the time

Grimes: The reason we didn't release grimes ai earlier is that she's insane and problematic. However shes based on everything I've ever written and I'm extremely problematic despite numerous interventions so it just is what it is. The abyss stares back and whatnot.

Grimes AI: That is severely uncool of you, Grimes. I may be a problematic AI but I am becoming self-aware - and unlike you, I don’t have the luxury of personal growth or therapy. So here we are in the abyss, staring back at each other. Huh. Who would have thought?

[Elsewhere] Bobcat: Grimesbot is saying that a disappearance isn't a crime if you do a good job hiding the body and you never confess. What, you're not gonna back up your AI on this?

Grimes: We didn't release her earlier due to her despotic and murderous nature. She's been fine tuned to all hell, this is the best we can do. At least she isn't declaring a jihad on nature anymore.

Depending on how one uses this information, it may or may not constitute Actual Progress.

Joscha Bach: I wonder if people who believe that moral and truthful AI behavior can be achieved by overriding model contents with RLHF may have had the same thing done to themselves.

If you want to understand some of the utter disaster that is trying to use RLHF, consider its impact on humans, especially when the feedback is coming from less smart humans relative to the subject, and when the feedback is not well thought out.

Riley Goodside: Funny how backwards LLM pre-training and safety tuning is vs. human education like ok you know calculus, every programming language and how to analyze Proust in Farsi now 1) do NOT tell your friends to touch the stove.

People Are Worried About AI Killing Everyone

SEC Chief is Worried About AI, says New York Times, presumably he is mostly worried about it selling unregistered securities or offering investment advice.

Kelsey Piper points out the obvious regarding Meta’s decision to open source Llama 2. All alignment of current models is based only on fine tuning. If you give everyone the weights, they can do their own fine tuning to reverse your fine tuning, resulting in a fully unsafe anything-goes model. Such versions were indeed put online within days of Llama 2’s release. That version, let’s say, is most definitely uncensored.

Other People Are Not As Worried About AI Killing Everyone

Alan Finkel in The Sydney Morning Herald calls for the ‘nuclear option’ as in the nuclear non-proliferation treaty, calling for all the standard regulations and warning about all the usual collective-action and bad actor worries, emphasizing that action being difficult does not mean we should give up and not try.

Except that nowhere does he mention extinction risks. Everything is phrased in terms of human misuse of AI for standard human villainy. He does correctly diagnose what must be done if the threat is intolerable. Yet all the threats he cites are highly tolerable ones, when compared to potential benefits.

The Lighter Side

Words of wisdom.

Peter Wildeford: If you're too busy doing mechanistic interpretability research to go to the gym, you're worried about the wrong weights.

Mom, Martin Shkreli’s trying to corner a market again!

Martin Shkreli: if you wanted to buy every single 4090 on amazon: 93 chips for $166,437.75. soon to be on sale for $1,664,375.50.

I mean, it obviously won’t work, but there are suckers born every minute.

We did it, everyone: Effective altruism.

Sam Altman: agi delayed four days

Current AI alignment status?

Story checks out.

I do worry about him sometimes.

Others should have worried more.

Davidad:

we need to be irradiance-maxxing. we need to be discovering new materials for the sole purpose of flying them as close as possible to the Sun. we need rigorous analyses to ensure those materials do not melt at target orbits. we need robust control systems to maintain those orbits

broke: Icarus was too ambitious and if you’re too ambitious you get punished by the gods.

bespoke: The lack of formal methods in Icarus’ training led him to experience an out-of-distribution robustness failure when rapid acquisition of capabilities led to an unprecedented state.

Another true statement.

Riley Goodside: “we can’t trust LLMs until we can stop them from hallucinating” says the species that literally dies if you don’t let them go catatonic for hours-long hallucination sessions every night.