43 Comments

I don't think it's good to use chatgpt to win arguments, so I asked chatgpt and it had this to say:

"Lack of Emotional Connection: If conversations are reduced to just point-by-point arguments, it may take away the emotional and empathetic aspect of resolving disagreements. Relationships thrive on emotional connection, not just logic and structure.

Over-reliance on External Input: Using an external tool like ChatGPT for every disagreement could create dependency, potentially undermining authentic, spontaneous communication.

Escalating Arguments: While well-structured arguments can be helpful, they can also make the conversation feel more like a debate or competition, which might intensify conflict rather than resolve it."

Expand full comment

On 𝕏 I frequently use ChatGPT to "win" arguments, but I use this in specific ways, and never for the direct benefit of my interlocutor, who has by the time I start posting screen shots from ChatGPT, has already demonstrated fallacious thinking that GPT is very good at detecting and calling out. Rather, it is for the benefit of the mostly-silent audience watching the exchange.

I do it by taking screen shots of the full conversational context and giving it to ChatGPT so it can craft a reply that is quite specific to the conversation at hand.

How does my interlocutor react? Usually with outrage and dismissal, accusing me of "resorting to using AI". I actually find this funny, because it's an actual deflection from what has happened: their fallacies have been called out, if not having their argument actually shredded. In fact, I have a standard rejoinder to this, which is a reply to the pinned post in my profile:

“I use Al to call out those with weak or fallacious arguments. I only post Al output that I agree with and where I am confident in its accuracy. If that upsets you, or you attack the tools l use, then you have already lost the argument. Argue with facts, not sources.”

And this is the crux of the matter: if the AI is right, and thorough at it, then it's right, and it doesn't matter that it's AI generated. The emotional detachment of the AI is actually a *benefit* in these cases, because my interlocutor has frequently already deployed arguments from emotion or ad hominem. Usually the reaction of my interlocutor exposes even deeper levels of bad faith, and I believe that is a *good* thing for public discourse.

Expand full comment

The issue is not using chatgpt to analyse arguments, that's fine.

The problem is using it as a judge of who's right. AI as it's trained today is sycophantic, it will literally try to please and agree with who's asking regardless of the truth of the argument.

It's the equivalent of calling your best friend that would never disagree with you and ask their opinion. Of course they'd back you! And of course it's not valid and you should be pissed off if someone does that.

Expand full comment

A general xrisk prevention thought experiment I am curious about: imagine you are dedicated to the goal of preventing nuclear weapons from becoming an existential threat to humankind. I will send you and a handful of others you nominate back to 1901. You will arrive within various major polities with levels of significant authority, and fluent in local languages and customs. You will be able to tell anyone you like anything true about nuclear weapons and nuclear power (including military, geopolitical, social and environmental impacts) and they are guaranteed to believe you, without requiring proof.

You are faced with the following constraints:

1. You cannot name the specific technology (you must refer to it as XY).

2. You cannot provide any specifics about how the technology functions.

3. You cannot provide any historical specifics (e.g., you could explain that global superpowers will emerge with massive stockpiles of weapons built on this new technology which are capable of obliterating the world multiple times over, but you cannot give any hint as to which powers you are referring to: folks will of course extrapolate but you can't correct or lead them to a specific 'correct' extrapolation).

4. You cannot provide any specifics as to timeframe except 'within the next 100 years'.

Given these constraints, what you would do to realistically achieve your goal? What goal would you consider realistically achievable?

Expand full comment

I find all comparisons of ASI to nuclear weapons fallacious. To say "The advent of ASI will be like giving everyone a nuclear bomb," is a proposition I will only grant if you grant the further proposition, "The advent of ASI will be like giving everyone a super nuclear bomb detector and defuser." I find the fact that xrisk mongers never think past the first proposition to the second to be curious, and hard to explain without a deeper theory about their motivations, such as confirmation bias or power seeking.

So in the case of your thought experiment, I would have to modify it from "nuclear weapons" to "nuclear energy" and discuss both the good and potentially destructive aspects of its use, and also describe the equilibrium likely to be formed when a huge number of actors with diverse interests all have this energy, and most of them still hold self-preservation as a cardinal value.

Expand full comment

Lol, AI defuses nothing about AI. Make a good deepfake detector first. Also extinction has no recovery.

Expand full comment

A primary reason is that many in X-risk believe that we don't know how to align an AGI basically at all. The most likely outcome is not that we'll have a dozen AGIs around whom are mostly aligned to humanity then one gets loose in a world prepared for that, which is what your "defuser" scenario would be. The most likely is that we create a system vastly smarter than us with very different goals, and this goes catastrophically poorly for us.

Even if we did have AGIs that we could turn to very useful work, the most likely method we would use is hobbling them with slower checks and balances, which makes them weaker against an unshackled system.

There's also the speed of takeoff. A decent amount in X-risk believe that once we reach a critical threshold we're likely to see rapid self improvement, then that greatly reduces the time for defensive systems to be produced. Other technologies have similar problems, but are far more recoverable.

Part of the whole problem is getting to a good equilibrium safely, which is why X-risk people would like regulation or treaties to be careful and slow. They still very much want the benefits of advanced AGI! They just think we have little reason to believe that our ability to align models is up to the large task ahead of them.

Expand full comment

Exactly!

Expand full comment

Would you be able to demonstate a small scale example of it? If they are guaranteed to believe you, it seems that nuclear winter alone will discourage in its direction.

Expand full comment

Given that our world came close to a nuclear WW3 multiple times but that we got through the cold war without one and now one seems very implausible, I'd honestly hesitate to change anything at all.

Maybe, after nukes are a known thing, try and enhance non-proliferation, stop countries like Pakistan getting any? I'd definitely stress in the 90s, after the fall of the USSR, that sanctions will *not* stop North Korea form getting nukes and that as horrifically costly as it might be, warfare is probably the only option.

Expand full comment

Podcast episode for: AI #86: Just Think of the Potential.

https://open.substack.com/pub/dwatvpodcast/p/ai-86-just-think-of-the-potential

Expand full comment

> We could move quickly down the cost curve, and enable rapid deployment. In theory yes, but I don’t think the timelines work for that?

It would be delightful on so many levels if AI turned out to be the Starlink of energy abundance, but I agree, this doesn't seem like enough of a demand increase (even just within large-scale / industrial demand) to make that kind of step-change difference.

Expand full comment

Even three (Microsoft, Google and Amazon) massive tech companies taking a serious direct interest in nuclear power construction is a seismic level event for this industry. All they need to do is for their army of lawyers and lobbyists to fundamentally change the landscape and create the paper trail precedent.

If just one of them can also establish a blue print for sensible construction and almost anyone else can copy it, that would be the icing on the cake.

Expand full comment

> I’m curious if anyone does have a half-decent version [of a relationship/NSFW chatbot] - or kind of what that would even look like, right now?

Because the intersection of "LLMs good enough to hold a conversation" and "LLMs whose owners are fine with using them as a publicly accessible relationship chatbot service" is basically zero, the state of the art is mostly found in imageboard posts right now, and involves using the OAI and Anthropic APIs directly via a few different confusing open-source UIs, and lots of arguing over the best "jailbreak" prompt template for each model or how best to write character descriptions.

Quality tends to vary wildly depending on how much effort has been put into a character description, but at their best, Claude 3.5 and GPT-4o girlfriend/boyfriend bots are...not terrible? They can maintain a consistent personality and opinions, they're not *immediately* repetitive, and they're certainly not dumb (unless you intentionally make them so). But there is the same tendency for them to be more agreeable and willing to go along with the user than the character description ought to imply, and it's very easy for Claude in particular to get locked into repetitive loops over the course of a long chat.

Expand full comment

I'm surprised they're not using Llama instead? Shouldn't ability to fine tune and not having to jailbreak around the restrictions, plus lower price, make up for the lower quality?

(Which you'd think would be the quality floor, but very obviously isn't.)

Expand full comment

Some people certainly do use Llama for the reasons you describe. I can't answer as to the quality outcomes there, but for GPT and Claude, fine-tuning seems unnecessary and jailbreaking is much easier than you might expect - easily shareable, relatively token-light prompt templates are sufficient for whatever behavior you want. As far as price goes, so far the cost over time of using top-end OAI and Anthropic models via my own API keys has been cheaper than a Netflix subscription (though I'm a very slow writer).

Expand full comment

Ziv. I appreciate the amount of work you put into your posts. Unfortunately, I do not have the time to read the entire articles and was hoping to use an AI to summarize the major points. Unfortunately, substack links don't seem to be accessible and I could not copy and paste the text. Any ideas?

Expand full comment

Literally Ctrl+A on a computer, Ctrl+V to dump it all into Google Docs, maybe delete the junk at the top if you care, should something like 90-10 it if that's what you want?

Expand full comment

Ziv. You are generous to reply. I'm very familiar with the computer key shortcuts, but unfortunately I read all of substack on a phone where the shortcuts don't work. As someone who is retired and likes to read broadly, I appreciate your In-Depth expertise. I suspect that there are others out there like me who would appreciate an abridged version of your posts. Maybe a skilled acolyte of yours could produce such a product! In any case, thank you again for all your insights.

Expand full comment

In Chromium mobile browsers, press and hold over any text. This should bring up options including Select All, and then should pop up a menu including Copy. Same method to Paste that into your favorite LLM.

If you're asking others to do that for you, sorry. If you're suggesting someone with the skills should write a plug-in that does all that with any web page, I'd like that too, but certainly not for Zvi's writing.

Expand full comment

I also use the app, but you can choose to receive the email version as well. It's a lot easier to feed the email to a summarizer.

Expand full comment

Two quick observations and one longer one:

1. I am missing a lot of context so maybe I have misunderstood a lot/everything, but is not OpenAI hiring the former CISO of Palantir the best (AI) news this week, month and maybe quarter? Again, context and lack thereof but if you had asked me to come up with organisations for which infosec was treated as an existential priority, Palantir would be on that list. If you asked me to come up with a list of people who solely by virtue of their role I would expect to be extremely resistant to intimidation and bullying in relation to their professional responsibilities, I do not think I would have thought of "CISO of Palantir" but in retrospect that certainly seems to fit the bill?

Of course, if he leaves in the next year or two, that would be even worse news than this is good news.

2. Eigengender goes all out on self-confidence, Roon makes a valid point that implies the real point which Eigengender seemed invested in not getting: at this stage I would not be comfortable backing my "thirty seconds of mild scrutiny" against Elon's ability to make it happen, did you think you would actually see in your lifetime a rocket booster coming gently down to earth to be caught by a robot gantry???

3. It is ironic that one of Daron Acemoglu's canonic examples is highly suspect: whilst he identifies former colonies with "non-extractive institutions" quite interestingly and carefully, and finds strong results even stripping out the obvious outliers (the Commonwealth), it is hard not to think that the result is even more heavily confounded with genetics than diet studies.

That idiosyncratic aside apart, the more subtle point from his work is that he postulates that extractive institutions survive despite obvious outcompetition because they are explicitly motivated to remain extractive to preserve factional power, because "businessmen" are an obvious potential competing source of power (see eg Latin America, but also Europe, China, America under Lina Khan, .... ).

Not hard to see how this could play out in the context of AI !!

Expand full comment

I suspect that Sheng Thao and London Breed are not using ChatGPT themselves. Rather, they hire a social media manager to tweet for them, who then secretly uses ChatGPT to do their job.

Expand full comment

"'Are you saying people should not use reasoning to deal with uncertainty?'"

There's nothing wrong with reasoning under uncertainty. Far, far too often reasoning under uncertainty in practice means "I have conceived of a possibility, therefore you should treat it as a certainty".

If you're going to reason under uncertainty - which you both should and must given the lack of certainty in virtually every domain - you should both admit that you're doing it and do your best to quantify that uncertainty. If you can't quantify it that's okay too, but you should admit it and adopt some epistemic humility.

Expand full comment

> “but with AI’s doing everything, how will humans have meaning? For that matter, how will they survive economically?”

People keep asking this question and keep forgetting the obvious answer: we'll ask AGI to for nanobots that will tweak our brains to give us the feeling of 'having meaning'. And it will give them to us, assuming its aligned - and meaning thou shall have.

Expand full comment

I think Scott’s Art Turing Test cheats by choosing AI-like human art, rather than typical examples.

Thus, we learn that..

- Some Renaissance artists were rubbish at perspective

- Jean-Auguste-Dominique Ingres does Neoclassicalism/Academicism in a way that’s so cold we think he’s a a robot. (The Deckard of painters).

- Really? That anime girl was painted by a human? It wasn’t very original.

Expand full comment

Re. The AI summarising breakup texts… I am expecting we will soon having the AI telling us “Dude, she’s just not that in to you.”

Expand full comment

“ I could be alone for a very long time, without interaction with other humans, so long as I had sufficient supplies, quite happily, if that was a Mr. Beast challenge or something.”

Or, I don’t know, maybe if the government ordered a Covid19 lockdown or something. Totally wild science-fiction idea of course, that would never really happen.

Expand full comment

"People are all right, I guess, if you like that sort of thing. Don't get me wrong, some of my best friends are people. Well, you know, ish." [waggles hand]

Expand full comment

You know how it is, you’re really into writing code for your startup, so you just order takeout food to be sent to your apartment. By the time you’re done, it’s fall 2020. You turn on the TV, and apparently there was a pandemic or something.

Expand full comment

To be honest, I consider myself an introvert and spend most of my time online anyway, but he Covid lockdowns were devastating to me. Turns out that even if I only need in-person contact once every week or two it *really* matters when it's gone, and online chats aren't a full replacement even at much higher volumes.

Also, trying to isolate without even online chats would probably drive me loopy within a month.

Expand full comment

Re. AI Dungeon being better than any of the current AI girlfriend offerings … yes, my highly subjective impression was that AI Dungeon was a better AI, and the trade-off here is that, should you so desire, you can get an AI girlfriend that is less smart, but (due to the lack of guardrails) more willing to do it with you.

A) AI Dungeon was pretty uncensored too, though they had a rough patch where the false positives were really annoying,

B) Some readers are probably wondering if this trade-off also exists, with, you know, real human beings.

Expand full comment

Zoo, I bought a new iPad Air 13" the first week they were available, and am very happy with it. It Just Works.

I can't advise on the suitability for testing Apple's AI features, but it seems to offer better value than the iPad Pros, with only slightly less powerful hardware.

Expand full comment

Excellent analysis of “The Player Of Games.” I think Amodei’s misreading of what happens there feeds directly into his to me highly dodgy thoughts re liberal democracy’s triumph over its rivals, how likely that is and the means he sees being employed to bring it about. I wonder how much “naïve readings of the Culture novels” has to answer for, in terms of potentially ruinous courses of action being embarked upon by very powerful tech people…

Expand full comment

"Liberal democracy" is almost an oxymoron - democracy and human rights are fundamentally in tension - *rights* are things that majorities ought not to be able to infringe.

I'm all for liberalism. Majoritarian tyranny is bad. Democracy's value is mostly as a relief valve for tensions that would otherwise lead to bloody civil war.

Be careful what you wish for.

Expand full comment

Totally agree. Liberal > democracy.

Expand full comment