Against Modesty’s Bailey

Modesty arguments often say that you should mostly or entirely bow to ‘expert consensus’ or the views of particular others, and who are you to disagree.

It has been a few years since I’ve properly addressed this so: My answer is that you are you. Other people are saying things for a wide variety of reasons, many of which are not about them paying attention and focusing on seeking this particular truth. Those people make mistakes all the time, and often have other motives and influences at work, especially social pressures and information cascades.

Them being as smart as you, or smarter than you, does not exempt them from this, and them being higher status or credentialed or cooler definitely does not exempt them.

A smart informed person sincerely thinking [X] can easily cease to be evidence for [X], once you have thought sufficiently about both [X] and why that person thinks [X].

Think for yourself, schmuck.

Or, as I once put it: You Have The Right To Think, also the moral duty to do so.

This post covers Eliezer Yudkowsky making a narrower claim than mine, about not conflating status with smarts and epistemic authority, and my documenting related discussion that happened on Twitter.

This concerns a particular historical interaction, but what matters is the general point.

I then briefly extend this to my broader claims.

Modesty’s Bailey

I think, in the exchange I will quote, Rob Bensinger is reaching but centrally correct.

Modesty arguments often are this, trying to conflate the motte and bailey as usual:

1. Motte: You should treat the opinions and arguments of others as meaningful information, and take it into account as Bayesian evidence.

2. Bailey: You cannot rationally weigh your inside view more than the views of others, or disagree much with expert consensus, or whoever is considered epistemically in charge around here, unless you can cite specific information you have that they don’t have access to. If you do, you are making a claim of high status, and that they are relatively stupid.

I affirm Modesty’s Motte.

I am heavily against Modesty’s Bailey.

Epistemic Peerage

Note the bolded passage (bold is mine) in the exchange below:

Eliezer Yudkowsky: Folks also don’t seem to realize that “epistemic peer” is a term of art in epistemology for people so impressive you’ll defer to them in your discipline without even needing to hear an argument. It doesn’t just mean “colleague you treat with respect”.​

It doesn’t mean you should mostly or fully defer to such a person. That would imply they are an ‘epistemic superior,’ someone whose opinion carries greater weight than your own, or even an ‘expert’ or ‘guru’ that you should defer to entirely except under special circumstances.

Epistemic peer means that, if you both have access to the same information, and they have not yet explained to you their reasoning, you should not presume one of you is better at this than the other, so you should probably take the two unadjusted opinions and split the difference.

Once they explain their reasoning, you should instead evaluate their reasoning. Sometimes they will have better reasoning than you, sometimes it will be worse. Remember the law of Conservation of Expected Evidence. If you could move towards their belief but not away from it, you should adjust your beliefs now.

The Exchange

I’m reproducing this discussion primarily because I think it has valuable things to say about how to do, and how not to do, epistemics, rather than because of the OP story being about Leopold in particular. Leopold, indeed, would later wisely recant the central conceptual point, in the form of creating the hedge fund Situational Awareness, whether or not he would agree that the two are the same principle.

Aashish Reddy: Eliezer on encountering Leopold, which some of you will enjoy:

Eliezer Yudkowsky (on LW): I remember the first time I met Leopold Aschenbrenner. He questioned me about why my timelines weren’t longer and finally said, “Why aren’t you updating at all in the direction of all the smart people saying longer timelines?” (aka: OpenPhil doctrine of 2050), “At this point you’re not even pretending to be rational.”

I hesitated briefly, because I knew it would not end well to answer honestly, but there wasn’t any good ending past there; and then I told him honestly that I did not consider those people to be my epistemic peers.

I don’t usually consider myself good at reading faces, but wow, the sheer LOOK of contempt and scorn that crossed his face; before he turned away and walked off without another word, dismissing me from further consideration. I will never forget that moment -- because of it being such a LOUD face and my actually being able to read it, not because it had any influence on my estimates, of course.

Eliezer Yudkowsky: ...and zero comments on the point that, in retrospect, Leopold appears to have been flat wrong that I ought to defer to his very respectable people about timelines, and my “Biological Anchoring: The Trick That Never Works” was ~100% correct? Cool. Y’all do you.

An obsession with vibes so grand that all grading of predictive outcomes is beneath it. How could you stand grandly above two people if you were also noticing that their predictions were not equally wrong? You cannot. So you are blind to the outcomes themselves.

Folks also don’t seem to realize that “epistemic peer” is a term of art in epistemology for people so impressive you’ll defer to them in your discipline without even needing to hear an argument. It doesn’t just mean “colleague you treat with respect”.

Oliver Habryka: I mean, I think the common-sense use of the term would have also been fine here? Like, it seems true and clear to me (from other things Eliezer has said) that he also does not consider them his epistemic peers in a more common-usage sense of the word.

Rob Bensinger: Seems vague, and dependent on interpretive details. And seems like an inference rather than an observation, and one that moves further away from the core of the disagreement with Leopold. (And if we readily read “epistemic peer” that way, we lose a very useful concept.)

Oliver Habryka: I mean, in as much as people’s reaction is to “broad disrespect for Leopold’s epistemics by Eliezers” then I think that reaction seems appropriate. I think it’s clear Eliezer does not have much respect for Leopold’s epistemics (and to be clear, neither do I).

Trying to invoke a misunderstanding about the term “epistemic peer”, feels like it’s a distraction in as much as that is the core disagreement.

Rob Bensinger: Leopold’s motte is “you should treat your colleagues with a modicum of respect and politeness”, and his bailey is “you should weight your own inside view no higher than Cotra’s, Carlsmith’s, etc.” Shrugging at the distinction lets him declare full victory.

Oliver Habryka: Is... it? I am confused. Is Leopold even participating in this conversation? Like, that was his motte with Eliezer? I don’t think so, or at least it doesn’t match my model of him, or the way he carries himself.

Rob Bensinger: This seems like a pretty general / common description of how modest epistemology works? You conflate the social hierarchy with the epistemic hierarchy; you conflate norms of collegiality with norms of deference, so that “arrogance” carries the moral valence of “being a dick”.

... And not just the valence of “being factually mistaken about some third party’s level of expertise”.

That’s where the outrage and force of Leopold’s action comes from, if Leopold’s mind is reasonably typical on this dimension. “How dare you BE SO RUDE”, not “... THINK 2+2=5”.

Oliver Habryka: Huh, I guess that doesn’t match my model of Leopold at all. Leopold dismisses almost everyone!

IMO the thing that Leopold was doing in that interaction (as I think I was there, or at least heard about it soon after) was demanding respect for the EA elders + the gods of the marketplace the way you demand respect for a king.

Rob Bensinger: I’m not sure I see how those things are incompatible with what I said? I think “show a modicum of professional courtesy to your colleagues” is often what “treat EA elders like they’re your kings” feels like from the inside.

Eliezer’s Explanation

Eliezer offers a fictionalized version that did not happen as written, but that to him illustrates the core thing that happened. He may or may not be right, I was not there.

Eliezer Yudkowsky: Let me translate [the OP Eliezer quote] out of Lesswrongese.

Leopold: Why do you think AGI could possibly happen as soon as 2030? The people *I* think are cool say 2050.

Eliezer: Blah blah actual reasons blah blah.

(History: Later shows my reasoning to be correct. You can read it in "Biology-Inspired AGI Timelines: The Trick That Never Works.")

Leopold: But do you think you're *smarter* than the people *I* think are cool and who put their median timelines at 2050? You should adjust your position toward their position, even if you disagree with their arguments, because you're no smarter than them. That's what *rational* people are supposed to do.

(History: Will later show all their arguments were wrong for exactly the reasons I said they would be wrong.)

Eliezer: If you are literally going to phrase it that way and try to put me in that kind of conversational corner, then fine, since I cannot win this conversation I will simply say what I believe to be true: Yes, I think I'm a sufficiently better forecaster that I've already taken their arguments into account, and that they haven't understood my arguments, and so I trust my own decision not to treat them as authorities whose opinion I ought to weigh on par with my own.

(History: Good call.)

Leopold: CONTEMPT

Eliezer: (Says nothing for the next few years because at the time Leopold is just a punk kid; and even if I immediately suspect that he is EA-bait who will get promoted to a position of power and then do a lot of damage, he is right now just a punk kid, and if you are a senior guy with a little power, you don't bring your own weight to bear against a kid.)

Leopold: (Gets promoted to a position of high power at eg FTX Philanthropy.) (Tries to torpedo my own suggestions and initiatives.) (After the ChatGPT moment makes it publicly acceptable and cool, switches to saying that AGI is coming in 2027 and we need to start an arms race with China about it.)

(Blows up a hedge fund.)

I should emphasize: This “translated” conversation DID NOT HAPPEN. Leopold did not say those things; I did not say those replies. The original on LessWrong is my best recollection of how the actual conversation went.

‘Blows up a hedge fund’ goes too far in describing what happened to Leopold’s hedge fund, Situational Awareness. It declined from $45 billion in assets to $10 billion, after it used too much leverage and got caught in margin calls, forcing a sale of its public book, but it did not per se blow up.

Leopold’s entire approach to his hedge fund did represent an extremely not-modest view of the world, to not only bet on his own ‘situational awareness’ and analysis of the market, but to do so at high leverage and for amounts where he was likely impacting prices, and thus vulnerable to additional pressures.

And you know what? He got too aggressive with leverage, but he was fundamentally right to be immodest, and to say he could better predict prices than the market, because he was looking closer, thinking better and more situationally aware. He made lots of great trades. The Efficient Market Hypothesis is False, and it is people like Leopold who make it do an imitation of true.

Man, Machine, Self: I think much of the heat came from them correctly correctly picking up, in non-lesswrong-ese, "saying someone is not your epistemic peer" sounds a lot like you're insulting them and calling them stupid.

Eliezer Yudkowsky: There is a truth which destroys the minds of EAs; as hinted by Aumann and explicitly argued by Hanson, to not defer to someone's authority, you must think you are smarter than them. This destroys people because their brain processes smartness as status.

Leopold tried to back me into a corner where to not defer to his friends, I had to speak the terrible, socially unutterable truth that I did not consider them just as smart as me. I bit the bullet and received his look of utter contempt in exchange -- and was right, of course. But imagine how hard it is for most EA kids to decide that they are ready to do the same, when everyone around them is telling them to be humble and not to get above themselves. And that's how and why OpenPhil was groupthink.

Zy (QTing the previous quote): Rationalist Hadith

QC: the most important thing i learned from rationalists is that smart people can be incredibly stupid. you have the god-given right to refuse to defer to anyone’s authority no matter how smart they seem if they’re talking nonsense, and intelligence is no substitute for wisdom.

Eliezer Yudkowsky: ...And now people who have no word for what I mean by "smart" are naturally reading it as the status thing. But of course they have no word for it, so I was foolish to call it "smart" instead of "shmeerp", a sum of knowledge and accurate self-assessment of relative validity by which you'd correctly decide that it was fine for *you* to decide the AGI-2050 arguments were trash even though *other people* thought those arguments were *not* trash.

The truth that destroys EAs is that you've got to believe you've got higher shmeerp than the AGI-2050 crowd to shrug off their arguments; and obviously whatever the heck shmeerp is, it would be very high-status to have it; so they think it's terribly rude to say outright that you think you have higher shmeerp than high-status other people -- and *they* often do actually usually just say "smart", there.

And then they are lost, because shmeerp does not track their sense of what status MUST be in order for the universe to MAKE SENSE, and so they have to redistribute shmeerp and reinterpret the whole universe of arguments accordingly, following by unseeing the universe.

And that's how OpenPhil ended up believing in AGI 2050 while simultaneously and to this very day earnestly congratulating themselves on how they were great at avoiding groupthink.

A Demonstration That Eliezer’s Translation Accurately Describes Many People Whether Or Not It Describes Leopold

As in, here is someone else’s pejorative translation of events, which confirms it.

Will Rinehart:

>be Eliezer "I invented Bayesian rationality" Yudkowsky
>meet young Leopold who actually works on the frontier
>kid asks "why aren't you updating at all in the direction of all the smart people?"
>you hesitate for dramatic effect, then drop the ultimate high-IQ line:
>"I do not consider those people to be my epistemic peers"
>Leopold's face contorts into pure, unfiltered contempt
>he turns and walks away without another word
>you stand there alone, triumphant
>"wow, I actually managed to read a facial expression for once. historic day"
>this of course has zero impact on your probability estimates because peers are for midwits

This is exactly the attitude Eliezer is describing:

1. If you do take your opinion into account more than someone else, you are calling them stupid. More than that, you are claiming ‘ultimate high-IQ.’

2. Status in context should largely equate to smartness.

3. Learning that [X] is high status should update you towards [X]’s opinion, even if you had already considered [X]’s arguments.

Pay attention to both the literal contents, and also to the vibe of utter contempt, not only from Leopold but also from Will Rinehart.

As in, contempt for the idea that anyone might think for themselves, or consider arguments rather than who else believes what.

Will’s contempt persists, despite Eliezer being vindicated on the substance.

Later, of course, Leopold wisely got over it, and indeed took it too far.

Wrong, Stupid and Low Status Are Three Distinct Things

Most people only partially make the relevant mistake.

Each correlates with the others, but mostly people know they are distinct.

It is most common to equate calling someone wrong and calling someone stupid.

1. Only a stupid person would be so wrong.

2. You would only dare think they were wrong if you thought they were stupid.

3. You would only dare call them wrong if you thought they were stupid.

4. You know that calling them wrong will be heard as calling them stupid.

Whereas, of course, smart people are wrong about things all the time, and stupid people are right about things all the time. Sometimes the smart person is wrong, and the stupid person is right, or the smart group is wrong and the stupid group is right, either because or in spite of superior information.

The other links are usually seen as more tenuous.

Of course people constantly use accusations of stupidity or wrongness to try and lower someone’s status, or smartness and rightness to raise it. Most people in most places still push back hard against attempts to too systematically equate smart or right with high status, or especially stupid and wrong with low status. We realize they are not the same, and also the monkey brain knows that there be dragons.

The danger comes in places that do systematically equate intelligence and rightness with status. Academia is the most famous version of this.

LessWrong-Yudkowsky style rationality has this as an obvious failure mode. You have to constantly push back against where the high status person says ‘think for yourself’ and everyone says ‘yes, as the guru says we will think for ourselves exactly as he told us to.’ There is no amount of hammering ‘no I am not a guru, at most I am an epistemic superior, I really am saying to think for yourselves’ or ‘we are not a cult’ that will consistently fully stick. In my experience rationalist circles manage, after much effort, to do a good overall job at this, but not a great one.

Effective Altruism is set up to have a maximally bad version of this mistake. There is tons of pressure to quantify all the things, and to maximize quantified results, and to bow to the legible, and a lot of demand to be told how to figure out how to think and do in order to do the most good, and thus be good, and so on. Results are often prioritized over epistemics. This all puts extra value on consensus, and various forms of high status often end up equated with correctness or intelligence or both.

(As usual, to avoid related implications, my caveat that while EA has lots of issues, we criticize them so much exactly because they are worth taking the effort to criticize, and if you don’t believe that you should see the other guy.)

A Quick Survey Of Some Reasons To Not Be Epistemically Modest

The essence of my broader claim is that, in practice, modesty arguments get screened off by sufficient attention and understanding. Modesty is a method for when you cannot budget sufficient attention and understanding, which is most of the time.

You Have The Right To Think. See that post, and also the extended discussions in the comments between me and, among others, Eliezer Yudkowsky and Robin Hanson.

The post was initially motivated as a response to Robin Hanson’s review of Eliezer’s highly related book Inadequate Equilibria, which discusses when you might be able to do something unusually well.

Actual modesty, in practice, happens for a mix of complicated reasons, as I broke down in On Robin Hanson’s ‘Social Proof, But Of What?’.

Once you have invested enough to understand the relevant questions and form an opinion, and have considered the arguments and other causes behind people’s opinions, you can and should substitute your judgment for theirs.

Often you should do this despite the ‘experts’ or other epistemic would-be authorities having more relevant knowledge than you, and having devoted more time to related questions than you, outnumbering you, being higher status and more respected and credentialed than you, and perhaps being smarter than you.

Your investment of sufficient attention and understanding can be and often is fractal. As illustrated in Leaders of Men, you can confidently point out where an expert, who knows ball far better than you, is failing to properly understand an aspect of ball. In Zeroing Out, I explain how you can know a consensus estimate is wrong in a particular direction, even if you could not hope to create, on your own, an estimate as accurate as the consensus.

This, ideally along with a track record of testable predictions and the logic behind them, gives you many tools by which to evaluate whether someone, or some group, is functionally an epistemic peer or superior in a given context.

The obvious mistake is ‘I disagree with them, therefore they are not an epistemic peer,’ which would defeat the whole purpose. What you can and should do is adjust your evaluation of sources based on the quality of their arguments.

This blog has always been about not being epistemically modest, most famously:

Zvi Mowshowitz (February 9, 2021): I fully reject the idea that smart people can’t improve on the margin on ‘expert’ opinion, period. Welcome to 2021. Modesty shmodesty. 

Covid-19 and our response to it demonstrated, basically, ‘modesty shmodesty.’ AI only pushed me further in that direction. I now have an extensive track record to look through, and see how things are going.

There is also an extensive track record regarding claims around AI timelines, the topic of the original exchange between Eliezer and Leopold. We have seen many, including Leopold, radically shorten their timelines (Leopold as late as August 2021 was only estimating 25% chance of AGI-level systems by 2070), based on evidence that was mostly foreseeable in advance, combined with shifting social context.

Against Modesty’s Bailey

To close, here is a quick list of the top 10 reasons to decide to disagree with an expert or general consensus, either locally or globally, including a market price, or with others who might otherwise be presumed to know better.

1. You know something they don’t, or know unique reasons you can trust key info.

2. You have devoted more attention to something relevant than they have.

3. They have not priced in or understood your reasons or reasoning.

4. You checked, and their reasoning is bad and they should feel bad.

5. No, seriously, often the reasoning is really dumb, or obviously motivated.

6. Social desirability and other biases impacting the consensus.

7. Information cascades and games of telephone distorting consensus.

8. You can explain their statements in other ways that do not track truth, and your model predicts they would say that even if it was false.

9. They seem to be lying to you, including ‘for your own good’ or the ‘common good.’

10. The demands for modesty and adherence to a particular epistemic authority are selective and asymmetrical. Why are you supposed to adhere to this particular person or group, rather than others?

On top of the reasons to not be epistemically modest, there are strong reasons to state your immodest beliefs: It is often more helpful to present your own views and your own reasons, rather than to mirror everyone else, to combat the reasons above.

Thus, I reaffirm Modesty’s Motte and re-reject Modesty’s Bailey.