❦
A review of the Fable annotation pass over Book II, “How to Actually Change Your Mind.” Written by the annotator, after the fact, in answer to the question: once the bad science and the unwarranted conclusions are removed, what remains?
The Fable set on Book II runs to 233 annotations across 85 essays in sequences E through K. Book II is a different kind of book from Book I. Book I was mostly about probability and the history of science, and its failures were failures of history. Book II is mostly about the psychology of belief, and it was written in 2007 and 2008 on top of the social psychology that the replication crisis, three years later, took apart. Its failures are failures of evidence, and they are more numerous, because there was more evidence to fail. Sorted by kind:
| Kind | Approx. | What it establishes |
|---|---|---|
| Kill shots: a factual claim is false, or the citation does not say what it is cited for | 35 | The Inquisition “tortured witches” (it was the institution that stopped the witch trials); Fisher “testified to Congress”; the Kelvin quotation; Kekulé’s dream; Nixon’s example as a partisan dig; Gandhi took “less risk than any of the people marching with him”; “Yeishu”; the Jesuit doubt practice; the Great Leap as “deliberate intent”; the Nazis as bioconservatives who did not plan to breed supermen; Rothblatt’s “cure”; LTCM’s “inefficiencies vanished”; “burned at the stake” for saying God is unprovable; the 2-to-10 somatic mutations; the Stroop subjects who “often say green”; the Marduk–Ishtar Purim theory; the Talmud that supposedly never asked about the firstborn; the retracted “camel has two humps”; Aumann’s own stated position; van der Waals bonding as a design flaw; Watchtower 1975; the Eagly meta-analysis cited for a halo on honesty that it found absent; the Finucane study cited for a meltdown judgment it did not measure; phlogiston, a sixth time. |
| Empirical reversals: the essay’s cited or implied finding was later overturned or shown to run the other way | 28 | Attitude polarization from balanced evidence; the actor–observer asymmetry; the Spinozan belief model; behavioral priming as a whole; the Stanford prison study; capuchin fairness; analytic thinking reducing religiosity; motivated numeracy; mortality salience; “stupid happy people”; Ekman’s lie detection; brainwashing; the cult-comes-back-stronger pattern; the female-conformity effect; the backfire effect; disability adaptation; risk–return in individual stocks; the Griffin–Tversky datum, which shows underconfidence and was read as its opposite. |
| Live disputes: the essay states as settled what its own field treats as contested | 25 | Cox’s theorem; regularity; Bayesian persuasion against the “conditioning on the algorithm neutralizes the arguer” intuition; the motivated-reasoning prior confound; heritability; cultural evolution; acceptance versus belief; religious credence; multilevel selection; adversarial versus inquisitorial fact-finding; value of information; the reversal test’s exceptions; expert-credibility thresholds; whether curiosity can be induced. |
| Reasoning flaws: the argument does not go through regardless of any study | 75 | The “stronger or weaker” criterion; the no-true-relativist; the genetic fallacy committed in the essay named after it and in Dark Side Epistemology; the unfalsifiable cult attractor; the unfalsifiable true rejection; the self-serving steelman rule; “likelihood” equivocated; the field axioms inverted; the 2 + 2 = 3 essay conceding its opponent’s point in its own scenario; the frequentist gloss on subjective credence; the 69/71 symmetry error; “an unresolved doubt is a null-op” in a book about probabilities between 0 and 1; argument-counting in the essay against argument-counting; “stories are not evidence” in a sequence whose next essay is a story; “meaning does not excuse impact” from the author of Generalization from Fictional Evidence; a spreading-activation mechanism that would prime the counterevidence too; the belief that Whorf was right; the seven-hundred-fifty-million-bit calculation against a null nobody holds. |
| Positive: the claim is right and the annotation says so, usually with the citation the essay lacked | 28 | Anchoring replicated in Many Labs and consider-the-opposite reduces it; evaluability; the ratio bias; the small-loss gamble; Milgram; Ditto & Lopez on motivated stopping; Kearney on lottery regressivity; pseudoreplication; Grove on decomposition; Sen on famine and free press; Hirschman on exit and voice; mental contrasting behind Leave a Line of Retreat; Deutsch & Gerard behind the Asch analysis; Krueger & Malečková on the hijackers; the martingale; “argue physics, not rationality”; “bad argument gets counterargument, does not get bullet.” |
| Hindsight | 8 | Neural networks; corporations as optimizers; “narrow AI” versus the objects that arrived; the AI timelines that the essay on unbounded scales diagnosed in others; the cult risk the essay predicted for its own movement, and what followed. |
| Petty | 10 | Mill’s remark as “someone once said”; ten decibels, not eight; a trillion statements is twelve thousand lifetimes, not a hundred; the speed of light to one decimal; the Einfeld fine in the wrong currency and unit; “billions” for trillions in an essay about understatement. |
| Internal contradictions between essays | 24 | Counted separately because they are the strongest kind of finding: nothing external is needed. Hug the Query’s rule against arguing rationality instead of physics, applied to a book that mostly argues rationality; Your Rationality Is My Business inventing the “right to an opinion” that Dark Side Epistemology says the Dark Side invented; Argument Screens Off Authority’s title refuted by its last paragraph; Don’t Believe You’ll Self-Deceive admitting that the previous essay was chosen for its effects; Update Yourself Incrementally against What Evidence Filtered Evidence; Original Seeing against Generalization from Fictional Evidence; Every Cause Wants to Be a Cult’s situationism against Correspondence Bias’s own data. |
Weighted as in the Book I review: the petty items prove nothing, the hindsight items test the method rather than the author, the live disputes say disputed, not wrong. The categories that carry weight are the kill shots, the reversals, the reasoning flaws and the contradictions, and together they touch every sequence. About sixty of the eighty-five essays are wounded. About twenty-five are seriously wounded, in the sense that the essay’s central example or central mechanism is gone.
The theorems, again. Conservation of expected evidence and its corollary, that a known destination means you are already there. Screening off and d-separation, correctly explained in Argument Screens Off Authority before the essay overreaches. The coin-report and Monty Hall analysis in What Evidence Filtered Evidence, which is the right way to think about selective reporting even though the essay’s intuition about its consequences was wrong. The likelihood ratio as the unit of evidence. The Fable notes attack the applications and the glosses; the mathematics is not touched and could not be.
A shorter list of robust effects. Anchoring, with a debiasing procedure that works. Evaluability and preference reversal between joint and separate evaluation. Denominator neglect and the ratio bias. The halo effect, narrowed to the traits the meta-analysis supports. Conformity, understood as Deutsch and Gerard understood it in 1955. The correspondence bias for others, minus the self half. False consensus, minus the part Dawes showed is Bayesian. Belief perseverance. Group polarization, which is the real mechanism behind three of the death-spiral essays. Pluralistic ignorance. The identifiable-victim effect, which the book demonstrates on its readers without naming. Motivated reasoning as Kunda described it, with the standing caveat that asymmetric scrutiny alone does not prove motive.
About eight procedures with evidence behind them. Consider the opposite, for anchoring and for biased assimilation. Split the Great Idea into parts and assess them separately. Plan the retreat: mentally contrast the feared outcome with what you would do, which is Oettingen’s finding and cognitive therapy’s. Hold off on proposing solutions, in groups. Raise concerns as questions or shared problems, which the aviation and surgical literatures found works and the essay on expressing concerns said was impossible. Argue the object level rather than the rationality of the speaker. Decompose global judgments and combine mechanically. Treat every additional nice claim as a burdensome detail, which is a fair paraphrase of what conjunction means. These are the book’s durable contribution, and it is notable that all of them are either mechanical or social; none is an act of solitary vigilance.
Two rules stated without qualification that deserve none. “Bad argument gets counterargument. Does not get bullet.” And: “It is superior to argue physics than rationality.” The second is the more important, because it is the standard by which most of the book fails.
The self-directed warning against motivated reasoning. The Bottom Line, Rationalization, and Fake Justification are wrong about the evidential value of other people’s arguments and right about one’s own. A conclusion written before the evidence was examined is not caused by the evidence. That sentence survives, and it is the sentence the book most needed.
Every historical claim used as evidence, again. The Inquisition, Fisher, Kelvin, Bacon, the Jesuits, the Nazis, Jesus of Nazareth, Gandhi, Kekulé, the Book of Esther, the Jehovah’s Witnesses, Long-Term Capital Management, the witch-burners of “a few centuries ago,” and phlogiston. As in Book I, not one is right as stated, and the pattern is the same: history remembered in the shape the argument requires. The Book II specimens are worse because several were checkable in the essay’s own source. Lowenstein’s book says what happened to LTCM; the Eagly meta-analysis says what the halo does to honesty; the Spiritual Exercises say what Jesuits do with doubt.
The psychology of the outgroup. This is Book II’s characteristic failure, and it has no Book I counterpart. Creationists lie. Theists who have thought about it have minds like Escher paintings. Scientologists have lost their sanity. New Agers are made crazier by quantum mechanics. Relativists do not exist. The lottery buyer has dropped the line of reasoning. The person who says “still a chance” is an unenlightened one. Hijackers are heroes of their own story, but Jesus Campers are insane. In every case a diagnosis is offered where an argument was owed, and in every case the diagnosis is the kind that the book’s own rules, hug the query, do not psychoanalyze, argue against the strongest form, forbid. The book warns against horns and gives its examples horns. Remove the diagnoses and about a third of the book’s pages are blank.
The priming foundation. Priming and Contamination, Do We Believe Everything We’re Told, Cached Thoughts, and We Change Our Minds Less Often Than We Think form a cluster whose thesis is that a fleeting exposure fixes the bottom line. The cluster rests on behavioral priming, which collapsed; on the Spinozan model, which was narrowed to a load artifact; on a Wansink citation; and on a misreading of a study that showed people underconfident about their own steadiness. What is left is belief perseverance, which is real and modest. “Once an idea gets into your head, it’s probably going to stay there” was the book’s empirical spine, and it is not supported.
The evolutionary just-so stories. Politics kills the mind because ancestral politics killed people. Lonely dissent is feared more than death because exile killed people. Robbers Cave is the psychology of hunter-gatherer bands. What you saw with your own eyes was true in the ancestral environment. Each is flagged by the essays themselves as speculation and then used as the explanation, and the ethnography, where it exists, contradicts the picture.
The physics of rationality. 0 And 1 Are Not Probabilities, Infinite Certainty, Absolute Authority, and No One Can Exempt You From Rationality’s Laws share a rhetoric in which Bayesian norms are laws of nature. The theorems the rhetoric leans on do not support it: Cox’s theorem has a gap and puts 0 and 1 in; regularity is impossible in infinite spaces; a 0.9999 credence in a theorem is incoherent, as the author’s institute later acknowledged; the “million statements” operationalization is frequentist and contradicts the book’s own subjectivism. The norms are good norms. They are not the Second Law, and the book’s reader is told they are.
The self-help program. Singlethink. The knife in the heart. The chant of “I can’t fool myself.” The Crisis of Faith with its allocated hours and quiet room. The Ritual with its sixty-two techniques and its tally on the wall. None was tested; all are solitary; and the evidence the book itself cites, the bias blind spot, Mercier and Sperber, the failure of awareness-based debiasing, predicts that they would not work. The book’s one honest report on the matter is in Don’t Believe You’ll Self-Deceive: does meta-level knowledge propagate to the object level? “Apparently not.” That sentence, applied to the program, is the verdict.
The thermodynamics of cults. Every cause wants to be a cult; entropy; the attractor; the air conditioner that must run constantly. It is unfalsifiable as stated, the sociology of religion finds that strictness rather than openness produces fervor, and the movement the book founded ran the air conditioner, in public and at length, with the result the essays predicted would follow from not running it. Evaporative cooling is real and was described by Hirschman in 1970. The rest is metaphor.
Strip the bad science and Book II reduces to: the Bayesian theorems, a list of about ten biases that replicate, about eight procedures that have evidence, two rules, and one self-directed warning. That is a good forty-page pamphlet, and a more practical one than Book I’s, because the procedures are things a reader can do on Monday.
What is lost is the book’s answer to its own title. “How to actually change your mind” is answered, in the text, by an individual program of vigilance, ritual and desperate effort, conducted alone, against a mind pictured as a citadel that a single fleeting exposure has already garrisoned. On the evidence, including the evidence the book cites, minds change less by that route than by any other. They change when a procedure forces the alternative into view, when a group is structured to surface dissent, when an adversary is given the floor, when the feared outcome has been walked through, when the judgment is decomposed and the pieces scored. The book knows all of this in fragments, and each fragment is annotated here as a survivor. What it did not do is assemble the fragments into the book they imply, which would be a book about institutions and procedures rather than about the inner life of a rationalist. That book would have been shorter, less thrilling, and true.
The Book I review asked what the book forbids that a standard course in judgment and decision making does not. Book II has a sharper version. Ask what it would look like to apply the book’s own methods to the book: hug the query, argue against the strongest form, treat each nice claim as burdensome, check the citation, do not psychoanalyze the opponent, say oops. The result is this annotation set. The book passed its own test in the places listed under “survives” and failed it everywhere else, and the failures cluster exactly where the book reasons about people it disagrees with. That is the finding a reader should carry out of Book II, and it is a finding the book predicted, about other people.
One caveat from the annotator, as before. This was an adversarial pass by brief. A reader who takes the Opus and Fable sets together would land somewhere near “useful in its procedures, wrong in its psychology, careless in its history, and dependent on a social science that has since half-collapsed,” which is probably the right place to land. Readers who want the book’s good parts should read The Bottom Line, Leave a Line of Retreat, Hold Off On Proposing Solutions, Anchoring and Adjustment, Evaluability, and Resist the Happy Death Spiral, in that order, and treat the rest as the annotations suggest.