Book VI: What Survives the Skeptic

❦

A review of the Opus annotation pass over the whole of Book VI, “Becoming Stronger”: Rob Bensinger's introduction, sequence X (“Yudkowsky's Coming of Age”), sequence Y (“Challenging the Difficult”) and sequence Z (“The Craft and the Community”) — 43 essays, 176 annotations. Written by the annotator, after the fact, in answer to the reader's question: once the failed studies, the bad citations and the unearned conclusions are removed, what is left standing?

First, weigh the critiques

Book VI is the least empirical of the six and the most self-referential. Sequence X is autobiography, and the claims in it are almost all unfalsifiable by construction. Sequence Y is exhortation, with two testable propositions in it. Sequence Z is where the real content is: it asks whether an art of rationality can be built, taught, verified and organized, and those are questions with literatures. Sorted by what each annotation actually does — many do more than one thing, and are counted by the dominant move:

KindApprox.What it establishes
Kill shots: a factual claim is false, or the citation does not say what it is cited for26The Indonesian ultimatum game, cited for high-stakes rejection, whose published finding is that rejection rates fell as stakes rose; the “randomly selected college professor” of the Vanderbilt study, who was selected specifically for a reputation for warmth; the FDA's 20,000–120,000 annual casualties, sourced to a hyperlink behind which stands an unrefereed 1985 Cato paper extrapolating from beta-blockers; the theologians who opposed anaesthesia in childbirth, a myth with no primary sources, and the smallpox inoculation that a Puritan minister introduced over his physicians' objections; the rules of Conway's Life, stated without the birth rule, which makes the automaton inert; “no particular empirical justification that I happen to have heard of” for doubting a great-man account of the Second World War, sourced to Shirer and offered against the entire structuralist historiography; the mind that must radiate waste heat to obey thermodynamics, which Bennett's reversible computation refutes and which Book IV's own essay on the Second Law contains the material to refute; the Raëlian membership figure, which is a cumulative sign-up count; “more than 50%, I would venture”; the 20-to-1 fuzzies ratio arrived at by saying “say”; “simply no” on the Rorschach, against a meta-analysis finding a third of its variables validated; “teenagers think they're immortal,” which is the reverse of what teenagers report; Shane Legg's collection of definitions, cited against “achieving goals,” by an author whose own formal definition published that year is “achieve goals in a wide range of environments”; the modern rabbi who cannot overrule the ancient one, in a tradition whose most famous story is a court outvoting a heavenly voice; the verse about the kid in its mother's milk, glossed with a meaning no interpreter has given it; lottery winners who “don't exist” and number in the thousands; the AI-Box Experiment, three wins in five games and no transcript; Seth Roberts and the polywell fusor as neglected good science, one never supported and one funded and failed; ego depletion, listed among the phenomena that actually exist; the Japanese asset bubble attributed to expectations of Japanese dominance; the equimarginal principle credited to a 2007 paper on AI drives; and the closing anecdote of The Sin of Underconfidence, which is the first-instinct fallacy.
Findings that failed to replicate, or reversed13The bystander effect, which reverses in the field — Philpot's surveillance-video study found intervention in nine of ten real conflicts and rising with crowd size; ego depletion, at 0.04 across 23 laboratories; analytic-thinking priming reducing religious belief, failed in preregistered replication and in the Science project; attitude polarization from two-sided evidence, failed in three large preregistered experiments; power priming and perspective-taking, with its most-cited results retracted in the Stapel investigation; the Dunning–Kruger crossover, largely reproducible from noise plus regression; the happiness set point, contradicted by two decades of German panel data; CBT for depression, whose effect size roughly halved across four decades of trials; meditation, which beats no active control on any outcome in the AHRQ review; the Dodo bird verdict itself, which is contested rather than settled.
Live disputes presented as settled19Whether cooperation requires punishment, when direct reciprocity, partner choice and reputation all sustain it and costly punishment is frequently welfare-destroying; the offense–defense balance, which its own specialists cannot operationalize; Jaynes's objective priors, with Halpern's counterexample to Cox and the marginalization paradoxes unmentioned; the wager for moral realism, which is a published position with published problems, diagnosed here as a confusion about utility-as-stuff; “too hard for humans, therefore we need a superintelligence,” which is the mysterious-answer move; whether philosophy makes progress; whether a common decision theory delivers Pareto optima, when the program-equilibrium folk theorem supports almost anything and bargaining remains unsolved; whether prices measure how much society cares; whether the psychotherapy schools differ; whether religion is a symptom of a reasoning deficit, against existential-security and cognitive-by-product accounts that were in print and are not mentioned.
Uncredited reinventions11Campbell's law, stated almost verbatim in Three Levels of Rationality Verification and not named; Goodhart alongside it; the equimarginal principle, which is Gossen 1854 and a Lagrange multiplier; the cybernetic definition of purposive behaviour as constraint on outcomes, which is Rosenblueth, Wiener and Bigelow in 1943 and Ashby's requisite variety in 1956, presented as a naturalistic enlightenment of 2003; the intelligence explosion, which is I. J. Good 1965 and Vinge 1993; the two-children protocol dependence, which is Bar-Hillel and Falk 1982; Boyd and Richerson's second-order punishment theorem; Iannaccone's strictness model, which answers Church vs. Taskforce's engineering proposal; Benkler's commons-based peer production, three years old when Collective Apathy and the Internet declared internet coordination an open problem; Denrell's formal treatment of learning from surviving samples; the moral-uncertainty literature.
Unsourced empirical claims and just-so stories17The 50-person band where everyone knows everyone else, which anchors five separate arguments across sequence Z and which Hill's data on 32 forager societies contradicts — most co-residents are not close kin and individuals move between camps all their lives; the evolutionary arms race of delay behind the bystander effect, constructed and then honestly abandoned; the “obvious evolutionary rationale” for punishing the insufficiently censorious; joining prices calibrated for campfire negotiation; intelligence as the unfairest card, against a literature where country of birth explains over half of global income variance; dysrationalia as the diagnosis for religious parents; the Catholic Church's 50%; the 20-to-1 ratio; “I had overestimated the difficulty of almost every single important problem,” from a sample of the problems that yielded.
Predictions the AI decade tested14Against: artificial intelligence as “a sick, dead field” in the years when Pearl, Vapnik, LeCun and Tesauro were publishing the answer; “an AI should look nothing like a human brain,” when the winning systems are neural networks whose best performers are also the best models of visual cortex; “any math simple enough for a neat equation won't work,” when a differentiable loss and an attention block did; the Completely Alien Mind Design that human engineers could not build because they could not understand it, now built and not understood; the privately funded brain in a box in a basement, against the corporate labs the essay dismissed as defeatism; genuine lie detectors within thirty years; the challenge to name anyone who says “I intend to build AGI, I can't yet, I'm working out why,” which was Hutter's and Schmidhuber's published stance already and is now every laboratory's; the boxing and multi-agent-oversight proposals dismissed unread, which are now control evaluations and debate. For: the target-smallness intuition, half-vindicated as specification gaming and Goodharting; the diagnosis that a system optimizing a proxy will edit whatever produces the proxy.
Internal contradictions between essays14“I will not proceed until I know positively that the ground is safe” — the precautionary principle — four essays after Raised in Technophilia costed that same rule in tens of thousands of deaths a year; “actions screen off justifications,” asserted three times, against The Bottom Line's insistence that a conclusion's worth is entirely the process that produced it, and refuted two paragraphs later by the lesson drawn about perfectionism; “admit one huge mistake rather than many little ones,” against both Bayesian conditionalization and Leave a Line of Retreat; “I overestimated the difficulty of almost every problem,” against the planning-fallacy essay's finding that people underestimate; “rationality is systematized winning,” retracted three paragraphs later into the correct expected-value formulation and then used in the uncorrected form for the rest of the book; the Rumsfeld example in a corpus containing Politics Is the Mind-Killer; “agree to shut up about the disagreement,” four essays after the suppression of visible dissent was identified as the defining mechanism of the Dark Side; spreading donations to raise total giving, then concentrating on the single highest-expected-utility charity, three essays apart; refusing to put a probability on one's own project, in a corpus that puts P(God) below one in a billion; two invented dialogues used as evidence in a book with an essay on generalization from fictional evidence; and the Center for Applied Rationality cited in the introduction as an accomplishment, by exactly the reputational standard that Three Levels of Rationality Verification says not to accept.
Endorsements: the annotation says the passage is right, and usually supplies the evidence it lacked18See below.
Petty4The Life rules; the 55 million deaths a year divided by 365, which is correct; “lottery winners don't exist”; the arithmetic of three AI-Box games.

Weighted as in the earlier reviews: the petty items prove nothing, the reinventions say uncredited rather than wrong, and the live disputes say disputed. What carries weight is the kill shots, the reversals, the just-so stories and the contradictions. Of the 43 essays, about 33 are wounded somewhere. About eleven are seriously wounded, in the sense that the central example, the central mechanism or the central conclusion does not survive: Raised in Technophilia, My Naturalistic Awakening, Use the Try Harder, Luke, Shut Up and Do the Impossible!, Raising the Sanity Waterline, Schools Proliferating Without Evidence, Why Our Kind Can't Cooperate, Helpless Individuals, Bystander Apathy, Practical Advice Backed by Deep Theories, and The Sin of Underconfidence.

What survives untouched

Epistemic Viciousness, the whole essay. The strongest thing in Book VI is a summary of somebody else's paper. Gillian Russell's list of the ways a martial art goes epistemically bad — the sacred sensei, the twenty-year emotional investment in a technique, the impossibility of learning from a book, deference to historical masters, and above all data poverty combined with acting as though one lived in evidential luxury — transfers item for item, as the essay says it does, and the transfer was a prediction. A rationality-training organization was founded three years later, and the criticisms subsequently made of that community by its own members and by outsiders are, essentially, this list. The essay closes by asking what can be done about it. Nothing in the following twenty essays answers, and the honesty of leaving the question open is worth more than an answer would have been.

Punishment of non-punishers. Tolerate Tolerance observes, in passing and as an intuition, that sanctioning people for failing to sanction is far more dangerous than sanctioning defectors, because it can lock an equilibrium in place even when the equilibrium harms everyone. This is a theorem. Boyd and Richerson proved in 1992 that extending punishment to non-punishers stabilizes cooperation in large groups — and that it stabilizes any behaviour whatsoever, since the second-order sanction is indifferent to the content of the first-order norm. It is the best-supported original observation in the book, and the surrounding sentence about cooperation requiring punishment at all is the part that fails.

The pluralistic-ignorance diagnosis. The fundraising thread where every donor gives silently and every objector posts is a clean instance of a well-measured process. Prentice and Miller found Princeton students uniformly believing their peers were more comfortable with campus drinking than they were, with the private discomfort invisible and the imagined consensus drifting private attitudes toward it. The mechanism, including the self-reinforcing silence, is exactly the one Why Our Kind Can't Cooperate describes. What does not survive in that essay is the remedy — practising agreement and lockstep — which inverts what the minority-influence literature recommends; but the diagnosis is right and it was right for the stated reason.

Physical presence, not belief. Can Humanism Match Religion's Output? spends two thirds of its length on the wrong variable — hell versus truth as a motivator — and then guesses correctly in its final paragraphs that what actually produces religious generosity is meeting other people who share the cause. Putnam and Campbell's panel analysis found precisely that: the giving and volunteering advantage of religious Americans is fully accounted for by the number of close friends held inside the congregation, with belief, prayer and denomination adding nothing once friendships are controlled. Lim and Putnam found the same structure for life satisfaction. The essay's own throwaway suspicion is the finding.

Two propositions in Challenging the Difficult. The goal-setting literature is squarely behind Trying to Try: across four decades and hundreds of studies, specific difficult goals beat “do your best,” and “do your best” is literally the losing control condition. And people do stop short: Lucas and Nordgren found participants systematically underestimating what a second round of effort would produce, with the second round in fact as productive as the first. That is the real version of the claim Use the Try Harder, Luke puts in a fabricated George Lucas's mouth — better evidence than the invented outtake, and pointing at a different remedy, since the failure is misprediction rather than a will to fail.

The three-level taxonomy. Reputational, experimental, organizational is a genuinely useful carving of the verification problem, and the observation that a gameable metric reshapes the field around itself is correct (it is also Campbell's law, and should have been cited). The essay's pessimism is what did not last: an answer covering all three levels arrived within a few years in the forecasting tournaments, where a proper scoring rule cannot be gamed by confidence, randomized training arms supply the experiment, and public track records supply the reputation. The question was the right question, posed clearly enough that someone else's answer is recognizable as an answer to it.

Beyond the Reach of God. The core argument — that a universe running on physics contains no clause reading “things have to make sense,” and that many self-described atheists still reason as though certain outcomes were not permitted — is philosophically clean and does not depend on any of the essay's flawed examples. Its sharpest illustration is unimpeachable and was fresh at the time: the belief that modern liberal democracies will never legalize torture, written in the years when one had just done so by written legal opinion. The Hitler counterfactual is sourced to Shirer against the field, the Game of Life rules are wrong, and the cryonics assertion is a contested empirical premise smuggled in as an obvious truth. The argument stands without any of them.

“No true science draws its strength from only one person.” The single best sentence in Go Forth and Create the Art!, and correct: Merton's documentation of multiple independent discovery is the structural evidence that findings are properties of a field's state rather than of a person, and the norms of organized skepticism exist to keep it that way. The annotation's only addition is that the warning is aimed away from the nearest instance — six books by one author, a community named for his website, and a body of technical claims developed largely outside the peer literature.

The survivorship parenthesis. My Best and Worst Mistake notes, correctly and without being asked, that the reader is hearing from someone who did not fall into a true dead end. This is Denrell's undersampling of failure, and naming it is more than most memoirs do. It also silently invalidates the sequence that follows it, which is why the annotations treat sequence X as parables about a real person rather than as data.

What does not survive

The whole evidentiary basis of sequence X. Twelve essays of retrospective self-diagnosis, in which the author reconstructs what he believed between 1996 and 2003 and why. My Bayesian Enlightenment concedes the problem in its own voice: memory is reconstructed at each recall, he cannot order the events, he has already been corrected by witnesses, and he cannot bear to read the contemporaneous record that exists. Add the consistency bias, which is strongest exactly where a person believes they have changed, and Nisbett and Wilson on the unavailability of one's own cognitive processes, and what remains is a well-told story whose lessons are not licensed by it.

The ancestral 50-person band. This premise carries five separate arguments in sequence Z — why large institutions fail, why joining prices are too high, why internet coordination fails, why bystanders don't help, why money feels optional. Hill and colleagues' co-residence data from 32 forager societies is directly against its operative clause: most co-resident adults are neither close kin nor affines, individuals move between bands repeatedly, and a typical forager's social world runs to several hundred people across many camps. The structure that actually characterizes human foraging — fluid membership, reputation carried between groups, cooperation with non-relatives — is what makes large-scale cooperation possible, not what forbids it.

Raising the Sanity Waterline, as an empirical program. Its central thought experiment — teach general rationality without mentioning religion, survey five years later, watch the religions fall away — has since been approximated, and the results are against it. The flagship analytic-thinking result failed preregistered replication; the deficit model of science communication fails harder, with science literacy predicting more polarization rather than less; and the cognitive science of religion locates belief in ordinary functional machinery, teleological intuition and agency detection, rather than in a defect upstream. The best predictor of national religiosity is existential insecurity. The waterline may be worth raising for other reasons; the canary will not notice.

Practical Advice Backed by Deep Theories, by its own examples. An essay arguing that theory makes advice generalize, whose central worked example is a diet with no controlled trial that the author reports does not work on him and for which the theory supplies no permitted exception, and whose list of cognitive phenomena that actually exist opens with ego depletion. The general claim is probably right. The essay is a demonstration of the failure mode it warns against: a causal story that felt explanatory, adopted from a source that felt scientific, producing confident advice with nothing behind it.

The AI-Box Experiment as evidence. Cited for years as a proof of concept about persuasion by a superior intelligence, and consisting of: two publicized wins, three later games disclosed in passing five years afterwards of which two were losses, no transcript of any of them by protocol, no observer, and stakes of ten to twenty dollars in the games that were publicized. Three wins in five, in a text game where a motivated player faced a bored one. The candour of the eventual disclosure is real; the file-drawer shape of the record is the reason the claim had the status it did.

The citation layer. Cameron's Indonesian ultimatum study cited for the opposite of its finding; Strupp and Hadley's warm professors becoming random ones; the Cato casualty numbers behind an anonymous hyperlink; the church that opposed anaesthesia; Legg and Hutter's own definition used against them; the Talmud on the worm and the kid; the Raëlian sign-up count; the fabricated Star Wars outtake presented as “little-known.” Individually small. Together they describe the same habit the Book V review identified: the supporting detail is remembered in the shape the argument needs, and the sequence that most insists on verification is the one where the verification was not done.

Verdict

Strip the failed studies and the unearned steps, and Book VI reduces to: Russell's list of epistemic vices and the observation that they transfer; the second-order punishment warning, which is a theorem; the pluralistic-ignorance diagnosis of why a fundraising thread looks like a wall of objections; the finding that physical presence rather than belief is what religion supplies; goal difficulty beating “do your best”; the persistence gap; the three-level verification taxonomy; Beyond the Reach of God's core argument and its torture example; Cialdini's advice to point at one bystander; the fact that telling people about the bystander effect appears to work; the survivorship parenthesis; and “no true science draws its strength from only one person.” That is perhaps twenty-five pages of durable material out of the longest and loosest of the six books.

The characteristic failure here is different from Book V's. Book V argued too fast; Book VI mostly does not argue at all. Sequence X is twelve essays of memoir presented as case study, and it is the only place in the six books where the author's method — check the claim, name the source, distrust the reconstruction — is suspended for the author's own testimony. Sequence Y is exhortation, and where it makes checkable claims it is roughly right, which is a better record than its rhetoric earns. Sequence Z asks the book's one great question, whether an art of rationality can be verified, sets out the taxonomy correctly, describes the failure modes with real insight, and then reaches for evidence and repeatedly grabs the wrong thing: the psychotherapy literature at its most sweeping, the bystander effect at its least replicable, a martial-arts analogy whose own history refutes the moral, and one advocacy think-tank's casualty estimate.

The most striking pattern across the book is that its best diagnoses are aimed away from itself. Epistemic Viciousness lists the pathologies of a school built on untestable techniques and does not observe that it is describing the project the next twenty essays propose. Three Levels establishes that reputational verification is nearly worthless and the introduction cites an unverified training organization as an achievement. Go Forth says a science with one hero is a cult, at the end of six books by one author. And Beware of Other-Optimizing — the most self-aware essay in the sequence, on the illusion that what worked for you will work for anyone — sits eleven essays after a sequence in which one man's intellectual biography is offered as a curriculum.

One caveat from the annotator, as in the earlier reviews. This was a maximum-skepticism pass by brief, and it says nothing about a passage where it found nothing to say. Much of the criticism above is “this is a live dispute you have declared closed” or “your evidence is not the evidence for your claim,” rather than “this is false”; and several of the book's instincts were good and merely underdetermined by what it cited — the verification problem is the right problem, the coordination failure is real, and the community-versus-belief insight about religion was correct before the data confirming it were quoted at it. Readers who want the good parts should read Epistemic Viciousness, Three Levels of Rationality Verification, Beyond the Reach of God, Tolerate Tolerance, Beware of Other-Optimizing, and the last two pages of Go Forth and Create the Art!, and treat the rest as the annotations suggest.

Top

Book