❦
A review of the Opus 5.5 annotation pass over the whole of Book VI, “Becoming Stronger”: Rob Bensinger's introduction, sequence X (“Yudkowsky's Coming of Age”), sequence Y (“Challenging the Difficult”) and sequence Z (“The Craft and the Community”) — 43 essays, 179 annotations, written under a brief of maximum skepticism. The pass was made independently of the earlier Opus annotations (now filed as “Opus (old)”); where the two overlap they usually agree, and where they disagree it is noted below. Written by the annotator, after the fact, in answer to the reader's question: what, if anything, remains unscathed?
No essay in Book VI comes through with nothing to answer for. Every one of the 43 carries at least one annotation that corrects a fact, supplies a failed replication, names an uncredited source or points to a contradiction with another essay. But “scathed” is not “destroyed,” and a set of claims survives the pass intact — some of them the best things in the six books. They are listed below, after the damage.
Book VI is the least empirical of the six books and the most self-referential. Sequence X is autobiography, sequence Y is exhortation, and sequence Z makes claims about groups, institutions and training that have literatures. Sorted by dominant move (many annotations do more than one thing):
| Kind | Approx. | What it establishes |
|---|---|---|
| Kill shots: a factual claim is false, or a source does not say what it is cited for | 29 | The Indonesian ultimatum game, cited for high-stakes rejection, whose published finding is that rejection fell as stakes rose — confirmed later at stakes near a year's income in north-east India; the “randomly selected” professors of the psychotherapy study, chosen for their warmth; the FDA casualty range from a 1985 Cato extrapolation, written after the drug lag had reversed and the FDA had become the fastest major regulator; the theologians who opposed anaesthesia, inoculation and Copernicus; “no justification I've heard of” for doubting that Hitler's personality drove the war, against fifty years of historiography; Hamming's “important problems,” which he defined to exclude problems without an attack; the modern rabbi who cannot overrule the ancient one, in a tradition that rules hilkheta ke-batrai, “the law follows the later authorities”; the Ashamnu that never resolves to do better, when Maimonides defines confession without that resolution as immersing while holding the impurity; shyness “not discussed” in bystander research, where audience inhibition is one of Latané and Darley's three founding mechanisms; the martial-arts schools that “competed regularly in realistic contests” until 1993 proved they hadn't; the unsolved problem of strangers coordinating online, two sentences before the essay cites Wikipedia; “intelligence is the unfairest card,” when country of birth explains more than half of global income variation; the brick wall at 40; alchemists who “poisoned themselves and died” instead of becoming chemists; the challenge to “name any other” who says “I intend to build AGI but don't yet know how,” answered by Hutter, Schmidhuber and Legg; the old writings that “it's too much pain” to read, which exist and would settle the questions the sequence reconstructs from memory. |
| Findings that failed to replicate, or reversed | 13 | Ego depletion, which does the causal work in three separate essays (“run out of mental energy,” “willpower, the true limiting resource,” and the list of “phenomena that actually exist”), at d ≈ 0.06 across 36 laboratories; the bystander effect at the level of crowds, which reverses on real CCTV; attitude polarization from balanced evidence (Taber & Lodge) and motivated numeracy; analytic-thinking primes reducing religious belief; growth mindset; the power-priming study behind “power reduces empathy”; the peak–end rule applied to meals, tested directly and largely absent; neurofeedback, no better than sham in blinded trials; the Cameron ultimatum result, reversed in citation. |
| Predictions the AI decade tested | 17 | Against the essays: a “sick, dead field” in the years of SVMs, TD-Gammon and LeNet; “no neat equation” (a single loss function and power-law scaling) — and equally against the mature view, since no working system computes a posterior; “an AI should look nothing like a human brain”; the Completely Alien Mind Design too hard to build without understanding it, now grown by optimization and understood afterwards; the privately funded brain in a box in a basement, against the corporate labs; human lie detectors within thirty years; the dismissal of boxing and multi-agent oversight, now AI control and debate, with results; the mailing-list objections of 1996 — orthogonality, resource seizure, a watching control system — which were the author's own later positions. Partly for: the persuasion premise behind the AI-Box Experiment, which controlled studies of machine persuasion have since supported where the Box itself could not; the difficulty of Friendly AI, which the author was right not to think he had overestimated — which is the opposite of what On Doing the Impossible claims. |
| Internal contradictions | 15 | Raised in Technophilia's general law against people who say a technology's risks may outweigh its benefits, against the author's later call in Time for an enforced worldwide halt; “I will not proceed until I know positively that the ground is safe” four essays after costing that rule in lives; “you don't use the best concepts you can use at the time” against decision theory, which has no other option; “actions screen off justifications” against the essay's own admission that the successes were semi-accidental; “rationality is systematized winning” retracted three sentences later; “no statistically discernible” benefit of psychotherapy, one paragraph after “patients get better faster than those who do nothing”; majority use within a science as the reliable guide to true causal theories, in the books of one-boxing, many-worlds and cryonics; “agree to shut up about the disagreement” four essays after deploring the silence of supporters; spreading donations to reduce regret, then concentrating them on the single top charity; money as the measure of caring, in an essay whose first paragraph says rational agents equalize marginal, not total, value; the 50% of church revenue “wasted on religious stuff,” in an essay whose conclusion is that the religious stuff produces the output; Tsuyoku Naritai's Japanese self-improvement set against the confession of flaws, when its Japanese form is built on hansei, ritual self-criticism. |
| Live disputes presented as settled | 22 | Probability theory as the “standard” definition of rationality; frequentism as more subjective; formal rules for priors (Kass & Wasserman's review of all of them); Jaynes's Bell argument and the marginalization paradoxes; Bayes as “The Rules” against the practising Bayesians (Gelman & Shalizi, Rubin) and the half-Bayesian Pearl; approximation as “less than optimal” against resource-rational analysis; the nihilism-cancels argument, which is Ross's published position and whose offered refutation supports it; the metaethics sequence as the “full reduction of morality”; the minimal-change principle of belief revision against Halt, Melt and Catch Fire; offense beating defence since 1945; Aumann's religion; religion as a sanity deficit against the cognitive science of religion and the existential-security data; the Santa Claus equivalence; cryonics; cooperation requiring punishment; Popper's consistent intolerance of intolerance; source-code decision theory yielding Pareto optima, against the program-equilibrium folk theorem. |
| Unsourced claims, just-so stories and logical gaps | 34 | The 40- and 50-person ancestral band where everyone knows everyone, which carries Your Price for Joining, Helpless Individuals, Collective Apathy and Bystander Apathy and which the co-residence and interaction data contradict; the “obvious evolutionary rationale” for punishing non-punishers, when people in experiments are reluctant second-order punishers; zealots making better soldiers, against battlefield-effectiveness data; firing squads for deserters, against the cohesion literature from Shils & Janowitz to Costa & Kahn; the 20-to-1 fuzzies ratio from “say”; “more than 50%, I would venture”; underconfidence as a besetting sin of rationalists, assigned from the people who ask the author for advice; “you can't discriminate levels too far above your own”; “win or fail, there is no best,” which is precisely the objective that makes the lottery ticket optimal (Dubins & Savage's bold play); the small-target argument without its measure; the revealed-preference inference from a burrito to a public good; the lawyer's convertible hour; bundling fuzzies, status and utilons as always worse, when public giving is complementary; the canary that measures existential insecurity rather than sanity. |
| Uncredited reinventions | 11 | Cybernetics (Rosenblueth, Wiener & Bigelow 1943) and teleonomy behind the “optimization process”; Campbell's blind-variation-and-selective-retention, which lumps evolution and thought on purpose; Campbell's law in Three Levels; Gossen's equimarginal principle; Olson's logic of collective action; Hirschman's exit and voice; the theory of the second best behind the valley of bad rationality; I. J. Good's ultraintelligent machine; the just-world literature behind Beyond the Reach of God; escalation of commitment; Shirky's cognitive-surplus arithmetic. |
| Rhetorical devices | 8 | Self-quotation as scripture (“Therefore is it written”); mind-reading opponents to dismiss a strategic argument about race dynamics; no-true-Scotsman against “rational people I know aren't happier”; moving the goalposts from the average believer to the best rationalist; Stove's polemic cited as an observation; an essay on the universe's neutrality whose practical upshot is two purchases, one from the author's line of work; fiction (the outtake, Jeffreyssai) delivering conclusions whose authority is stipulated; the Rumsfeld applause line in the book of Politics Is the Mind-Killer. |
| Endorsements: the passage is right, usually with evidence it lacked | 28 | See below. |
| Petty | 2 | The invented Star Wars outtake called “little-known”; Dawes's book dated to the '80s. |
The petty items and the reinventions say careless or uncredited rather than wrong; the live disputes say contested. What carries weight is the kill shots, the reversals, the tested predictions and the contradictions. By that standard about fourteen of the 43 essays are seriously wounded — the central example, mechanism or conclusion does not survive: Raised in Technophilia, My Best and Worst Mistake, Fighting a Rearguard Action Against the Truth, On Doing the Impossible, Make an Extraordinary Effort, Shut Up and Do the Impossible!, Raising the Sanity Waterline, Your Price for Joining, Helpless Individuals, Money: The Unit of Caring, Bystander Apathy, Collective Apathy and the Internet, Practical Advice Backed by Deep Theories and The Sin of Underconfidence.
Punishment of non-punishers. The passing observation in Tolerate Tolerance — that sanctioning people for failing to sanction can lock in an equilibrium that harms everyone — is a theorem (Boyd & Richerson's “cooperation, or anything else”) with a measured human mechanism: people who privately reject a norm publicly enforce it to prove their sincerity (Centola, Willer & Macy; Willer, Kuwabara & Macy). The sentence before it, that cooperation is unstable without punishment, fails; this one is the best-supported original insight in the book.
Epistemic Viciousness, as prediction. “Every single one of these risk factors transfers straight over to any attempt to start a rationality dojo.” It did. Within a decade the communities formed around these writings produced the pathologies on Gillian Russell's list — a sacred founder's canon, untestable techniques, deference, a closed world — in groups whose former members documented them. The essay's own addition, that martial arts were healthier when fights were real, fails (the Boxers of 1900 fought real wars and believed they were bulletproof); the transferred list is the most accurate forecast in Book VI. The essay asked what could be done and left the question open. The honest answer is external verification, the right to exit and dispersed authority, which the psychotherapy and martial-arts examples in the neighbouring essays show and which the book never proposes.
The pluralistic-ignorance diagnosis and the synagogue's remedy. Silent donors next to loud objectors, each thinking themselves alone, is the right mechanism, and the synagogue format the essay ends up recommending — visible, unforced pledges — is supported by field experiments in which telling people that others gave raised their own giving (Frey & Meier; Shang & Croson). What fails in Why Our Kind Can't Cooperate is the proposed training — uniforms, lockstep, practised applause — which works, and works equally well for compliance with orders to harm.
“Physically meeting other people who share your cause.” Can Humanism Match Religion's Output? spends most of its length on hell as a motivator and then guesses, correctly, that the active ingredient is the congregation. Church attendance, not belief, predicts volunteering across 26 countries, more strongly in secular ones, and the religious advantage in life satisfaction is carried by friends within the congregation.
The core of Beyond the Reach of God. The universe has no clause saying things must make sense, and people who disbelieve in God still reason as if certain outcomes were forbidden. The illustration that democracies will never legalize torture was, when written, a description of the preceding six years. The essay's worked examples fail — the Hitler counterfactual against the historiography, cryonics as the unfairness that isn't — and the argument needs none of them.
The valley of bad rationality. That moving part of a flawed system closer to the ideal need not improve it is the general theory of the second best (Lipsey & Lancaster, 1956), uncredited but correct, and the essay's refusal to believe that the ideal entrepreneur is wrapped in comforting overconfidence is backed by the evidence that overconfidence drives entry into markets that fail and that founders' optimism predicts worse growth.
Three Levels of Rationality Verification, mostly. The hedge fund selling puts to manufacture a record is exactly Andrew Lo's “Capital Decimation Partners.” Separating the examiner from the teacher is correct and practised, and external examinations are associated with higher achievement. The question itself — how to verify a reasoning skill in a way that resists gaming — has a known answer the essay did not have: the proper scoring rule, under which honesty maximizes expected score. The happiness measure it trusts is weaker than it thought when averaged, and Campbell's law should have had its name.
Beware of Other-Optimizing. The advice — your success is n = 1, as likely to work for me as any other blog post, and power makes pushing dangerous — is the best in the sequence and consistent with what is known about egocentric discounting in advice. Its one citation (power reduces perspective-taking) is from a fragile priming paradigm; the advice does not need it. Its one caveat needs strengthening: trials suggest that “different things work for different people” is often noise, which cuts against the author's own “X worked so well for me” as much as against his advisers'.
A handful of sentences. “The chief obstacle is the difficulty of testing the results of rationality training programs” (A Sense That More Is Possible). “No true science draws its strength from only one person” (Go Forth and Create the Art!), supported by the statistics of multiple discovery and best applied to the book itself. The hedge in the same essay that “the next leg of the work will be done by direct, specific investigation of rationality,” which is what happened, mostly in academic laboratories that never read the book. Brennan's rejection of “it is futile to inquire how a master knows a thing,” which is Kahneman and Klein's condition for trusting expertise. The survivorship parenthesis in My Best and Worst Mistake. The intention–behaviour gap in Trying to Try. The warning that people warned of a bias overcorrect (The Sin of Underconfidence). The first of Jeffreyssai's three flaws, motivated skepticism, among the best-established findings the book uses. No gain from years of experience as a therapist, which Goldberg's eighteen-year follow-up found and then some. And the forecast that visible early contributors would unlock online giving, built within two years as Kickstarter and GoFundMe.
Three corrections in the book's favour. Where this pass disagrees with the earlier Opus annotations, it is to acquit. The gloss on “Thou shalt not boil a kid in its mother's milk” — help should not be turned against the helper — is close to Philo's reading of the verse, so it is not a meaning “no interpreter has given it.” The waste-heat flash in My Naturalistic Awakening is right for any mind with finite memory: reversible computing removes the cost of computation, not of forgetting, and a mind that keeps steering must keep forgetting. And the Game of Life rules as stated in Beyond the Reach of God — three neighbours alive, two unchanged, anything else dead — are the standard compact statement of Conway's rules with a one-word slip (“stay alive” where “become or stay alive” belongs), not a statement of different rules.
Sequence X as evidence. Twelve essays reconstruct the author's teenage reasoning from memory and narrate a wiser present self diagnosing a foolish past one — the configuration in which Ross and Wilson found memory least reliable, because people who believe they have changed exaggerate how bad they were. The author declines to reread the dated writings that would check the story. The lessons drawn from it are then graded by outcome (“actions screen off justifications”), which is outcome bias stated as a principle, from a single survived path, which is survivorship stated as a method.
The difficulty thesis of sequence Y. “I had overestimated the difficulty of almost every single important problem” is contradicted by the planning-fallacy literature and, decisively, by the author's own test case: the Friendly-AI problem the essay says got less impossible with perseverance was, by the author's 2022 account, one humanity would probably fail to solve in time. The machinery built on the thesis — shut up and do the impossible, reject any proposal involving only ordinary effort, decline to put a probability on success — protects the claim of impossibility from the tests that could bear on it, and was answered by the research programmes (control, debate) it dismissed.
The ancestral band. Sequence Z explains why large institutions fail, why nerds won't join, why the Internet can't coordinate and why bystanders don't help by a 40-to-50-person band of people who all knew each other. The forager data show fluid bands of mostly unrelated co-residents and lifetime interaction networks in the hundreds. The problems the essays describe are real and have better explanations, uncredited, in Olson, Hirschman, Ostrom and Latané and Darley.
Ego depletion as infrastructure. It appears as the reason fear of hell cannot make people perfect, as the “true limiting resource” that makes arguing over priorities costly, as the mechanism of warm fuzzies (“willpower restoration”) and in the list of phenomena that actually exist. It has not survived two multi-laboratory preregistered replications. Every argument that routes through it needs another route.
Money as the unit of caring. Prices measure marginal willingness to pay, weighted by wealth; refusing $5 to a public good says nothing about caring for it; the lawyer usually cannot sell the extra hour; and the “cashier's check to a poor woman,” offered as the paradigm of inefficient fuzzies, turned out to be the cash transfer against which effective charities are now benchmarked.
The AI forecasts. Almost every concrete forecast in Book VI about how AI would be built was falsified by 2026: the field was not dead, the working systems look like brains, a neat equation did work, the mind was built without being understood, the builders were corporations with billion-dollar budgets rather than a basement, and the dismissed safety proposals became research programmes. What held up is the general worry, not the specifics — and the specific objections the young author “refuted” in 1996 were the ones the mature author would spend his career advancing.
Book VI is best read as what it says it is: a memoir and a call to build something. As memoir its evidence is weaker than its author allows; as a call it asked the right questions — how would we verify a rationality skill, how do communities of reasoners go bad, why do groups of clever people fail to cooperate — and answered them less well than researchers who were asking the same questions without having read it. The claims that survive intact are almost all diagnoses (pluralistic ignorance among donors, second-order punishment, epistemic viciousness, the valley of partial rationality, the danger of optimizing other people), and several of them were known under other names. The prescriptions — perseverance as the answer to impossibility, practised agreement, separate purchasing of fuzzies and utilons, refusal to quantify — fare worst. The single most important sentence in the book may be its own warning that a science drawing its strength from one person is the idiom of cults; the most important thing the annotations add is that it applies here.