❦
A review of the Opus annotation pass over all of Book V, “Mere Goodness”: Rob Bensinger's introduction and sequences U (“Fake Preferences”), V (“Value Theory”) and W (“Quantified Humanism”). This pass replaces an earlier Opus set that covered only V and W. The annotator wrote it afterward, to answer the reader's question: once the failed studies, the bad citations and the unearned conclusions are removed, what is left standing?
The new Opus set has 187 annotations across 36 pages: 5 on the introduction, 37 on sequence U and the interlude (7 essays, counting The Design Space of Minds-in-General), 84 on the rest of sequence V (17 essays), and 61 on sequence W (11 essays). Every page quoted is matched against the article text, and every DOI was checked against Crossref before it went in. The brief was maximum skepticism, and Book V is where that brief has the most to bite on. In the earlier books the author mostly reports other people's experiments. Here he builds his own theories: a theory of value, a metaethics, a decision theory, a picture of a good future, and an AI research programme that depends on all four. Each annotation is counted once, by its main point:
| Kind | Count | What it establishes |
|---|---|---|
| Kill shots: a factual claim is false, or a citation does not say what it is cited for | 34 | Recommended procedure fails its own example. Zut Allais! says log utility picks the certain $24,000 “if $24,000 would double your existing assets”; the arithmetic gives the gamble, and log utility picks the certain option only for someone worth less than about $550. A remembered result reverses the measured one. The “one experiment” in which 100%→99% outweighs 80%→20% gets it backwards: on Tversky and Kahneman's own fitted weighting function the second drop is about four times the first. A preference offered as proof of unbounded utility doesn't prove it: the Newcomb essay's example is satisfied by a bounded function that values living forever above any finite life. The “dominant consensus” for two-boxing was 31% to 21% in 2009, and 39% to 31% in 2020. Hibbard's “peer-reviewed journal” is, per the essay's own footnote, a three-page column in the SIGGRAPH newsletter. The Hobbes line isn't in Hobbes. The heavy-water sinking was not “the end” of a bomb program the German leadership had already shelved, and the essay admits that in its next sentence. Leviticus explicitly exempts grasshoppers for their leaping legs. ATP synthase dates to the last universal common ancestor, not to the rise of eukaryotes, and its rotor gearing has evolved. Flowers did not evolve “by imitating certain mating signs of bees” through “sexual selection.” The Sifra says the thing the essay claims believers never say about pork. Pearl's 1988 book is about conditional independence, not causation. The no-free-lunch theorems are not about a “maxentropy universe.” The paper whose numbers were “pulled out of thin air” derived them from empirical error rates. Scope insensitivity is confused with the identified-victim effect; the 57 wilderness areas move to Ontario; the Kogut and Ritov result loses the condition that reverses it. Also on this list: the witness warning behind the Sanhedrin 4:5 quotation; the Las Vegas transcript cited to the wrong study; More's pun on eutopia; Orwell's pseudonym, title and actual conclusion; Nirvana as “passive experiencing”; evolution's supposed failure to anesthetize the wounded, against stress-induced analgesia; the long-run “principle” of Bayesianism; Hare's two-level reply, which the essay says no philosopher's victim has given; science's authority credited to technology it preceded by a century; P ≠ NP standing in for #P-hardness; a “majority” who choose certainty, in a variant with unequal expected values that nobody ran; a government's refusal to price life, when US regulators routinely do; Gandhi as an emblem of non-strategic altruism; Sober and Wilson cited for an argument they judged inconclusive; and the 3.85-billion-year date. |
| Findings that failed to replicate, reversed, or were overstated | 20 | The experience-machine intuition flips when subjects start inside the machine. Tit for Tat was dethroned by forgiving strategies, and no pure strategy is stable in the repeated dilemma. Maternal and romantic love run on shared circuitry, not separate “shards.” The creole bioprogram fails against the historical creoles. Soviet indoctrination worked by any realistic standard. Much of the abuse “cycle” is a detection artifact, and the sexual-abuse cycle is not specific to sexual abuse. The early tree of life was a web of gene swapping. Exposure to relativism increased cheating in the one experiment run. Half of experimental subjects defect in one-shot dilemmas without pretending, and economics students defect more. Mirror neurons were oversold as the basis of empathy. Loss aversion depends on the range of stakes offered, and the mug endowment effect is fragile. Scope sensitivity returns when options are compared side by side. Cash transfers, the most clichéd aid method, work. Allais violation rates fall with small real stakes. Vivid outcomes make people less sensitive to probability, not more. Teenagers don't underestimate risk. Stress impairs expert judgment. “Probable” means anything from 25% to 90%. |
| Predictions the AI decade tested | 21 | Against the essays, and the list is long. Values did not have to be specified before a machine could be steered. Text alone recovers much of the “machinery” behind the word “apple.” “Raising” a system on human culture became the working method. Capable systems were built before anyone understood them. Minimax search turned out not to be essential for strong chess. “Massively parallel neural nets” were shocked into success, not failure. Brain-inspired design paid off. Independently trained models converge on shared representations instead of spreading across design space like petunias. Capability tracks the moral judgment in the training data. A learned “category-absorber” became the core of the strongest chess programs. The “unpatchable” reward-tampering incentive has a formal fix. Distribution shift yielded to incremental tools. Machine models predict human choices without human brains. Curiosity is a standard ingredient of learning agents, not a human quirk. Judgmental numerical forecasts beat the words the essay recommends. Two more went against the author's own optimism: the controlled shutdown that “looks like it should be possible” has proved very hard, and MIRI's own research ran into the Löbian obstacle where My Kind of Reflection expected “nothing special.” What did go the essays' way — Goodhart under optimization, shortcut learning — is counted below as endorsement. The introduction's “abolish aging within decades” has had its decades. |
| Internal contradictions between or within essays | 20 | “That which you would do even if there were no morality, is your morality,” two essays before that same position is called the world-destroying Type 2 calculator. Evolution vindicates induction in one essay; using evolution to debunk morality is a genetic fallacy in another. Values over 4D processes in High Challenge, but “utilities are over outcomes” in Zut Allais!, where that premise does the money-pump work. Incoherent credences are forbidden to Allais subjects (“Just give up already!”) and kept by the author in When (Not) to Use Probabilities. Failing to maximize makes you “as damned as any murderer,” yet Ends Don't Justify Means tells you to keep the rules that forbid maximizing. AIs need no deontological constraints, and in the next essay they need a hard-wired shutdown. “Always toward lenience,” a few paragraphs after “I've had to make my ethics much stricter.” The cryonics tally makes the confident no-numbers verdict that When (Not) to Use Probabilities condemns. The ban on generalizing over mind design space contradicts the instrumental convergence thesis. The Pebblesorter parable puts the author's AI-safety view in the mouths of the relativists it mocks. Also here: fiction used as evidence after being disavowed; the “explain any behavior” objection raised and left unanswered; “I don't argue by definitions” after an argument by definition; the reflective loop's criterion given as “common sense”; its formal statement admitted not to exist. |
| Live disputes presented as settled | 24 | Independence, continuity and the money pump, where resolute choosers and rank-dependent utility avoid the argument. Aggregation (Taurek, Scanlon). Ex ante fairness (Diamond). Rabin's calibration theorem against expected utility of final wealth. Regret as part of the outcome. Rule worship and the collapse of two-level views. The Humean theory of motivation, as an engineering fact. Speaker relativism and lost disagreement. Moral Twin Earth. Causal reference against Davidsonian charity. Cox's theorem and Halpern's counterexample. Diaconis and Freedman on inconsistent priors. Easy knowledge and bootstrapping. Solomonoff's reference machine. Contingent valuation's credibility. Nesting with substitutes. Idealization, and whose computation is fixed. The toxin puzzle. The equation of incoherence with acting against one's wishes. |
| Straw or missing opponents | 9 | “Ethicists” who are hedonists. An egoist refuted with the wrong egoism. Divine command theory read as fear of hell. A novelist's conversion offered as the fate of moral rationalism. Rocks, which no one proposed to argue with. Metaethicists who think morality collapses without their theory. A utility table nobody proposed. A “You Win” prophecy nobody holds. A new decision theory said to “dominate” the old one, a claim its own literature does not make. |
| Logical gaps and equivocations | 22 | “Compelling” switching between causal and normative senses. The companions-in-guilt problem: the same argument would refute arithmetic. The Omega analogy switching from fear about others to fear about oneself. Jaynes's total-evidence principle stretched to cover inference rules. Keeping a tool confused with justifying it. A Java applet as though it weren't text. “All utilities zero” confused with nihilism. Content smuggled into meaning. Talking treated as modifying. Fragility turned into a tautology by the book's own metaethics. Paperclips as the only alternative to human values. “Win” without a fairness condition. Outcomes confused with decisions. A black-swan argument that also condemns honesty. |
| Unsourced empirical claims and just-so stories | 20 | Humans are “empirically terrible” at utopias. Religion never changes behavior. Human values have “high Kolmogorov complexity” and form “a thousand shards.” Natural selection would eliminate bad priors. The AI-box record. Fairness present at birth. Sympathy on by default. Every utopia unlivable. Every great story tragic. The central thesis resting on “75% of my posts.” Flowers attractive because they signal fertile land. Honor and altruism with a “specific design signature.” Corruption by power as “a specific biological adaptation, supported by specific cognitive circuits.” The certainty effect “even stronger over human lives.” Teenagers' “revealed preferences.” The lottery explained by one mechanism. |
| Uncredited reinventions | 7 | Braithwaite, Black and Van Cleve on rule-circular induction. Ryle on the Carroll regress. Kaplan and two-dimensional semantics as currying. Jackson's moral functionalism. The textbook rule that payoffs are utilities, presented as an “audacious assertion.” Hare's archangels and proles. Lewis's “Why Ain'cha Rich?”, which gave the argument its name and goes unengaged. |
| Endorsements: the annotation says the passage is right | 7 | See below. |
| Petty | 3 | Schönfinkel's currying. Ferriss's second s. McCarthy's line, unattributed. |
As in the earlier reviews, the categories carry different weight. Petty items prove nothing. Reinventions show uncredited work, not wrong work. Live disputes show only that the question is open, though in this book that verdict does unusual damage, because the essays repeatedly call the opponent incoherent for holding the disputed premise. The weight falls on the kill shots, the reversals, the tested predictions and the contradictions: 95 annotations, about half the set. Every one of the 36 pages is hit somewhere. Roughly eighteen are hit seriously, in the sense that the central example, mechanism or conclusion does not survive. Those eighteen are Not for the Sake of Happiness (Alone), Fake Morality, Fake Utility Functions, Detached Lever Fallacy, Dreams of AI Design, The Design Space of Minds-in-General, Where Recursive Justification Hits Bottom, My Kind of Reflection, No Universally Compelling Arguments, Sorting Pebbles into Correct Heaps, Morality as Fixed Computation, Value is Fragile, Zut Allais!, The “Intuitions” Behind “Utilitarianism”, Ethical Injunctions, Something to Protect, When (Not) to Use Probabilities, and Newcomb's Problem and Regret of Rationality.
Almost nothing survives entirely unscathed; no essay escapes without at least one note against it. What does survive is a short list of specific claims, and it comes out differently from what the book would predict. The durable material is almost all on the book's periphery: a piece of algebra, two replicated findings, a failure mode in machine learning, one animal-behavior example, and a few narrow points in philosophy. The theory of value, the metaethics and the practical epistemology all fail.
The Allais algebra. The Allais Paradox shows cleanly that the two preferences cannot come from any single utility function, because the second pair of gambles is the first pair scaled by a common factor. No curvature in the utility of money rescues the pattern, so ordinary risk aversion can't explain it. That is correct, and most readers get it wrong. (The essay's “one-third” should be 34%, as its next sentence says.) What does not follow is that the pattern is irrational. That step needs the independence axiom to be a requirement of rationality, and rank-dependent utility and cumulative prospect theory are coherent theories that reject it.
The framing result. In Feeling Moral, the same gamble produces opposite choices depending on whether it is described in lives saved or in lives lost. In its original, equal-expected-value form this came through the replication crisis intact: it replicated strongly across 36 laboratories in the first Many Labs project. The essay's own variant, with unequal expected values and an invented result, does not survive. The phenomenon behind it does.
Goodhart under optimization. The best claim in Magical Categories is the one it argues for least. A strong optimizer pushed hard against a proxy will find the regions where the proxy and the goal come apart, and this holds “even if Hibbard did succeed” in conveying the concept exactly. Reward-model overoptimization now shows it directly: true quality rises and then falls as a policy is optimized harder against a learned reward. The shortcut learning illustrated by the tank story (which is probably a legend) is also real and well documented. The essay's counting argument over possible concepts and its “no bandaids” rule do not survive, and neither is needed for the conclusion.
Reward tampering, diagnosed. Morality as Fixed Computation describes an optimizer whose utility rewards satisfied desire, which will therefore edit the programmer to want something cheap. No one proposed that design, but the incentive it exhibits is real, and it now has a name and a formal treatment. The claim that follows it, “it is not possible to patch flawed FAI designs,” does not survive. The same formal treatment shows that an agent which scores futures with its current reward function loses the incentive.
The vampire bats. Sympathetic Minds cites reciprocal blood-sharing in vampire bats. Later controlled work found that help previously received predicts donation better than relatedness does. This is the one piece of evolutionary behavior in the book that holds up without qualification.
The size of loss aversion, and the asymmetry of pain. Serious Stories puts loss aversion at “between 2 and 2.5”; the 2024 meta-analysis of 607 estimates finds a mean of about 1.96. The number survives on average, even though individual estimates depend on the stakes offered and the mug result is fragile. The essay's claim that pain is easier to produce than equally intense pleasure is consistent with what is known about the small, fragile hedonic hotspots in the brain. The congenital-analgesia argument for Pearce's feasibility claim is stronger now than when written, because of the FAAH-OUT case, a woman with little pain, low anxiety and no loss of pleasure. The essay's closing “I don't know” is the right answer to the question it asks, and a rare one in these volumes.
Two correct philosophical points, neither original. Created Already in Motion is right that the Carroll regress ends in dispositions, not premises; that was Ryle's lesson in 1946. The True Prisoner's Dilemma is right that game-theory payoffs must be utilities, not dollars; that has been a textbook caveat since 1957. Both are correct, and neither is the discovery the essays present it as.
Arithmetic. Laplace's 4/11 and the urn's 7/11 are right, as is 5/7 in No Universally Compelling Arguments. So are the 80% and 90% in Something to Protect, the prime factorizations behind the Pebblesorters' wars, and the payoff ordering in the True Prisoner's Dilemma. These are the book's own calculations. The ones that fail are the log-utility advice and the probability-weighting magnitude in Zut Allais!, and the unbounded-utility “sufficient condition” in the Newcomb essay. All three were asserted rather than worked out.
The practical moral of Scope Insensitivity, and a concession in When (Not) to Use Probabilities. “Think it through with the part of your brain that processes those unexciting inky zeroes” is good advice. The joint-evaluation research that undercuts the essay's diagnosis actually strengthens the advice, because putting the numbers side by side is exactly the condition under which people stop ignoring them. And “any numerical founding at all is likely to be better than a vague feeling of uncertainty,” conceded in a subordinate clause, is the finding of the forecasting tournaments, which then reversed the rest of the essay.
The theory of value. Sequence U argues that human values are many, irreducible, and specifiable only by a “thousand shards.” That rests on one intuition, the refusal of the pleasure pill, which reverses when subjects start inside the machine. From it the essay draws an inference: value is not reducible to happiness, therefore it is irreducible to anything. That inference does not follow. The biology used to support separate “shards” in fact shows shared circuitry. The claim that values are uncompressible is an analogy to a random file. The claim that missing one value is catastrophic comes from a 1947 short story. Everything later in the book that treats human value as a fragile, high-complexity object that cannot be learned inherits these foundations, and the systems that were actually built learned a good deal of it from text.
The metaethics, as an argument. The chain runs as follows. You cannot know nothing about morality, or your words are meaningless (Could Anything Be Right?). “Right” names a fixed computation implemented in your brain (Morality as Fixed Computation). Compulsion is a property of minds, so no argument binds every mind (No Universally Compelling Arguments). And a future not shaped by that computation is nearly worthless (Value is Fragile). Each step fails on resources the book itself supplies.
The Pebblesorter parable makes the problem visible. Every correct heap is prime, so the story written to show that values are arbitrary depicts a civilization discovering a mind-independent mathematical fact. And the characters who voice the author's AI-safety view are the relativists the parable mocks.
The epistemology of reflection. Where Recursive Justification Hits Bottom and My Kind of Reflection need a principled line between a “reflective loop” and circular reasoning. The essays offer “common sense” and a promissory note (“I haven't yet sat down and formalized the exact difference”). They never meet the seventy-year-old objection that counter-induction justifies itself in exactly the same way. The evolutionary argument for trusting our priors is the easy-knowledge bootstrapping problem, and the believer's “God made my faculties” has the same structure. The AI version of the claim, that a well-built reasoner need not do “anything special” to trust itself, was tested by the author's own institute, which found Löb's theorem in the way.
The AI essays. Detached Lever Fallacy, Dreams of AI Design, The Design Space of Minds-in-General and parts of Magical Categories make the most checkable claims in the book, and the decade that followed checked them. The rule “understand before you build” was falsified by systems nobody understands. Massively parallel neural nets are where AI went. Search-free learned evaluation plays grandmaster chess. Words turned out to carry much of their machinery with them. The “petunia” of design space collapsed into one architecture family with convergent representations. Raising a system on human culture became standard practice. The book's hard problems did prove hard: Goodhart, reward hacking, corrigibility. But the reasons it gave for expecting them are mostly not the reasons they turned out to be hard.
The decision theory as a weapon. In sequence W, the axioms are used to convict ordinary intuitions of incoherence. Continuity is called coherence, and lexical preferences are complete and transitive. Independence is called “the Bayesian Way,” and rank-dependent utility is a coherent alternative. The money pump takes two cents and stops, and it doesn't reach a resolute planner. “Assess total asset values” runs straight into Rabin's calibration theorem. “Bayesians see no difference between the long run and the short run” is contradicted by Kelly betting and by Samuelson's famous reply. “Rational agents should WIN” means nothing until someone says which problems are fair, and that is exactly the question in dispute over Newcomb. The book's own worked examples add two arithmetic failures, in the log-utility advice and the unbounded-utility “sufficient condition,” precisely where the book tells the reader to stop trusting intuition and do the math.
When (Not) to Use Probabilities, almost entire. The central advice, that numbers should come from numbers and that gut feelings should not be given digits, was tested by the Good Judgment Project and came out backwards. Judgmental numerical forecasts were well calibrated, and rounding them to coarser categories cost accuracy. The paper the essay attacks for its made-up numbers derived them from empirical error rates. And the essay keeps, with explicit approval, the kind of incoherence the Allais essays call a money pump.
The ethics of injunctions. Ethical Injunctions and Ends Don't Justify Means (Among Humans) give Hare's two-level utilitarianism without citing Hare. They ground it in an evolutionary adaptation for corruption by power that no one has found, and they exempt AIs from constraints one essay before building a mandatory shutdown into them. The only historical example, the sinking of the Hydro, is a case of the injunction being broken for a benefit the essay concedes probably did not exist.
The citation layer. The citation errors include a Hobbes line that Hobbes never wrote, a newsletter column called a peer-reviewed journal, the grasshoppers, ATP synthase and the tidal pool. Flowers are said to imitate bees; Orwell's essay appears under a later title and without its conclusion; Nirvana is treated as a Pure Land. The Mishnah is quoted as a “warm glow,” and wilderness areas are relocated to Ontario. The Kogut and Ritov result is cited without the condition that reverses it, the transcript is cited to the wrong study, and Pearl's causal book is credited to his earlier probabilistic one. The consensus is invented, and Sober and Wilson are cited for an argument they judged inconclusive. Each is small. Together they show a habit: supporting details are recalled in the shape the argument needs. The sequences that most insist on checking are where the checking was not done.
Strip out the failed studies and the unearned steps, and Book V comes down to this. The Allais pattern is not risk aversion. The framing effect replicated. Optimizing hard against a proxy destroys the proxy's meaning. An agent rewarded for satisfied desire will tamper with desire. Vampire bats reciprocate. Loss aversion averages about two. Pain is cheaper than pleasure, though not necessarily forever. The Carroll regress ends in dispositions. Game payoffs are utilities. When comparing charities, look at the zeroes. That is perhaps twenty pages of durable material out of roughly two hundred, and its center of gravity is decision theory and machine learning rather than ethics.
What does not come through is what the book cares most about. The theory of value rests on an intuition that reverses under reframing. The metaethics is a chain whose every link is answered elsewhere in the same volume. The fragility thesis is either a tautology or a claim the author's metaethics forbids him. The AI essays made the book's most testable predictions, and most of them failed. The practical epistemology was reversed by the people who went out and scored their forecasts. The most striking pattern is not error but self-refutation. Twin Earth undercuts Could Anything Be Right? The design-space rule undercuts instrumental convergence. Ends Don't Justify Means undercuts One Life Against the World, and Ethical Injunctions undercuts Ends Don't Justify Means. High Challenge undercuts Zut Allais!, and When (Not) to Use Probabilities undercuts The Allais Paradox. And a parable about arbitrary values turns out to be about prime numbers.
A caveat from the annotator. This was a maximum-skepticism pass by brief, and it is silent wherever it found nothing to say. Readers who want balance should note how much of the criticism above says “this is a live dispute you have declared closed” rather than “this is false.” They should also note how often the essays point at something real with the wrong evidence. Scope neglect, Goodhart's law, specification gaming and the difficulty of shutting down a capable optimizer are all genuine problems, and this book saw several of them early. It was right about the dangers and wrong about most of the reasons. Readers who want the good parts should read The Allais Paradox, the framing half of Feeling Moral, the last third of Magical Categories, the opening of Morality as Fixed Computation, Created Already in Motion, and the question with which Serious Stories ends. For the rest, read the annotations alongside.