Inadequate Equilibria: Where and How Civilizations Get Stuck book by Eliezer Yudkowsky

Partial Excerpts (antilibrary)

2017

Preface and Acknowledgments

Inadequate Equilibria is a book about a generalized notion of efficient markets, and how we can use this notion to guess where society will or won’t be effective at pursuing some widely desired goal

We can ask an analogous question, however, about the “efficiency” of other human endeavors.

How “efficient,” or adequate, should we expect civilization to be at various tasks?

I think that generalized notions of efficiency are the primary key to good reasoning about these real-life questions

So generalized notions of efficiency and inefficiency take

The second half of this book will then delve into background questions of mindset and methodology going into inadequacy analysis, and ways that this analysis can go wrong, particularly for the underconfident

1. Inadequacy and Modesty

This is a book about two incompatible views on the age-old question: “When should I think that I may be able to do something unusually well?”

These two viewpoints tend to give wildly different, nearly cognitively nonoverlapping analyses of questions

Where should they look to find things that our civilization isn’t already doing about as well as can be done?

When I think about problems like these, I use what feels to me like a natural generalization of the economic idea of efficient markets. The goal is to predict what kinds of efficiency we should expect to exist in realms beyond the marketplace, and what we can deduce from simple observations. For lack of a better term, I will call this kind of thinking inadequacy analysis.

Toward the end of this book, I’ll try to refute an alternative viewpoint that is increasingly popular among some of my friends, one that I think is ill-founded. This viewpoint is the one I’ve previously termed “modesty

I’ll open with a cherry-picked example that I think helps highlight the difference between these two viewpoints

i.

I once wrote a report, “Intelligence Explosion Microeconomics,” that called for an estimate of the economic growth rate in a fully developed country—that is, a country that is no longer able to improve productivity just by importing well-tested innovations

2. An Equilibrium of No Free Energy

I am now going to introduce some concepts that lack established names in the economics literature

First, I want to distinguish between the standard economic concept of efficiency (as in efficient pricing) and the related but distinct concepts of inexploitability and adequacy, which are what usually matter in real life

i.

In a pre-market economy, when you offer somebody fifty carrots for a roasted antelope leg, your offer says something about how impressed you are

This is the only instinctive notion people start with for what a price could mean: a personal interaction between Alice and Bob reflecting past friendships and a balance of social judgments

In contrast, the economic notion of a market price

is an “is” statement rather than an “ought” statement, an observation and not a wish.

In particular, an efficient market, from an economist’s perspective, is just one whose average price movement can’t be predicted by you.

3. Moloch’s Toolbox

There’s a toolbox of reusable concepts for analyzing systems I would call “inadequate”—the causes of civilizational failure, some of which correspond to local opportunities to do better yourself

I shall, somewhat arbitrarily, sort these concepts into three larger categories:

Decisionmakers who are not beneficiaries;

Asymmetric information;

and above all,

Nash equilibria that aren’t even the best Nash equilibrium, let alone Pareto-optimal.

I will then play fast and loose with these concepts in order to fit the entire Taxonomy of Failure inside them.

For example, “irrationality in the form of cognitive biases” wouldn’t obviously fit into any of these categories, but I’m going to shove it inside “asymmetric information” via a clever sleight-of-hand

i. For want of docosahexaenoic acids, a baby was lost

As reasoning styles, modest epistemology and inadequacy analysis depend on a mix of explicit principles and implicit mental habits

In inadequacy analysis, it’s one thing to recognize in the abstract that we live in a world rife with systemic inefficiencies, and quite another to naturally perceive systems that way in daily life

my goal here won’t be to unkindly stick the label “inadequate” to a black box containing the world; it will be to say something about how the relevant systems actually operate

For our central example, we’ll be using the United States medical system, which is, so far as I know, the most broken system that still works ever recorded in human history

In previous years, I would use the case of central-line infections as my go-to example of medical inadequacy. Central-line infections, in the US alone, killed 60,000 patients per year,

Central-line infections were also known to decrease by 50% or more if you enforced a five-item checklist that included items like “wash your hands before touching the line.”

I discovered while re-Googling in 2015 that wider adoption of hand-washing and similar precautions are now finally beginning to occur, after many years—with an associated 43% nationwide decrease in central-line infections. After partial adoption.1

So my new example is infants suffering liver damage, brain damage, and death in a way that’s even easier to solve, by changing the lipid distribution of parenteral nutrition to match the proportions in breast milk.

Background: Some babies have digestion problems that require direct intravenous feeding. Long ago, somebody created a hospital formula for this intravenous feeding that matched the distribution of “fat,” “protein,” and “carbohydrate” in breast milk.

Just like “protein” comes in different amino acids, some of which the body can’t make on its own and some of which it can, what early doctors used to think of as “fat” actually breaks down into metabolically distinct elements

So what happens if your infant nutrition was initially designed based on the concept of “fat” as a natural category, and all the “fat” in the mix comes from soybean oil?

Since 1961, parenteral nutrition has used soybean oil as its source of fat.[6] And for decades, babies on parenteral nutrition have suffered devastating liver and brain damage. The death rate on soybean oil is 30 percent by age four

The death-or-liver-transplant rate was reduced from 37 percent with soybean oil to 9 percent with fish oil.2

When Jaminet and Jaminet wrote the above, in 2012, there was a single hospital in the United States that could provide correctly formulated parenteral nutrition, namely the Boston Children’s Hospital

This formulation was illegal to sell across state lines.

A few years after the Boston Children’s Hospital developed their formula—keeping in mind the heap of dead babies continuing to pile up in the meanwhile—there developed a shortage of “certified lipids” (FDA-approved “fat” for adding to parenteral nutrition). For a year or two, the parenteral nutrition contained no fat at all which is worse and can kill adults

The FDA eventually solved the problem with the shortage of US-certified lipids, by… allowing US hospitals to import parenteral nutrition bags from Europe. And it only took them two years’ worth of dead patients to figure that out!

As of 2016, if your baby has short bowel syndrome, and has already ended up with liver damage

your doctor can apply for a special permit to use a non-FDA-approved substance for your child on an emergency basis. After this, you can buy Omegaven and hope that it cures your baby and that there isn’t too much permanent damage and that it’s not already too late.

4. Living in an Inadequate World

For a fixed amount of inadequacy, there is only so much dysfunction that needs to be invoked to explain it

Every possible dysfunction is competing against every other possible dysfunction to explain the observed data

i.

I mean, it’s not like the only possible way one might notice that the US health care system is a vast, ill-conceived machine that is broken and also on fire is to understand microeconomics and predict a priori that aspects of this system design might promote inadequate equilibria

In real life, one notices the brokenness by reading economists who blog about the grinding gears and seas of flame, and listening to your friends sob about the screams coming from the ruins.

Then what good does it do to understand Moloch’s toolbox? What’s the point of the skill?

I suspect that for many people, the primary benefit of inadequacy analysis will be in undoing a mistake already made, where they disbelieve in inadequacy even when they’re looking straight at it

There are people who would simply never try to put up 130 light bulbs in their house

Often the most commonly applicable benefit from a fancy rational technique will be to cancel out fancy irrationality

Whether you’re trying to move past modesty or overcome the Free Energy Fallacy:

Step one is to realize that here is a place to build an explicit domain theory—to want to understand the meta-principles

iv.

Going beyond picking the right horse in the race and becoming a horse yourself, inventing your own new personal solution to a civilizational problem, requires a much greater investment of effort.

events like that should be rare in a given person’s life

Above all, reaching the true frontier requires picking your battles.

Thinking that you’ve searched a large system and found one new exploit is one thing. Thinking that you can exploit arbitrary lines of code is quite another

No matter how broken academia is, no one can improve on arbitrary parts of the modern academic edifice. My own base frequency for seeing scholarship that I think I can improve upon is “almost never,” outside of some academic subfields dealing with the equivalent of “unusually bad code.”

Similarly, you do not generate a good startup idea by taking some random activity, and then talking yourself into believing you can do it better than existing companies