(2024-04-04) The Unaccountability Machine Why Do Big Systems Make Bad Decisions

The Unaccountability Machine — why do big systems make bad decisions? Waiting to board a flight recently... A fellow passenger had been refused boarding and was pleading with the airline attendant. The attendant was obviously sympathetic — but it was down to company policy. We were witnessing what Dan Davies calls an “accountability sink”: a situation in which a human system delegates decision-making to a rule book rather than an identifiable individual. If something goes wrong, no one is held to account.

The starting point of Davies’ entertaining, insightful book is that the uncontrolled proliferation of accountability sinks is one of the central drivers of what historian Adam Tooze calls the “polycrisis” of the 21st century.

the story, Davies continues, is not so simple. Seen from another perspective, accountability sinks are entirely reasonable responses to the ever-increasing complexity of modern economies. Standardisation and explicit policies and procedures offer the only feasible route to meritocratic recruitment, consistent service and efficient work

The question, therefore, is not how to return to a world without non-human decision-making systems — it’s how to ensure that they’re open to feedback, able to adapt and improve

The Unaccountability Machine scrutinises two intellectual frameworks that aim to supply an answer. The first will be well known to most FT readers: economics. Davies judges it hopelessly deficient.

Instead, Davies argues, there exists a second, better way — one less familiar to most of us. This is “management cybernetics”,

Davies’ hero here is the management consultant, information theorist and philosopher Stafford Beer,

Beer conceived of companies as forms of non-human intelligence — what the science-fiction writer Charlie Stross calls “very old, very slow AIs”. (AGI)

So how about practical recommendations? Davies argues that in the corporate world, the most obvious one is to defang the leveraged buyout (LBO) industry. Here, the cybernetic solution is to require private equity investors to guarantee the debts of takeover targets, sealing off the great accountability sink of limited liability.

Beyond this, though, Davies is coy about specifics.