Economists end up solving this problem by denying it (a mechanism that Stafford Beer looked at in detail, concluding that ignorance is a kind of information processing system of last resort). Where analysis fails, ideology steps in, and the solution the economists decided on was to fantasise an equivalent system for producers — a profit-maximising firm with an understanding of its market, a style production process and a manageable set of decisions. It ended up going disastrously wrong, but the original intellectual sin may have been the failure to respect the integrity of the black box.

Stafford Beer’s cybernetics tells us that in these cases, while people’s opinions are important, the facts of the organisational outcomes are what we need to work with. In his most pithy formulation of the principle, he expands the black box principle to a rather more uncomfortable statement.

The purpose of a system is what it does.

An organisation does things, and it systematically does some things rather than others. But that’s as far as it goes. Systems don’t make mistakes — if they do something, that’s their purpose. But it also works the other way around. Systems don’t have inner desires, so they don’t do things intentionally either. There’s just a network of cause and effect. We might think they’re conspiring, but they’re working within structures that made the outcome inevitable. Or we might see everything as a terrible cock-up, but we don’t understand that the outcome was the inevitable result of the way the system works.

Stafford Beer takes an approach to this problem that helps to mark the distinction between information theory and cybernetics. That solution is to say that information and action are one and the same; variety coming in from the environment, or being transferred from one system to another, only counts as ‘information’ if it has a causal role in decision-making. Otherwise it’s just ‘data’ — collections of facts that hang around on disk drives, waiting to be erased* or for the format to become obsolete.

*There are a lot of people is Silicon Valley who might do well to consider how much money they have invested in ‘data’ without bearing this distinction in mind.

Warren Buffett similarly highlighted the effects of the social default in his 1984 letter to Berkshire Hathaways’s shareholders:

Most managers have very little incentive to make the intelligent-but-with-some-chance-of-looking-like-an-idiot decision. Their personal gain/loss ratio is all too obvious: if an unconventional decision works out well, they get a pat on the back and, if it works out poorly, they get a pink slip. (Failing conventionally is the route to go; as a group, lemmings may have a rotten image, but no individual lemming has ever received bad press.)

Lemmings might make small changes, sure, but not the changes they need in order to make an outsize impact.