This is known as the “alignment” problem. It’s discussed at length in other excellent books, like Superintelligence, Stuart Russell’s Human Compatible, and Brian Christian’s The Alignment Problem, so I won’t go into it in depth here.

What this means is that when you have to think about connections between parts, rather than just counting the individual parts of a system, the number of possibilities grows very rapidly; the potential combinations multiply, rather than just adding up. Very quickly, they multiply up to astronomically huge numbers, spelling absolute death to any hope of knowing the entire state of the system.

In fact, knowing only a few of the feedback circuits can be actively misleading, if you rely too greatly on your partial information. It is a sobering thought, for example, that despite employing some of the best and brightest* analysts in the world, the advice given by the US State Department over the last fifty years could comfortably have been outperformed by a parrot that had been trained to repeat the phrase, ‘Don’t start a war.’ The repeated failures of the State Department are not the consequence of ignorance; they are the consequence of having very good and deep — but not total — knowledge of an extremely complicated situation, in which facts outside of that information set turned out to be crucial. Knowing a great deal of detail about a subset of a system has a habit of increasing your confidence in your opinions disproportionately from their reliability.

*The phrase ‘the best and brightest’ is often used by people who don’t know that its original context was ironic. It entered the language as the title of David Halberstam’s book about the policy mistakes of the Vietnam War.

Trying to build things meant that everything had to be kept specific, but the theory of information couldn’t straightforwardly be applied to massively connected systems. You needed to find a way of describing things that was both rigorous and representative of reality. As economists will tell you, this isn’t an easy thing to do.

This law states that anything which aims to be a ‘regulator’ of a system needs to have at least as much variety as that system. In order to understand what that means, we need to get a short way into the theory of cybernetics. But for the meantime, for a sort of high-level metaphorical understanding, * think about how you steer various kinds of vehicles. A train can really only go forwards and backwards, so it only needs a single handle. A car can make turns, so its control system requires a steering wheel to represent the added dimension. And an aeroplane needs a joystick rather than a steering wheel, because it can make two kinds of turns. Let’s see where cyberneticians took this idea.

Which to be honest, is often all you really need in these contexts. The whole damn thing* in cybernetics is made of newspapers and high-level analogies. This isn’t computer science, where you can point to the states, bits and circuits on a piece of silicon.

The overlapping of different systems — and the tendency of individuals to have different roles at different levels of abstraction — is a key part of Beer’s theory, and one of the main reasons why his diagrams got so complicated. He claims that every ‘viable system’ needs to have all five of the functions described so far in order to be capable of long-term survival, but that every such system can also be seen as System I within a larger system. Similarly, since we defined System I as part of an organisation that could in principle be a viable separate organisation, the internal management of System I needs to have its own equivalents to systems 2, 3, 4, and 5; it needs internal regulation, optimisation and intelligence, and a balancing, identity-preserving function of its own.

Often, when you’re trying to diagnose why a system is failing, you need to consider both the larger system in which it’s embedded and the organisation within its operations. A great source of management problems, for example, is that organisations often fail to identify some of their operations as distinct systems, and so they lack their own internal ‘higher functions’. A division of this sort will generally be a ‘problem child’; unable to absorb its own environmental variability, it will bounce from crisis to crisis, taking up disproportionate time and effort on the part of the middle managers to which it has been assigned.