Strangely enough, the science they are trusting in is about three hundred years behind the times. If their science conforms with what their senses tell them, they are subscribing to Newtonian views developed during the seventeenth century. This mechanistic science leads us to view humans as machines that respond to internal and external stimuli, each living in a separate corner of a larger machine: the physical universe. Such a view leads us to believe that we are completely separate, self-animating beings. Our bodies house brains, but our thoughts are only side products of our physical machines; consciousness, free will, divine purpose, and Essence are superfluous at best. The mechanistic view has led us to try and predict and control nature rather than harmonize with it. We strive rather than surrender.

Much the same will be true for the issues that I cover in this book. I’m not saying that we should be confident that value lock-in or major catastrophe will occur this century. What I am saying is that their chance of occurring is very real—certainly more than 1 percent, and certainly greater than many everyday risks, like dying in a car crash. When combined with how much is at stake, the expected value of trying to ensure a good future is enormous. When we’re applying the significance, persistence, and contingency framework, we should therefore be thinking about expected significance, expected persistence, and expected contingency.

For this reason, even if you think that the absence of future generations is not a moral loss or that the end of civilisation would be a good thing (issues that I discuss in Part IV), it’s still very important to avoid AI takeover or the lock-in of bad values. There will be future generations of intelligent beings either way, and by preventing the takeover of the world by an AI with bad values, you are changing how good or bad the future is over the course of civilisation’s life span. That’s the main effect, rather than any impacts on civilisation’s life expectancy.

This last point—that we need to structure global society so that cultural evolution guides the world towards better values and better societal structures—highlights an issue facing the design of a morally exploratory world that I’ll call the lock-in paradox. We need to lock in some institutions and ideas in order to prevent a more thoroughgoing lock-in of values. One challenge is that these institutions and ideas will be morally controversial; for example, from many fundamentalist religious perspectives, the idea that we would encourage or even allow a diversity of worldviews might be regarded as abominable. Similarly, the idea that the path to the correct moral view is via reflection and good-faith debate, rather than studying the scripture of a holy book, is not one that everyone would accept.

The lock-in paradox thus resembles the familiar paradox of tolerance— the necessity for liberal societies to defend themselves against intolerant views that would undermine their freedom, even if doing so requires curtailing the very tolerance they want to preserve.

There’s one sense in which the recording engineers’ maxim is more applicable, though. These are problems of human organisation, and the overwhelming majority of environmental variety of created by humans. The knowledge that something is an organisational problem means that the source of its complexity is the human beings in the organisation. That, in turn, means that the human beings in the organisation have sufficient variety and capability to match the complexity of the problem; the problem can’t be bigger than them, because their own misdirected efforts created it. Only problems that aren’t cybernetic can genuinely be insoluble.*

*One implication of this proposition is that it gives us a rigorous deduction of what would constitute environmental Armageddon — it’s the singularity point at which the problem of climate change ceases to be soluble purely by changing human behaviour and organisation.