Much the same will be true for the issues that I cover in this book. I’m not saying that we should be confident that value lock-in or major catastrophe will occur this century. What I am saying is that their chance of occurring is very real—certainly more than 1 percent, and certainly greater than many everyday risks, like dying in a car crash. When combined with how much is at stake, the expected value of trying to ensure a good future is enormous. When we’re applying the significance, persistence, and contingency framework, we should therefore be thinking about expected significance, expected persistence, and expected contingency.

Would we wield such unprecedented power responsibly? Worryingly, the pursuit of value lock-in has been common throughout history. We saw that when the Qin took control of China, they undertook a programme to systematically eradicate competing schools of thought; similarly, the Han systematized Confucian teachings to the detriment of competing schools. The Mohists, too, desired to lock in their own values indefinitely, if only they had the power. They saw moral disagreement as the biggest problem in the world and thought that the solution was to ensure that everyone had the same values.

When we look at history, we see that the predominant culture in a society tends to entrench itself, eliminate the competition, and take steps to replicate itself over time. Indeed, many moral views regard their own lock-in as desirable.

But if we are guilty of gross moral errors ourselves, then locking in our present values would also be a disaster.

Instead, we should try to ensure that we have made as much moral progress as possible before any point of lock-in. Political philosophers often argue over what an ideal state would look like. I think we should accept that we don’t know what the ideal state would be; the primary question is how we can build a society such that, over time, our moral views improve, people act more often in accordance with them, and the world evolves to become a better, more just place.

This last point—that we need to structure global society so that cultural evolution guides the world towards better values and better societal structures—highlights an issue facing the design of a morally exploratory world that I’ll call the lock-in paradox. We need to lock in some institutions and ideas in order to prevent a more thoroughgoing lock-in of values. One challenge is that these institutions and ideas will be morally controversial; for example, from many fundamentalist religious perspectives, the idea that we would encourage or even allow a diversity of worldviews might be regarded as abominable. Similarly, the idea that the path to the correct moral view is via reflection and good-faith debate, rather than studying the scripture of a holy book, is not one that everyone would accept.

The lock-in paradox thus resembles the familiar paradox of tolerance— the necessity for liberal societies to defend themselves against intolerant views that would undermine their freedom, even if doing so requires curtailing the very tolerance they want to preserve.