I now believe the world’s long-run fate depends in part on the choices we make in our lifetimes. The future could be wonderful: we could create a flourishing and long-lasting society, where everyone’s lives are better than the very best lives today. Or the future could be terrible, falling to authoritarians who use surveillance and AI to lock in their ideology for all time, or even to AI systems that seek to gain power rather than promote a thriving society. Or there could be no future at all: we could kill ourselves off with biological weapons or wage an all-out nuclear war that causes civilisation to collapse and never recover.

It’s not just technology that has improved people’s lives; moral change has done so, too. In 1700, women were unable to attend university, and the feminist movement did not exist.If that well-off Brit was gay, he could not love openly; sodomy was punishable by death. In the late 1700s, three in four people globally were the victims of some form of forced labour; now less than 1 percent are. In 1700, no one lived in a democracy. Now over half the world does.

For these reasons changing values has particularly great significance from a longterm perspective. Looking to the past, we see that such changes have had an enormous impact on the lives of billions of people. Looking to the future, if we can improve the values that guide the behaviour of generations to come, we can be pretty confident that they will take better actions, even if they’re living in a world very different from our own, the nature of which we cannot predict.

Slavery is so abhorrent that, before getting to grips with the historical scholarship on the topic, I assumed that abolition must have been inevitable. But now I’m not at all sure. Though it’s impossible to know for certain, it’s entirely plausible to me that, were the tape of history rerun a hundred times with slightly different starting conditions, in a significant proportion of those reruns, there would still be legal slavery in many or most countries in the world, even at today’s level of technological development.

For this reason, even if you think that the absence of future generations is not a moral loss or that the end of civilisation would be a good thing (issues that I discuss in Part IV), it’s still very important to avoid AI takeover or the lock-in of bad values. There will be future generations of intelligent beings either way, and by preventing the takeover of the world by an AI with bad values, you are changing how good or bad the future is over the course of civilisation’s life span. That’s the main effect, rather than any impacts on civilisation’s life expectancy.