This is known as the “alignment” problem. It’s discussed at length in other excellent books, like Superintelligence, Stuart Russell’s Human Compatible, and Brian Christian’s The Alignment Problem, so I won’t go into it in depth here.

To put its wisdom simply, one could say the fundamental human challenge is this:

It’s hard to learn if you already know.

Unfortunately, we are hardwired to feel as if we know—as if we see reality itself rather than a version of reality filtered through our biases, backgrounds, or expertise. But we can unlearn the habit of knowing and reinvigorate our curiosity.

Even when we get the big decisions directionally right, we’re not guaranteed to get the results we want.

We don’t think of ordinary moments as decisions. No one taps us on the shoulder as we react to a comment by a coworker to tell us that we’re about to pour gasoline or water onto this flame.

Inertia keeps us doing things that don’t get us what we want. It operates in our subconscious largely undetected until its effects are too hard to counter.

I asked, “If you could pick one trait that would predict how someone would turn out, what would it be?”

“That’s easy,” he said. “How willing they are to change their mind about what they think they know.”

The most valuable people, he continued, weren’t the ones with the best initial ideas, but the ones with the ability to quickly change their minds. They were focused on outcome over ego. By contrast, he said, the people most likely to fail were those obsessed with minute details that supported their point of view.