We would want the solution to the safety problem before somebody figures out the solution to the AI problem.

In the next century, we will be inventing radical new technologies - machine intelligence, perhaps nanotech, great advances in synthetic biology and other things we haven't even thought of yet. And those new powers will unlock wonderful opportunities, but they might also bring with them certain risks. And we have no track record of surviving those risks. So if there are big existential risks, I think they are going to come from our own activities and mostly from our own inventiveness and creativity.

We would want the solution to the safety problem before somebody figures out the solution to the AI problem.

We should not be confident in our ability to keep a super-intelligent genie locked up in its bottle forever.

Human nature is a work in progress.

How can we trace out the links between actions that people take today and really long-term outcomes for humanity - outcomes that stretch out indefinitely into the future? I call this effort macrostrategy - that is, to think about the really big strategic situation for having a positive impact on the long-term future. There's the butterfly effect: A small change in an initial condition could have arbitrarily large consequences.

The challenge presented by the prospect of superintelligence, and how we might best respond is quite possibly the most important and most daunting challenge humanity has ever faced. And-whether we succeed or fail-it is probably the last challenge we will ever face.