Our approach to existential risks cannot be one of trial-and-error. There is no opportunity to learn from errors. The reactive approach - see what happens, limit damages, and learn from experience - is unworkable. Rather, we must take a proactive approach. This requires foresight to anticipate new types of threats and a willingness to take decisive preventive action and to bear the costs (moral and economic) of such actions.

Machine intelligence is the last invention that humanity will ever need to make.

We would want the solution to the safety problem before somebody figures out the solution to the AI problem.

Knowledge about limitations of your data collection process affects what inferences you can draw from the data.

There are some problems that technology can't solve.

In the next century, we will be inventing radical new technologies - machine intelligence, perhaps nanotech, great advances in synthetic biology and other things we haven't even thought of yet. And those new powers will unlock wonderful opportunities, but they might also bring with them certain risks. And we have no track record of surviving those risks. So if there are big existential risks, I think they are going to come from our own activities and mostly from our own inventiveness and creativity.

Our approach to existential risks cannot be one of trial-and-error. There is no opportunity to learn from errors. The reactive approach - see what happens, limit damages, and learn from experience - is unworkable. Rather, we must take a proactive approach. This requires foresight to anticipate new types of threats and a willingness to take decisive preventive action and to bear the costs (moral and economic) of such actions.