You have a very precisely defined goal and you build a machine that's superhuman in its capabilities for achieving goals. If it turns out that the subsequent behavior of the robot in achieving that goal was not what you want, you have a real problem.

To my knowledge nobody - no one who is publishing papers in the main field of AI - is even working on consciousness. I think there are some neuroscientists who are trying to understand it, but I'm not aware that they've made any progress.

The robot is not going to want to be switched off because you've given it a goal to achieve and being switched off is a way of failing - so it will do its best not to be switched off. That's a story that isn't made clear in most movies but it I think is a real issue.

When people talk about the singularity, when people talk about superintelligent AI, they're not talking about sentience or consciousness. They're talking about superhuman ability to make high-quality decisions.

AI's ability to recognize visual categories and images is now pretty close to what human beings can manage, and probably better than a lot of people's, actually. AI can have more knowledge of detailed categories, like animals and so on.

What AI could do is essentially be a power tool that magnifies human intelligence and gives us the ability to move our civilization forward. It might be curing disease, it might be eliminating poverty. Certainly it should include preventing environmental catastrophe. If AI could be instrumental to all those things, then I would feel it was worthwhile.

Everything we have of value as human beings - as a civilization - is the result of our intelligence.