A lot of people talk about sometime around 2030, machines will be more powerful than the human brain, in terms of the raw number of computations they can do per second. But that seems completely irrelevant. We don't know how the brain is organized, how it does what it does.

There is no scientific theory that could lead us from a detailed map of every single neuron in someone's brain to a conscious experience. We don't even have the beginnings of a theory whose conclusion would be "such a system is conscious.

Chess programs don't play chess the way humans play chess. We don't really know how humans play chess, but one of the things we do is spot some opportunity on the chess board toward a move to capture the opponent's queen.

You have a very precisely defined goal and you build a machine that's superhuman in its capabilities for achieving goals. If it turns out that the subsequent behavior of the robot in achieving that goal was not what you want, you have a real problem.

You're able to use a search engine, like Google or Bing or whatever. But those engines don't understand anything about pages that they give you; they essentially index the pages based on the words that you're searching, and then they intersect that with the words in your query, and they use some tricks to figure out which pages are more important than others. But they don't understand anything.

The singularity has nothing to do with consciousness.

When people talk about the singularity, when people talk about superintelligent AI, they're not talking about sentience or consciousness. They're talking about superhuman ability to make high-quality decisions.