Once the reality sets in of a superhuman intelligence being in control of every aspect of the infrastructures we rely upon for everyday life, we will simply have to try to come to an accommodation with that entity.
John L. CastiMy own guess is that quite quickly the machine intelligence will start dreaming machine dreams and thinking machine thoughts, both of which would totally incomprehensible to us. This would then lead to each species, we and the machines, moving off on to its own separate life trajectory.
John L. CastiI do not believe that there is any person, method, or tool that can consistently and reliably predict specific human events, X- or otherwise.
John L. CastiIf you ask which of the scenarios I think is most dangerous, though, I will give a different answer. In that form of the question, I regard a nuclear attack, terrorist-generated or otherwise, as the most threatening combination of likelihood and long-term damage to modern life today.
John L. CastiA good analogy is stretching a rubber band. You can stretch and stretch and even feel the tension increase in the muscles in your hands and arms as the gap from one end of the band to the other widens. But at some point you reach the limits of elasticity of the band and it snaps. The same thing happens with human systems.
John L. CastiWe see an ever-increasing move toward inter and trans- disciplinary attacks upon problems in the real world ... The system scientist has a central role to play in this new order, and that role is to first of all understand ways and means of how to encode the natural world into "good" formal structures.
John L. Casti