To illustrate, if the first generation of a computer program that is able to broadly match the effectiveness of an AI researcher can rewrite its algorithms and double its speed or capabilities in six months, then the second-generation program is expected to take three calendar months to perform a similar chunk of work.

Ī third source of concern is that a sudden ' intelligence explosion' might take an unprepared human race by surprise. In contrast, skeptics such as computer scientist Yann LeCun argue that superintelligent machines will have no desire for self-preservation.

Many researchers believe that a superintelligence would resist attempts to shut it off or change its goals (as such an incident would prevent it from accomplishing its present goals) and that it will be extremely difficult to align superintelligence with the full breadth of important human values and constraints. Two sources of concern are the problems of AI control and alignment: that controlling a superintelligent machine, or instilling it with human-compatible values, may be a harder problem than naïvely supposed. In 2022, a survey of AI researchers found that some researchers believe that there is a 10 percent or greater chance that our inability to control AI will cause an existential catastrophe (more than half the respondents of the survey, with a 17% response rate). Concerns about superintelligence have been voiced by leading computer scientists and tech CEOs such as Geoffrey Hinton, Alan Turing, Elon Musk, and OpenAI CEO Sam Altman. The probability of this type of scenario is widely debated, and hinges in part on differing scenarios for future progress in computer science. Just as the fate of the mountain gorilla depends on human goodwill, so might the fate of humanity depend on the actions of a future machine superintelligence. If AI surpasses humanity in general intelligence and becomes ' superintelligent', then it could become difficult or impossible for humans to control.

The existential risk ('x-risk') school argues as follows: The human species currently dominates other species because the human brain has some distinctive capabilities that other animals lack. Existential risk from artificial general intelligence is the hypothesis that substantial progress in artificial general intelligence (AGI) could result in human extinction or some other unrecoverable global catastrophe.

Game changer book alphazero