[summary: A 'sufficiently advanced' Artificial Intelligence is one smart enough that we need to start thinking about some potential difficulty being discussed.
For example: We probably don't need to worry about an AI that tries to prevent us from pressing its off-switch, until the AI knows that (a) it has an off-switch and (b) pressing the off-switch will prevent the AI from achieving its other goals.
In turn, this knowledge is a special case of Big-picture strategic awareness, which might follow from learning many facts about many domains via Artificial General Intelligence.
See Advanced agent properties for a list of some different ways a cognitive agent or algorithm could be smart enough that we would need to start doing AI alignment theory to it.]
A 'sufficiently advanced Artificial Intelligence' is a cognitive agent or cognitive algorithm with capabilities great enough that we need to think about it in a qualitatively different way from robotic cars; a machine intelligence smart enough that we need to start doing AI alignment theory to it.
For example, we probably don't need to worry about an AI that tries to prevent us from pressing its off-switch, until the AI knows that (a) it has an off-switch and (b) pressing the off-switch will prevent the AI from achieving its other goals.
In turn, this knowledge is a special case of Big-picture strategic awareness, which might follow from learning many facts about many domains via Artificial General Intelligence.
The page on advanced agent properties starts to list out some of the different ways that an AI could be 'smart enough' in this sense, along with the particular problems that might be encountered with an AI that smart.