[summary: With respect to the theory of constructing sufficiently advanced AIs in ways that yield good outcomes:

• Direction is when it's okay for the AI to compute plans, because the AI will end up choosing rightly;
• Limitation is shaping an insufficiently-aligned AI so that it doesn't run a computation, if we expect that computation to produce bad results;
• Opposition is when we try to prevent the AI from successfully doing something we don't like, assuming the AI would act wrongly given the power to do so.

For example:

• A successfully directed AI, given full Internet access, will do beneficial things given that Internet access;
• A limited AI, suddenly given an Internet feed, will not do anything with that Internet access, because its programmers haven't [whitelisting whitelisted] this new domain as okay to think about;
• Opposition is airgapping the AI from the Internet and then putting the AI's processors inside a Faraday cage, in the hope that even if the AI wants to get to the Internet, the AI won't be able to, say, produce GSM cellphone signals by modulating its memory accesses.]

'Directing' versus 'limiting' versus 'opposing' is a proposed conceptual distinction between 3 ways of getting good outcomes and avoiding bad outcomes, when running a sufficiently advanced Artificial Intelligence:

• Direction means the AGI wants to do the right thing in a domain;
• Limitation is the AGI not thinking or acting in places where it's not aligned;
• Opposition is when we try to prevent the AGI from successfully doing the wrong thing, assuming that it would act wrongly given the power to do so.

For example:

• A successfully directed AI, given full Internet access, will do beneficial things rather than detrimental things using Internet access, because it wants to do good and understands sufficiently well which actions have good or bad outcomes;
• A limited AI, suddenly given an Internet feed, will not do anything with that Internet access, because its programmers haven't [whitelisting whitelisted] this new domain for being okay to think about;
• Opposition is airgapping the AI from the Internet and then putting the AI's processors inside a Faraday cage, in the hope that even if the AI wants to get to the Internet, the AI won't be able to produce GSM cellphone signals by modulating its memory accesses.

A fourth category not reducible to the other three might be stabilizing, e.g. numerical stability of floating-point algorithms, not having memory leaks in the code, etcetera. These are issues that a sufficiently advanced AI would fix in itself automatically, but an insufficiently advanced AI might not, which causes problems either if early errors introduce changes that are reflectively stable later, or if we are intending to run the AI in insufficiently-advanced mode.

Not running adversarial searches

Under the [agents_as_searches] view, we can see an agent as embodying computations that search out actions within a domain. The non-adversarial principle then says that we should not be running a search which is looking for ways to hurt us.

We can achieve this desideratum in one of two ways:

• The AI is sufficiently nice / value-aligned that it's okay for the AI to think about this domain / run searches here; whatever the AI is searching for, we don't mind if that search succeeds.
• The AI is not aligned in this domain and therefore should not be running searches here--we didn't finish the alignment here, and therefore, we shouldn't be running this computation.

If both of these measures fail, then as a [strictly fallback second line of defense], we could try to make a bad search return a null result (the AI doesn't find any way to do the harmful thing).

For example: The non-adversarial principle implies that if you build a shutdown button for the AI that suspends the AI to disk when pressed, you must also ensure:

• That the AI wants there to be a shutdown button;
• That the AI wants the state of this shutdown switch to reflect the dynamic decisions of the human operators;
• That the AI does not want to influence the operators to press the switch or not press it;
• That the AI does not want to cause anything besides an orderly suspend-to-disk to happen after this switch is pressed.

Or:

• That the AI doesn't want to think about the shutdown button at all (e.g. because it's not inside a [whitelisting whitelisted cognitive domain]).

The first set of measures would be 'direction'--if those measures work, it's okay for the AI to think here because we think those thoughts are pointed in the right direction. The second approach would be limitation--we don't think the AI's computations are aligned inside this domain yet, so until then, we shouldn't be running computations there.

As a final, strictly fallback line of defense, you could put a big red lever in the computing center that shut off all the electricity even if the AI was failing to suspend to disk. (But you're not relying on this working; it's just there as a maybe-it-might-work measure that might buy you a little more probable safety even if the primary lines of defense failed.)

Relation to other non-adversarial ideas

The direction/limitation/opposition distinction can help state other ideas from the AI safety mindset. For example:

The principle niceness is the first line of defense can be rephrased as follows: When designing an AGI, we should imagine that all 'oppositional' measures are absent or failed, and think only about 'direction' and 'limitation'. Any oppositional measures are then added on top of that, just in case.

Similarly, the Omnipotence test for AI safety says that when thinking through our primary design for alignment, we should think as if the AGI just will get Internet access on some random Tuesday. This says that we should design an AGI that is limited by [whitelisting not wanting to act in newly opened domains without some programmer action], rather than relying on the AI to be unable to reach the Internet until we've finished aligning it.