Value alignment problem

https://arbital.com/p/value_alignment_problem

by Eliezer Yudkowsky May 15 2015 updated Feb 2 2017

You want to build an advanced AI with the right values... but how?


Disambiguation: For the research subject that includes the entire edifice of how and why to produce good AIs, see AI alignment.

[summary: The 'value alignment problem' is to produce sufficiently advanced machine intelligences that want to do beneficial things and not do harmful things. The largest-looming subproblem is 'value identification' or 'value learning' (sometimes considered synonymous with value alignment) but this also includes subproblems like Corrigibility, that is, AI values such that it doesn't want to interfere with you correcting what you see as an error in its code.]

The 'value alignment problem' is to produce sufficiently advanced machine intelligences that want to do beneficial things and not do harmful things. The largest-looming subproblem is 'value identification' or 'value learning' (sometimes considered synonymous with value alignment) but this also includes subproblems like Corrigibility, that is, AI values such that it doesn't want to interfere with you correcting what you see as an error in its code.