""turn the universe into pap..."

https://arbital.com/p/410

by Nebu Pookins Jun 4 2016


Since human beings don't have "utility functions" \(coherent preferences over probabilistic outcomes\), the notion of "rescuing the utility function" is itself a matter of rescue\. Natively, it's possible for psychology experiments to expose inconsistent preferences, but instead of throwing up our hands and saying "Well I guess nobody wants anything and we might as well turn the universe into paperclips\!", we try to back out some reasonably coherent preferences from the mess; which is, arguendo, normatively better than throwing up our hands and turning the universe into paperclips\.

"turn the universe into paperclips" is an in-reference and might not be suitable in a short article meant to introduce "rescuing the utility function" in isolation. At the very least, we should turn that into a link to an article on paper clip maximizers so that unfamiliar readers can know what the heck that sentence is supposed to mean. Alternatively, we could use a different example that doesn't rely on that background knowledge.