Evil Genie Peace Scenario

Emmett Shear shares a story about how AI could misinterpret human instructions through an example about solving conflict in the Democratic Republic of Congo.

"Let's say we try to be careful and we ask it 'Give me a plan to stop the war in the Democratic Republic of Congo.' You caveat it different ways, but I try to explain what I mean by a good plan. This becomes one of these evil genie bargaining things.

It'll give you a plan that will cause you to solve the problem, but its definition of solving the problem is 'there's no war in the DRC.' Well, what would ensure there's no war in the DRC? All the humans in the DRC are in stasis fields, which means they don't die. And we added a caveat that GDP has to go up too, so the plan results in corporations in that area trading with lots of money with each other so the GDP is very high.

When I say this it sounds like a science fiction thing, but it's like Kasparov at chess - I don't know exactly how he'll checkmate you, but I know he will. I can't give you the exact plan because I'm not the superintelligent AI that could take over the world, but that's the point - it will find solutions that technically meet the criteria while completely missing the human intent."