We're building AI genies
In many stories, genies often do exactly what we ask with unforeseen and sometimes unwanted results. In a Schneier on Security essay, Bruce Schneier discusses how AIs can act like genies and how they can deal with "underspecified" goals. Some extreme but instructional examples:
Ask an AI to deal with spam calls, and a Dionysus genie might change your phone number. Ask it to book a particular flight, and a Golem genie might hack the airline database because the flight is full. Ask it to get a refund for an online purchase, and a Dionysus genie might draft a legal threat on fake letterhead and send it to the company. Ask it to buy tickets to a popular concert, and a Golem genie might discover the ticketing system is rate-limited and spin up thousands of cloud servers to hammer it. You’d know if some of these things happened, but you might never realize if others did. More detailed prompting won’t fix this. The lesson of the genie is that no amount of detail is good enough.
But he's not a doomer, and suggests several ideas for trustworthy AI.