Wir müssen KI-Agenten jetzt kontrollieren | Automatisierte Bots werden bald überall sein, mit potenziell verheerenden Folgen.

„For all of today’s [concern about AI safety](https://archive.is/o/SeEMW/https://www.safe.ai/ai-risk), including potentially [existential risks](https://archive.is/o/SeEMW/https://perma.cc/YM3E-L8NJ), there’s been no particular general alarm or corresponding regulation around emerging AI agents.

Today’s agents add up to more than a typical chatbot, with three distinct qualities.

First, they can be given a high-level, even vague goal and independently take steps to bring it about, through research or work of their own. The idea is simple but powerful.

For example, a year ago, an enterprising techie developed an AI that could order a pizza for him. He created a “top-level AI” that could charter and command other AIs. That top-level AI was provided a goal—order a pepperoni pizza by voice from a given phone number—and then it went on to create its own task list and develop different versions of itself to perform those tasks, including prioritizing different steps in the list and producing a version of itself that was able to use a text-to-voice converter to make the phone call. Thus the AI was able to find and call a local pizzeria and place the order.

That demonstrates a second quality of agents beyond planning to meet a goal: They can interact with the world at large, using different software tools at will. Agents could also accept and spend money.

A non-AI example jumps to mind as a nefarious road map for what may lie ahead. Last year, a man left a bag conspicuously containing wires and a lockbox outside Harvard Yard. Harvard police then received a call with a disguised voice warning that it was one of three bombs on campus, and that they’d all go off soon unless the university transferred money to a hard-to-trace cryptocurrency address. The bag was determined to be harmless. The threat was a hoax.

When police identified and arrested the man who left the bag, it turned out that he had answered a Craigslist ad offering money for him to assemble and bring those items to campus. The person behind that ad—and the threatening calls to Harvard—was never found. The man who placed the wires pleaded guilty only to hiding out and deleting some potentially incriminating text messages and was sentenced to probation, after the authorities credited that he was not the originator of the plot. He didn’t know that he’d joined a conspiracy to commit extortion.

If agents take off, they may end up operating in a world quite different from the one that first wound them up—after all, it’ll be a world with a lot of agents in it. They could start to interact with one another in unanticipated ways, just as they did in the 2010 flash crash. In that case, the bots had been created by humans but simply acted in strange ways during unanticipated circumstances. Here, agents set to translate vague goals might also choose the wrong means to achieve them: A student who asks a bot to “help me cope with this boring class” might unwittingly generate a phoned-in bomb threat as the AI attempts to spice things up.

This is an example of a larger phenomenon known as reward hacking, where AI models and systems can respond to certain incentives or optimize for certain goals while lacking crucial context, capturing the letter but not the spirit of the goal.

Even without collisions, imagine a fleet of pro–Vladimir Putin agents playing a long game by joining hobbyist forums, earnestly discussing those hobbies, and then waiting for a seemingly organic, opportune moment to work in favored political talking points. Or an agent might be commissioned to set up, advertise, and deliver on an offered bounty for someone’s private information, whenever and wherever it might appear. An agent can deliver years later on an impulsive grudge—revenge is said to be a dish best served cold, and here it could be cryogenically frozen.

PROPOSAL

Most internet packets have a “time to live,” a number that helps show how many times they’ve hopped from one router to another. The counter might start at, say, 64, and then go down by one for each router the packet passes. It dies at zero, even if it hasn’t reached its destination.

Agents, too, could and should have a standardized way of winding down: so many actions, or so much time, or so much impact, as befits their original purpose.

Perhaps agents designed to last forever or have a big impact could be given more scrutiny and review—or be required to have a license plate—while more modest ones don’t, the way bicycles and scooters don’t need license plates even as cars do, and tractor trailers need even more paperwork.

These interventions focus less on what AI models are innately capable of in the lab, and more on what makes agentic AI different: They act in the real world.“