
Deemerwha studio // Shutterstock
Last quarter, the VP of engineering at a large gaming company explained to Oso how they’d built an AI site reliability engineering agent to help resolve incidents and fix production issues. For weeks, it worked beautifully—triaging alerts, identifying root causes, even suggesting fixes their team would have taken hours to develop.
Then one day, it launched a denial-of-service attack on their internal monitoring system.
The agent had permissions to query their monitoring APIs. It was supposed to use them to gather context for incident response. But when it decided those APIs might hold the answer to a particularly thorny issue, it started hammering them with requests until the system fell over.
They shut the agent down, obviously. But unplugging the agent is a blunt instrument—it means losing all the goodness they were getting before.
An agent is a system. To secure any system, you need the right mental model to reason about it. The industry doesn’t have that mental model for agents yet, and that’s a problem.
Without a shared mental model of what an agent is, people can’t decompose it. And if it can’t be decomposed, security can’t be designed around it. The disasters make headlines. More commonly, though, concerns about agent security are leading to agents so locked down they can barely do anything.
Nondeterminism is both the promise and the peril of agents. An AI agent behaves in nondeterministic ways because it is given the agency to determine how it executes tasks. You can’t remove that autonomy without gutting the agent—but you can mitigate the risks. The most fundamental control is permissions.
Consider Wade Foster’s sharp post on the “AI Automation Spectrum” and prior work by Anthropic, Tines, and Simon Willison. If you can categorize what kind of system you’re building, you can reason about what could go wrong and how to prevent it. Many organizations want to move from left to right on a spectrum of autonomy, but most are stuck because they can’t reason about what agents might do. Taxonomy can be a diagnostic tool. Know what’s nondeterministic, and you’ll know where the risk is and what controls to apply.
Agent Taxonomy
Imagine you’re a retailer. When you get customer feedback, you want to ask happy customers to leave reviews and fix issues for unhappy ones, but you want to automate this. You could build a straightforward automated workflow, or you could move from left to right on this spectrum of autonomy.
Read more