What is agent governance?

Summary slides

Why an agent needs governance beyond what a model or a single system already has

A model that only answers questions carries a bounded kind of risk: at worst it produces a wrong or misleading response, and a human downstream still decides whether to act on it. An agent is built specifically to close that gap, to take the next step itself, calling a tool, modifying a record, sending a message, initiating a transaction, without necessarily waiting for a human to approve each individual step along the way. That shift, from a system that only informs a decision to a system that can execute one, is exactly what makes governance for agents a distinct concern from governance for the underlying model powering them, since a well-governed model plugged directly into a poorly governed agent framework can still take consequential, and effectively irreversible actions that nobody explicitly, deliberately decided the agent should ever be allowed to take in the first place.

Guardrails and access control, covered as their disciplines elsewhere, constrain what an agent can technically reach at any moment, but agent governance sits one level above that: it’s the structure that decides which agents exist in the first place, what scope of autonomy each one has been approved for, and how that scope is reviewed and adjusted as the agent’s real-world track record accumulates. An organization can have excellent guardrails on every individual agent it runs and still have no meaningful agent governance at all, if nobody is tracking how many agents are running, what each one is really authorized to do, or whether any agent’s demonstrated behavior still matches the scope it was originally, formally approved under.

How agent governance defines what an agent is allowed to do

The starting point for governing any agent is a written definition of its intended scope, not a vague description of its general purpose, but the actual boundaries: which tools it can invoke, which systems and data it can reach, what categories of action it can take without a human in the loop, and what categories of action always require explicit human approval regardless of how routine they might otherwise seem. An agent whose scope is defined only in general terms, “helps with customer support” or “manages scheduling,” leaves enormous room for the agent’s actual behavior to drift into territory nobody explicitly considered when it was first approved.

This scope definition has to distinguish clearly between an agent’s reasoning and its actions, since an agent can reason about a very wide range of topics without that reasoning translating into a correspondingly wide range of permitted actions. An agent handling customer inquiries might reasonably need to reason about billing, account status, and product details to answer a question well, while its actual permitted actions might be limited to drafting a response for a human to send, or issuing a refund only below a dollar threshold, with anything above that threshold routed to a human regardless of how confident the agent’s reasoning happens to be. Governance that only constrains what an agent can talk about, without separately and explicitly constraining what it can do, misses the distinction that matters most for real-world risk.

How approval works for a new agent before it goes live

Before an agent is given any autonomy in production, it needs to pass through an approval process that specifically evaluates the agent as an autonomous actor, not merely as a wrapper around an already-approved model. This means testing the agent against realistic scenarios that carefully probe not just whether it performs its intended task correctly, but whether it recognizes the boundaries of its scope, whether it correctly, reliably escalates situations that fall outside what it was designed to handle on its own, and whether it degrades safely rather than improvising when it encounters a situation its designers didn’t anticipate.

The approval process also needs to specifically, deliberately examine the agent’s actual failure modes, not merely its success cases, since an agent that performs its intended task well in the overwhelming majority of situations can still cause serious harm through the comparatively rare cases where it fails, particularly and specifically when a failure involves the agent taking an action rather than merely producing a wrong answer someone downstream still has to evaluate. An agent that occasionally misclassifies a support ticket produces a comparatively minor problem; an agent that occasionally takes an unauthorized action because it misjudged a boundary case produces a materially more serious one, and the approval process has to weight these differently even when the underlying error rate looks similar.

A newly approved agent typically deserves a narrower initial scope than its long-term intended one, with autonomy expanding only gradually as its actual production behavior demonstrates it operating reliably within whatever boundaries it was first, deliberately approved under. Starting an agent directly at its eventual full scope on the theory that testing had already, sufficiently proved it trustworthy, skips the important step where production conditions, considerably messier and more varied than any test scenario could ever fully capture get a chance to reveal problems that testing simply never managed to surface on its own.

How ongoing oversight differs once an agent is operating autonomously

Approval is a single gate; oversight is what happens continuously afterward, and the two require different mechanisms. An agent approved for a scope of autonomy needs ongoing monitoring that specifically tracks whether its actual behavior in production continues to match that approved scope, catching both outright violations, the agent taking an action it was never authorized to take, and subtler drift, the agent increasingly handling edge cases on its own that were originally meant to be escalated, simply because escalation adds friction and the agent’s design quietly biases it toward resolving things independently when it can.

This oversight needs a concrete mechanism for a human to review a representative sample of an agent’s actual decisions and actions, not merely aggregate statistics about how often it succeeded or failed overall, since aggregate success rates can look reassuring even while individual decisions, examined closely, reveal the agent reasoning its way to a correct outcome through a process that would have produced a badly wrong outcome under only slightly different circumstances than the ones it happened to encounter. A support agent that closes tickets correctly nine times out of ten looks fine in aggregate, but reviewing the actual transcripts might reveal it’s succeeding for the wrong reasons in several of those cases, getting lucky rather than reasoning soundly, a pattern that aggregate metrics alone will never surface.

Oversight also needs a clear, low-friction path for the humans working alongside an agent day to day to flag a concern, since the people closest to an agent’s actual output, a support lead carefully reviewing its tickets, an operations manager closely watching its scheduling decisions, will often notice a problem well before it ever shows up in any formal metric, and a governance structure with no working channel for that frontline observation to reach the people who can act on it wastes the earliest and most useful warning signal available to it.

How to structure escalation so an agent knows when to stop and ask

An agent’s scope definition is only as good as its actual ability to recognize when a situation falls outside that scope, and this recognition deserves explicit design attention rather than being left as an implicit side effect of the agent’s general reasoning ability. An agent should be built with concrete triggers for escalation: a dollar amount above a threshold, a request involving a category of data or action explicitly marked as requiring human review, a level of uncertainty in the agent’s assessment that crosses a defined line, or simply a request that doesn’t clearly resemble any of the situations the agent was designed and tested to handle.

Escalation has to be an available option, not a theoretical one that adds so much friction the agent effectively never uses it. An escalation path that requires a slow, multi-step process while the agent’s default path forward is comparatively immediate action creates a structural incentive, even entirely without any explicit design choice deliberately pushing in that direction, for the agent’s actual behavior to gravitate steadily toward resolving things on its own rather than escalating, simply because escalation is consistently the harder, more effortful path and independent action is consistently the easier one. Making escalation at least as easy to trigger as independent action removes this quiet structural bias.

The design also has to account for an agent’s confidence being an unreliable signal on its own, since a model can express high confidence in a wrong conclusion, meaning escalation triggers built purely around the agent’s self-reported certainty will miss exactly the cases where the agent is confidently wrong rather than uncertainly wrong. Combining confidence-based triggers with independent, externally defined criteria situations that always, unconditionally escalate regardless of how certain the agent itself happens to claim it is, gives escalation a backstop that doesn’t depend entirely, or even primarily, on the agent accurately, correctly judging its reliability in the moment it matters most.

How agent governance scales as the number of agents grows

A single agent handling one well-defined task is comparatively easy to govern through direct, close oversight, but an organization running dozens or hundreds of agents across many different teams and functions can’t sustain that same close, individualized oversight for each one without the governance function itself becoming an organizational bottleneck. This is where agent governance has to build infrastructure: a central inventory of every agent currently operating, what each one is approved to do, who owns it, and when it was last reviewed, since without this inventory an organization can lose track of how many agents are running and what any one is authorized to do.

Scaling governance also means tiering the actual depth of oversight to the actual risk of what an agent does, rather than applying identical, undifferentiated scrutiny to an agent that merely drafts internal meeting summaries and an agent that can initiate financial transactions on the organization’s behalf. A risk-tiered approach lets the governance function concentrate its most intensive, human-heavy review on the agents whose failures would matter most while lighter-weight, more automated monitoring covers agents whose worst-case failure is low-consequence, preserving oversight capacity for where it matters most instead of spreading it thin and evenly across agents that don’t pose comparable risk.

As the actual number of agents grows, governance also needs a clear answer to what happens when two separate agents’ actions interact in ways neither one was individually, deliberately designed to anticipate, since an organization running many independently governed agents can still quite easily end up with emergent behavior at the level of the whole combined system that no single agent’s individual approval process was ever positioned to catch on its own.

How agent governance connects to access control and permission boundaries

Agent governance and access control are closely related but distinct: access control enforces what an agent can technically reach at the moment it tries to reach it, while governance decides what an agent should be allowed to attempt in the first place and reviews whether that decision continues to hold up over time. The two need to stay tightly coupled in practice, since an agent’s governed scope is only meaningful if the underlying permission system enforces it, and a permission system with no connection to governance can drift out of sync with whatever the governance process has decided an agent should be allowed to do.

This coupling matters most at the moment an agent’s scope changes, whether expanding after a successful review period or contracting after a concerning incident, since a governance decision to adjust an agent’s scope is only meaningful if it actually, promptly propagates into the technical permissions that agent operates under. A governance process that formally approves a scope reduction while the agent’s actual credentials remain unchanged for days or weeks afterward has, in practical terms, not reduced anything, regardless of what the governance documentation now says.

How multi-agent systems change what governance has to cover

An agent that operates entirely on its own is easier to reason about than a system where multiple agents coordinate, hand off work to each other, or operate on shared resources, since a multi-agent system introduces failure modes that don’t exist for any individual agent considered in isolation. One agent can take an action based on another agent’s output without a human ever reviewing the intermediate step, meaning an error introduced early in a chain of agent-to-agent handoffs can propagate and compound through several further steps before it ever surfaces somewhere a human is looking.

Governance for a multi-agent system needs explicit, deliberate attention paid directly to these handoff points, treating each place where one agent’s output becomes another agent’s input as a boundary worth its dedicated scrutiny, rather than simply assuming that because each individual agent was separately, independently approved, the composed system built from several of them working together automatically inherits the same safety properties without anyone checking. A multi-agent system also needs a clear answer to which agent, if any, has the authority to override or correct another agent’s output before it propagates further, since a system with several agents but no defined hierarchy or check between them can let a single early mistake move through the whole chain unchallenged.

How incident response works when an agent causes harm

Agent governance needs its explicit connection to incident response, distinct from how an organization handles a conventional software incident, because an agent incident often involves an action already taken rather than merely a system that behaved incorrectly, and the response has to account for whatever real-world consequence that action already produced, not just the technical bug that caused it. This means an agent incident response plan needs a fast well-rehearsed path to immediately suspend an agent’s autonomy, not merely to patch its underlying logic at some later point afterward, since the actual priority in the moment is stopping any further unauthorized action from occurring, with proper root-cause analysis and a fix following only once the immediate exposure has been contained.

A post-incident review for an agent failure has to trace not just the proximate technical cause but the actual, deeper governance question sitting quietly behind it: was this agent operating within its formally approved scope when the incident occurred, and if so, does the scope itself need to change, or if not why did the enforcement mechanism that was supposed to keep it within scope fail to do so. Treating every single agent incident purely as an engineering bug to be patched, without ever asking whether the underlying governance approval itself needs to be revisited, misses exactly the cases where the deeper lesson is that the agent should never have been approved for that particular scope of autonomy in the first place, regardless of how the immediate technical fix ultimately gets resolved.

How to keep agent governance from becoming stale as agents evolve

An agent’s underlying model, tools, and prompts change over time, often more frequently and less formally than the infrequent, deliberate review cycle that originally approved the agent’s scope, which means a governance approval granted against one version of an agent can quietly stop reflecting the agent as it exists in production. A meaningful change to an agent’s underlying model, its available tools, or its core instructions deserves to trigger a fresh governance review of its own rather than being treated as a routine, minor update that doesn’t require revisiting the original approval decision at all, since the version of the agent that was carefully tested and formally approved may no longer, in practice, be the actual version that’s running in production right now.

This connects agent governance directly to the same living-documentation discipline that model governance already depends on: an agent registry that’s updated as a natural, unavoidable part of the deployment workflow itself, rather than as a separate, easily-forgotten task someone has to actively remember to do stays trustworthy in a way that a registry only ever revisited during periodic audits eventually, inevitably stops being, and periodic audits still deserve a concrete place within the broader practice specifically to catch whatever drift the day-to-day workflow inevitably, quietly misses along the way.

How agent governance shapes whether the people working alongside an agent trust it

An agent’s governed scope and its escalation triggers aren’t purely internal control mechanisms, they directly shape whether the humans working alongside an agent day to day trust it enough to rely on it appropriately, and this trust matters in both directions: too little of it leads people to quietly re-check or override everything the agent does, eroding whatever efficiency the agent was meant to provide, while too much of it leads people to stop scrutinizing an agent’s output even in the cases governance specifically intended for human review to remain meaningful. Governance that’s invisible to the humans working with an agent, existing only as an internal policy document that nobody who touches the agent day to day has ever really seen, can’t meaningfully calibrate this trust in either direction, since people form their sense of how much to trust an agent from its observed behavior, not from a governance document they’ve never read and likely never will.

Making an agent’s actual scope and its escalation behavior visible to the people working alongside it, so a support lead reviewing an agent’s tickets understands what the agent is and isn’t authorized to decide on its own, gives that person a basis for calibrating their trust correctly rather than guessing. This visibility also creates a feedback loop back into governance itself, since the people closest to an agent’s actual output are best positioned to notice when its actual behavior no longer matches what governance says it should be doing, and a governance structure that keeps its decisions opaque to exactly the people best positioned to catch a problem early is quietly discarding one of its most valuable sources of ongoing signal.

Trust calibrated this way also has to account for how an agent’s reliability can vary considerably across different categories of the work it does, since a person who’s seen an agent handle routine requests flawlessly can reasonably extend that same confidence to a novel situation the agent has never been tested against, a form of trust transfer that governance needs to actively, deliberately guard against by making an agent’s actual tested scope, and the boundaries of what it’s been shown to reliably handle legible and visible, rather than simply assuming people will correctly, accurately infer those boundaries entirely on their own from ordinary, casual day-to-day observation alone.

How to handle accountability when an agent’s action was technically within scope but still wrong

Not every problematic agent action falls outside its approved scope, and this particular category deserves its dedicated governance attention precisely because it’s so easy to overlook entirely: an agent can take an action that was fully authorized under its governance approval and still be the wrong action for the situation, a case where the scope itself was reasonable in general but the agent’s judgment within that scope produced a bad outcome anyway. An agent authorized to issue refunds under a certain threshold might issue a technically permitted refund to a customer whose pattern of requests, visible in hindsight, should have prompted more scrutiny than the agent’s narrow, transaction-by-transaction view of its authority ever surfaced.

Governance that only asks whether an agent stayed within its approved boundaries misses this entire category of failure, since the honest answer in these particular cases is very often yes, the agent did exactly what it was authorized to do, and the problem lies instead in whether that authorization was too broad, too context-blind, or too disconnected from patterns that only become visible when someone looks across many of the agent’s decisions together, rather than at any single one considered entirely in isolation. This is part of why the periodic review of a sample of an agent’s actual decisions matters as much as monitoring for outright scope violations, since a pattern of individually authorized but collectively concerning decisions will never show up as a violation in any single transaction, only as a trend across many of them considered together.

Accountability in these particular cases also has to resist the temptation to treat the agent itself as the responsible party, since an agent has no accountability in any truly meaningful sense, it’s the governance structure, the scope that was approved, and the humans who approved and continue to oversee that scope, who bear responsibility for what an agent operating within its authorized boundaries goes on to do. A postmortem that simply concludes the agent made a bad call and stops right there, without ever asking whether the scope that made that call possible was itself the right scope to have approved, only treats the surface symptom while leaving the actual, underlying governance gap completely untouched.

How agent governance handles changes an agent makes to its working context

Some agents are designed to modify their working materials as part of normal operation, updating a memory store, adjusting a set of stored preferences, refining a prompt template based on what’s worked well in past interactions, and this category of self-modification introduces a governance question that a purely static agent never raises: whether the agent’s accumulated changes to its working context can, over time, shift its effective behavior well outside whatever scope was originally approved, even though nobody ever changed the agent’s underlying code, model, or explicit permissions.

An agent that gradually, continuously adjusts its stored guidance based on accumulated interactions can end up behaving quite differently after months of operation than it did on the actual day it was originally approved, not because anyone ever deliberately, explicitly expanded its scope, but because the agent’s incremental, quiet self-modification drifted its effective behavior there one small adjustment at a time, each one individually, reasonably defensible on its own, with no single moment anyone would have flagged as a scope change worth reviewing in isolation. Governance for agents with this kind of self-modifying capability needs a mechanism specifically, deliberately designed to catch this slow, quiet, cumulative drift, since neither the original approval process nor conventional monitoring of overt, external actions is well positioned to notice a change that happened entirely, quietly within the agent’s internal working state, invisible to anything watching only its outward behavior.

This typically means periodically, deliberately comparing an agent’s current, actual effective behavior directly against its originally approved baseline, rather than merely checking whether its explicit code or formal permissions have technically changed, treating the question of whether the agent still behaves like the agent that was originally approved as an entirely distinct check from whether its underlying configuration still technically matches what was formally deployed. An agent whose configuration hasn’t technically, formally changed at all can still have drifted meaningfully in its actual behavior, and governance built only around configuration and permission checks alone will entirely miss this failure mode every single time it occurs.

Common mistakes teams make around agent governance

A few patterns recur often enough across organizations building agentic systems that they’re worth naming directly, since recognizing them in advance is considerably easier than untangling them after an agent has already caused a problem in production.

1. Treating an agent’s initial approval as permanent rather than as a scope that needs ongoing re-justification as the agent’s production behavior accumulates.

2. Defining an agent’s scope only in terms of its general purpose rather than the tools, data, and categories of action it’s permitted to touch.

3. Constraining what an agent can reason about while leaving what it can do far less tightly constrained, missing the distinction between reasoning and action that matters most for risk.

4. Building an approval process that tests an agent’s success cases thoroughly while giving comparatively little attention to how it behaves in its actual failure modes.

5. Launching a newly approved agent directly at its eventual full intended scope rather than starting narrower and expanding as production behavior earns it.

6. Monitoring only aggregate success metrics rather than reviewing a sample of an agent’s actual individual decisions and actions.

7. Building an escalation path that’s slower or more effortful than the agent’s default path to independent action, creating a structural bias toward resolving things on its own.

8. Relying primarily on an agent’s self-reported confidence as the trigger for escalation, missing the cases where the agent is confidently wrong rather than uncertainly wrong.

9. Having no central inventory of which agents are running, what each one is approved for, and who owns it, leaving the organization unable to answer basic governance questions about its agent population.

10. Applying identical, undifferentiated oversight to every agent regardless of how consequential its worst-case failure is, spreading governance attention too thin to matter where it counts most.

11. Letting a governance decision to change an agent’s scope lag behind the actual technical permissions that agent operates under, leaving a documented decision that isn’t enforced.

12. Approving agents individually in a multi-agent system without ever examining the handoff points between them, missing failure modes that only exist at the level of the composed system.

13. Treating an agent incident purely as a technical bug to patch without asking whether the underlying governance approval itself needs to be revisited.

14. Letting an agent’s underlying model, tools, or instructions change without triggering a fresh governance review, leaving the approved version and the running version quietly out of sync.

15. Maintaining an agent registry that’s only updated during occasional audits rather than as a natural part of the ongoing deployment workflow, letting it drift out of date exactly when it matters most.

16. Concluding, after an agent takes a harmful but technically authorized action, that the agent alone made a bad call, without asking whether the underlying scope that made the action possible was ever the right scope to approve in the first place.

17. Overlooking slow, cumulative drift in a self-modifying agent’s effective behavior because neither its underlying code nor its formal permissions ever technically changed, even as its actual day-to-day behavior moved well outside its originally approved baseline.

18. Keeping an agent’s actual governed scope invisible to the people working alongside it every day, leaving them to calibrate their trust in the agent purely from casual observation rather than from any understanding of what it’s authorized and tested to handle.

What connects all eighteen of these mistakes is a single underlying pattern: treating an agent’s autonomy as something decided once at approval time rather than as something that has to be continuously, actively re-earned against its actual, observed behavior in production. An agent is fundamentally different from a static system precisely because its behavior can shift, through drift in how it handles edge cases, through changes to its underlying model, through the accumulation of small decisions that individually look reasonable but collectively, quietly move it outside the scope anyone deliberately intended, and governance that isn’t built around that ongoing re-justification will eventually, inevitably end up governing the agent it approved months ago rather than the agent that’s running today.

The deeper principle underneath all of this is that autonomy and oversight aren’t opposing forces to be traded off against each other, but rather that durable autonomy for an agent is only ever earned through the continuous, ongoing presence of oversight, since an agent operating without any meaningful check on whether its actual behavior still matches its approved scope isn’t demonstrating trustworthy autonomy at all, it’s simply operating unobserved, and an organization that mistakes the mere absence of visible problems for the presence of reliability is building its entire confidence on a foundation it has never properly verified and may never get the chance to verify before that gap eventually matters.