What is AI governance?

Summary slides

Why AI governance is broader than any single practice underneath it

Model governance decides which models an organization formally approves and continuously monitors over time. Agent governance decides what autonomous systems are allowed to do in practice. Access control enforces what a system can technically reach at any moment. Responsible AI practice builds accountability and fairness directly into how a system is designed and used day to day. Each of these individual disciplines solves a real problem in its own right, but none of them, even taken all together, automatically produces a coherent answer to the much broader question an organization eventually, inevitably has to face: across everything the organization is doing with AI, spanning many separate systems, many separate teams, and many separate models, is the organization’s overall, aggregate AI use aligned with its stated values, its actual legal obligations, and its honest tolerance for risk.

AI governance exists specifically and deliberately to answer that broader question, providing the actual organizational structure, the policy, and the decision rights that let leadership understand and actively steer the organization’s full AI footprint, rather than having that footprint exist only as the sum of many separate, individually reasonable decisions made by individual teams working in isolation with no single person or body ever holding the complete picture. An organization can have excellent model governance, excellent agent governance, and excellent access control implemented independently within several different teams, and still quite easily, have no AI governance at all, if nobody at the actual organizational level is tracking what all of this work adds up to, whether it’s internally consistent across every team, and whether it satisfies obligations that exist at the level of the whole organization rather than any single, isolated system within it.

How AI governance starts with a clear inventory of where AI is used

An organization can’t govern what it doesn’t know exists, and the starting point for AI governance is a maintained inventory of every system across the organization that uses AI in some meaningful way, what it’s used for, what data it touches, and who owns it. This sounds like an obvious first step, but it’s one many organizations quietly skip, in part because AI adoption inside a large organization often happens gradually and quite unevenly across many teams, with individual teams adopting a tool or building a system without any centralized process ever formally recording that adoption anywhere, leaving leadership with an incomplete, and often steadily outdated picture of how extensively AI is being used across the organization as a whole.

Building this inventory requires considerably more than a single, one-time survey, since new AI use continues to appear well after the inventory is first built, whether through a team adopting a new vendor tool, a developer building a small internal system that quietly starts relying on a model without much fanfare, or an existing system gradually, incrementally expanding its use of AI well beyond its originally intended, approved scope. An inventory needs an ongoing process for capturing new AI use as it appears, ideally built into procurement and development workflows themselves, so that adopting a new AI capability naturally triggers an inventory update rather than depending on someone remembering to report it separately after the fact.

The inventory itself also needs to carefully distinguish between meaningfully different categories of AI use, since a simple chatbot answering general customer questions and a system making automated decisions about a person’s credit or employment carry very different governance implications, and an inventory that simply, lazily lists “uses AI” without capturing what the AI specifically does and what it’s concretely deciding provides far too little detail for governance to act on in any meaningful way.

How AI governance connects to legal and regulatory compliance specifically

AI governance and legal compliance overlap substantially but aren’t identical, and treating them as the same thing tends to leave gaps in both directions. Compliance asks whether the organization is meeting its actual legal obligations, obligations that vary considerably by jurisdiction and by industry and that continue to evolve as new regulation specifically targeting AI use comes into force. Governance is broader, encompassing not just what’s legally required but what the organization has decided it wants to hold itself to regardless of whether a law currently requires it, since regulation in this area tends to lag meaningfully, considerably behind the actual pace of AI adoption across industries, and an organization that governs itself only to the strict letter of current law will likely find itself exposed to risks that regulation hasn’t caught up to yet and may not for quite some time to come.

This connection matters in practice because an AI governance function needs a working relationship with legal counsel, not merely treating legal review as a separate, disconnected gate an AI system technically passes through once before launch, but building ongoing legal awareness directly into the governance process itself, so that a new regulation, a new enforcement action taken against another organization in a similar industry, or a new interpretation of an existing law reaches the people making governance decisions in something close to real time, rather than being discovered only once a system is already deep into active development or, considerably worse, already live in production.

Different, distinct categories of AI use also carry very, meaningfully different regulatory exposure depending on context, and governance needs to reflect that variation rather than simply, lazily applying a single, uniform compliance checklist across every different situation the organization encounters. A system making decisions that materially affect a person, employment, credit, insurance, healthcare, typically sits under considerably more regulatory scrutiny than a comparatively low-stakes internal productivity tool, and an organization’s governance structure needs the same kind of risk-tiering here that model governance already applies to its model approval process, concentrating the most careful legal and compliance review specifically and deliberately on exactly the uses that carry the most legal exposure for the organization as a whole.

How to structure AI governance roles and decision authority

AI governance needs assigned decision authority at the organizational level, not merely a set of principles everyone is generally expected to follow. This typically means a defined governance body, whether a formal committee or a smaller group with clear organizational standing, that holds authority to set organization-wide AI policy review and approve categories of high-risk AI use, and intervene when a system’s use of AI has drifted outside what’s truly acceptable. Without this authority in place, AI governance tends to exist only on paper, as a set of stated values with no actual mechanism forcing any team’s decisions to align with them in day-to-day practice, a gap that tends to widen quietly over time until something eventually forces it back into view.

This governance body needs honest representation from well beyond engineering alone, since the risks AI governance is meant to manage, legal exposure, reputational risk, fairness and equity concerns, operational risk span far more of the organization than merely the teams building AI systems themselves, and a governance structure staffed only by the people building the systems tends to systematically underweight exactly the risks those particular people simply aren’t specifically positioned to see clearly from where they sit. Legal, compliance, security, and the actual business functions using AI in their day-to-day work all have a stake in how governance decisions get made, and excluding any single one of them from the actual decision-making structure leaves governance blind to whatever that excluded perspective would have otherwise caught before it became a problem.

The authority this body holds also needs teeth behind it, in the exact same way that model governance and agent governance both need assigned accountability rather than merely diffuse, unaccountable committee ownership: an answer to who can concretely block a high-risk AI system from launching in the first place, who can order an existing system’s AI use restricted once a problem surfaces, and who’s accountable when a governance decision turns out, in hindsight, to have honestly been wrong. A governance body holding responsibility for AI risk across the whole organization but with no actual power to act on what it finds provides oversight only in name, never in any substantive sense that matters when a problem eventually surfaces.

How AI governance classifies risk across very different AI use cases

Not every AI system an organization builds or adopts carries comparable risk, and treating a low-stakes internal tool with the same governance intensity as a system making consequential decisions about people wastes governance attention where it matters least while potentially under-scrutinizing where it matters most. A risk classification framework, sorting AI use into distinct tiers based on factors like whether a system makes or materially influences a decision about a person, what harm a failure could realistically cause, and how much human oversight sits between the system’s output and any real-world consequence that follows from it, lets governance allocate its most intensive review specifically to the uses that warrant it most.

This classification has to be applied consistently and fairly across the whole organization rather than simply left entirely to each individual team’s judgment about how much scrutiny its system deserves, since a team building a system tends to have both limited visibility into how that system’s risk compares to others across the whole organization and, quite often, a natural, understandable incentive to classify its work as lower-risk than a fully objective, honest assessment might conclude, simply because a lower-risk classification typically means a considerably faster less burdensome path all the way to actual launch, with far less scrutiny along the way. An organization-wide centralized classification framework, applied consistently by people without that exact same structural incentive, produces considerably more consistent and considerably more trustworthy risk tiering than simply leaving each individual team to grade its homework entirely on its own terms.

Risk classification also isn’t a one-time judgment made only at a system’s initial launch, since a system’s actual risk can shift meaningfully over real time as its scope quietly expands, as it’s applied to new use cases nobody anticipated when it first, originally launched, or as the population it affects grows in ways that change the aggregate consequence of even a comparatively low individual failure rate. Governance needs a working mechanism for reclassifying a system’s risk tier as its actual use evolves over time, rather than permanently, rigidly anchoring itself to whatever tier it was originally assigned back when governance first reviewed it and nothing more recent has ever been checked.

How AI governance intersects with an organization’s broader data governance

AI systems are fundamentally built on data, and an organization’s AI governance can’t be meaningfully separated from its broader data governance practice, since decisions about what data an AI system can access, how that data was originally collected, and what consent or legal basis covers its use directly determine much of the actual risk an AI system carries. A model trained or actively operating on data originally collected for one stated purpose but now being used for a meaningfully different one entirely raises a data governance question that AI governance simply can’t resolve entirely on its own, and the two practices need a working connection between them, rather than operating as separate, disconnected reviews that each, mistakenly, assume the other has already fully handled the relevant data-related concerns.

This intersection matters most concretely and practically at the exact point where a new AI system is first proposed, since the underlying data governance question, is this data available for this use needs answering before the separate AI governance question, is this use of AI itself appropriate, can be meaningfully, honestly evaluated at all in any sense. An AI governance review that formally approves a system’s intended use without confirming the underlying data governance supports that use is building its entire approval on an assumption that nobody has ever verified in the first place, leaving the approval resting on nothing more solid than an unexamined guess.

How to build AI governance policy that people follow

Governance policy that exists only as a lengthy document few people have read tends to get quietly routed around by teams under delivery pressure, not necessarily out of any deliberate disregard for governance, but because a policy that’s hard to find, hard to understand, or badly matched to how teams work naturally gets treated as a formality to satisfy after the fact rather than a part of how work gets done. Governance that gets followed tends to be built directly into the workflows teams already, actively use, a governance checkpoint embedded within the existing project approval process itself rather than a separate, standalone process nobody remembers exists until it’s already too late to matter, and written in plain, concrete language that practically helps a team make a good decision rather than merely stating an abstract principle with no concrete guidance about what it means for the situation in front of them.

This also means AI governance has to be resourced adequately, since even a well-designed, well-embedded policy fails if the governance function itself lacks the actual capacity to review the volume of AI use an organization is generating, leaving teams waiting weeks for a review that was supposed to be a lightweight checkpoint, which in turn creates exactly the pressure that pushes people to route around the process entirely. A governance function whose review capacity hasn’t scaled alongside the organization’s actual AI adoption becomes, through simple, entirely predictable bottleneck pressure alone, a structural incentive for exactly the behavior it was originally, deliberately built to actively prevent from happening at all.

How AI governance scales as AI use spreads across more of the organization

AI governance for an organization with a handful of AI systems, built and maintained by one team, looks very different from AI governance for a large organization where dozens of teams are independently adopting AI tools and building AI-powered systems, and a governance structure designed for the former tends to buckle under the volume and diversity of the latter if it isn’t deliberately redesigned as adoption spreads. Scaling AI governance means building the same kind of tiered, risk-proportionate review that model governance and agent governance both already depend on, reserving the governance body’s direct attention specifically for high-risk cases while providing considerably lighter-weight, more self-service tools, clear practical guidelines, pre-approved reusable patterns, automated risk-classification questionnaires, that let teams handle their low-risk AI adoption without needing to route every single decision through a central bottleneck.

Scaling also means investing considerably in AI literacy across the whole organization, not merely within the governance function itself, since a governance structure that depends entirely on a small, central team catching every single problem simply doesn’t scale to an organization where AI adoption is happening in dozens of separate places simultaneously, while an organization where the people building and using AI systems understand the basic risk categories governance cares about catches many potential problems considerably earlier, closer to where they originate, well before they ever even need to reach central governance at all.

How AI governance connects to external stakeholders beyond the organization itself

AI governance ultimately has to account for people outside the organization who are affected by its AI use, customers, job applicants, patients, anyone whose experience or outcomes are shaped by a system they had no direct part in building or approving, and this external accountability needs its explicit place within governance rather than being treated as something that’s automatically covered once internal risk and compliance review is complete. This typically means building actual channels for external concerns to reach the governance function directly, a way for someone affected by an automated decision to understand that AI was involved and to meaningfully contest or appeal an outcome they honestly believe was wrong, since an organization whose AI governance only ever hears from people already inside the organization is missing exactly the perspective of the actual people its AI systems most directly affect out in the real world.

This external dimension also shapes how transparent an organization chooses to be about its AI use, and while the level of disclosure appropriate varies by context and by what a use case warrants, governance benefits from treating transparency as a deliberate choice made in light of what stakeholders reasonably need to know, rather than as a question answered by default through whatever minimal disclosure legal requirements happen to specify, since trust built through honest transparency tends to prove considerably more durable over the long run than trust that exists only precariously, because a problem simply hasn’t yet become publicly, visibly known.

How to measure whether AI governance is working

A governance structure that exists on paper but has no way of measuring its effectiveness can’t distinguish between managing risk well and simply not yet having encountered the failure that would reveal a gap. Meaningful measurement includes tracking how many AI systems are going through governance review relative to the full inventory of AI use the organization honestly believes exists, since a large and steadily growing gap between the two suggests governance coverage that looks comprehensive on paper but is quietly missing a meaningful share of the organization’s AI footprint. It also includes actively tracking how governance decisions age over real time, how quickly a flagged concern gets properly resolved, and how often a system operates for an extended period without its required periodic review ever happening on schedule, since a governance process that formally requires ongoing review but rarely delivers it when scheduled provides considerably less protection than its written documentation would otherwise suggest to an outside observer.

Honest measurement also has to look considerably beyond mere process metrics to actual outcomes wherever possible, actively tracking incidents connected to AI use across the whole organization, whether an incident happened inside a system that had gone through proper governance review beforehand, and whether the review process itself would plausibly have caught the problem that ultimately occurred in practice. An incident occurring inside a properly reviewed system suggests the review process itself has a gap worth investigating further, while an incident occurring inside a system that had skipped review entirely points instead to a coverage problem rather than a review-quality problem, and honestly distinguishing between these two meaningfully different diagnoses matters considerably for deciding what needs to change going forward.

How AI governance handles tools adopted outside any formal build process

A meaningful share of an organization’s actual AI use often doesn’t come from systems the organization deliberately designed and built, but from individual employees or teams adopting a third-party AI tool directly, a writing assistant, a coding assistant, a general-purpose chat interface, without that adoption ever passing through anything resembling a formal review process at all. This particular category of use, often called shadow AI when it happens entirely without governance’s knowledge, poses a distinct challenge because the inventory-building and risk-classification practices described earlier all assume governance already knows a system exists in the first place, an assumption this category of use directly, fundamentally violates by its very nature.

Addressing this well requires governance to actively treat its visibility gap as a problem to deliberately solve, rather than an unfortunate but supposedly unavoidable side effect of how large organizations work, building lightweight accessible channels for employees to honestly disclose the tools they’re using, making that disclosure easy enough that it doesn’t create the exact same friction that already pushes formal AI adoption to quietly route around governance in the first place. Pairing this approach with reasonable default guidance about what data is and isn’t appropriate to share with an external AI tool, rather than relying solely on after-the-fact, reactive discovery once something has already gone wrong, gives governance a chance to actively shape this entire category of use proactively and thoughtfully, instead of merely reacting only once a problem has already, quietly surfaced somewhere the organization wasn’t watching.

This particular category of use also benefits from a deliberately, meaningfully different governance posture than formally built systems typically receive, since the actual goal with shadow AI usually isn’t to eliminate it entirely, employees adopting these tools are often honestly solving a legitimate problem for themselves in their work, but rather to bring it into a lightweight governance conversation that can flag risky use, sensitive data being shared with an external tool, or a tool being used for a decision it was never appropriate for in the first place, without treating every individual employee’s ordinary use of a general productivity tool as requiring the exact same intensity of review as a system the organization is deliberately, formally building and deploying to make consequential decisions about other people.

How AI governance connects to training and organizational change management

Written policy alone doesn’t change how people behave, and AI governance depends on an investment in helping the people across the organization who are building and using AI systems understand not just what the policy says but why it exists and what risk it’s meant to address. Training that only recites written policy in the abstract, general sense tends to produce compliance in name only people who can recite the rule perfectly without ever understanding when it applies to the messy situation in front of them in the moment, while training grounded in concrete, realistic examples of what governance is deliberately trying to prevent gives people the judgment needed to apply the underlying principle correctly even in novel situations the written policy never explicitly, specifically anticipated in advance.

This training also needs to reach considerably beyond the people directly building AI systems to the much broader population making decisions informed by AI output day to day, since a manager using an AI-generated recommendation to inform a decision about a person carries responsibility for understanding that system’s limitations, even though that same manager never wrote a single line of code or ever reviewed even one model evaluation themselves. Governance that invests training entirely, exclusively in technical teams while leaving downstream decision-makers without any understanding of what the actual AI systems they’re relying on can and can’t reliably do leaves a gap open precisely, exactly where AI output translates into a real-world consequence for someone, which is exactly the point in the whole chain where that gap matters most.

Common mistakes organizations make around AI governance

A number of patterns show up often enough across organizations building out AI governance that naming them directly is more useful than leaving them to be discovered the hard way.

1. Building model governance, agent governance, and access control independently within different teams while never establishing any organization-wide structure that ties them together into a coherent whole.

2. Never building a maintained inventory of where AI is used across the organization, leaving leadership with an incomplete and steadily aging picture of the organization’s true AI footprint.

3. Treating legal compliance and AI governance as identical, either assuming compliance alone is sufficient or building governance principles disconnected from what the organization is legally required to do.

4. Establishing a governance body with responsibility for AI risk but no assigned authority to block, restrict, or reshape a system’s use of AI.

5. Staffing governance decisions almost entirely with the engineering teams building AI systems, leaving legal, compliance, and the affected business functions without representation in the actual decisions.

6. Applying identical governance scrutiny to every AI use case regardless of actual risk, exhausting governance capacity on low-stakes systems while under-reviewing the cases that matter most.

7. Letting individual teams classify their systems’ risk level, creating a structural incentive toward underclassification that an objective, centralized assessment would avoid.

8. Treating risk classification as a one-time judgment made at launch rather than something that needs revisiting as a system’s actual scope and use evolve over time.

9. Reviewing AI systems for appropriateness of use without first confirming the underlying data governance supports that use.

10. Writing governance policy as a lengthy document disconnected from the workflows teams use, virtually guaranteeing it gets routed around under delivery pressure.

11. Failing to scale the governance function’s review capacity alongside the organization’s actual AI adoption, creating bottleneck pressure that pushes teams to bypass review entirely.

12. Building no channel for people outside the organization to raise concerns about or contest an AI-driven outcome that affected them.

13. Treating transparency about AI use as a minimal legal-compliance question rather than a deliberate choice made in light of what stakeholders need to know.

14. Having no way to measure whether governance is catching problems, leaving the organization unable to distinguish risk management from simply not yet having encountered the failure that would expose a gap.

15. Diagnosing every AI incident as a review-quality problem without first checking whether the system involved had gone through governance review at all.

16. Ignoring or failing to actively surface shadow AI use, tools individual employees or teams have adopted entirely on their own, leaving a meaningful share of the organization’s AI footprint completely outside governance’s visibility.

17. Investing AI governance training almost entirely in the technical teams building systems while leaving the much broader population of downstream decision-makers, the people relying on AI output to make decisions, without any understanding of what those systems can and can’t reliably do.

What connects all seventeen of these mistakes is a single underlying pattern: treating AI governance as a document or a principle rather than as a living, organization-wide practice that has to be continuously resourced, enforced, and honestly measured against outcomes to mean anything at all. An organization’s stated AI principles are only ever as good as the actual structure that connects them to the decisions being made across every team building or adopting AI, and governance that exists mainly as a statement of values, entirely disconnected from authority, capacity, and measurement, provides only the comfort of having addressed AI risk without the actual substance of having done so.

The deeper principle beneath all of this is that AI governance is ultimately about maintaining an accurate organization-wide picture of where accountability sits for a technology whose behavior can be difficult to predict in advance and whose consequences often land on people well outside the team that built the system in the first place, and an organization that can’t answer, clearly and specifically, who’s accountable for any piece of its AI use hasn’t truly achieved governance in any meaningful sense, no matter how detailed and thorough its written policies happen to look on paper to an outside observer.