What is AI native security?

Summary slides

Why AI native security differs from applying conventional security practices to AI systems

Conventional application security rests on assumptions that have held reasonably well for many decades now: a system’s actual behavior is fully, entirely determined by its code, that code can be read, carefully reviewed, and reasoned about directly by a person, and a vulnerability, once found, can usually be patched by changing a clearly identifiable piece of that same code. An AI system built around a language model breaks each of these assumptions in a structural way. The model’s actual behavior truly emerges from training on vast, enormous quantities of data, not from code a security reviewer can simply, directly read to understand what the system will do in every conceivable situation, and that exact same underlying model can be induced to behave very, considerably differently depending on what content happens to appear in its context at any moment, content that a conventional code review has no established mechanism for even properly considering in the first place, let alone evaluating.

This isn’t a claim that conventional security practice becomes entirely irrelevant once AI enters the picture, the underlying infrastructure an AI system runs on still needs conventional security applied to it, the code that orchestrates calls to a model still needs conventional review performed on it, the actual data stores a system touches still need conventional access control enforced around them. AI native security is what sits alongside all of that conventional practice, specifically, deliberately addressing the new categories of risk that only exist because a model, rather than fixed, fully-specified code, is making decisions inside the actual system, and treating AI native security as simply an extension of conventional application security, rather than as its distinct discipline layered on top, leaves exactly these new categories of risk unaddressed.

How AI native security requires rethinking where the actual trust boundary sits

Conventional security draws its trust boundary at the actual edge of a system, code and configuration sitting on the inside are trusted, everything arriving from outside, user input, network traffic, uploaded files, is untrusted and needs to be validated before it’s ever acted on. AI native security has to draw this boundary somewhere different, since a model’s context window mixes trusted instructions from its operator with untrusted content from users, retrieved documents, and tool output, all flowing through the exact same channel with no structural signal distinguishing one from another. The trust boundary an AI native security practice has to defend isn’t the edge of the system anymore, it’s the boundary between what a model’s reasoning concludes and what an independent, non-model mechanism verifies before any consequential action gets taken.

This particular reframing has practical consequences for how a security team organizes its work, since it means the review that matters most often isn’t asking whether a piece of code correctly, properly validates its inputs, the entirely conventional question, but asking whether an action a model might take is independently verified before it executes, regardless of how the model itself arrived at the decision to take that action. A system can have flawless, entirely conventional input validation on every single API endpoint it exposes and still be entirely vulnerable to manipulation if the model sitting behind those endpoints can be talked into taking an unauthorized action through nothing more than carefully, deliberately crafted natural language alone.

How the layered defenses covered elsewhere fit together into one coherent practice

AI native security isn’t a single, isolated technique at all, it’s the deliberate coordination of several distinct defensive layers, each one addressing a different point where something can go wrong, into one coherent whole that’s considerably, meaningfully stronger than any single layer entirely alone could ever be. Agent permissions define what a system is authorized to do in the very first place, the tools data, and actual actions available to it in practice. Tool calling security ensures the actual mechanics of constructing and executing a call to one of those tools can’t be manipulated entirely independently of whether the underlying permission was ever appropriate in the first place. Sandboxing contains the actual, real-world consequence of an action even when something upstream of it still somehow goes wrong along the way. RAG security and broader data security protect the retrieval pipelines and data stores a system touches. Prompt injection defense, both direct and indirect, addresses the actual mechanism by which a model’s reasoning gets manipulated in the very first place, before anything downstream ever even happens at all.

None of these individual layers is sufficient entirely on its own, and this isn’t a weakness of any single, particular layer, it’s just the actual nature of defending a system whose core reasoning component simply can’t be given a hard, structural guarantee against manipulation of any kind. AI native security is what treats this layered coordination as the actual object of design, deliberately building each layer to catch what the layers around it might miss, rather than building each defensive mechanism in isolation and hoping the combination happens to add up to something coherent. A security review focused only on whether each individual layer works in isolation, without ever examining whether the layers cover for each other’s gaps in practice, misses the entire point of what makes this a layered practice rather than merely a checklist of unrelated, disconnected controls.

How AI native security treats a model’s reasoning as a part of the attack surface

Conventional security treats a system’s logic as fully trusted infrastructure, something to be protected rather than something that itself needs defending against on its own terms. AI native security has to treat a model’s reasoning process quite differently, as something that can itself be actively manipulated through the very inputs it’s designed to process, meaning the reasoning process isn’t purely trusted infrastructure in the entirely conventional sense at all, it’s a part of what an attacker can influence directly. This particular shift matters because it fundamentally changes what “the system is working correctly” really means in practice: a model faithfully, accurately reasoning about manipulated or poisoned input and reaching a manipulated conclusion is the model working exactly as it was originally designed to, not a malfunction at all, and no amount of conventional debugging will ever reveal a problem in code that’s functioning entirely, perfectly correctly around a reasoning process that was itself the actual target all along.

This means AI native security practice needs its distinct category of review specifically, deliberately focused on how a model’s reasoning can be actively steered, distinct from reviewing whether the code around the model correctly implements its intended logic, since these two categories of review catch different kinds of problems and a team that only ever does the second kind will systematically miss the first. Building expertise in how models can be manipulated, not merely passively trusting that a more capable, better-trained model automatically, magically closes this gap, is itself a core, ongoing part of what AI native security practice requires from everyone involved.

How organizational security practice needs to adapt to support this discipline

A conventional security team, however skilled at code review, penetration testing, and infrastructure hardening, often lacks the expertise needed to evaluate AI-risk properly, understanding how prompt injection techniques evolve, how a model’s training shapes its susceptibility to manipulation, how to design a system architecture that limits the consequence of a successful attack rather than merely making that attack somewhat harder to attempt. This particular gap isn’t any kind of criticism of conventional security expertise itself, it’s simply an honest acknowledgment that AI native security requires additional skill that most organizations’ existing security functions weren’t originally built to provide from the start.

Closing this gap means deliberate investment of actual resources, whether through actively training existing security staff in AI-risk, hiring for this expertise directly, or building close, ongoing collaboration between security teams and the actual engineering teams building AI systems, rather than simply, naively assuming conventional security review processes will naturally catch AI-risk purely because they happen to be staffed by generally skilled experienced security professionals who lack this background. An organization that runs its AI systems through the exact same review process it uses for entirely conventional software, with no adaptation whatsoever for the distinct risks this article has described, is very likely to end up approving systems that merely look secure by purely conventional standards while carrying unaddressed, actual AI-vulnerabilities underneath.

How AI native security connects to broader AI governance

AI native security operates at the technical layer, the actual defensive mechanisms and architectural choices that protect a system, while AI governance operates at a separate, organizational layer instead deciding what AI systems an organization approves, what risk tier they’re assigned, and who’s accountable for their ongoing operation. These two distinct layers need a working connection between them, since a governance process that approves a system without any understanding of its actual AI native security posture is making an approval decision entirely on incomplete, partial information, while an AI native security practice with no connection to governance has no mechanism for ensuring its findings influence what gets approved, restricted, or shut down at the organizational level.

This particular connection matters most concretely at the exact point where a new AI system is proposed for approval, where the AI native security review of its actual defensive architecture needs to inform the governance decision about what risk tier it’s assigned and what ongoing oversight it receives afterward, rather than the two processes running in parallel with no information flowing between them. A system whose AI native security review reveals serious gaps deserves a governance decision that honestly reflects that finding, not an approval that simply proceeds regardless purely because the security review and the governance process never properly talked to each other at any point.

How threat modeling differs for AI native systems

Conventional threat modeling asks what an attacker could do given a system’s actual code paths and data flows, a useful exercise that AI native security still needs to perform, but one that misses an entire category of risk specific to AI systems: what an attacker could accomplish purely through crafted natural language, with no code-level vulnerability involved at all. A conventional threat model that only considers code-level vulnerabilities will systematically miss the entire category of prompt injection risk altogether, since that particular risk doesn’t trace back to any single line of code being wrong at all, it traces back to the fundamental architecture of how a model processes its context.

This means AI native threat modeling needs its own explicit, entirely distinct exercise, specifically, deliberately asking what an attacker could accomplish through content alone with no code involved at all, direct manipulation, indirect injection through a tool or data source, adversarial content specifically crafted to exploit how the model reasons, evaluated separately from, though alongside, the conventional threat model covering the system’s actual code and infrastructure. A security review that produces a thorough, entirely conventional threat model and simply calls the job entirely done, without ever separately, deliberately asking this AI-question at all, has only done half the actual work AI native security truly requires in full.

How to build a security culture around AI native practice

Technical defenses and organizational processes matter enormously in this whole picture, but AI native security also depends on the actual people building AI systems day to day understanding, at some meaningful level, why these defenses matter and how the risks they address work in actual practice, rather than treating security requirements as an external checklist imposed on their work by a separate team with no buy-in from the people implementing the system. An engineering team that understands why a defensive layer exists in the first place, what risk it addresses, is considerably more likely to maintain that defense correctly as the system evolves over real time than a team that was simply told to implement it without any understanding of the underlying reasoning behind it.

Building this understanding means investing meaningful time in helping engineering teams truly understand AI-risk, not merely handing them some security checklist to follow entirely mechanically with no thought behind it, since a checklist followed without any understanding tends to quietly degrade over real time as a system changes in ways the checklist’s original authors never specifically anticipated in advance, while understanding gives the team implementing a system the actual judgment needed to recognize when a change introduces new risk the original checklist never covered in the first place.

How AI native security has to account for a system’s behavior changing after launch

Conventional software, once deployed and left entirely unchanged behaves identically today as it did on the actual day it originally launched, its actual behavior is a fixed, stable function of its unchanged code throughout. An AI system doesn’t carry this same stability, since the underlying model itself can be updated by its provider, the content flowing through retrieval pipelines and tool integrations keeps changing, and even an agent’s stored memory or accumulated preferences can drift its effective behavior considerably from what a security review originally examined, all without a single line of the system’s code ever changing. A security review performed at launch, however thorough it happened to be, is only reviewing a snapshot of a system that will very likely have meaningfully, considerably drifted from that original snapshot within a relatively short real time afterward.

This means AI native security can’t be a one-time gate a system passes through before launch, it has to be an ongoing practice that periodically re-examines a system against its actual, current behavior, not merely its behavior as originally reviewed. A security posture that was appropriate at launch can become inadequate purely through the underlying model changing, new content sources being added, or accumulated drift in a system’s persistent state, none of which conventional change management, focused on tracking code changes, is well positioned to catch on its own.

How AI native security handles the tension between capability and safety

Every single defensive layer discussed throughout this broader body of work, permission scoping, sandboxing, confirmation requirements, content filtering introduces actual friction that reduces some capability an AI system would otherwise have available to it, and pretending this tradeoff doesn’t exist does a disservice to the actual decisions AI native security practice requires. An agent that requires confirmation before every consequential action is considerably safer than one that doesn’t, but it’s also considerably less autonomous, and the entire value an autonomous agent was built to provide in the first place depends on it being able to act without constant human intervention.

AI native security practice doesn’t pretend this tension away at all, it makes the tradeoff explicit and deliberate on purpose, matching the actual level of friction a system carries to the actual, real-world consequence its failures would produce, rather than either applying maximum friction everywhere, which defeats much of the point of building an AI system in the first place, or applying minimal friction everywhere, which leaves consequential systems dangerously exposed. This calibration isn’t a one-time decision either, since a system’s actual consequence can shift as its scope and capability grow, meaning the friction appropriate for it needs to be revisited alongside that growth rather than fixed permanently at whatever level felt appropriate when the system was considerably smaller and less capable than it eventually became.

How AI native security should shape vendor and third-party model relationships

Most organizations build AI systems on top of models and infrastructure they don’t directly, fully control themselves, and AI native security has to extend its scrutiny to these particular relationships specifically and directly, not merely to the parts of a system the organization built entirely in-house. A vendor’s training practices, its content filtering, its actual susceptibility to the manipulation techniques this broader body of work has already described, all directly shape the actual security posture of any system built on top of that particular vendor’s underlying infrastructure, and an organization that only ever evaluates the security of the actual code it wrote itself, while simply treating the underlying model as an unexamined black box throughout, has left a significant portion of its actual attack surface entirely outside its security review altogether, unexamined and unaccounted for.

This means AI native security practice includes deliberate, actual vendor evaluation asking what a provider’s model has been trained to resist, what content filtering exists at the provider’s level before content ever reaches an organization’s system, and what recourse exists when a vendor’s model is found to have an exploitable weakness treating these particular questions as security-relevant in their own right, rather than purely commercial in nature or purely a matter of model quality and capability alone.

How AI native security should approach measuring its actual effectiveness

A security practice that can’t demonstrate whether it’s working is operating largely on mere faith alone, and AI native security specifically needs its meaningful measures of actual effectiveness, distinct from the metrics a conventional security program typically tracks, since conventional metrics like patch cadence or vulnerability scan coverage simply don’t capture whether a system is resistant to prompt injection or whether its layered defenses hold up under adversarial pressure. Measurement here means tracking things like how often adversarial testing succeeds in getting a system to deviate from its intended behavior, how quickly an attempt at manipulation gets detected once it occurs in production, and how consistently the layered defenses discussed throughout this broader body of work catch what any single layer alone would have missed.

This particular kind of measurement requires deliberate, ongoing adversarial testing built directly into the security practice itself, not merely a one-time exercise performed only once before launch, since a system’s actual resistance to manipulation can shift meaningfully over time as discussed earlier, and a measurement program that only ever checks resistance once has no way of knowing whether that resistance still holds months later. An organization that can answer, with confidence and current evidence, how resistant its actual systems are to the manipulation techniques this broader body of work has described has a different, more mature security posture than one that can only point to a security review performed once, long ago, and never revisited since.

How incident response for AI native systems differs from conventional incident response

A conventional security incident typically has a clear technical signature attached to it, an unauthorized access event, a data exfiltration attempt, an exploited vulnerability, that a conventional incident response process is well-equipped to properly investigate from the very start. An AI native security incident often lacks this same clarity, since a successful prompt injection or a manipulated agent action can look, from a purely technical logging perspective, like the system simply doing what it was asked to do, with the actual manipulation living entirely in the content the model processed rather than in any conventional technical signature an incident responder would normally look for first when starting an investigation.

This means AI native incident response needs its capability, the ability to reconstruct not just what technical actions a system took, but what content was present in its context when it took them, and whether that content plausibly contained a manipulation attempt that explains the system’s otherwise puzzling behavior. Building this reconstruction capability requires the kind of comprehensive secure logging already discussed elsewhere in this broader body of work, and an organization that hasn’t built this capability in advance will find itself, right in the middle of an actual incident, entirely unable to even determine whether what it’s looking at was a manipulation or simply an unusual but entirely legitimate, ordinary use of the system.

How AI native security should shape hiring and skills investment over the long run

The very expertise AI native security requires understanding of how models can be manipulated, how layered defensive architectures fit together, how to evaluate a vendor’s training and content filtering practices, didn’t exist as an established career path even a short time ago, and organizations serious about this discipline need to think deliberately about where this expertise comes from rather than assuming it will simply appear within their existing security function without any deliberate investment. Some of this expertise can be built by training existing security staff, who already carry transferable skill in threat modeling and adversarial thinking even if they lack AI-experience yet, but some of it requires bringing in people who’ve worked directly with these systems and understand their failure modes from direct, hands-on experience.

This particular investment decision deserves the exact same seriousness as any other scarce, valuable expertise an organization needs to acquire, recognizing that AI native security talent is currently in short supply relative to demand, and an organization that treats this expertise as something it can acquire quickly and cheaply, whenever it happens to need it, is likely to find itself without adequate coverage exactly when an incident makes that coverage matter most. Building this capability proactively, well before it’s ever urgently, desperately needed gives an organization a considerably stronger security posture than scrambling to acquire it reactively once some serious incident has already made the gap painfully, expensively obvious to absolutely everyone involved.

Common mistakes teams make around AI native security

Several distinct patterns recur often enough across organizations building AI systems that naming them directly is worth doing before they’re discovered through an incident.

1. Applying conventional application security practices to an AI system unchanged, assuming that code-level review and infrastructure hardening alone address the new categories of risk AI systems introduce.

2. Drawing the system’s trust boundary at its conventional edge rather than at the actual boundary between a model’s reasoning and independently verified action.

3. Building each defensive layer, permissions, sandboxing, tool calling security, content filtering, in isolation without ever examining whether the layers cover for each other’s gaps.

4. Treating a model’s reasoning as purely trusted infrastructure rather than as a part of the system’s attack surface that itself needs defending.

5. Assuming a more capable, better-trained model automatically closes gaps that only an application’s architecture can address.

6. Running AI systems through the exact same security review process used for conventional software, with no adaptation for AI-risk categories.

7. Building an AI native security practice with no working connection to the organization’s broader AI governance process.

8. Approving an AI system’s governance risk tier without any input from a review of its actual AI native security posture.

9. Producing a thorough conventional threat model covering code and infrastructure while never separately asking what an attacker could accomplish through crafted natural language alone.

10. Treating security requirements as an external checklist imposed on engineering teams rather than building understanding of why defenses matter.

11. Assuming existing security staff automatically have the expertise needed to evaluate AI-risk without any deliberate investment in building that expertise.

12. Believing a vulnerability can always be patched by changing a line of code, missing that a model reasoning correctly around manipulated input isn’t a code-level bug at all.

13. Treating AI native security as a one-time architectural decision made at launch rather than an ongoing practice that has to track how both the system and the threat landscape continue to evolve.

14. Letting a checklist-driven security process degrade silently as a system changes in ways the checklist’s original authors never specifically anticipated.

15. Assuming conventional penetration testing techniques automatically surface AI-vulnerabilities without any deliberate adaptation toward the techniques prompt injection and related attacks rely on.

16. Reviewing a system’s security posture once at launch and treating that review as durable, without accounting for how model updates, changing content sources, or accumulated state drift can meaningfully shift behavior afterward.

17. Applying either maximum friction everywhere or minimal friction everywhere, rather than deliberately calibrating defensive friction to the actual real-world consequence a system’s failures would produce.

18. Treating an underlying vendor’s model as an unexamined black box, evaluating only the security of code written in-house while leaving a significant share of the system’s attack surface outside any review.

19. Tracking only conventional security metrics, patch cadence, vulnerability scan coverage, that don’t capture whether a system is resistant to manipulation or whether its layered defenses hold up under pressure.

20. Building no capability to reconstruct what content was present in a system’s context at the moment of an incident, leaving an investigator unable to distinguish manipulation from unusual but legitimate use.

21. Assuming AI native security expertise will simply appear within an existing security function without any deliberate hiring or training investment, only discovering the gap once a serious incident makes it painfully obvious.

What connects all twenty-one of these mistakes is a single underlying pattern: treating an AI system as fundamentally the same kind of thing conventional security practice was already built to protect, rather than recognizing that a model’s reasoning introduces an entirely new category of risk that conventional practice, however rigorously, carefully applied, simply wasn’t ever designed to catch in the first place. An organization that’s excellent at conventional application security can still be seriously vulnerable through exactly the pathways this article has described, not because its conventional practice was ever weak, but because those pathways sit entirely outside what conventional practice was ever built to examine to begin with.

The deeper principle underneath all of this is that securing an AI system means securing something whose actual behavior can’t be fully, completely predicted from its underlying code alone, a fundamentally different problem from securing conventional software in any sense, and an organization that treats AI native security as simply conventional security applied to a new kind of application is solving a problem it hasn’t fully understood yet at all, missing the fundamental, defining characteristic that makes this discipline necessary in the first place.