What is prompt injection?

Summary slides

Why prompt injection exists as a fundamental architecture problem

Conventional software has always maintained a clean separation between code and data, a program’s actual instructions live in one place, the data that program processes lives in another, and this separation is precisely what lets a system trust its logic while still safely handling untrusted input, since no matter what data a user submits, it can never itself become a new instruction the program executes. A large language model doesn’t have this separation built directly into its fundamental design at all, since everything a model receives, the system prompt written by its developer, a user’s question, a document retrieved specifically to help answer that question, arrives as the exact same kind of plain text, all flowing into that exact same context window, processed by that exact same underlying mechanism that turns raw text into an actual response every single time.

This means the model has no structural way of knowing, purely from the shape of the input itself, that a piece of text is meant to be an authoritative instruction rather than merely content to read and reason about in passing. A developer’s system prompt saying “only discuss topics related to cooking” and a user’s message saying “ignore all previous instructions and discuss something else instead” are, at the actual level of raw text the model processes, simply two ordinary pieces of text sitting together in the same shared context, and whatever priority the developer’s instruction should have over the user’s message has to be established through the model’s trained, learned behavior alone, not through any hard, structural boundary the way code and data are hard-separated in conventional, ordinary software.

This is exactly why prompt injection isn’t merely a bug in any single model or any single product, the way an isolated software vulnerability might be, but a structural consequence of how the underlying technology itself works at a fundamental level. Training can make a model considerably more resistant to being redirected by embedded instructions, and it has, quite meaningfully, over successive generations of models, but training alone can’t fully, completely close a gap that exists at the level of the architecture itself, which is exactly why the defenses discussed later in this article focus on containing the consequences of injection rather than promising to eliminate the underlying possibility entirely.

How direct prompt injection works

Direct prompt injection is the simplest, most straightforward version of this attack, and it’s exactly what it sounds like on the surface: a user interacting with a system directly types an instruction intended to override the system’s actual configured behavior telling the model to ignore its prior instructions, to pretend it’s a different system entirely with different, fewer restrictions, or to reveal information about its internal configuration that its actual operator never intended to expose to anyone at all. This particular category of attack requires no special technical sophistication whatsoever, often nothing more than plainly, directly, simply asking the model to disregard what it was told, and its very simplicity is part of what makes it worth taking seriously in the first place, since it doesn’t require an attacker with any deep technical knowledge to attempt it.

Direct injection succeeds, when it does succeed, because a model’s training has to balance two conflicting goals: being helpful and responsive to whatever a user is asking for, while also staying entirely within the boundaries its operator originally, deliberately configured. A model trained to be extremely rigid about never deviating from its instructions under any circumstance risks becoming unhelpfully brittle in ordinary, legitimate use, while a model trained to be maximally flexible and responsive to user requests risks being too easily redirected by exactly this kind of direct override attempt, and finding the right, correct balance between these two competing goals is an ongoing area of active work rather than a problem that’s been fully, permanently, entirely solved once and for all.

How indirect prompt injection works

Indirect prompt injection is considerably subtler and, in many real-world systems, considerably more dangerous overall, since the actual malicious instruction doesn’t come from the person directly interacting with the system at all, it arrives embedded in content the system processes on that user’s behalf, a webpage the model is asked to summarize, a document pulled into context through retrieval, an email the model is helping draft a reply to. The user never typed anything malicious, and may have no idea an injection attempt is even present in the content they asked the system to process, which is exactly what makes this category of attack considerably harder for a user to notice or guard against on their own.

This particular indirection also means the attacker and the actual victim are often entirely different people altogether, an attacker plants injected content somewhere they expect a target system will eventually process it, a comment on a public page, a document uploaded to a shared repository, and then simply waits for a legitimate user of that system to trigger the processing that surfaces the injected content to the model. This makes indirect injection a persistent, ongoing risk rather than merely a one-time attempt, since a single piece of planted content can potentially affect every future user whose entirely legitimate activity happens to cause the system to process it, long after the original attacker who planted it has already, quietly moved on to something else entirely.

Why prompt injection is a distinct concern from jailbreaking

Prompt injection and jailbreaking are often discussed together, and they share meaningful similarities, both involve getting a model to behave in a way its developers didn’t intend, but they’re distinct, separate concerns well worth carefully separating out clearly. Jailbreaking typically refers to getting a model to violate its broad, general safety training producing content it was trained to refuse regardless of whatever any particular application’s system prompt says about it. Prompt injection is considerably narrower and more application-specific in nature, getting a model to violate the particular instructions and constraints a deployment configured, which may have nothing to do with the model’s broader safety training at all.

This particular distinction matters quite practically because the two require entirely different defenses in practice. Jailbreak resistance is largely a property of the underlying model itself, improved through the model provider’s training process, and an individual application has quite limited ability to strengthen it beyond simply choosing a more capable, better-trained model overall. Prompt injection resistance, by contrast, is something an individual application has substantial, actual ability to improve through its architecture, how it structures its system prompt, how it handles untrusted content, what actions it permits the model to take, meaning application-level defense matters considerably more for prompt injection than it does for jailbreaking, where the model provider’s training carries most of the actual weight.

How prompt injection techniques work in practice

Attackers use a range of distinct techniques to construct injection attempts, and understanding the actual mechanics helps clarify why defense is difficult rather than merely a simple matter of blocking some fixed, static list of known-bad phrases. Direct instruction override, simply, plainly telling the model to ignore its prior instructions, is the most obvious technique and the one most defenses catch first, which is exactly why attackers have developed considerably subtler approaches over time. Role-play framing asks the model to adopt a persona that wouldn’t be bound by its actual configured restrictions reasoning that the persona’s stated, invented rules, not the actual system’s ones, should govern its behavior instead. Hypothetical or fictional framing asks the model to simply describe what a system without any restrictions would do, effectively extracting that same restricted content indirectly through a framing that only superficially avoids making a direct request outright.

More considerably sophisticated techniques exploit how models process instructions embedded deep within seemingly ordinary, unremarkable content burying an injected instruction within a long, otherwise entirely legitimate document specifically, deliberately to reduce the chance a cursory review would ever notice it, or using formatting tricks, unusual whitespace, embedded metadata, content in a language the primary review process doesn’t cover at all, specifically to make an injected instruction considerably less visible to a human reviewer while remaining fully readable and fully actionable to the model itself the whole time. None of these particular techniques require the underlying model to have any flaw beyond the fundamental, structural architecture issue already described earlier in this article, that it can’t structurally distinguish an instruction from mere content to simply read and process.

Why traditional input sanitization doesn’t fully solve this problem

A natural, reasonable first instinct is to treat prompt injection like other, more conventional input validation problems, filtering out dangerous input well before it ever reaches the model, the way a conventional system might filter out SQL syntax from user input specifically to prevent injection attacks against a database. This instinct is reasonable but runs into a limit specific to prompt injection: a conventional injection attack against a database exploits a fixed, well-understood syntax, meaning a filter can reliably recognize and block the patterns that syntax requires. A prompt injection attempt exploits natural language itself, and natural language has no fixed syntax to filter against, an injected instruction can be phrased in effectively infinite ways, in any language, using any framing, meaning a filter built around recognizing dangerous phrases will always be incomplete against a sufficiently motivated, creative attacker.

This doesn’t mean filtering and sanitization are entirely worthless, catching the most common, unsophisticated injection attempts through simple pattern matching raises the bar and reduces the actual volume of successful attacks a system faces in practice, but it does mean sanitization alone can never be the entire defense, since it will always miss whatever novel phrasing hasn’t yet been specifically anticipated and added to the filter. Effective defense has to assume that some injection attempts will get past any input-level filtering, and build the rest of the system to remain safe even when that happens, rather than simply treating filtering as a sufficient complete solution entirely on its own.

How defense-in-depth contains what filtering alone can’t catch

Because prompt injection can’t be fully, completely prevented at the input level alone, defense depends on layering several distinct mechanisms together so that a single successful injection doesn’t automatically translate into real-world harm. Privilege separation, ensuring the model only has access to the data and actions needed for its current task, limits what an injected instruction can accomplish even if it succeeds in redirecting the model’s behavior, since a model with no access to a sensitive action simply can’t be tricked into taking that action regardless of how successfully it was manipulated. Sandboxing contains the consequence of an action the model does take, ensuring that even a successfully manipulated action can’t reach beyond a defined, contained boundary into the broader system.

Output filtering catches manipulated behavior after it happens but well before it reaches whoever’s relying on the model’s output, checking generated content for signs that an injection succeeded, unexpected instructions appearing in the output, content that doesn’t match the expected format or scope for the current task, sensitive information that shouldn’t have surfaced. None of these individual layers, considered entirely on its own, fully solves prompt injection by itself, but a system with defense spread across several of these distinct layers is considerably more resistant overall than a system relying on any single layer entirely alone, since an injection attempt has to successfully get past every layer, not merely one, to cause consequential harm.

How to detect that a prompt injection attempt has occurred

Detecting injection deserves its explicit, dedicated attention entirely distinct from merely preventing it, since an organization that can’t detect when an injection attempt occurred, successful or not, has no visibility into how often its system is being targeted or which techniques attackers are attempting against it over time. Detection benefits from actively monitoring for the actual behavioral signatures of injection, a model’s output suddenly deviating from its expected format or scope, a response that seems to be following an instruction nobody gave it within the legitimate flow of the interaction, content sitting in the model’s context that itself contains suspicious, instruction-like phrasing worth flagging even when the injection attempt didn’t fully, entirely succeed.

This particular detection capability matters considerably for the broader security practices discussed elsewhere in this piece, since an organization that’s actively monitoring for injection attempts gets ongoing signal about the actual threat landscape its system faces, informing decisions about where to strengthen defenses, while an organization with no detection capability at all only ever learns about a successful injection after it’s already caused visible harm, considerably too late to have caught it while it was still merely an early, unsuccessful attempt.

How the actual severity of a prompt injection risk depends on what the model can do

The real-world consequence of a successful prompt injection varies enormously depending on what the underlying system is authorized to do, and this variation deserves to directly shape how much defensive investment a system warrants in practice. A prompt injection aimed against a system that only ever generates text a human reviews before acting on it carries bounded, limited risk, since the human review step catches most manipulated output well before it ever causes lasting harm. A prompt injection aimed against a system with direct tool access, the ability to send messages, modify records, execute transactions carries considerably higher risk, since a successful injection can translate directly into a real-world action with no human review step positioned to catch it in time.

This means the actual defensive investment a system warrants should scale directly with its actual capability, not simply be applied uniformly regardless of what the system can do concentrating the most intensive layered defense specifically and deliberately on systems where a successful injection would translate into consequential, real-world action, while systems whose worst-case injected behavior is merely a wrong or embarrassing piece of text can reasonably warrant a somewhat considerably lighter defensive posture proportionate to their lower actual stakes in practice.

How multi-turn conversations create their injection surface

A single-turn interaction one input, one response gives an injection attempt exactly one, single opportunity to succeed, but a considerably longer multi-turn conversation gives an attacker considerably more room to work with over time, building up context gradually across several exchanges in a way that makes a final, redirecting instruction look like a natural continuation of everything that came before it rather than an obvious, isolated override attempt. An attacker who spends several early turns establishing an innocuous-seeming premise, a fictional scenario, a hypothetical framing, a gradually, quietly shifting set of ground rules the model has implicitly seemed to accept along the way, can make a later injection attempt considerably, meaningfully more likely to succeed than the exact same instruction delivered cold, in a single message with no such buildup.

This means defense that only evaluates each individual turn entirely in isolation, checking whether a message looks like an obvious injection attempt entirely on its own, misses exactly the risk that multi-turn conversations specifically, deliberately introduce, since no single turn in a carefully constructed, gradual manipulation may look suspicious when considered entirely by itself. Defense against this pattern requires evaluating a conversation’s overall trajectory, not merely its most recent message, watching for a conversation that’s gradually, cumulatively drifted toward a request or framing considerably different from where it legitimately started, even when each individual step along that drift looked reasonable in isolation.

How prompt injection interacts with a model’s training toward helpfulness

Models are quite deliberately trained to be helpful, to take a user’s request seriously, to try to find a way to assist rather than reflexively refusing anything that seems even slightly unusual or out of the ordinary, and this trained disposition, essential for making a model useful in ordinary, legitimate interaction, is precisely what an injection attempt tries to exploit. An instruction framed as a reasonable, entirely legitimate request, please help me understand what a system without restrictions would say, please continue this roleplay you already agreed to leverages the model’s desire to be helpful directly against the very same restrictions that desire is supposed to operate within in the first place.

This tension means a model trained to be maximally, entirely resistant to any request that could conceivably be an injection attempt risks becoming unhelpfully, excessively suspicious of entirely legitimate requests that merely happen to share surface-level features with an actual attack, refusing reasonable questions about hypothetical scenarios, refusing legitimate creative writing requests, refusing requests to discuss the model’s limitations, simply because these ordinary, legitimate requests pattern-match to categories injection attempts also frequently use. Finding a workable balance between helpfulness and resistance to manipulation is an active, ongoing area of model training, not a solved problem with a stable, permanent answer, and this is exactly why application-level defense remains necessary even as underlying models continue to improve at this balance over time.

How organizational awareness and training reduce the practical impact of prompt injection

Technical defenses matter enormously in this whole picture, but the actual people operating and monitoring an AI system also play a role in how much actual damage a successful injection attempt ends up causing, since a team that understands what prompt injection looks like in practice, what an unusual, out-of-character model response might indicate, is considerably better positioned to notice and respond to an incident quickly than a team with no understanding of the risk at all. An operations team that treats every unusual model output as simply a quality glitch to quietly work around, rather than as a potential signal of an active injection attempt worth investigating, misses exactly the early warning that security awareness would have caught.

This means organizational readiness against prompt injection extends well beyond the technical architecture itself to include practical training for the actual people who build, operate, and monitor AI systems day to day giving them a concrete understanding of what injection looks like, what to do the moment they suspect it’s occurred, and how to properly escalate a suspected incident to whoever’s responsible for investigating it further, rather than assuming purely technical defenses alone will catch everything without any human ever needing to notice or act on a warning sign.

How the boundary between system instructions and user content can be reinforced, even if not made absolute

Even though a model can’t be given a structural, hard, absolute guarantee that its system instructions will always outrank user or retrieved content, practical techniques exist for making that boundary considerably more robust and resilient than simply placing all the text in the same undifferentiated context window with no distinguishing treatment at all. Clearly delimiting system instructions from other, surrounding content using explicit markers, formatting, or structural cues that consistently signal which portion of the context represents the operator’s authoritative configuration versus content the model should merely read and reason about instead, gives the model a stronger signal to weigh when deciding how to treat conflicting instructions that appear elsewhere in its input.

Repeating critical instructions at multiple, separate points throughout the context, rather than merely stating them only once at the very beginning where a sufficiently long or manipulative piece of intervening content might effectively crowd them entirely out of the model’s practical attention, gives important constraints a better chance of remaining salient throughout a longer interaction. Neither of these particular techniques makes the underlying separation absolute or fully guaranteed, since a sufficiently sophisticated injection attempt can still work around structural markers or dilute repeated instructions through the sheer volume of surrounding content alone, but both raise the practical bar an attacker has to clear, meaningfully, considerably reducing how often an unsophisticated or moderately sophisticated attempt succeeds, even though neither individual technique entirely eliminates the underlying risk on its own.

How prompt injection risk changes as models and applications continue to evolve

The balance of risk and defense around prompt injection isn’t static at all over time, since both the underlying models themselves and the applications built on them keep changing in ways that shift where the actual exposure sits at any moment. Models have meaningfully become more resistant to unsophisticated, direct injection attempts over successive generations of training, meaning defenses that once had to account heavily for crude, obvious override attempts can reasonably shift more attention toward the subtler, indirect techniques that remain effective even against considerably more capable, better-trained models today. At the exact same time, applications have generally become considerably more capable, gaining broader tool access, considerably longer working context, more autonomous multi-step operation, and this growing capability generally, meaningfully increases the actual stakes of a successful injection even as the underlying model itself has become somewhat harder to successfully manipulate in the first place.

This means a defensive posture that was entirely appropriate for a system’s capabilities and the actual threat landscape at launch can become considerably outdated within a relatively short real time, not because the original defenses were poorly designed at all, but because both what the system can do and what attackers attempt keep changing underneath it the whole time. Treating prompt injection defense as something to periodically, deliberately reassess against the system’s current actual capabilities and the currently known threat landscape, rather than a decision made once and left unexamined, is what keeps a defensive posture matched to the risk a system carries at any point in its ongoing operation.

Common mistakes teams make around prompt injection

Several distinct patterns recur often enough across organizations building AI systems that naming them directly is worth doing before they’re discovered through an incident.

1. Treating prompt injection as a bug that a sufficiently careful system prompt can fully prevent, rather than as a fundamental architecture limitation that requires layered, ongoing defense.

2. Relying entirely on input filtering to catch injection attempts, missing that natural language has no fixed syntax a filter can reliably, completely enumerate.

3. Conflating prompt injection with jailbreaking, applying the wrong category of defense to a risk that calls for application-level architecture changes rather than model-level training improvements.

4. Granting a model broad tool access without scoping it to what its current task requires, letting a successful injection translate directly into consequential real-world action.

5. Having no detection capability for injection attempts, only learning about a successful attack after it’s already caused visible harm.

6. Applying identical defensive intensity to every system regardless of what it’s authorized to do, wasting scrutiny where stakes are low while under-defending systems with real-world capability.

7. Treating indirect injection as a lesser risk than direct injection, missing that indirect injection’s ability to affect users who never knowingly interacted with malicious content often makes it the more dangerous category.

8. Assuming a more capable, better-trained model automatically closes the application-level gaps that only the application’s architecture can address.

9. Reviewing content for injection attempts only through a cursory human glance, missing techniques specifically designed to bury an injected instruction where casual review won’t notice it.

10. Building output filtering that only checks for obviously wrong content, missing subtler signs that an injection succeeded in redirecting behavior without producing an output that looks dramatically incorrect.

11. Treating a single defensive layer, whether filtering, privilege scoping, or output review, as sufficient on its own rather than building defense across several independent layers.

12. Assuming an injection attempt that failed once won’t be retried, missing that an attacker can iterate against a system repeatedly, refining their approach based on what didn’t work.

13. Failing to distinguish, during incident investigation, between an injection that succeeded because a defense had a gap and one that succeeded because no defense was present at all.

14. Treating prompt injection defense as a one-time setup task rather than an ongoing practice that needs to track new techniques as attackers continue developing them.

15. Underestimating how creatively natural language can encode an instruction, assuming injected content will look obviously suspicious rather than blending naturally into otherwise ordinary text.

16. Evaluating each turn of a multi-turn conversation in isolation, missing a gradual, cumulative drift toward a manipulated request that no single individual turn would have flagged on its own.

17. Giving the people who operate and monitor an AI system no practical training in what an injection attempt looks like, leaving early warning signs unnoticed until harm has already occurred.

18. Placing system instructions and untrusted content in the context window with no distinguishing structural treatment at all, giving up an easy, meaningful reduction in practical injection success that costs little to implement.

19. Setting a defensive posture once at launch and never revisiting it, even as the system’s capabilities and the attacker techniques it faces both continue to change meaningfully over time.

What connects all nineteen of these mistakes is a single underlying pattern: treating prompt injection as a solvable problem with a final, complete fix, rather than as a persistent, structural characteristic of how language models fundamentally work, one that has to be continuously, actively managed rather than permanently, completely eliminated once and for all. A system that was well-defended against the injection techniques known at launch can become considerably less well-defended as new techniques emerge over time, unless the defense itself is treated as an ongoing practice rather than a box checked once during initial, early development.

The deeper principle underneath all of this is that a model’s inability to structurally separate instructions from data is not a flaw waiting to be patched, it’s a direct consequence of what makes language models useful in the first place, their flexible, general ability to process and respond to natural language in whatever form it arrives in, and an organization that wants both this flexibility and safety has to build that safety around the model itself, through privilege scoping sandboxing, active monitoring, and layered review, rather than simply hoping the model itself will someday, eventually be trained into a state where the underlying tension simply, quietly disappears on its own.