What is LLM security?

Summary slides

Why LLM security is a distinct layer from securing the application built around a model

Application-level AI security, already covered elsewhere as agent security, tool calling security, and prompt injection defense, addresses how a system uses a model, what tools it grants, what content it exposes the model to, how it handles the model’s output. LLM security addresses something more fundamental: the actual properties of the model itself, independent of any particular application built on top of it. A model with weak resistance to jailbreaking, or one whose actual weights can be easily extracted through careful, sustained querying carries that weakness into every single application ever built on top of it, regardless of how carefully any individual application’s architecture was designed around it in the first place.

This distinction matters because the two layers require different actors and different levers to address. An application team can build excellent tool calling security and excellent permission scoping around a model, and none of that architectural care does anything to fix a model that’s fundamentally, structurally easy to extract or manipulate at the model level itself. Fixing that requires action at the level of the model’s training, fine-tuning, and deployment, work that typically sits with whoever trains or hosts the model rather than with the team building an application on top of it, which is exactly why LLM security deserves its distinct treatment rather than being folded entirely into application-level practice.

How model extraction attacks work

Model extraction refers to an attacker systematically, methodically querying a model observing its outputs, and using those observations to reconstruct a functionally similar model without ever gaining any direct access to the original weights at all. This is a risk specifically because a model represents a substantial investment in training data, compute, and expertise, and an attacker who successfully extracts a close approximation gains much of that value without paying any of the original cost, while also potentially gaining insight into the original model’s weaknesses that a defender never intended to expose.

Defending against extraction means treating a model’s API access with the exact same seriousness as any other valuable, sensitive asset an organization owns, rate limiting queries to make large-scale extraction attempts prohibitively slow, monitoring for query patterns that look more like systematic probing than ordinary, legitimate use, and watermarking or otherwise fingerprinting model outputs in ways that make it possible to later detect whether a suspicious competing model was trained on extracted outputs. None of these defenses fully eliminates extraction risk, a sufficiently patient, well-resourced attacker willing to operate within rate limits over an extended period can still make progress, but together they raise the cost of extraction considerably, turning what might otherwise be a cheap, fast attack into a slow, expensive, and more detectable one.

How jailbreaking works and why it differs from the prompt injection risks covered elsewhere

Jailbreaking refers to getting a model to violate its broad, general safety training producing content it was specifically, deliberately trained to refuse, and this is distinct from prompt injection, which targets an application’s configured instructions rather than the model’s underlying safety training itself. A model can be extremely well jailbreak-resistant while still being highly vulnerable to prompt injection in a poorly designed application, and the reverse is equally true, a model with weaker safety training deployed within an extremely well-architected application. LLM security is specifically concerned with the first half of this pairing, how resistant the model itself is to being talked out of its trained behavior, regardless of what any application built on top of it does.

This particular resistance is shaped primarily through the model’s training process, techniques that specifically, deliberately expose the model to adversarial attempts during training and reinforce refusal of harmful requests even when those requests are creatively, indirectly framed to get around it. An organization deploying a third-party model has limited direct ability to improve this underlying resistance itself, but it has ability to evaluate it, testing a candidate model’s actual jailbreak resistance as part of model selection, rather than assuming every model available on the market carries comparable resistance simply because they’re all broadly similar in general capability.

How adversarial inputs can manipulate model behavior beyond ordinary prompt injection

Beyond prompt injection’s reliance on natural, readable language alone, LLM security also has to account for more technically sophisticated adversarial inputs sequences of tokens or characters, sometimes not even meaningful as ordinary language to a human reader, that have been specifically discovered or engineered to reliably shift a model’s behavior in a particular direction. These particular adversarial sequences are typically found through systematic search conducted against a model’s actual behavior, sometimes requiring direct access to the model’s internals to construct efficiently and reliably, and their existence reflects the fact that a model’s decision boundary, the actual mathematical function determining its output, contains regions that don’t correspond to anything a human would naturally consider meaningful language but that still reliably trigger exploitable behavior.

Defending against this category of adversarial input is harder than defending against ordinary, natural-language prompt injection, precisely because these inputs don’t need to look like anything a content filter built around recognizing suspicious natural language would ever flag. This means LLM security benefits from input preprocessing that normalizes or restricts input in ways that specifically disrupt these kinds of adversarial token sequences, alongside monitoring that watches for the kind of dramatic, discontinuous shift in model behavior that a successful adversarial input attack typically produces, a response that looks nothing like what the model would ordinarily generate for similar, ordinary requests.

How model supply chain risk shapes LLM security specifically

A model’s actual behavior is shaped by absolutely everything that went into training and fine-tuning it originally, and this entire process represents a supply chain an organization needs to honestly evaluate, not merely the final, deployed model in isolation. Training data poisoning, where an attacker who can influence what data a model trains or fine-tunes on introduces content specifically, deliberately designed to create a hidden, exploitable behavior, a backdoor that activates only under chosen conditions the attacker controls, represents one of the more insidious risks in this category, precisely because a poisoned model can pass every ordinary evaluation while still carrying a hidden vulnerability nobody happened to test for.

Organizations that fine-tune their models on internal data need honest confidence in the actual integrity of that fine-tuning data pipeline, since a compromised data pipeline feeding fine-tuning is, in effect, a direct path for an attacker to shape the model’s trained behavior. Organizations relying on third-party models need honest confidence in that provider’s training practices and their actual supply chain security overall, since a compromise at the provider’s level propagates directly into every downstream deployment of that model regardless of how carefully any individual customer’s application was built around it.

How model versioning and rollback capability matter for LLM security

A model that’s discovered to have a security weakness, whether a jailbreak vulnerability, a poisoned fine-tuning dataset, or simply a behavioral regression introduced by an update needs a practical path to being replaced or rolled back, and this capability deserves deliberate, upfront design rather than being figured out reactively in the middle of an actual incident. An organization that’s tightly coupled its entire application architecture to one single model version, with no mechanism for swapping to a different version or a different model entirely faces limited options when a security issue with that model surfaces, often forced to accept ongoing risk simply because the practical cost of changing models was never seriously planned for in advance.

This means LLM security considerably benefits from an architecture that deliberately treats the underlying model as a swappable component right from the very start of a project maintaining the actual ability to test candidate replacement models against that exact same evaluation suite originally used for the original model, and keeping a practiced rollback path ready to use, rather than simply assuming, without ever verifying it, that the current model will remain adequate indefinitely into the future. An organization that’s rehearsed swapping models, even just once, under low-stakes conditions, is considerably better positioned to do it quickly and correctly under the pressure of an actual security incident than one attempting it for the very first time in the middle of an emergency.

How red-teaming an LLM specifically differs from red-teaming a broader application

Red-teaming a broader application, actively attempting to manipulate it through prompt injection, through tool misuse, through the various techniques already covered elsewhere in this broader body of work, is necessary but meaningfully distinct from red-teaming the underlying model itself directly, which specifically probes the model’s raw behavior independent of whatever application context it’s normally deployed within. This particular model-level red-teaming asks questions like how resistant the model honestly is to jailbreaking across a wide, systematic range of creative framings, how it behaves when given adversarial or out-of-distribution input that has nothing to do with any application’s intended use, and whether its outputs reveal anything about its training data or its internal configuration that shouldn’t be exposed.

This particular model-level red-teaming deserves its dedicated practice, ideally informing which model an organization chooses to deploy in the very first place, rather than being folded entirely into the application-level testing performed after a model has already been selected and integrated. An organization that only ever red-teams its complete, deployed application, never separately evaluating the underlying model’s raw behavior in isolation risks missing weaknesses that are honestly properties of the model itself, weaknesses no amount of careful application architecture can ever fix, since they exist one entire layer below where that architecture operates day to day.

How output validation protects downstream systems from a compromised or manipulated model

Even with investment already made in the model-level defenses this article has already described, LLM security benefits considerably from a final, entirely independent layer that validates a model’s actual output well before it’s trusted by whatever downstream system consumes it, checking generated content against expected formats, expected value ranges, and known patterns that would indicate the model’s behavior has been successfully manipulated despite every upstream defense. This output validation functions as a backstop specifically because it doesn’t depend on correctly anticipating every possible way a model might be manipulated in advance, it simply checks whether the actual output looks like what a correctly functioning model should have produced for this kind of request.

This particular validation deserves to be built entirely independently of the model itself using conventional, non-model logic wherever possible to do so, since a validation layer that itself depends on another model’s judgment inherits the same fundamental vulnerability to manipulation that the original output validation was built to catch in the first place. A validation check that verifies a generated number falls within an expected, reasonable range, or that a generated action matches one of a small, explicitly permitted set provides a more reliable backstop than a check that simply asks a second model whether the first model’s output merely looks correct on the surface.

How LLM security connects to the broader practice of AI native security

LLM security is one distinct layer within the broader AI native security practice already discussed elsewhere in this body of work, sitting below the application-level defenses, permissions, sandboxing, tool calling security, prompt injection defense, that a deployment builds around a chosen model. Getting LLM security right doesn’t ever make application-level defense entirely unnecessary, a well-secured model deployed with no application-level scoping still carries risk from an application that grants it more authority than its actual task requires, and getting application-level defense right doesn’t ever compensate for a fundamentally weak underlying model, since application-level architecture simply has no lever to fix a weakness that exists at the model layer itself entirely.

This layered relationship means an organization’s security posture depends entirely on both layers being addressed together, with deliberate attention paid to which team or process owns each layer, since model-level weaknesses often require action from whoever trains or provides the model, while application-level weaknesses require action from whoever builds the system using that model, and an organization where neither team believes the other layer’s risks are its responsibility ends up with the exact gap that a coordinated AI native security practice is specifically meant to close.

How model inversion attacks attempt to reconstruct training data directly from a model

Model inversion is a distinct risk from model extraction, since extraction aims to reconstruct a functionally similar model, while inversion instead aims to reconstruct pieces of the actual training data a model was originally built on, exploiting the fact that a model’s parameters can encode more information about individual training examples than its designers ever intended to expose. An attacker who successfully performs model inversion against a model trained on sensitive data can potentially recover fragments of that original data directly from the model’s behavior, without ever needing any actual access to the original training dataset itself.

This particular risk deserves particular attention for models trained or fine-tuned on sensitive data, personal information, proprietary internal content of any kind, anything an organization would consider a serious problem if it were exposed through the model’s outputs rather than through a conventional data breach. Defending against inversion means deliberately limiting how much of any single training example a model is allowed to memorize during the training process itself, techniques that trade off some raw model performance for reduced memorization, and specifically testing a trained model for actual susceptibility to inversion before deploying it on data whose exposure would carry consequences.

How membership inference attacks reveal whether data was used in training

Membership inference is a subtler variant of the training data risk that inversion represents, aiming not to reconstruct training content itself but merely to determine whether a known piece of data was part of a model’s training set at all, a question that can itself carry sensitivity even without any content ever being reconstructed directly. Confirming that a particular individual’s data was included in a training set, for instance and by way of a concrete example, can itself constitute a privacy violation, regardless of whether any of that individual’s actual information is ever recovered through the attack.

This particular risk matters most for organizations training or fine-tuning models on data where the mere fact of inclusion carries sensitivity of its medical records, legal case data, anything where confirming someone’s data was used would itself represent a disclosure the organization never intended to make. Defending against membership inference overlaps considerably with defending against inversion, since both exploit that exact same underlying tendency of models to behave subtly, meaningfully differently on data they were trained on versus data they’ve honestly never seen before at all, meaning techniques that reduce this behavioral gap defend against both risks simultaneously.

How LLM security should account for the risks introduced by open-weight models

Models whose actual weights are publicly available introduce a distinct category of LLM security consideration entirely of their own, since an attacker with direct access to a model’s weights can perform analysis and manipulation that’s simply impossible against a model only reachable through a query interface, examining the model’s internal structure directly, fine-tuning it toward specifically harmful behavior without any of the safety constraints the original provider built in, or extracting information about its training process that a query-only interface would never reveal. This doesn’t mean open-weight models are inherently less secure in every respect, they offer transparency benefits a closed model simply can’t provide, but it does mean the threat model for an open-weight model differs meaningfully from a closed one, and LLM security practice needs to account for that difference rather than applying identical assumptions to both categories.

Organizations choosing to deploy an open-weight model specifically need to carefully evaluate their actual exposure to this distinct threat model, considering whether the model’s safety training can be meaningfully removed or degraded by anyone with access to its weights, including their internal staff or anyone who might gain unauthorized access to their infrastructure, and whether that risk is acceptable given what the model is being used for within their deployment.

How LLM security should treat a model’s confidence and calibration as a security-relevant property

A model that expresses high confidence in a wrong answer creates a distinct kind of risk from a model that expresses low confidence and is simply, honestly wrong instead, since a downstream system or human relying on that confidence to decide how much scrutiny an output deserves will be systematically misled by a model whose confidence doesn’t track its accuracy. This calibration gap, the difference between how confident a model sounds and how correct it is, deserves explicit attention within LLM security practice, since an attacker specifically aiming to get harmful or manipulated content past whatever review process exists benefits directly from a model that sounds confidently correct regardless of whether it is.

Evaluating a model’s actual calibration, not merely its raw accuracy alone means specifically testing whether its expressed confidence, whether stated explicitly or implied through the sheer fluency and apparent certainty of its language correlates with correctness across a representative, broad range of tasks. A model that’s well calibrated gives downstream systems and human reviewers a useful signal for allocating their scrutiny, while a poorly calibrated model, however capable in raw accuracy terms, actively works against the review processes an organization has built specifically to catch its mistakes.

How LLM security intersects with the risks of multimodal models

Models that process images, audio, or other non-text input directly introduce their distinct category of LLM-level security consideration entirely, since each additional input modality represents another channel through which adversarial content, whether a jailbreak attempt, an adversarial input specifically engineered to manipulate behavior, or an attempt at model inversion, might be delivered. A model’s safety training and its resistance to manipulation, evaluated thoroughly for text input, doesn’t automatically transfer with the same strength to every other modality that model can also process, and an organization that only tests text-based jailbreak resistance while deploying a model that also accepts images has only tested half of its model’s actual attack surface.

This means LLM security for a multimodal model needs its dedicated evaluation performed across every single modality that model accepts as valid input, not merely the modality that happens to be easiest to test or the one an application’s primary use case happens to emphasize most heavily. A model that’s been extensively red-teamed for text-based jailbreaking but never evaluated for image-based manipulation attempts carries a gap in its actual security evaluation, one that a sufficiently motivated attacker specifically targeting the untested modality can potentially exploit precisely because nobody thought to look there first.

How LLM security should shape an organization’s ongoing relationship with its model provider

Choosing a model provider just once, right at the start of a project, and simply treating that relationship as settled, static infrastructure afterward misses the fact that a model provider’s security posture, their training practices, their disclosure practices when a new vulnerability is discovered, continues to matter for the entire lifespan of an organization’s dependence on that model. A provider that discloses a newly discovered jailbreak vulnerability quickly and transparently, along with a remediation timeline, deserves to be treated differently, in terms of the ongoing trust an organization extends to that relationship, than a provider who’s known to sit on disclosed vulnerabilities or to communicate about them only vaguely, well after the fact.

This means LLM security benefits considerably from treating the provider relationship itself as an ongoing, continuous security consideration, not merely a one-time procurement decision made once and then simply forgotten about entirely, actively tracking how a provider responds when new vulnerabilities in their models are discovered, whether by the provider itself, by independent researchers, or by the customer’s internal red-teaming efforts. An organization with visibility into how its provider behaves during a security event is considerably better positioned to make an informed decision about whether to continue relying on that provider, or to invoke whatever rollback and replacement path was built specifically for exactly this kind of situation.

How LLM security testing should evolve alongside the broader field’s discovery of new attack techniques

The techniques used to jailbreak, extract, or otherwise manipulate large language models have continued evolving quite rapidly ever since these systems first became widely, broadly deployed, and a testing regime built around the techniques known at the time a model was first evaluated will fall behind as new techniques are discovered and published by the broader security research community. This means LLM security testing simply can’t be a fixed, static test suite run only once and then considered entirely complete, it needs an actual, ongoing process for incorporating newly discovered attack techniques as they become known over time, testing already-deployed models against them retroactively rather than only ever, narrowly applying new techniques to whatever model happens to be under evaluation next in line.

This particular ongoing evolution also means an organization considerably benefits from staying connected to the broader security research community working on these problems day to day, rather than developing its testing techniques entirely in isolation, since a technique discovered and published by an outside researcher against a different model may quite well generalize to the model an organization has deployed in production, and an organization with no visibility into this broader research landscape will simply never know to test for it until it’s already, quietly been exploited against their live production system.

Common mistakes teams make around LLM security

Several distinct patterns recur often enough across organizations working with large language models that naming them directly is worth doing before they’re discovered through an actual incident.

1. Treating application-level security as sufficient on its own, without any separate evaluation of the underlying model’s raw resistance to jailbreaking or extraction.

2. Assuming every model available on the market carries comparable safety training simply because they’re broadly similar in general capability, without testing candidate models against adversarial scenarios.

3. Providing no rate limiting or query monitoring on model access, leaving a model extraction attempt free to operate at whatever pace an attacker chooses.

4. Building content filters that only recognize suspicious natural language, missing adversarial token sequences that don’t resemble anything a human would consider meaningful language at all.

5. Fine-tuning a model on internal data without confidence in that data’s integrity, leaving a path open for training data poisoning to introduce a hidden, exploitable behavior.

6. Relying on a third-party model provider without any evaluation of that provider’s training practices and supply chain security.

7. Coupling an application’s entire architecture to a single model version with no practiced path for swapping to a different version or model when a security issue surfaces.

8. Discovering, only in the middle of an actual incident, that rolling back or replacing a compromised model is considerably harder and slower than anyone had planned for.

9. Red-teaming only the complete, deployed application without ever separately evaluating the underlying model’s raw behavior in isolation.

10. Letting model selection happen before any red-teaming, rather than letting model-level red-teaming inform which model gets chosen in the first place.

11. Building output validation that itself depends on another model’s judgment, inheriting the same fundamental vulnerability to manipulation the validation was meant to catch.

12. Having no independent, non-model output validation at all, trusting a model’s generated output without any backstop that doesn’t depend on the model having behaved correctly.

13. Assuming LLM security and application-level AI security are the same discipline, missing that they require different actors, different levers, and different ownership.

14. Leaving neither the model provider nor the application team clearly responsible for a category of risk, creating a gap neither side believes is theirs to address.

15. Treating LLM security as a static property evaluated once at model selection, rather than something that needs ongoing attention as new extraction, jailbreak, and adversarial input techniques continue to be discovered.

16. Deploying a model trained or fine-tuned on sensitive data without specifically testing it for susceptibility to model inversion before that data’s exposure through the model becomes a live risk.

17. Overlooking membership inference risk in contexts where the mere fact of an individual’s data being included in training would itself constitute a disclosure, even without any content being reconstructed.

18. Applying the same threat model to open-weight and closed models, missing that direct access to weights introduces a distinct category of manipulation and analysis a query-only interface never allows.

19. Evaluating a model’s raw accuracy without ever testing whether its expressed confidence correlates with correctness, leaving downstream reviewers systematically misled by confidently wrong output.

20. Testing jailbreak resistance only against text input for a model that also accepts images or audio, leaving an entire untested attack surface exactly where a motivated attacker would think to look.

21. Treating a model provider relationship as settled infrastructure after initial procurement, without tracking how that provider responds when new vulnerabilities in their models are discovered.

22. Running a fixed, static test suite built around attack techniques known at initial model evaluation, without any ongoing process for incorporating newly discovered techniques as the broader field continues to evolve.

What connects all twenty-two of these mistakes is a single underlying pattern: treating the model as a trusted, fixed component whose internal security properties don’t need any independent evaluation, rather than recognizing that the model itself carries distinct risk that exists prior to, and entirely independent of whatever application gets built around it. A model’s weaknesses don’t announce themselves through obviously broken application behavior, they surface only when someone specifically, deliberately looks for them at the model layer itself, and an organization that never looks there is trusting a component it has never properly verified at all.

The deeper principle underneath all of this is that a large language model isn’t a fixed, inert piece of infrastructure the way a conventional database or a conventional API might be, it’s a complex, trained artifact whose actual behavior carries its distinct attack surface, shaped directly by its training data, its fine-tuning, and its underlying architecture, and an organization that secures everything around the model while treating the model itself as an unexamined black box has left exactly the foundation everything else is built on entirely, completely unverified and unexamined.