What is AI native software?

Summary slides

“AI native software” sits as the single broadest term in this knowledge base’s overall coverage of the AI native concept — systems, applications, platforms, architecture, and the underlying stack are all more lenses on that same underlying phenomenon, while “AI native software” names the underlying phenomenon itself and, particularly, the actual practice of building it well. This article focuses specifically and deliberately on that practice: what changes about the day-to-day discipline of software engineering when the software being built is AI native, as distinct from the more structural and product-level concerns the other, closely related articles in this section already cover.

Why traditional software engineering practice doesn’t transfer cleanly

Decades of software engineering practice — version control, automated testing, code review, continuous integration, incident response — were built around a foundational assumption that no longer fully holds for AI native software: that a system’s behavior is deterministic, meaning the same input reliably produces the same output, and that behavior can therefore be fully specified, fully tested, and fully verified as either correct or incorrect. Traditional software engineering’s entire toolkit is built to exploit that determinism — a unit test asserts an exact expected output because a deterministic function will produce it every time; a code review can reason precisely about what a change will do because the change’s effect is fully determined by the code itself.

AI native software breaks this assumption at its core, because the model components at the center of these systems are probabilistic rather than deterministic — the same input can produce meaningfully different outputs across different requests, and a system’s behavior depends not just on its own code but on a model whose exact internal behavior isn’t fully specified or fully controllable by the team building around it. This isn’t a minor wrinkle a team can simply work around with a few extra test cases bolted onto an otherwise unchanged process — it requires different practices across nearly every stage of the software development lifecycle, which is exactly what the rest of this article covers, one stage at a time, in turn.

How requirements and specification change when behavior can’t be fully pinned down

Traditional software requirements are typically expressed as precise specifications: this input, produce exactly this output. AI native software requirements need a different shape, because a model’s output for an input isn’t a single fixed value to specify exactly, but a distribution of plausible outputs, some acceptable and some not. Requirements for AI native software are better expressed as intent and acceptance criteria — what outcome the software needs to achieve, and what range of outputs would satisfy that outcome — rather than as an exact specification of the output itself.

This shift has tangible practical consequences for how a team writes and reviews requirements from day to day. A traditional requirement like “return the customer’s account balance” has one correct answer, fully specified by the underlying data. An AI native requirement like “summarize the customer’s account history for a support agent” doesn’t have one correct answer — it has a range of acceptable summaries and a range of unacceptable ones, and the requirement needs to specify what separates the two, typically through concrete examples of good and bad output, rather than through a single expected result the way a traditional specification would. Teams that write AI native requirements the traditional way, as a single exact expected output, tend to produce requirements that are either impossible to satisfy precisely or that quietly narrow the software’s actual usefulness by over-specifying a single acceptable phrasing when many different phrasings would have served the underlying intent equally well.

How testing shifts from binary correctness to statistical evaluation

Traditional software testing is built around binary assertions: a test passes or fails, and a passing test suite is generally taken as reasonably strong evidence the software behaves correctly. AI native software testing, echoing the evaluation-and-guardrail pattern discussed in the related article on AI native design patterns, needs a fundamentally different approach, because there’s no single correct output to assert against for most open-ended AI-driven behavior, and a single test case passing or failing tells a team very little about how the system will behave across the full range of inputs it will encounter in production.

AI native testing instead relies on evaluation against a representative set of cases, measuring an aggregate rate of acceptable versus unacceptable output across that set rather than a binary pass or fail for the system as a whole. This shift changes what “good test coverage” even means: rather than covering every code path with an exact expected output, good coverage for AI native software means having a representative, sufficiently large, and periodically refreshed set of realistic cases that reflects the variety of inputs the system needs to handle well, weighted toward the cases that matter most for the system’s actual stakes. A small, narrow, unrepresentative evaluation set gives a team false confidence in exactly the same way a traditional test suite with poor coverage does, but the failure mode is subtler, because a small AI evaluation set can show a comfortably high pass rate while still missing entire categories of real-world input the system handles poorly.

This testing shift also changes what a “regression” means. A traditional regression is usually sharp and unambiguous — a change that used to work now produces a clearly wrong result. An AI native regression is often a shift in the aggregate rate of acceptable output across an evaluation set, sometimes without any single dramatic failure a team would notice from a quick manual check — which is exactly why continuous, automated evaluation, run on every meaningful change to prompts, retrieval logic, or the underlying model, matters considerably more for AI native software than manual spot-checking ever could, since a small but real degradation is easy to miss entirely without a systematic measurement catching it.

How version control and change tracking need to expand beyond code

Traditional version control tracks code, and for traditional software, code is essentially the complete determinant of behavior — two deployments running the identical code will behave identically. AI native software’s behavior depends on at least three things beyond the code itself: the prompts and instructions to a model, the data retrieved and included in a request, and the version of the underlying model being called, any one of which can change a system’s real-world behavior even when the code orchestrating them hasn’t changed at all.

Building AI native software well means extending version control discipline to cover all three of these dimensions, not just the code. Prompts need to be versioned and reviewed with the same rigor as code changes, since a small wording change to a prompt can shift behavior as significantly as a meaningful code change would. The data a system’s retrieval layer draws on needs its own change tracking, so a team can trace a behavior shift back to a data update if one turns out to be the actual cause. And model version changes — whether an explicit upgrade a team chooses deliberately or a provider’s silent update to a model behind an API — need to be tracked and, ideally, evaluated deliberately before being treated as safe, echoing the versioning and change management discussion in the related article on AI native architecture. Teams that only version their code, treating prompts and model versions as informal, untracked configuration, routinely struggle to explain why a system’s behavior changed on a day, because the actual cause of the change often lies outside the one dimension they were tracking.

How deployment and rollback practices need to account for external model changes

Traditional software deployment assumes a team controls, or at minimum fully observes, every change that could affect a system’s behavior — a deployment is a discrete, team-initiated event, and rolling back a bad deployment reliably returns the system to its previous, known-good behavior. AI native software introduces a new complication: a model provider can update the underlying model behind an API at a time and in a way the adopting team doesn’t fully control, which means a system’s behavior can shift without any deployment from the team’s side at all.

This has practical implications for how AI native software teams need to operate. Monitoring needs to watch not just for the team’s deployments but for unexplained behavior shifts that might trace back to an external model update, which means the continuous evaluation infrastructure discussed earlier needs to run routinely, not just triggered by the team’s deployment events, since a meaningful behavior shift can occur on a day with no team-initiated deployment at all. Rollback practices need an answer for the case where a regression traces back to an external model change rather than the team’s code — sometimes this means pinning to a stable model version rather than automatically receiving a provider’s latest update, trading some access to ongoing improvement for more predictable, controllable behavior, a tradeoff worth making deliberately rather than accepting by default simply because automatic updates are the path of least resistance.

How code review practices need to expand to cover prompts and evaluation criteria

Traditional code review focuses on code: does the logic do what it’s meant to do, is it maintainable, does it introduce risk. AI native software needs code review to expand its scope to cover artifacts a traditional review process was never built to evaluate — prompts, whose exact wording can shift model behavior in ways that aren’t always obvious from reading the prompt alone, and evaluation criteria, whose quality directly determines whether the team’s testing process will catch problems before they reach production.

Reviewing a prompt change well requires a different kind of scrutiny than reviewing a code change — rather than reasoning deterministically about what a change will do, as a traditional code reviewer can for most traditional logic, a reviewer of a prompt change often needs to run the change against a representative evaluation set and review the resulting output, since the effect of a prompt change frequently isn’t obvious from reading the prompt text alone the way the effect of a traditional code change usually is from reading a diff. Reviewing evaluation criteria well requires asking whether the criteria capture what “good” means for the task at hand, since a technically well-implemented evaluation system built around the wrong criteria gives a team confident, well-measured, but ultimately misleading assurance about quality. Teams that treat prompts and evaluation criteria as informal artifacts outside their normal review discipline, reviewed casually or not at all compared to the rigor applied to code, tend to accumulate exactly the kind of undetected quality erosion the evaluation-and-guardrail pattern discussed elsewhere in this knowledge base is meant to prevent.

How team composition and required skills broaden beyond traditional software engineering

Traditional software teams are built primarily around software engineering skill — the ability to design, write, test, and maintain deterministic code. AI native software teams need that skill too, but also need skills traditional software teams frequently didn’t require at all: data curation and management expertise, since the data and knowledge layer discussed in the related article on AI native architecture disproportionately determines a system’s real-world quality; evaluation design expertise, since building a representative, well-structured evaluation set is its own specialized skill distinct from traditional test writing; and domain expertise specific to whatever the software does, since defining what “good” output looks like for a task — the acceptance criteria discussed earlier in this article — usually requires subject-matter knowledge a general-purpose software engineer doesn’t automatically have.

This broadening of required skill doesn’t necessarily mean every AI native software team needs to hire entirely new specialists for each of these areas — smaller teams often distribute this expertise across existing roles, with a software engineer developing evaluation-design skill over time, or a product manager with strong domain knowledge taking direct ownership of defining acceptance criteria rather than leaving that responsibility to engineers without the relevant domain background. What matters is that the expertise exists and is applied somewhere on the team, rather than assuming traditional software engineering skill alone is sufficient to build AI native software well, an assumption that tends to produce software that’s technically well-engineered in the traditional sense while still falling short on the specifically AI-native dimensions of quality this article has covered.

How documentation practices need to capture behavior that isn’t fully determined by code

Traditional software documentation can, in principle, be fully derived from the code itself, since the code fully determines the software’s behavior — good documentation is a convenience that saves a reader from tracing through the code directly, not a separate source of truth. AI native software documentation needs to serve a somewhat different function, because a meaningful share of the software’s actual behavior isn’t determined by code that a reader could trace through at all, but by a model’s learned behavior, which isn’t directly inspectable the way traditional code logic is.

Useful documentation for AI native software tends to include things traditional documentation rarely needed: a record of what evaluation criteria the system is held to and why those criteria were chosen, a record of known failure modes and edge cases the system doesn’t handle well, discovered through evaluation and production experience rather than derivable from reading the code, and a record of the model version, prompt version, and data version a deployment depends on, given how much behavior those three dimensions determine as discussed earlier in this article. Teams that document AI native software the traditional way — describing the code’s structure and API surface, without capturing the behavior and limitations that live outside the code itself — tend to leave future maintainers, including their future selves, without the information needed to safely modify or extend the system, since the code alone no longer tells the whole story of how the system behaves.

How incident response and debugging differ when a failure isn’t a clean bug

Traditional software incident response is built around a comparatively clean investigative process: a failure occurred, the failure has a traceable cause somewhere in the code or infrastructure, and finding that cause usually means reading logs, reproducing the failure, and tracing execution until the responsible line of code or configuration is identified. AI native software breaks this clean process in a recurring way: a failure — a wrong answer, an inappropriate action, a hallucinated fact — often doesn’t have a single traceable line of code responsible for it in the way a traditional bug does, because the failure emerged from the model’s reasoning over a combination of prompt, retrieved context, and input that happened to produce a bad result, without there necessarily being an identifiable bug in the traditional sense to find and fix.

This changes what debugging AI native software looks like in practice. Rather than searching for a single root cause the way a traditional debugging process would, an AI native incident investigation more often needs to ask a different set of questions: was the retrieved context sufficient and relevant for this request, or did the retrieval layer surface something misleading or incomplete; was the prompt’s phrasing ambiguous or misleading in a way that a slightly different wording would have avoided; is this failure an isolated, rare edge case, or does it represent a broader pattern that a properly constructed evaluation set should have caught before the change reached production at all. Answering these questions well depends heavily on the observability infrastructure discussed in the related article on AI native architecture — without a structured record of exactly what was retrieved, what was sent to the model, and what the model returned for the failing request, an AI native incident investigation has very little to work with beyond guesswork, in a way a traditional incident investigation, with its more directly traceable execution path, generally doesn’t struggle with nearly as much.

A further complication specific to AI native incident response is that the same investigation, repeated later against the same input, may not reproduce the same failure at all, given the underlying non-determinism discussed throughout this article — a traditional bug can usually be reliably reproduced by replaying the same input, but an AI native failure sometimes can’t be reproduced this cleanly, which means an AI native incident response process needs to rely more heavily on the captured record of what happened during the original failure than on the ability to reliably reproduce it after the fact, a meaningfully different investigative posture than traditional software incident response typically requires.

How estimating effort and scoping work differs for AI native software projects

Traditional software estimation, however imperfect in practice, is built around a reasonably well-understood process: break a feature into discrete pieces of deterministic logic, estimate the effort each piece requires, and sum the estimates into a project timeline. AI native software estimation runs into a complication this traditional process doesn’t anticipate: a meaningful share of the actual effort in building AI native software well doesn’t come from writing deterministic logic at all, but from the comparatively unpredictable, iterative work of getting a model’s behavior to meet the acceptance criteria discussed earlier in this article — work whose duration depends heavily on how well-suited the underlying model is to the task, something a team frequently can’t know with confidence until they’ve already invested effort trying.

This unpredictability tends to catch teams new to AI native software off guard specifically because the code-writing portion of a project — building the orchestration logic, the data pipeline, the interface — often estimates reasonably well using traditional methods, creating a misleading sense that the whole project is similarly estimable, right up until the team discovers that the harder, less predictable work of achieving acceptable model behavior on their task takes considerably longer, or reveals that the task needs to be scoped down, in ways the traditional code-focused estimate never accounted for. Teams that estimate AI native software projects well tend to separate these two kinds of work explicitly in their planning — estimating the more traditional, deterministic engineering work with traditional methods, while treating the model-behavior work as a more uncertain, iterative effort deserving its separate timeline, explicit checkpoints for reassessing scope based on what early evaluation results show, and enough built-in flexibility that discovering the task needs to be narrowed doesn’t blow up the entire project’s schedule the way an unanticipated discovery like that would for a more rigidly, traditionally estimated plan.

How security practice needs to expand to cover model-risks

Traditional software security practice is built around a well-established set of concerns: input validation, access control, protecting against injection attacks that exploit a system’s logic against it, and securing data in storage and in transit. AI native software needs all of this traditional discipline, and additionally needs to account for a category of risk traditional security practice was never built to address: risks specific to a model reasoning over untrusted input, where the very flexibility that makes a model useful also creates new ways for that flexibility to be misused.

The most significant of these is a category of attack where crafted input attempts to manipulate a model into ignoring its intended instructions or constraints, producing behavior the system’s designers never intended — a fundamentally different kind of security vulnerability than a traditional injection attack, since it exploits the model’s reasoning rather than a flaw in surrounding code, and can’t be fully closed off with the kind of input sanitization that reliably defeats traditional injection attacks. Defending against this category of risk, discussed in the guardrail and safety tooling covered in the related article on AI native stack, requires layered defenses rather than any single fix — constraining what actions a model is permitted to take regardless of what it’s been convinced to attempt, validating a model’s proposed actions against expected boundaries before executing them, and treating any content a model processes from an untrusted source, including content retrieved from a system’s knowledge base if that knowledge base can be modified by untrusted users, as a potential vector for this kind of manipulation rather than as inherently safe simply because it originated from within the system’s infrastructure.

A further security consideration specific to AI native software is data exposure through model output itself — a model that has access to sensitive information as part of its retrieved context can inadvertently reveal that information in a response to a user who shouldn’t have access to it, even without any traditional access-control failure occurring anywhere in the surrounding code, simply because the model’s reasoning combined information in a way nobody explicitly authorized. Defending against this requires access control to be enforced at the data and retrieval layer itself, ensuring a model is never given access to information a user shouldn’t see in the first place, rather than relying on the model’s judgment to withhold information it technically has access to but shouldn’t disclose — a distinction worth being precise about, since the second approach depends on the model behaving exactly as intended in every case, which is precisely the kind of guarantee non-deterministic software can’t reliably provide.

How cost engineering becomes a software engineering concern rather than a purely operational one

Traditional software engineering treats infrastructure cost largely as an operational concern, addressed after the fact through infrastructure scaling and provisioning decisions rather than through the code itself, since the marginal cost of an individual traditional software operation is typically negligible enough not to warrant engineering-level attention. AI native software breaks this separation, because the cost of a single request — a model call, a retrieval operation — is often significant enough, and variable enough depending on how the request is engineered, that cost becomes a software engineering concern properly addressed in the code and design itself, not merely an operational afterthought handled downstream through infrastructure provisioning.

This shows up concretely in decisions that look like ordinary engineering choices but carry measurable cost implications specific to AI native software: how much context gets included in a model call, echoing the progressive context-loading pattern discussed in the related article on AI native design patterns; which model handles a task, echoing the multi-model routing discussed in the related article on AI native stack; and how many separate model calls an orchestrated workflow triggers, echoing the earlier discussion of not over-applying the orchestrator-worker pattern beyond what a task’s complexity requires. Engineering these decisions well requires the same kind of measurement discipline discussed throughout this article applied specifically to cost — tracking what a code path or design decision costs in production, not just whether it produces acceptable output — treating cost as a measured quality dimension of AI native software rather than a downstream operational number a team only discovers after a change has already shipped.

Common mistakes teams make when applying traditional software practice to AI native software

The single most common mistake, observed across teams at nearly every level of experience, is applying traditional binary testing discipline directly to AI native software without adapting it to the statistical evaluation approach discussed earlier — writing a small number of exact-match test cases and treating a passing test suite as meaningful evidence of quality, the same way it would be for traditional deterministic code. This mistake tends to produce a false sense of security, since a small set of exact-match tests can pass consistently while the system’s aggregate real-world quality, measured properly against a representative evaluation set, is considerably worse than the passing tests would suggest.

A second mistake, closely related to the first but with its distinct root cause, is under-tracking the non-code dimensions of AI native software — prompts, data, and model versions — treating them as informal configuration rather than first-class, versioned artifacts deserving the same discipline applied to code. Teams that make this mistake routinely lose the ability to explain why a system’s behavior changed on a day, because the actual cause frequently lies in one of these under-tracked dimensions rather than in the code changes their version control system was capturing, and reconstructing what changed after the fact, without that tracking already in place, becomes a difficult forensic exercise rather than a quick lookup.

A third mistake, more organizational than technical but no less consequential for the software that gets built, is assuming traditional software engineering skill alone is sufficient to build AI native software well, without deliberately building or acquiring the data curation, evaluation design, and domain expertise discussed earlier in this article. Teams making this mistake often produce software that’s well-engineered by traditional standards — clean code, good architecture, reliable infrastructure — while still underperforming on the specifically AI-native dimensions of quality that traditional software engineering skill alone was never built to address, a gap that’s especially easy to miss internally precisely because everything the team knows how to measure well looks healthy.

A fourth mistake, one that tends to surface only after an incident forces the issue, is applying traditional security practice without accounting for the model-risks discussed above, treating input validation and access control as fully solved by the same techniques that reliably protect traditional software, without recognizing that a model’s reasoning introduces attack surfaces those traditional techniques were never designed to close. Teams making this mistake often have strong traditional security practice everywhere except the places where a model processes untrusted input or has access to sensitive retrieved context, which is precisely where the actual risk in an AI native system tends to concentrate.

A fifth mistake, subtler than most of the others but no less costly over time, is treating cost as a purely operational concern to be addressed only after launch through infrastructure scaling, rather than as an engineering decision made throughout the design and implementation of the software itself, as discussed above. Teams making this mistake often discover, only once a system is already in production and serving volume, that design choices made early — how much context gets included in each call, how many model calls a workflow triggers — are driving disproportionate cost relative to the value they contribute, at which point correcting those choices requires revisiting code that’s already shipped rather than having been engineered with cost as a first-class consideration from the start.

A sixth mistake, tied closely to the incident response discussion above and often the last of these lessons a team internalizes, is treating AI native debugging exactly the same way traditional debugging is treated — assuming every failure has a single, traceable root cause waiting to be found through the same kind of methodical code tracing that reliably resolves traditional bugs — and becoming stuck or frustrated when that process doesn’t yield a clean answer the way it typically would for deterministic software. Teams that instead build the observability and structured investigation practice discussed earlier, tailored specifically and deliberately to the non-deterministic, context-dependent nature of AI native failures, tend to resolve incidents considerably faster and with a more accurate, complete understanding of what happened than teams still reaching, out of habit, for a purely traditional debugging mental model.

What ultimately connects all six of these mistakes, taken together as a whole, is applying a mental model built for deterministic software directly onto software whose core behavior is non-deterministic, without recognizing that this difference isn’t a minor wrinkle requiring a few adjustments, but a foundational shift that touches requirements, testing, version control, deployment, review, team composition, documentation, security, cost engineering, and incident response, all at once, and all in ways that reinforce and compound with each other rather than staying neatly separate. Teams that recognize this shift explicitly and early, and adapt each of these practices deliberately rather than simply assuming traditional software engineering discipline transfers unchanged into this new context, tend to build AI native software that’s reliable in ways that traditional software engineering rigor, applied without this deliberate adaptation, consistently and predictably fails to achieve over time.