The Problem With "Done" When the Output Is Probabilistic
AI acceptance criteria must specify not just what output the model produces but under what input distribution, at what confidence threshold, measured by which eval metric, and re-evaluated on what cadence — otherwise "done" means different things to every team member reviewing the same pull request. That is the entire problem, stated plainly. Everything below is the operational framework for solving it.
Traditional software acceptance criteria assume determinism. Given input X, the system returns Y. Pass or fail. AI systems don't work that way. A language model that classifies support tickets correctly 91% of the time on Tuesday may drop to 84% by Friday if the incoming message distribution drifts. Neither number is a bug in the classical sense. But one of them might be a release blocker — and without written criteria, nobody agrees on which.
Why Traditional Pass/Fail Acceptance Criteria Fail for Probabilistic Outputs
Deterministic criteria test behavior; probabilistic criteria test distributions. When a QA engineer writes "the search feature returns the correct result," there is a single ground truth to verify. When a product manager writes "the AI summarizer produces accurate summaries," that sentence contains at least four undefined variables: what counts as accurate, accurate across which inputs, measured how, and compared against what baseline.
The failure mode is predictable. Engineering ships a model that hits 88% precision on the internal test set. QA manually reviews ten outputs, finds two they disagree with, and flags a defect. Product escalates because the demo looked clean. No one is wrong — they simply evaluated different things. The sprint closes without resolution, technical debt accumulates in the form of undocumented model behavior, and the next iteration starts with the same ambiguity baked in.
Probabilistic output acceptance criteria have to be written differently from the start. The spec itself must acknowledge that a model is being evaluated, not a function.
The Four Components Every AI Acceptance Criterion Must Contain
Good AI acceptance criteria engineering follows a four-part structure. Every criterion — at the story level, not just the epic — should answer all four.
1. Input Scope
Define the distribution the model is expected to handle. "English-language customer support tickets under 500 tokens, excluding attachments" is a scope. "Customer messages" is not. Input scope tells QA exactly which test cases are in-distribution and which represent edge cases that require a separate acceptance decision.
2. Eval Metric
Name the metric explicitly. Precision, recall, F1, BLEU, ROUGE, mean average precision, exact match, human preference rate — each measures something different and optimizing for the wrong one produces misleading acceptance signals. A document retrieval system optimized for recall will surface more relevant results but also more noise. Whether that trade-off is acceptable is a product decision, not an engineering default.
3. Threshold
State the minimum acceptable score, and state it as a range or floor, not a target. "Precision ≥ 0.87 on the defined input scope, measured against the held-out evaluation set" is a threshold. "The model should be highly accurate" is a wish. The threshold is also where the 20–60% cost advantage of a well-staffed offshore engineering model gets tested in practice — a lower threshold maintained by cheaper labor is not the same product.
4. Re-Evaluation Trigger
Specify when the criterion gets re-tested after initial acceptance. Time-based triggers ("re-evaluate monthly"), data-drift triggers ("re-evaluate when incoming distribution shifts by more than X% on feature Y"), and deployment triggers ("re-evaluate before any prompt or weight update") all serve different risk profiles. Without this component, acceptance is a snapshot, not a contract.
Aligning Product, QA, and Engineering on "Good Enough" Before a Model Ships
The alignment conversation has to happen before story grooming, not during QA sign-off. The single most effective structural change a team can make is a pre-sprint AI criteria review that includes all three roles — product, QA, and engineering — explicitly negotiating threshold and metric before any model work begins.
Product owns the business case for the threshold. If 85% precision is acceptable for a low-stakes categorization task but 97% is required for a medical intake routing model, that is a product decision with downstream engineering cost implications. Product needs to be in the room when the number gets written.
Engineering owns feasibility. A model that hits 97% precision on a biased training set may perform at 79% in production. Engineering needs to surface that gap at criteria-writing time, not at demo time. This is where InWork's model of US CTO oversight on every engagement pays dividends — a senior technical voice in the criteria conversation prevents threshold inflation that no model can realistically achieve.
QA owns test set integrity. The evaluation set used to verify the acceptance threshold must not overlap with training data, must reflect the defined input scope, and must be version-controlled alongside the model artifact. If QA doesn't own that discipline explicitly, the acceptance test is circular.
Document the agreed criteria in the same place as the rest of the story acceptance criteria. Not in a separate spreadsheet. Not in a Confluence page three links deep. In the ticket, where the pull request will be reviewed against it.
Integrating AI Acceptance Criteria Into Sprint Ceremonies Without Slowing Velocity
The concern is legitimate: adding four-component criteria to every AI story sounds like it doubles grooming time. In practice, it front-loads two hours to save eight hours of post-merge ambiguity.
Definition of Done for AI stories should be a team-level standard, not reinvented per ticket. Establish a standing AI sprint definition of done template that pre-fills the framework: input scope, eval metric, threshold, re-eval trigger. Teams fill in values, not structure. Grooming time for a well-understood model type drops to minutes once the template is internalized.
In sprint review, demo the eval results alongside the feature. Show the precision curve, show the confusion matrix, show the drift delta from the prior sprint. Product sees numbers, not vibes. QA has artifacts to archive. Engineering has a defensible ship decision. That is what AI QA criteria product and engineering alignment looks like operationally.
Retrospectives should include one standing AI-specific question: did our acceptance criteria catch the failure mode we actually encountered this sprint? If not, update the template. The criteria framework should evolve with the model.
Fine-Tuned Models vs. Prompted Foundation Models: Different Criteria, Different Risk Profiles
How to write acceptance criteria for AI models diverges meaningfully depending on whether the model is fine-tuned or prompt-driven.
A fine-tuned model's behavior is locked at training time. Its acceptance criteria can be more precise because the input-output relationship is more stable. Threshold drift between sprints is a signal of data contamination or distribution shift, not expected variance. Re-evaluation triggers are typically data-drift-based or deployment-based.
A prompted foundation model's behavior is coupled to the prompt and the underlying model version. Every prompt change is effectively a model change and requires re-validation against the acceptance threshold. Every provider model update — a new GPT version, a new Claude release — is a potential regression event. Re-evaluation triggers must include provider update events, and the acceptance criteria should specify which foundation model version the threshold was validated against.
This distinction matters at the contractual level too. If a client expects consistent behavior and the underlying foundation model updates silently, the acceptance criteria are the only artifact that documents whether the new behavior still meets the original definition of done.
For teams operating under SOC2-aligned practices, HIPAA-aware architectures with BAA available, or GDPR-aware architecture requirements, the re-evaluation cadence also intersects with compliance review cycles — another reason the trigger component of acceptance criteria is not optional.
Writing Criteria That Outlast the Sprint
The teams that ship reliable AI products aren't the ones with the most sophisticated models. They're the ones with the most disciplined definitions of what reliable means. AI acceptance criteria engineering is ultimately a communication discipline — it forces every stakeholder to agree on the contract before the code is written, and it gives that contract enough specificity to be enforced when the PR lands.
The framework above is not complicated. Four components, written in the ticket, reviewed by all three roles before grooming closes. The difficulty is cultural: it requires product to commit to a number, engineering to surface feasibility constraints early, and QA to own test set integrity as a first-class artifact. Teams that build that discipline find that their AI sprint definition of done becomes a competitive differentiator — not because the framework is novel, but because most teams still haven't written it down.