AI QA extends conventional QA to systems whose outputs can be probabilistic, context-dependent, and sensitive to data, prompts, and model configuration. Engineering discipline is especially important here: testing cannot be replaced by a subjective feeling that an answer "seems fine."
How AI QA differs from conventional QA
- →Model responses may be nondeterministic: identical requests do not always produce identical results.
- →Quality depends on data, prompts, retrieval, guardrails, and model configuration as well as code.
- →Assess safety, robustness, cost, and behavioral degradation alongside correctness.
- →Some checks take the form of evaluations rather than conventional binary pass/fail tests.
The foundations needed for AI QA
- →A strong conventional QA foundation: risks, test design, requirements, APIs, and systems thinking.
- →Understanding of HTTP/APIs, logging, data, and integrations, because most AI products are built around services.
- →The ability to define quality criteria when outcomes are not perfectly binary.
- →A basic understanding of LLMs, embeddings, RAG, prompt engineering, and model limitations.
What to test in AI systems
- →Correctness and relevance: how well an answer matches the task and context.
- →Safety: toxicity, data leakage, jailbreaks, unsafe recommendations, and policy violations.
- →Robustness: behavior with noisy, incomplete, adversarial, and edge-case requests.
- →Latency and cost: quality must not come at an unacceptable speed or expense.
- →Observability: whether you can understand why the system produced a particular answer and where a failure occurred.
Useful AI QA artifacts
- →An evaluation dataset with explicit quality criteria rather than only impressive demo prompts.
- →A risk matrix covering safety, privacy, hallucinations, and business-critical errors.
- →A change log for prompts, models, and retrieval configuration.
- →Golden scenarios for smoke and regression checks after every change to the chain.
Common mistakes
- →Evaluating only a handful of happy-path prompts and treating them as proof of quality.
- →Failing to separate checks of the model, orchestration logic, and external data.
- →Ignoring cost and latency while focusing only on an answer's apparent intelligence.
- →Relying on subjective impressions instead of agreed evaluation criteria.