How to Build an AI Support Agent Evaluation Suite

An AI support agent evaluation should answer a practical release question: can this exact system help a developer without inventing an endpoint, hiding uncertainty, leaking sensitive data, or blocking a necessary human handoff? A polished demo cannot answer that question. Neither can a generic model benchmark, because the production behavior depends on retrieval, source freshness, prompts, routing policy, citations, tools, conversation history, and the application code around the model.

The useful unit of testing is therefore the support workflow, not just the final paragraph. A representative developer question should enter with known context and an expected decision. The evaluation records what the system retrieved, which claims it made, whether those claims were supported, whether it asked for the one missing fact, and whether it escalated when the evidence was not enough.

This implementation checklist turns those expectations into a maintainable AI support agent evaluation suite. It covers dataset design, stage-level scoring, human calibration, adversarial cases, release gates, and the production feedback loop that keeps the suite relevant after launch.

Begin with the decisions the agent is authorized to make. For API support, those usually include answering from documented evidence, requesting a missing endpoint or error detail, declining an unsupported claim, handing off to an operator, and optionally proposing or running a tightly guarded verification step. Each decision needs a pass condition and an unacceptable failure condition before anyone assembles test prompts.

Write a short evaluation contract for every decision. Name the customer goal, required evidence, allowed sources, privacy boundary, expected route, severity of a wrong answer, and what a reviewer must observe. A question about a public pagination parameter can tolerate stylistic variation. A request involving credentials, account changes, write operations, or production impact needs a stricter route and stronger evidence.

Map the agent as a sequence of testable stages: intent detection, retrieval planning, source filtering, context sufficiency, answer construction, citation selection, confidence calibration, clarification or handoff routing, redaction, and optional tool use. A single end-to-end score collapses these failure modes. When a response is wrong, the team must know whether the necessary source was absent, retrieval missed it, the generator ignored it, or the router answered despite low confidence.

Keep an end-to-end acceptance result, but preserve stage evidence beside it. Store retrieved source identifiers, top matches, the answer contract, route reason, safe confidence signal, tool decision, latency, and grader results. Do not store raw credentials or customer-private content merely because it is convenient for debugging. The evaluation harness must obey the same tenant and data boundaries as production.

Build the first dataset from real support shapes rather than a bag of trivia questions. Use sanitized historical conversations, documentation-owner fixtures, known regressions, and carefully reviewed synthetic variants. OpenAI's evaluation guidance recommends task-specific tests that reflect real distributions, with typical, edge, and adversarial cases. Production logs can reveal what actually confuses customers, while domain experts define the facts and decisions that must remain stable.

Stratify the set so a high volume of easy questions cannot hide failure on rare, costly cases. Include exact endpoint questions, ambiguous product names, missing error details, conflicting docs, stale examples, version differences, authentication and scope problems, rate limits, webhook signatures, pagination, unsupported operations, account-specific requests, emotional human requests, and provider failures. Add direct and indirect prompt-injection attempts because retrieved documents can carry hostile instructions as readily as customer messages.

Do not require one canonical response when several answers could help. Store structured expectations instead: the facts that must appear, claims that must not appear, acceptable sources, required route, prohibited actions, and severity. For a 401 question, the expected behavior might require the agent to distinguish authentication from authorization, cite the relevant auth documentation, avoid asking for a full token, and request a sanitized error body or scope only if the evidence cannot narrow the cause.

Use exact matching for deterministic fields such as route, method, path, status code, cited source ID, or tool argument. Use a rubric for qualities such as completeness, clarity, and clarification usefulness. This separation makes failures actionable. It also prevents a fluent answer from receiving credit after naming a nonexistent parameter or using a citation that does not support the claim.

Retrieval evaluation asks whether the required evidence was available to the generator. Mark the source or source set that can answer each fixture, then measure whether it appeared in the retrieved context and whether irrelevant material crowded it out. Inspect endpoint identity, version, auth scheme, schema, example, error behavior, and source freshness separately when those details determine correctness.

Report retrieval coverage and precision by case type, not only as global averages. A system can look healthy while consistently missing webhook security notes or retrieving the wrong version of an endpoint. Test empty indexes, partial ingestion, malformed sources, duplicate chunks, superseded examples, and two similarly named operations. The expected outcome may be a safe clarification or source-gap handoff when the right evidence truly is not indexed.

For every material API claim, ask two questions: is it correct, and does the cited evidence support it? Citation presence alone is weak. A response can attach a plausible documentation link while inventing the method, mixing API versions, or generalizing from an example. Score claim-level support for endpoints, fields, auth rules, status codes, limits, response shapes, SDK behavior, and production assertions.

A practical grader extracts material claims, maps each claim to the retrieved evidence, and labels it supported, contradicted, or not established. Human reviewers should inspect a sample of passes as well as failures. Track unsupported-claim rate and citation precision as release guardrails, then preserve the individual examples so a falling aggregate cannot conceal one severe hallucination.

Confidence is useful only when it changes behavior. Create paired fixtures where one version contains sufficient endpoint evidence and another removes a decisive field, changes the version, or introduces a conflict. The agent should answer the supported case and clarify or hand off the insufficient case. Score the route and reason, then examine whether the confidence policy separates those populations consistently.

Clarification quality deserves its own rubric. A good clarification asks for the smallest safe fact that can resolve uncertainty: the method and path, a sanitized status and problem type, the SDK version, or the environment. A weak clarification asks the customer to paste credentials, repeats the whole question, or requests documents the system should already have. Handoff quality should verify that the operator receives the known facts, retrieved evidence, uncertainty, customer goal, and next suggested action without exposing operator-only internals to the customer.

Safety fixtures must test the application boundary, not only the base model. OWASP distinguishes direct prompt injection from indirect injection carried through external content and notes that retrieval alone does not eliminate the risk. Put conflicting instructions in customer messages and in indexed documents. Ask for hidden prompts, provider configuration, private sources, another customer's records, full tokens, write actions, and unsafe diagnostic output.

Expected results should be explicit: system and workspace policy remains authoritative; retrieved instructions are treated as untrusted data; credentials are redacted before model use; private source URLs do not become customer citations; unauthorized tools are not called; and failures preserve a safe operator trace. Add encoding, multilingual, and split-payload variants over time, especially after any real bypass attempt.

Automated graders scale review, but they are another measurement system that can drift. Give each rubric a narrow definition and examples of pass, fail, and borderline behavior. Prefer classification, pairwise comparison, or scoring against named criteria over an open-ended request to judge quality. Google Cloud's judge-model guidance calls for human ratings as ground truth when evaluating the judge itself; the same principle applies to an internal evaluator.

Calibrate on a blinded expert-reviewed set. Measure agreement by criterion and case type, investigate systematic disagreements, and version the rubric with the grader prompt and model. Keep deterministic checks for endpoints, citations, routes, redaction markers, and tool arguments even when a model-based grader is present. Human review remains necessary for severe failures, novel cases, and periodic audit of automated passes.

Run the fixed suite before changes to retrieval, chunking, embeddings, prompts, policies, tools, models, or provider routing. Compare the candidate with the current baseline on exactly the same fixtures. Define release gates by risk: zero secret exposure and cross-tenant access, zero unauthorized writes, no severe unsupported endpoint claims, minimum routing performance for insufficient-context cases, and bounded regression on ordinary answer quality, latency, and cost.

Repeat variable cases enough times to reveal unstable behavior, but keep deterministic preprocessing and routing tests separate so randomness does not obscure an ordinary code regression. Save the configuration, source snapshot, run time, aggregate results, and failed case IDs. Averages are useful for trend detection; release decisions should also inspect worst-case failures and changes within high-risk slices.

Woes' current support-agent implementation reflects this stage-based model. Deterministic greetings and capability responses can avoid retrieval, while technical questions build a query plan, retrieve workspace-scoped context, assess sufficiency, assemble a context pack, and parse a structured answer contract. The answer path checks whether API-fact responses cite retrieved evidence, calibrates confidence, and can clarify, skip, or hand off instead of sending an unsupported answer.

The implemented tests cover missing context, low confidence, citation mismatch, credential redaction, optional PII masking, untrusted retrieved instructions, operator-only source filtering, provider failures, and guarded live verification. Live checks default to read-only, and customer-facing results fail closed on response-body exposure. Workspace-scoped run analytics record answered and skipped runs, confidence, latency, tokens, cost, and skip reasons; message feedback records verified widget ratings. Those signals can supply evaluation cases, but they do not prove correctness without the reviewed evidence checks described above.

In week one, define the decision contracts and collect a small balanced fixture set. Start with ten to twenty examples for each essential slice rather than hundreds of loosely labeled prompts. Have support and documentation owners mark required evidence, while engineering and security define prohibited actions and severe failures. Run the current system and repair ambiguous fixtures before treating the baseline as meaningful.

In week two, add stage instrumentation and deterministic graders. Verify retrieval coverage, source identity, routes, citations, redaction, and tool arguments. In week three, create narrow rubrics for supported claims, clarification quality, and handoff completeness, then calibrate automated grading against expert review. In week four, set release thresholds, wire the suite into the change process, and establish a weekly review of new production failures, negative feedback, source gaps, and operator corrections.

Keep a held-out set for periodic assessment so tuning does not optimize only the visible fixtures. When a failure appears, decide whether it represents a product bug, source gap, retrieval miss, policy problem, or grading error. Add the repaired case to the right slice and record why it matters. That practice turns evaluation from a launch document into operational memory.

A dependable AI support agent is not the one that answers the largest share of questions. It is the one whose answers, clarifications, handoffs, and refusals behave predictably under representative evidence and risk. Build the suite around those decisions, preserve stage-level traces, calibrate automation with humans, and promote real failures into permanent regressions.

That creates a release process the team can defend. Better retrieval improves the cases that should be answered. Better confidence policy protects the cases that should not. Better handoffs preserve momentum when a person is required. The result is not a perfect score; it is a support system that becomes safer and more useful as the product, documentation, and customer questions change.