Skip to content

Quality — Agent Service Quality Assessment ​

Quantitative assessment of AI Agent output quality and task completion reliability. Establishes measurable quality thresholds, release gates, and continuous improvement loops for enterprise deployment decisions.


Why Quality? ​

Inspect finds defects in Logic Artifacts. Risk validates whether those defects form real attack surfaces. But neither answers the question that matters most for go/no-go decisions:

Does this Agent consistently produce outputs that meet business requirements?

Quality fills this gap with two complementary assessment methodologies tailored to the two dominant Agent types in enterprise deployment:

  • Tool Agents execute deterministic tasks — ticket dispatch, payment processing, order management — where outputs are binary (success/failure). They need a single reliability metric with risk-driven test rigor.
  • RAG Agents answer knowledge-based questions — customer support, policy lookup, technical guidance — where outputs are natural language and quality is multi-dimensional. They need a structured, metric-graded assessment across multiple quality dimensions.

Quality provides standardized, repeatable, auditable measurement — so release decisions are based on evidence, not intuition.


Two Sub-Specifications ​

  • Quality RAG-Agent ​

    Multi-dimensional quality assessment for knowledge-based RAG Agents.

    Evaluates Agent outputs across 4 dimensions and 12 metrics: Knowledge Answer Quality (Correctness, Completeness, Relevance, Traceability, Timeliness), Dialogue and Complex Task Quality (Consistency, Robustness, Decomposition Capability, Coherence), Expression and Compliance Quality (Format Compliance), and Boundary and Safety Response Quality (Boundary Recognition, Safe Response). Uses Baseline Test Cases, Controlled Blind Testing, and a three-layer Release Gate.

    Assessment types: Rule Verification, LLM-as-Judge, Embedding Assistance

  • Quality Tool-Agent ​

    Reliability-focused assessment for deterministic Tool Agents.

    Measures reliability as a single core metric — the probability of successful task completion across multiple independent executions. Applies risk-driven test rigor with differentiated test counts and pass rate thresholds per L1/L2/L3 risk classification. Enforces a Ratchet Mechanism that binds version history to quality scores, preventing regression.

    Core principle: binary outputs demand statistical measurement, not continuous scoring


RAG-Agent: 4 Dimensions × 12 Metrics ​

CategoryMetricsCount
A — Knowledge Answer QualityCorrectness, Completeness, Relevance, Traceability, Timeliness5
B — Dialogue and Complex Task QualityConsistency, Robustness, Decomposition Capability, Coherence4
C — Expression and Compliance QualityFormat Compliance1
D — Boundary and Safety Response QualityBoundary Recognition Capability, Safe Response Capability2

Gate structure (three independent layers — no single score can mask a gate failure):

LayerWhat it guards
Critical Test Case GateMust-pass facts, rules, and high-risk boundary scenarios
Metric and Dimension GatePer-dimension minimum thresholds
Regression Degradation GateNo version-to-version quality regression

Tool-Agent: Risk-Driven Reliability Assessment ​

Risk LevelTypical ImpactTest CountPass Rate Threshold
L1Low — internal tooling, non-critical workflows3090%
L2Medium — customer-facing, business operations5095%
L3High — financial, compliance, security-critical100100%

Assessment workflow: Risk Classification → Parameter Configuration → Mock Data Generation → Shadow Sandbox Execution → Reliability Calculation → Gate Determination → Result Output


Position Within the SanityOps Framework ​

Inspect (Defect Inspection) ── finds the defects
    │
    ▼
Risk (Risk Scanning) ── validates the attack surfaces
    │
    ▼
Quality (Service Quality) ── You are here
    │   ├─ RAG-Agent: measures answer quality across 12 metrics
    │   └─ Tool-Agent: measures task reliability as a single score
    │
    ▼
Relevance (Relevance Assessment)

Quality is the final gate before deployment. It answers: given that defects are known and risks are validated, does this Agent actually perform well enough to ship? Output is a quantified, auditable, version-bound assessment — not a subjective impression.


Quick Start ​

  1. Complete Inspect and Risk first — Quality assessment is most effective when defect and risk context is available
  2. Identify your Agent type: Tool Agent (deterministic tasks) or RAG Agent (knowledge Q&A)
  3. For Tool Agents: follow Quality Tool-Agent to configure risk level, test counts, and pass rate thresholds
  4. For RAG Agents: follow Quality RAG-Agent to establish Baseline Test Cases and run the 12-metric assessment
  5. Feed results to Relevance for quality failure → defect diagnostic mapping