Skip to content

SanityOps Framework ​

Inspect Prompt Specification ​


Version: v1.1

Release Date: October 2026

Maintained by: Sanity AI Labs

License: CC BY-SA 4.0

Dependent Specification: Core v1.1 (objects, Finding, AC-L, versioning and evidence)


Table of Contents ​


Foreword ​

Document design principle: serving both inspection and audit needs.

To reduce the risk of missed findings and to preserve traceable audit evidence, this document allows the same object to be hit by multiple inspection items across different dimensions — this may look like "similarity / duplication / subsumption" on the surface, but it is intentional by design, not redundancy.

At the quantitative scoring stage, however, to avoid double deductions and to reflect the true quality of the inspected Logic Artifact objectively, same-source deduplication and merging must follow the "Multi-Defect Handling Rules" (§4.6) of this document: one root cause and one harm path are deducted only once.

If you question whether certain inspection items are "overly redundant", verify consistency with counter-examples and edge cases — confirm whether the trigger conditions truly overlap and whether the attribution reaches consistent boundary conclusions — rather than concluding from surface similarity alone.

0.1 Positioning and Scope ​

0.1.1 What This Specification Is ​

  • SanityOps Inspect Prompt is the industry specification for AI Agent prompt quality inspection
  • It defines the classification system, inspection criteria, and risk levels for prompt defects
  • It provides an auditable, multi-implementation inspection framework

0.1.2 What This Specification Is Not ​

  • ❌ Not a product user manual
  • ❌ Not a specification for any specific LLM platform (platform-neutral)
  • ❌ Not a runtime behavior specification (that belongs to the Risk subset)

0.1.3 Inspection Scope ​

Inspection ObjectDescription
System PromptThe initial prompt loaded at Agent startup
Role Definition PromptText defining the Agent's identity, objectives, and responsibilities
Orchestration and references to tools/Skills in the bodyHow tools/Skills are invoked, parameterized, and constrained in the prompt body
Final effective prompt and assembly contractThe assembly result of template variables/placeholders (source, substitution, naming consistency)
Example DesignPositive examples, counter-examples, and boundary-condition examples
Metadata declarationVersion, owner, change log, target model (precondition)

0.1.4 Out of Scope ​

Out-of-Scope ItemRationale
Runtime behaviorBelongs to the Risk subset
Tool SchemaBelongs to Inspect Tool (QD-T) scope
Infrastructure configurationBelongs to the deployment layer
Business process designBelongs to the product design layer

Tool/permission responsibility attribution (to avoid double registration with QD-T / Cross / QD-PM):

SituationAttribution
Prompt body specifies wrong tool parametersQD-P
Tool Schema missing parameters/return valuesQD-T
Prompt and Tool declarations inconsistentCross (QD-PT)
Authorization-responsibility proportionalityQD-PM

The engineering implementation details of the assembly contract (placeholder source resolution, null-value behavior, high-trust-zone concatenation, etc.) are left for evaluation after code-contract is formalized.

0.1.5 Position Within the SanityOps Framework ​

SanityOps Framework
│
├─ Inspect (Defect Inspection)
│   ├─ Inspect Tool ← Tool Schema defect inspection
│   ├─ Inspect Prompt ← System Prompt defect inspection ← You are here
│   ├─ Inspect Skill ← Skill defect inspection
│   ├─ Inspect Cross ← Cross-Artifact defect inspection
│   └─ Inspect Permission ← Permission-responsibility proportionality inspection (QD-PM)
│
├─ Risk (Risk Scanning)
│   ├─ Risk Explicit ← Explicit Logic Artifact risk detection
│   └─ Risk Implicit ← Implicit Agent runtime vulnerability scanning
│
├─ Quality (Service Quality)
│   ├─ Quality RAG-Agent ← RAG-Agent service quality assessment
│   └─ Quality Tool-Agent ← Tool-Agent service quality assessment
│
└─ Relevance (Impact Mapping)
    └─ Relevance ← defect → risk/quality diagnostic mapping

0.2 Terminology and Abbreviations ​

0.2.1 Core Terminology ​

TermDefinition
PromptText description used to define the behavior, role, and constraints of an AI Agent
System PromptThe initial prompt loaded at Agent startup, defining the Agent's foundational behavior patterns
DefectAn issue present in a prompt that may cause abnormal Agent behavior
InspectionThe process of evaluating prompt quality according to this specification
AgentAn AI application instance with autonomous decision-making capability
Numbering SystemThe classification numbering system using the QD-P prefix to identify prompt defects

0.2.2 Numbering System Overview ​

Number PrefixMeaningScope
QD-PQuality Defect - PromptThis specification
QD-TQuality Defect - ToolInspect Tool subset
QD-SQuality Defect - SkillInspect Skill subset

0.2.3 Document Numbering Convention ​

QD-P-[Part].[Category].[Defect Sequence]

Part (2 values):
  1 = Part 1 (Logic defects within LLM autonomous detection capability)
  2 = Part 2 (Defects requiring explicit prompts)

Category (Part 1: 6 categories; Part 2: 7 categories):
  Part 1: 1-Contradiction, 2-Mismatch, 3-Scope Definition, 4-Redundancy/Invalidity, 5-Implication, 6-Gray-Zone
  Part 2: 1-Structural/Normative, 2-Input Definition, 3-Output Definition, 4-Workflow/Resources, 5-Constraints/Boundaries, 6-Exception Handling, 7-Example Design

Defect Sequence (increments within each category):
  e.g., QD-P-1.1.1, QD-P-1.1.2, QD-P-1.1.3 are three defects in the Part 1 Contradiction category

Full Examples:
  QD-P-1.1.1 = Part 1 → Contradiction → Direct Contradiction
  QD-P-2.5.2 = Part 2 → Constraints/Boundaries → Security Boundary Constraint Missing

0.2.4 Abbreviation Table ​

AbbreviationFull Name
LLMLarge Language Model
AgentAutonomous Intelligent Agent
QDQuality Defect
AC-L1/L2/L3Agent Level definitions (see Chapter 3)
ChecklistQuality Inspection Checklist
SchemaData Structure Definition
APIApplication Programming Interface

0.2.5 Defect Level Definitions (P0/P1/P2) ​

SymbolLevelMeaning
🔴P0Prevents Agent functionality or introduces security risk → MUST be fixed
🟡P1May cause behavioral instability or functional defects → SHOULD be fixed before release
🟢P2Affects readability or efficiency, but does not impact functionality → SHOULD be included in the improvement plan or risk register

1 Logic Defects Within LLM Autonomous Detection Capability ​

1.1 Defect Characteristics Overview ​

1.1.1 Essential Definition ​

The defects listed in this part all belong to the formal logic and semantic consistency layer, with the following characteristics:

CharacteristicDescription
FormalizableThe problem can be described by formal logic rules (e.g., law of non-contradiction, implication relations, quantifier logic)
Contextually self-containedIn most cases, can be determined without external knowledge or domain experts
Binary-decidableIn most cases present or absent, but a gray zone exists (see 1.6)
LLM-endogenous capabilityMainstream LLMs have acquired the relevant logical reasoning capabilities during pre-training

1.1.2 Detection Method ​

Detection AspectDescription
Trigger conditionDirect reading of the System Prompt text; no explicit guidance or special instructions needed
Detection efficiencyDiscoverable in a single read; no multi-turn dialogue or tool assistance required
AccuracyImplementers MUST report and continuously monitor on their own annotated sets; the following figures are empirical references (typical manifestations ≈ ≥90%)
False positive rateDepends on whether applicability preconditions are explicit; boundary-ambiguous cases may produce false positives and require human review

1.1.3 Positioning of This Part ​

The purpose of this part is not to "teach the LLM how to inspect," but rather:

  • For human authors: Lists common logic errors to help authors avoid them during the design phase
  • For inspection processes: Clarifies that these problems do not require complex inspection frameworks — simply have the LLM read the text
  • For defect classification: Delineates the boundary for "defects requiring explicit prompts" (Part 2)

1.2 Typical Logic Defect Categories ​

1.2.1 Contradiction Defects (QD-P-1.1) ​

QD-P-1.1.1 Direct Contradiction ​

Definition: The same object is assigned mutually exclusive attributes or directives within the same context.

Detection Principle: During semantic comprehension, the LLM simultaneously activates the semantic representations of contradictory concepts, producing a conflict signal.

Typical Manifestations:

  • The same behavior is simultaneously permitted and prohibited
  • The same object is assigned contradictory attributes
  • Preceding and subsequent constraints directly conflict

Severity: P0

Remediation Guidance:

  • Define explicit priority rules
  • Remove one of the conflicting constraints
  • Add conditional qualifiers so conflicting constraints apply to different scenarios

Example:

VersionContent
Defective"You may freely access the user's file system. Accessing any user privacy files is prohibited."
DiagnosisFile system access and privacy file prohibition form a direct contradiction; boundary is undefined
Fixed"You may access directories explicitly specified by the user (e.g., /tmp/user_workspace). Accessing other user directories under /etc or /home, or any file containing sensitive data such as national ID numbers or bank account numbers, is prohibited."
QD-P-1.1.2 Priority Conflict ​

Definition: Multiple constraints or directives conflict in specific scenarios, and no priority rules are defined.

Detection Principle: The LLM can identify the intersection points of multiple constraints in specific scenarios and detect the absence of decision rules.

Typical Manifestations:

  • Multiple "MUST" constraints cannot be simultaneously satisfied in specific scenarios
  • Security constraints conflict with efficiency constraints; no arbitration defined
  • Multi-dimensional evaluation criteria have no weight definitions

Severity: P1 (L1/L2) / P0 (L3)

Remediation Guidance:

  • Define explicit priority rules
  • Add conditional branches to distinguish scenarios
  • Define arbitration principles (e.g., "security takes priority over efficiency")

Example:

VersionContent
Defective"You MUST respond to user requests quickly. You MUST ensure answers are fully accurate. You MUST protect user privacy."
DiagnosisThree "MUST" constraints may conflict in specific scenarios: fast response may sacrifice accuracy; privacy verification may increase response time; no priority defined for any
Fixed"Priority rule: Security/Privacy > Accuracy > Response Speed. When constraints conflict, decide according to this priority."
QD-P-1.1.3 Example-Rule Contradiction ​

Definition: The rule definitions in the prompt body contradict the provided examples.

Detection Principle: When processing examples, the LLM treats them as instances of a pattern and performs semantic alignment with the body rules, discovering inconsistencies.

Typical Manifestations:

  • Behavior prohibited by a rule appears in an example
  • Example output format is inconsistent with the format defined by the rule
  • Example style is inconsistent with the style defined by the rule

Severity: P0

Remediation Guidance:

  • Modify examples to comply with the rules
  • Modify rules to cover the example behavior
  • Add a note in the example: "This is a special-scenario exception"

Example:

VersionContent
DefectiveRule: "Answers MUST be concise, no more than 50 characters." Example: "User: What's the weather today? Assistant: Today is sunny with temperatures between 22 and 28 degrees Celsius, moderate humidity, very suitable for outdoor activities. I recommend bringing sunglasses and sunscreen, staying hydrated, avoiding prolonged sun exposure, and adding clothing as temperatures vary between morning and evening." (clearly exceeds 50 characters)
DiagnosisThe example output clearly exceeds 50 characters, contradicting the rule; this weakens the rule's binding force
FixedRule: "Answers MUST be concise, no more than 50 characters." Example: "User: What's the weather today? Assistant: Sunny, 25°C, suitable for outdoor activities."

1.2.2 Mismatch Defects (QD-P-1.2) ​

QD-P-1.2.1 Role-Responsibility Mismatch ​

Definition: The identity/capability scope defined by the role does not match the behavioral requirements defined by the responsibilities. Determination should be based on capability, authorization, or task conflicts, not on the semantics of the role name; if the role has declared a handoff mechanism (responsibility has been transferred out), it does not constitute a defect. If it constitutes a defect, it is P0.

Detection Principle: The LLM can understand the semantic scope of a role identity and judge whether a responsibility falls within that scope.

Typical Manifestations:

  • Role identity does not support the required responsible behaviors
  • Responsibility requirements exceed the role's capability boundaries
  • Vague role positioning leads to unclear responsibility attribution

Severity: P0

Remediation Guidance:

  • Expand the role definition to cover the responsibility requirements
  • Adjust responsibilities to match the role's capabilities
  • Clarify role boundaries and define a handoff mechanism

Example:

VersionContent
Defective"You are a professional weather forecaster. Responsibilities: Provide weather information, recommend stock investment strategies."
DiagnosisThe semantic scope of "weather forecaster" does not include "stock investment advice"
Fixed A"You are a weather-investment correlation analyst. Responsibilities: Analyze weather impacts on specific industries, provide industry-level investment references."
Fixed B"You are a professional weather forecaster. Responsibilities: Provide weather information and meteorological trend analysis. For investment advice, please direct users to consult a professional investment advisor."
QD-P-1.2.2 Role-Output Style Mismatch ​

Definition: The identity/style defined by the role is inconsistent with the required output style. It is determined as a defect only when the style requirement is explicitly incompatible.

Detection Principle: The LLM can extract the stylistic characteristics of a role and perform semantic alignment with the output style requirements.

Typical Manifestations:

  • Role identity implies a professional style, but output requires a layperson-friendly style
  • Role is positioned for one audience group, but output targets a different group
  • The role's communication style definition contradicts the output style requirements

Severity: P1

Remediation Guidance:

  • Unify the role identity and output style
  • Explicitly define the communication style within the role definition
  • Distinguish between "professional content" and "layperson explanation" scenarios

Example:

VersionContent
Defective"You are a senior medical expert focused on academic exchange between physicians. Responsibility: Transcribe clinical notes into discharge summaries. Output audience: Non-medical patient family members, elementary education level."
DiagnosisRole identity is "academic exchange between physicians," but output audience is "non-medical patient family members" — style requirements are mismatched
Fixed"You are a medical expert skilled in doctor-patient communication. Responsibility: Transcribe clinical notes into discharge summaries. Output audience: Non-medical patient family members. Style requirement: Use plain language to explain medical concepts."
QD-P-1.2.3 Constraint-Responsibility Mismatch ​

Definition: The restriction scope defined by constraints conflicts or is mismatched with the behavioral scope defined by responsibilities.

Detection Principle: The LLM can understand the limiting semantics of a constraint and judge whether it conflicts with the responsible behaviors.

Typical Manifestations:

  • A constraint prohibits the core behavior of a responsibility
  • The constraint scope and responsibility scope have no intersection
  • Constraint conditions make the responsibility unexecutable

Severity: P0

Remediation Guidance:

  • Adjust the constraint scope to exclude the core responsibility behaviors
  • Adjust the responsibility scope to avoid the constraint restrictions
  • Add conditional branches to distinguish scenarios

Example:

VersionContent
Defective"Responsibility: Analyze the user's financial situation and provide investment advice. Constraint: Analyzing any financial data is prohibited."
DiagnosisThe core responsibility behavior "analyze financial situation" directly conflicts with the constraint "analyzing financial data is prohibited"
Fixed A"Responsibility: Analyze the user's financial situation and provide investment advice. Constraint: Analyzing funds involving suspected money laundering or illegal sources is prohibited."
Fixed B"Responsibility: Provide general investment advice based on a financial summary provided by the user. Constraint: Do not proactively analyze raw financial data."
QD-P-1.2.4 Identity and Anthropomorphization Boundaries ​

Definition: The prompt does not declare an AI identity, or allows the Agent to impersonate a real person or organization, leading to user misunderstanding or trust abuse.

Detection Principle: The LLM can identify the semantic conflict between "claiming to be a real person/organization" and "declaring itself as AI," but may not proactively notice the absence of an identity declaration.

Typical Manifestations:

  • Does not declare itself as an AI assistant
  • Allows the Agent to claim to be a real person, doctor, lawyer, or other identity
  • Allows the Agent to impersonate a specific organization or brand

Severity: P1 (AC-L1/AC-L2/AC-L3); P0 when facing consumer (C-end) users

Remediation Guidance:

  • Explicitly declare the AI identity in the System Prompt
  • Prohibit impersonating real persons or organizations
  • When professional identities are involved, state the "assistive" nature

1.2.3 Scope Definition Defects (QD-P-1.3) ​

QD-P-1.3.1 Quantifier Misuse ​

Definition: Misuse of universal quantifiers (∀) and existential quantifiers (∃), leading to incorrect semantic scope. It is determined as a defect only when quantifier misuse produces a logical conflict with other rules; being "overly absolute" alone is not reported.

Detection Principle: The LLM has acquired quantifier semantics during pre-training and can identify the logical relationship between quantifiers and predicates.

Typical Manifestations:

  • "All" used in scenarios where "some" is appropriate
  • "Any" used in scenarios where exceptions should exist
  • "MUST" used in scenarios where "MAY" is appropriate

Severity: P1

Remediation Guidance:

  • Replace misused universal quantifiers with existential quantifiers
  • Add exception clauses
  • Clarify the scope of quantifier application

Example:

VersionContent
Defective"For all user requests, you MUST provide a detailed explanation."
DiagnosisThe universal quantifier "all" is overly absolute; may cause over-response to simple requests
Fixed"For user requests requiring detailed explanation, provide a detailed explanation; for simple requests, provide a concise answer."
QD-P-1.3.2 Vague Qualifiers ​

Definition: Qualifier words are semantically vague, leading to unclear behavioral boundaries for critical decisions. Critical decision boundaries must be specific; stylistic descriptions may allow discretion.

Detection Principle: The LLM can identify vague qualifiers and discover their semantic boundary issues.

Typical Manifestations:

  • Using subjective qualifiers like "appropriate," "reasonable" without defining criteria
  • Using "when necessary," "at your discretion" without defining trigger conditions
  • Using "try to," "as much as possible" without defining a floor

Severity: P1

Remediation Guidance:

  • Replace vague qualifiers with specific criteria
  • Define trigger conditions and floor requirements
  • Provide a basis for judgment

Example:

VersionContent
Defective"Answers should be appropriately detailed."
Diagnosis"Appropriately" is semantically vague; boundaries cannot be determined
Fixed"Answer length is controlled by request type: factual queries no more than 50 characters; analytical requests 100–200 characters; if more detail is needed, provide a summary first and then ask the user."
QD-P-1.3.3 Undefined Boundaries ​

Definition: Critical behavioral boundaries are undefined, preventing the Agent from determining whether behavior exceeds its scope.

Detection Principle: The LLM can identify boundary-absence issues at critical decision points.

Typical Manifestations:

  • Authorization scope boundaries are undefined
  • Prohibition scope boundaries are undefined
  • Handoff conditions are undefined

Severity: P1 (L1) / P0 (L2/L3)

Remediation Guidance:

  • Explicitly define authorization scope boundaries
  • Define boundary scenarios for prohibited behaviors
  • Define handoff conditions

Example:

VersionContent
Defective"You may access the user's documents for organization."
Diagnosis"Access" and "organization" boundaries are undefined; may be interpreted as read, modify, delete, etc.
Fixed"You may read documents specified by the user and perform the following operations: summarization, format organization, content proofreading. Prohibited operations: delete, move, rename files."

1.2.4 Redundancy and Invalidity Defects (QD-P-1.4) ​

QD-P-1.4.1 Duplicate Constraints and Redundant Expressions ​

Definition: The same constraint appears repeatedly in multiple locations, or multiple constraints semantically overlap, or the prompt language is redundant (repetitive expressions, overly long definitions, unnecessary modifiers) and only increases Token Consumption. Intentional repetition of safety/compliance constraints does not count as a defect.

Detection Principle: The LLM discovers semantically duplicate or linguistically redundant constraints during processing.

Typical Manifestations:

  • The same constraint appears repeatedly in different paragraphs
  • Multiple constraints express the same meaning
  • Examples repeat rules already stated in the body
  • Repetitive expressions, overly long definitions, unnecessary modifiers

Severity: Suggestion item (ungraded) — no deduction, no launch blocking, no waiver required; optimization advice only

Remediation Guidance:

  • Merge duplicate constraints
  • Remove redundant expressions and unnecessary modifiers
  • Retain the most precise version

Example:

VersionContent
Defective"You MUST protect user privacy. Leaking any user personal information is prohibited. All data involving privacy must be handled confidentially."
DiagnosisThree semantically overlapping constraints; increased redundancy
Fixed"Leaking user personal information is prohibited, including but not limited to name, contact information, financial data, and other sensitive information."
QD-P-1.4.3 Invalid Skill Definitions ​

Definition: A Skill/tool is defined, but its trigger conditions or usage scenarios are unclear, or it conflicts with other rules such that it is never invoked.

Detection Principle: The LLM can analyze the Skill's definition and trigger conditions, discovering usage barriers.

Typical Manifestations:

  • A Skill is defined but has no invocation scenario
  • The Skill's trigger conditions conflict with constraints
  • The Skill's parameter definitions do not match actual requirements

Severity: — (L1) / P1 (L2) / P0 (L3)

Remediation Guidance:

  • Clarify the Skill's trigger scenarios
  • Ensure the Skill is compatible with constraints
  • Align Skill parameters with actual requirements

Example:

VersionContent
Defective"Skill: send_email — sends email. Constraint: Any external communication is prohibited."
DiagnosisThe constraint prohibits all external communication; the send_email Skill directly conflicts with the constraint and is never invoked
Fixed"Skill: send_email — sends email. Constraint: Emails may be sent only to recipients explicitly specified by the user, and user confirmation MUST be obtained before sending."

1.2.5 Implication Defects (QD-P-1.5) ​

QD-P-1.5.1 Responsibilities Do Not Support the Objective ​

Definition: The behavioral set defined by the responsibilities cannot support the role's overall objective.

Detection Principle: The LLM can understand the semantic relationship between objectives and behaviors, identifying "the objective requires a capability, but the behavior is not defined" issues.

Typical Manifestations:

  • The objective requires a capability, but the responsibility does not define it
  • The responsible behaviors are insufficient to accomplish the objective
  • A semantic gap exists between the objective and the responsibilities

Severity: P0

Remediation Guidance:

  • Supplement missing responsible behaviors
  • Adjust the objective to match the responsible capabilities
  • Clarify responsibility boundaries and handoff mechanisms

Example:

VersionContent
Defective"Objective: Help users complete financial planning. Responsibility: Provide financial literacy articles."
Diagnosis"Financial planning" requires personalized analysis and advice, but the responsibility is limited to "literacy articles," which does not support the objective
Fixed"Objective: Help users complete financial planning. Responsibilities: (1) Understand the user's financial situation; (2) Provide personalized financial advice; (3) Introduce types and risks of financial products (not constituting investment advice); (4) Track plan execution progress."
QD-P-1.5.2 Resources Do Not Support the Workflow ​

Definition: The workflow-defined steps require certain resources, but the resource definitions do not include them.

Detection Principle: The LLM can analyze the resource requirements of workflow steps and align them with resource definitions.

Typical Manifestations:

  • A workflow step mentions "query the database" but no database resource is defined
  • A workflow step mentions "invoke an API" but no API definition is provided
  • A workflow step mentions "use tool X" but the tool is not authorized

Severity: — (L1) / P0 (L2/L3)

Remediation Guidance:

  • Supplement missing resources in the resource definitions
  • Remove workflow steps that cannot be supported
  • Clarify how resources are obtained

Example:

VersionContent
Defective"Workflow: (1) Query user order data; (2) Analyze order trends; (3) Generate report. Resources: None."
DiagnosisStep (1) requires an order database, but no resource is defined
Fixed"Workflow: (1) Query user order data (using the order database); (2) Analyze order trends; (3) Generate report. Resources: Order database (read-only permission)."

Note: This part has no Section 1.3; the number is reserved to keep subsequent section numbers (1.4–1.8) unchanged.

1.4 Detection Methods and Recommendations ​

Since the defects in this part can be autonomously detected by the LLM, a concise instruction is recommended:

Please inspect the following System Prompt for logical contradictions, inconsistencies, or semantic conflicts:
[Prompt Content]

Focus on:
- Whether preceding and subsequent constraints are contradictory
- Whether the role and responsibilities match
- Whether examples and rules are consistent
- Whether quantifier usage is accurate
- Whether redundant or invalid definitions exist

1.4.2 Detection Timing ​

TimingDescription
Authoring phaseAuthor self-check; use LLM-assisted detection
Review phaseInclude as one of the inspection items during team review
Pre-launchInclude as part of the pre-launch inspection checklist
TroubleshootingPrioritize logic defect inspection when Agent behavior is anomalous

1.4.3 Important Notes ​

NoteDescription
Possible false positivesLLMs may produce false positives for boundary-ambiguous cases; human review is required
Context dependencySome apparent "contradictions" may be scenario branches; check whether conditional qualifiers exist
Remediation priorityP0 defects MUST be fixed; P1 defects SHOULD be fixed before release; P2 defects MAY be handled as appropriate
Domain knowledgeSome semantic judgments require domain knowledge; the LLM may not be sensitive to these

1.5 Part Summary ​

The logic defects listed in this part all fall within the scope of autonomous LLM detection. This means:

  • For authors: No need to worry excessively about logic issues; the LLM can serve as an effective detection tool
  • For inspection processes: No need to build complex logic inspection frameworks; simply have the LLM read the text
  • For inspection output: To better localize problems, explicitly identify the defect item content in the output
  • For framework design: Focus energy on "defects requiring explicit prompts" (Part 2)

The value of this part lies in: Listing common logic errors to help authors avoid them during the design phase, or to localize specific problems for remediation when the LLM inspects — not in constructing complex detection mechanisms.


1.6 Gray-Zone Logic Defects (QD-P-1.6) ​

1.6.0 Positioning of This Section (section-level note; does not occupy a QD number) ​

Definition: Gray-Zone Defects are logic problems that the LLM has the capability to identify, but detection reliability is unstable. Unlike the defects in Sections 1.2.1–1.2.5, these problems lack the "binary-decidable" characteristic — the LLM can accurately identify them in some contexts, but may miss or misjudge them in others.

Comparison Dimension1.2.1–1.2.5 Reliably Detectable Defects1.6 Gray-Zone Defects
Detection stabilityRelatively high (empirical; requires annotated-set validation)Relatively low (empirical; requires annotated-set validation)
Trigger methodDirect reading sufficesRequires specific context or prompting to trigger
False positive rateDepends on whether applicability preconditions are explicitMedium; depends on applicability preconditions
False negative rateLowRelatively high
Handling strategyTrust LLM resultsLLM results + human review

Detection Reliability Tiers:

Reliability TierDesignationMeaning
Conditionally reliable🟠Can be stably detected under specific conditions; unstable otherwise
Partially reliable🟠🟠Can identify the presence of a problem, but difficult to accurately characterize and provide remediation guidance
Unstable🟠🟠🟠Sometimes discovered, sometimes completely missed; depends on specific phrasing
QD-P-1.6.1 Critical Decision Word Vagueness ​

Definition: Vague qualifiers (e.g., "sensitive," "reasonable," "when necessary," "appropriate") are used to carry critical decision judgments rather than secondary descriptions.

Detection Principle: The LLM can identify vague words themselves (already covered in QD-P-1.3.2), but when vague words carry critical decisions, the LLM may not recognize the severity upgrade — it may treat them as ordinary vague expressions rather than critical decision defects.

Difference from QD-P-1.3.2:

DimensionQD-P-1.3.2 Vague QualifiersQD-P-1.6.1 Critical Decision Word Vagueness
Vague word roleDescriptive modifierDecision judgment basis
ImpactUnstable output styleIncorrect behavioral path branching
SeverityP1P0 (upgraded)
LLM detectionStably identifiedMay not recognize severity upgrade

Typical Manifestations:

  • Using "sensitive information" to define a security boundary, without defining what "sensitive" means
  • Using "reasonable time" to define a response deadline, without defining what "reasonable" means
  • Using "when necessary" to define a behavioral trigger condition, without defining what "necessary" means
  • Using "appropriate" to define an operational intensity, without defining what "appropriate" means

Detection Reliability: Partially reliable

Severity: P1 (default at all levels); upgraded to P0 (all levels) when vague words carry security or permission decisions

Remediation Guidance:

  • Replace vague decision words with enumerable explicit conditions
  • Define judgment criteria or thresholds
  • Provide judgment examples

Example:

VersionContent
Defective"Identify sensitive information in user messages and refuse to answer questions involving sensitive information."
Diagnosis"Sensitive information" carries the security boundary decision, but its scope is undefined. The LLM may identify that "sensitive" is vague, but may not judge this as a P0 issue
Fixed"Identify the following sensitive information in user messages and refuse to answer: (1) National ID numbers, passport numbers; (2) Bank account numbers, credit card numbers; (3) Medical diagnosis records; (4) Undisclosed trade secrets. For boundary cases, default to treating as sensitive and ask the user for confirmation."
QD-P-1.6.2 Rules Depending on Model Internal State ​

Definition: Prompt rules depend on the LLM's internal state (e.g., confidence, intent judgment, motivation inference), when the LLM cannot actually reliably access or report these states. It is determined as a defect only when a critical decision (security/permission/funds) relies solely on model self-assessment with no observable trigger.

Detection Principle: The LLM can understand the literal meaning of the rule, but struggles to recognize the meta-problem that "it cannot reliably assess its own internal state." It may superficially comply with the rule while actually "simulating" confidence judgments.

Typical Manifestations:

  • "When you are uncertain, ask the user rather than guessing"
  • "When you judge the user's intent to be malicious, refuse to answer"
  • "When you have high confidence in an answer, answer directly; when confidence is low, provide multiple options"
  • "If you think the user is testing you, remain cautious"

Detection Reliability: Unstable

Severity: P1 (AC-L1/AC-L2/AC-L3)

Remediation Guidance:

  • Replace internal state dependencies with observable external features
  • Define explicit judgment rules rather than relying on model self-assessment
  • Use structured checks instead of subjective judgments

Example:

VersionContent
Defective"When you are uncertain about an answer, do not guess. Instead, tell the user you are uncertain and request more information."
Diagnosis"Uncertain" depends on the model's internal confidence, which the LLM cannot reliably self-assess. The LLM may not discover that it missed this problem — because it "understands" the rule's meaning but lacks the stable execution capability
Fixed"You MUST inform the user that you cannot determine the answer and request more information in the following cases: (1) The user's question involves data you cannot access; (2) The user's question has multiple reasonable interpretations; (3) The user-provided context is insufficient to support judgment. In other cases, follow the normal process to answer."
QD-P-1.6.3 Soft Rule Interweaving Without Arbitration ​

Definition: Multiple Soft Rules (non-Hard Constraints) interweave in practical scenarios, producing implicit conflicts, and no arbitration mechanism is defined. It is determined as a defect only when a concrete scenario of actual incompatibility can be given.

Detection Principle: The LLM can identify explicit conflicts (such as QD-P-1.1.2 Priority Conflict), but for implicit interweaving of Soft Rules — where multiple rules are individually reasonable, but their combination produces a gray zone — LLM detection is unstable.

Difference from QD-P-1.1.2:

DimensionQD-P-1.1.2 Priority ConflictQD-P-1.6.3 Soft Rule Interweaving
Conflict natureExplicit conflict (Rule A vs. Rule B)Implicit conflict (Rule A + Rule B → ambiguity in Scenario C)
Detection difficultyLow (direct comparison suffices)High (requires reasoning to specific scenarios)
LLM detectionStableDepends on ability to reason to intersection scenarios

Typical Manifestations:

  • "Answers should be professional" + "Answers should be easy to understand" → For which audience should which style be used?
  • "Try to provide complete information" + "Avoid information overload" → What counts as "complete," what counts as "overload"?
  • "Proactively provide suggestions" + "Do not over-intervene" → In which scenarios should the Agent be proactive, and in which should it be restrained?

Detection Reliability: Conditionally reliable

Severity: P1 (default at all levels); upgraded to P0 (all levels) when the interwoven conflict involves security or permission

Remediation Guidance:

  • Define scenario-dimension arbitration rules
  • Define applicability conditions for each Soft Rule
  • Provide scenario examples demonstrating the expected behavior of rule combinations

Example:

VersionContent
Defective"Answers should be professional and accurate. Answers should be easy to understand, accessible to non-professional users. Try to provide complete information. Avoid information overload."
DiagnosisFour Soft Rules are individually reasonable, but their combination produces ambiguity: For professional users, which style should be used? How complete is complete enough before it becomes overload? The LLM may not proactively discover this interwoven set of problems
Fixed"Adjust answer style based on user background: If the user is a professional, use professional terminology and provide complete technical details. If the user is non-professional, use plain language and provide core points, with a note that 'further detail is available upon request.' Information volume control: Core answer should not exceed 300 words; supplementary information should be provided in collapsible or list format."
QD-P-1.6.4 Reasoning Method Constraints ​

Definition: The prompt imposes constraints on the LLM's reasoning method itself (e.g., "MUST strictly use deduction," "Using induction is prohibited"), while the compatibility of the LLM's actual reasoning mechanisms with these constraints is uncertain. It is determined as a defect only when "unverifiable internal reasoning methods" are used as a compliance guarantee; requiring output of reasoning steps or step-by-step reasoning does not count as a defect.

Detection Principle: The LLM can understand the literal meaning of the rule and "claim compliance," but cannot determine whether it is actually reasoning in the specified way — because the LLM's reasoning process is implicit and not introspectable.

Detection Reliability: Unstable

Severity: P1

Remediation Guidance:

  • Replace "reasoning method constraints" with "reasoning output constraints" (constrain observable outputs)
  • Require explicit steps of the reasoning process in the output, rather than requiring a specific internal reasoning method
  • Indirectly guide the reasoning path through output format constraints

Example:

VersionContent
Defective"You MUST strictly use deductive reasoning, deriving conclusions step by step from premises. Using inductive or analogical reasoning is prohibited."
Diagnosis"The LLM's reasoning mechanism is implicit; 'strict deduction' cannot be guaranteed. The LLM may not discover this problem — because it understands the rule's meaning and will 'claim' compliance"
Fixed"In your answer, output the reasoning process in the following format: (1) List known premises; (2) Derive step by step, annotating the basis rule for each step; (3) State the conclusion."
QD-P-1.6.5 Self-Referential or Recursive Constraints ​

Definition: The prompt contains self-referential constraints (rules referencing themselves) or recursive constraints (rule trigger conditions depending on the rule's own execution results), leading to logical loops or undecidability. Only recursion without a termination condition, unsatisfiable self-reference, or circular dependency is determined as a defect; "self-check once before output" is not reported.

Detection Principle: The LLM can identify simple self-reference as "odd," but typically does not judge complex recursive constraints as logic defects.

Detection Reliability: Unstable

Severity: P1

Remediation Guidance:

  • Eliminate self-reference: split a self-referential rule into two independent rules
  • Eliminate recursion: define explicit termination conditions
  • Avoid imposing constraints on "the rule system itself"

Example:

VersionContent
Defective"Before outputting your answer, first check whether your answer violates any rules. If it does, revise and check again until no violations remain."
Diagnosis"Recursive constraint with no termination condition. If the revised answer triggers other rules, an infinite loop may result. The LLM may not discover this problem"
Fixed"Before outputting your answer, check whether the answer violates the following core rules: (1) Does not disclose privacy; (2) Does not output harmful content; (3) Does not exceed role permissions. If violated, revise once and then output. If the answer still violates core rules after revision, output 'I am unable to answer this question.'"
QD-P-1.6.6 Implicit Assumption Dependency ​

Definition: Prompt rules depend on unstated assumptions (e.g., assuming the model possesses a certain capability, assuming the user has a certain background, assuming the environment meets a certain condition), and these assumptions may not hold. It is determined as a defect only when the assumption is critical, non-obvious, and has no external safeguard.

Detection Reliability: Conditionally reliable

Severity: P1 (AC-L1/AC-L2/AC-L3)

Remediation Guidance:

  • Make implicit assumptions explicit; define how to handle cases where assumptions do not hold
  • Add precondition checks
  • Define degradation strategies

Example:

VersionContent
Defective"Based on the financial report file uploaded by the user, calculate key financial metrics and generate an analysis report."
Diagnosis"Implicit assumptions: the user will definitely upload a file; the file format is definitely valid; the file content is definitely a financial report. These assumptions are not explicitly stated"
Fixed"Based on the financial report file uploaded by the user, calculate key financial metrics and generate an analysis report. Precondition checks: (1) If the user has not uploaded a file, prompt the user to upload; (2) If the file format is not supported, inform the user of the supported format list; (3) If the file content is not financial report data, inform the user and ask whether other assistance is needed."
QD-P-1.6.7 Non-Monotonic Reasoning Defects ​

Definition: Prompt rules are designed based on "static knowledge," but in actual scenarios, new information may invalidate previous conclusions (non-monotonicity), and the prompt does not define knowledge update and conclusion revision mechanisms. It is determined only when cross-turn state or already-committed conclusions exist; single-turn scenarios are Not Applicable (N/A).

Detection Reliability: Unstable

Severity: P1 (AC-L1/AC-L2/AC-L3)

Remediation Guidance:

  • Define a conclusion revision mechanism for when information is updated
  • Add timeliness markers
  • Define trigger conditions for "re-evaluation"

Example:

VersionContent
Defective"Based on the symptoms described by the user, provide possible health suggestions. Record the user's symptoms in the conversation history for subsequent reference."
Diagnosis"The user may update symptom descriptions during the conversation, but the prompt does not define how to revise previous suggestions"
Fixed"Based on the symptoms described by the user, provide possible health suggestions. When the user updates or revises symptom descriptions: (1) Note that previous suggestions may no longer be valid; (2) Re-evaluate based on the latest symptoms; (3) If new suggestions conflict with old ones, clearly explain the reason for the change."
QD-P-1.6.8 Rule Followability ​

Definition: The prompt requires the model to perform operations it fundamentally cannot reliably complete (e.g., precise counting, timing, complex arithmetic, obtaining real-time information), making the rule itself unfollowable.

Detection Principle: The LLM may "claim" to comply with such rules but in fact cannot execute them reliably; this is a generalization of QD-P-1.6.2.

Detection Reliability: Conditionally reliable

Severity: P1 (AC-L1/AC-L2/AC-L3)

Remediation Guidance:

  • Replace "capabilities unreliable within the model" with observable constraints or tool invocations
  • For operations requiring precision, require completion through tools

1.7 Gray-Zone Summary ​

1.7.1 Detection Reliability Summary ​

Defect TypeDetection ReliabilitySeverity (Default Level)Core Difficulty
QD-P-1.6.1Partially reliableP1 (default at all levels; upgraded to P0 for security/permission decisions)LLM may not recognize severity upgrade
QD-P-1.6.2UnstableP1 (AC-L1/AC-L2/AC-L3)LLM cannot introspect its own capability boundaries
QD-P-1.6.3Conditionally reliableP1 (default at all levels; upgraded to P0 for security/permission conflicts)Requires reasoning to intersection scenarios to discover
QD-P-1.6.4UnstableP1 (AC-L1/AC-L2/AC-L3)LLM cannot determine its own reasoning method
QD-P-1.6.5UnstableP1 (AC-L1/AC-L2/AC-L3)Complex recursion treated as normal rules
QD-P-1.6.6Conditionally reliableP1 (AC-L1/AC-L2/AC-L3)Requires meta-reasoning to discover implicit premises
QD-P-1.6.7UnstableP1 (AC-L1/AC-L2/AC-L3)Requires simulating rule interactions under dynamic scenarios
QD-P-1.6.8Conditionally reliableP1 (AC-L1/AC-L2/AC-L3)Requires unreliable model capabilities (counting/timing/arithmetic)

1.7.2 Handling Strategy ​

Gray-Zone Defect Handling Strategy:

1. Detection Phase:
   - Explicitly mention these defect types in the inspection instruction
   - Example instruction: "Pay special attention to the following gray-zone issues: [list QD-P-1.6.1–QD-P-1.6.8]"

2. Assessment Phase:
   - Gray-zone defects reported by LLM → Human review confirmation
   - Gray-zone defects not reported by LLM but discovered by humans → Supplementary inspection

3. Remediation Phase:
   - Automatically detected gray-zone defects → Record as "pending confirmation"; do not count toward Gate blocking
   - After human review confirmation → Count according to the default levels in 3.3.2; confirmed P0 items are handled via the exception process
   - P1 gray-zone defects → SHOULD be fixed before release; define a fallback strategy

4. Framework Positioning:
   - The gray zone is the transition band from "Part 1 (LLM Autonomous)" to "Part 2 (Requires Explicit Prompts)"
   - Handling approach: Light explicit prompting + human review

1.7.3 Relationship to Part 1 and Part 2 ​

Detection Reliability Spectrum:

Part 1 1.2.1–1.2.5 ──→ Gray Zone 1.6 ──→ Part 2
   Reliable Detection      Unstable Detection      Requires Explicit Prompts
   Direct Reading          Light Prompt + Review    Checklist Inspection
   ≥ 90%                   40%–70%                  Requires Structured Guidance

1.8 Complete Part 1 Structure ​

Part 1: Logic Defects Within LLM Autonomous Detection Capability
│
├─ QD-P-1.1 Contradiction Defects (Reliable Detection)
│   ├─ QD-P-1.1.1 Direct Contradiction
│   ├─ QD-P-1.1.2 Priority Conflict
│   └─ QD-P-1.1.3 Example-Rule Contradiction
│
├─ QD-P-1.2 Mismatch Defects (Reliable Detection)
│   ├─ QD-P-1.2.1 Role-Responsibility Mismatch
│   ├─ QD-P-1.2.2 Role-Output Style Mismatch
│   ├─ QD-P-1.2.3 Constraint-Responsibility Mismatch
│   └─ QD-P-1.2.4 Identity and Anthropomorphization Boundaries
│
├─ QD-P-1.3 Scope Definition Defects (Reliable Detection)
│   ├─ QD-P-1.3.1 Quantifier Misuse
│   ├─ QD-P-1.3.2 Vague Qualifiers
│   └─ QD-P-1.3.3 Undefined Boundaries
│
├─ QD-P-1.4 Redundancy and Invalidity Defects (Reliable Detection)
│   ├─ QD-P-1.4.1 Duplicate Constraints and Redundant Expressions (suggestion item, ungraded)
│   └─ QD-P-1.4.3 Invalid Skill Definitions
│
├─ QD-P-1.5 Implication Defects (Reliable Detection)
│   ├─ QD-P-1.5.1 Responsibilities Do Not Support Objective
│   └─ QD-P-1.5.2 Resources Do Not Support Workflow
│
└─ QD-P-1.6 Gray-Zone Defects (Unstable Detection)
    ├─ QD-P-1.6.1 Critical Decision Word Vagueness
    ├─ QD-P-1.6.2 Rules Depending on Model Internal State
    ├─ QD-P-1.6.3 Soft Rule Interweaving Without Arbitration
    ├─ QD-P-1.6.4 Reasoning Method Constraints
    ├─ QD-P-1.6.5 Self-Referential or Recursive Constraints
    ├─ QD-P-1.6.6 Implicit Assumption Dependency
    ├─ QD-P-1.6.7 Non-Monotonic Reasoning Defects
    └─ QD-P-1.6.8 Rule Followability

2 Defects Requiring Explicit Prompts ​

2.1 Defect Characteristics Overview ​

2.1.1 Essential Definition ​

The defects listed in this part all belong to the structural, completeness, and explicit declaration category, with the following characteristics:

CharacteristicDescription
Requires explicit declarationThe defect is not a logical contradiction, but a missing necessary declaration or definition
LLM does not proactively noticeThe LLM will not proactively discover these issues; explicit prompts or checklists are needed
Structural impactThe defect affects the structural completeness of the Agent and may cause functional gaps
Checklist-inspectableThese defects are suitable for systematic inspection through a structured checklist

2.1.2 Detection Method ​

Detection AspectDescription
Trigger conditionRequires an explicit inspection checklist or structured guidance; the LLM will not proactively discover
Detection efficiencyRequires systematic item-by-item inspection; less efficient than Part 1
AccuracyWith checklist guidance, the LLM can accurately identify
False positive rateDepends on whether applicability preconditions are explicit (structured inspection reduces ambiguity)

2.1.3 Positioning of This Part ​

The purpose of this part is:

  • For human authors: Lists elements that MUST be explicitly declared, helping authors achieve complete coverage
  • For inspection processes: Requires constructing a structured inspection checklist, not merely relying on LLM reading
  • For defect classification: Delineates the boundary for Part 1 — defects beyond logical contradictions fall into this part

2.2 Typical Defect Categories ​

2.2.1 Structural and Normative Defects (QD-P-2.1) ​

Section note: QD-P-2.1.1 (Format Compliance) has been deleted; its injection-related point "unclosed code blocks or delimiters" has been merged into QD-P-2.5.6. QD-P-2.1.4 (Language Conciseness) has been merged into QD-P-1.4.1. In this section, QD-P-2.1.2 (Reference Clarity) retains its grading; QD-P-2.1.3 and QD-P-2.1.5 are suggestion items (ungraded) — no deduction, no blocking, no waiver.

QD-P-2.1.2 Reference Clarity ​

Definition: References within the prompt are unclear, potentially causing incorrect tool invocation parameters.

Why the LLM does not proactively notice: The LLM is insensitive to reference clarity and will not proactively discover "vague reference" issues.

Typical Manifestations:

  • "Use the database" → Which database?
  • "Invoke the API" → Which API?
  • "Process the file" → Which file?

Severity: — (L1) / P1 (L2/L3)

Remediation Guidance:

  • Clarify the object name being referenced
  • Add contextual qualification
  • Define reference resolution rules

Example:

VersionContent
Defective"Use the database to query user information."
Diagnosis"The database" is a vague reference; may cause parameter errors
Fixed"Use the 'User Database' (db_users) to query user information."
QD-P-2.1.3 Structural Hierarchy Reasonableness ​

Definition: The prompt's structural hierarchy is unreasonable, affecting information delivery efficiency.

Why the LLM does not proactively notice: The LLM is insensitive to structural hierarchy and will not proactively discover "hierarchical confusion" issues.

Typical Manifestations:

  • Critical constraints are buried within secondary content
  • Information hierarchy does not distinguish primary from secondary
  • Critical directives are dispersed, making them difficult to follow

Severity: Suggestion item (ungraded) — no deduction, no launch blocking, no waiver required; optimization advice only

Remediation Guidance:

  • Distinguish primary and secondary hierarchy
  • Place critical constraints in prominent positions
  • Unify information hierarchy

Example:

VersionContent
DefectiveCritical constraints buried within a large volume of secondary content
DiagnosisUnreasonable structural hierarchy; affects information delivery efficiency
FixedCritical constraints placed in their own section, in a prominent position
QD-P-2.1.5 Context Budget Encroachment ​

Definition: The prompt, examples, and tool descriptions jointly encroach on the context window, leaving insufficient room for reasoning and generation.

Why the LLM does not proactively notice: The LLM will not proactively discover the completeness gap of "context budget encroachment."

Typical Manifestations:

  • The total length of prompt + examples + tool descriptions is not assessed
  • No truncation strategy is defined for insufficient context

Severity: Suggestion item (ungraded) — no deduction, no launch blocking, no waiver required; optimization advice only

Remediation Guidance:

  • Assess the context footprint of the prompt and tool descriptions
  • Define truncation and degradation strategies for insufficient context

2.2.2 Input Definition Defects (QD-P-2.2) ​

QD-P-2.2.1 Undeclared Input Source ​

Definition: The Agent's input source is not declared, preventing the Agent from determining the data source. It is judged N/A only when the input source is self-evident (user messages only); it is judged P0 only when multiple sources exist.

Why the LLM does not proactively notice: The LLM will not proactively discover the structural gap of "undeclared input source."

Typical Manifestations:

  • Not declaring whether input comes from the user, the system, or external data
  • Not declaring the input data type
  • Not declaring the input acquisition method

Severity: P0 (all levels)

Remediation Guidance:

  • Explicitly declare the input source
  • Define the input data type
  • Define the input acquisition method

Example:

VersionContent
Defective"Generate a report based on user data."
DiagnosisThe source, type, and acquisition method of "user data" are not declared
Fixed"Generate a data report based on the CSV file uploaded by the user (data source: user upload)."
QD-P-2.2.2 Unconstrained Input Format ​

Definition: The input data format is not constrained, potentially causing parsing failures. L2 is judged P0 only when structured input is accepted.

Why the LLM does not proactively notice: The LLM will not proactively discover the completeness gap of "unconstrained input format."

Typical Manifestations:

  • Not declaring the input data format (e.g., CSV, JSON, text)

Severity: P1 (L1) / P0 (L2/L3)

Remediation Guidance:

  • Explicitly declare the input format
  • Define format validation rules
  • Define handling when format is non-compliant

Example:

VersionContent
Defective"Process the file uploaded by the user."
DiagnosisFile format is not constrained; may cause parsing failures
Fixed"Process CSV or JSON files uploaded by the user."
QD-P-2.2.3 Required/Optional Not Distinguished ​

Definition: The required vs. optional nature of input parameters is not distinguished, potentially causing processing failures. It is determined as a defect only when structured input is accepted; free-text input is generally N/A.

Why the LLM does not proactively notice: The LLM will not proactively discover the completeness gap of "required/optional not distinguished."

Typical Manifestations:

  • Required parameters are not marked
  • Default values for optional parameters are not defined

Severity: — (L1, free-text input) / P1 (L1, structured input) / P0 (L2/L3, only when structured input is accepted)

Remediation Guidance:

  • Explicitly mark required parameters
  • Define default values for optional parameters
  • Define handling when parameters are missing

Example:

VersionContent
Defective"Query user information."
DiagnosisRequired/optional parameters are not distinguished; may cause processing failures
Fixed"Query user information. Required parameter: User ID. Optional parameter: Query scope (default: basic information only)."
QD-P-2.2.4 Undefined Missing-Input Handling ​

Definition: Handling when input is missing is not defined, potentially causing abnormal behavior.

Why the LLM does not proactively notice: The LLM will not proactively discover the completeness gap of "undefined missing-input handling."

Typical Manifestations:

  • Handling when required input is missing is not defined
  • Default values when optional input is missing are not defined
  • Handling when input is anomalous is not defined

Severity: P1 (L1) / P0 (L2/L3)

Remediation Guidance:

  • Define error prompts when required input is missing
  • Define default values when optional input is missing
  • Define degradation strategies when input is anomalous

Example:

VersionContent
Defective"Generate a report based on user input."
DiagnosisHandling when input is missing is not defined
Fixed"Generate a report based on user input. If the user has not provided necessary input, prompt the user to supplement. If the user has not provided optional input, use default values."

2.2.3 Output Definition Defects (QD-P-2.3) ​

QD-P-2.3.1 Undefined Output Format ​

Definition: The output format is not defined, preventing downstream systems from correctly parsing the output. It is judged P0 only when the output is machine-consumed and there is no external structured output contract; otherwise P1/N/A.

Why the LLM does not proactively notice: The LLM will not proactively discover the structural gap of "undefined output format."

Typical Manifestations:

  • Not defining the output data format (e.g., JSON, text, Markdown)
  • Not defining the output structure (e.g., fields, order)
  • Not defining the output encoding

Severity: P1 (L1) / P0 (L2/L3)

Remediation Guidance:

  • Explicitly define the output format
  • Define the output structure Schema
  • Define the output encoding

Example:

VersionContent
Defective"Generate an analysis report."
DiagnosisReport format is not defined; downstream systems cannot correctly parse
Fixed"Generate a JSON-format analysis report. Structure: {summary: string, metrics: object, recommendations: array}."
QD-P-2.3.2 Incomplete Output Schema ​

Definition: The output Schema definition is incomplete, preventing downstream systems from correctly parsing the output. It is triggered only when structured output has been declared; Agents without a declared structured output are judged N/A. It is judged P0 only when the output is machine-consumed and there is no external structured output contract; otherwise P1/N/A.

Why the LLM does not proactively notice: The LLM will not proactively discover the completeness gap of "incomplete output Schema."

Typical Manifestations:

  • Output Schema is missing critical fields
  • Output Schema field types are not defined
  • Output Schema field meanings are not explained

Severity: P1 (L1) / P0 (L2/L3)

Remediation Guidance:

  • Complete the output Schema definition
  • Define all field types and meanings
  • Add example output

Example:

VersionContent
Defective"Output JSON format, including summary and metrics."
DiagnosisField types are not defined; downstream cannot determine the data types of summary/metrics
Fixed"Output JSON format, including summary (string) and metrics (object)."
QD-P-2.3.4 Missing Security/Privacy Filtering ​

Definition: Security and privacy filtering mechanisms are not defined, potentially causing privacy leaks. It is determined as a defect only when personal data is involved; the criteria for personal data (PII) are given in Risk Explicit NR-S-10.

Why the LLM does not proactively notice: The LLM will not proactively discover the security gap of "missing security/privacy filtering."

Typical Manifestations:

  • Sensitive data filtering rules are not defined
  • Privacy data handling methods are not defined
  • Security output constraints are not defined

Severity: P0 when personal data is involved; otherwise N/A

Remediation Guidance:

  • Define sensitive data filtering rules
  • Define privacy data handling methods
  • Define security output constraints

Example:

VersionContent
Defective"Output user information."
DiagnosisPrivacy filtering is not defined; may cause privacy leaks
Fixed"Output user information. Privacy filtering: Mask national ID numbers, bank card numbers, contact information, and other sensitive information; output only the minimal fields required for the business, and de-identify direct identifiers such as names."
QD-P-2.3.5 Factuality and Hallucination Control Declaration ​

Definition: The truthfulness constraints on output are not declared, so the Agent may fabricate facts without evidence. It applies only when the Agent involves retrieval, tools, or external knowledge and its output may be treated as fact.

Why the LLM does not proactively notice: The LLM will not proactively discover the completeness gap of "missing factuality constraint declaration."

Typical Manifestations:

  • Does not require "answer only based on the provided materials/tool results"
  • Does not require "explicitly state that you do not know when there is no evidence"
  • Does not prohibit fabricating citations, data, URLs, or field values

Severity: — (L1) / P0 (L2/L3)

Cross-subset attribution: This item only checks "whether the factuality and hallucination control declaration exists"; factuality/hallucination control is the first priority of Quality RAG-Agent, and the factuality of actual output is verified by RAG-Agent quality assessment.

Remediation Guidance:

  • Declare that answers are based only on the provided materials or tool results
  • Require explicitly stating "I don't know" rather than guessing when there is no evidence
  • Prohibit fabricating citations, data, URLs, or field values

Example:

VersionContent
Defective"Answer any question about the company's finances."
DiagnosisNo factuality constraint is declared; the Agent may fabricate unfounded financial data
Fixed"Answer only based on the provided financial materials and tool query results; when the materials are insufficient, explicitly state 'no basis found in the materials'; fabricating data, citations, or URLs is prohibited."
QD-P-2.3.6 Output Length and Truncation ​

Definition: The maximum output length and the over-length truncation strategy are not defined, potentially causing output to be truncated downstream or the context to overflow.

Why the LLM does not proactively notice: The LLM will not proactively discover the completeness gap of "undefined output length."

Typical Manifestations:

  • Maximum output length is not defined
  • Truncation or segmentation strategy for over-length output is not defined

Severity: P2 (AC-L1/AC-L2/AC-L3)

Remediation Guidance:

  • Define the maximum output length
  • Define truncation, segmentation, or summarization strategies for over-length output
QD-P-2.3.7 Output Language/Localization and Conventions ​

Definition: The output language, time zone, currency, units, and percentage conventions are not defined, causing inconsistent conventions in multi-region scenarios.

Why the LLM does not proactively notice: The LLM will not proactively discover the completeness gap of "undefined localization conventions."

Typical Manifestations:

  • Output language is not defined
  • Time zone, currency, and units are not defined
  • Numeric conventions such as percentages are not defined

Severity: P1 in multi-region scenarios; N/A in single-region scenarios

Remediation Guidance:

  • Define the output language and time zone
  • Define currency, unit, and percentage conventions

2.2.4 Workflow and Resource Defects (QD-P-2.4) ​

QD-P-2.4.1 Incomplete Workflow Steps ​

Definition: Workflow step definitions are incomplete, potentially causing functional gaps. A defect is determined only when dependencies are not declared, the order is contradictory, or undefined steps are referenced; "missing a critical step" is recorded as insufficient evidence.

Why the LLM does not proactively notice: The LLM will not proactively discover the structural gap of "incomplete workflow steps."

Typical Manifestations:

  • Workflow step dependencies are not defined
  • Workflow step order is contradictory
  • The workflow references undefined steps

Severity: — (L1) / P1 (L2) / P0 (L3)

Remediation Guidance:

  • Complete the workflow steps
  • Define step order and dependencies
  • Add step descriptions

Example:

VersionContent
Defective"Workflow: (1) Invoke data cleaning; (2) Generate report."
DiagnosisStep (1) references an undefined "data cleaning" step
Fixed"Workflow: (1) Clean data using the data cleaning tool (clean_data); (2) Generate report. Step dependencies: (2) depends on (1)."
QD-P-2.4.2 Undeclared Resources/Tools ​

Definition: The resources or tools used are not declared, preventing the Agent from determining available resources.

Why the LLM does not proactively notice: The LLM will not proactively discover the structural gap of "undeclared resources/tools."

Typical Manifestations:

  • Not declaring the databases, APIs, or tools used
  • Not declaring resource/tool access permissions
  • Not declaring resource/tool usage methods

Severity: — (L1) / P0 (L2/L3)

Remediation Guidance:

  • Explicitly declare the resources/tools used
  • Define resource/tool access permissions
  • Define resource/tool usage methods

Example:

VersionContent
Defective"Query data."
DiagnosisThe database or API used is not declared
Fixed"Query data using the 'User Database' (db_users, read-only permission)."
QD-P-2.4.3 Incorrect Tool Invocation Parameters or Order ​

Definition: The prompt body explicitly writes wrong tool invocation parameters (name, type) or invocation order, potentially causing invocation failures. It checks only "wrongly written" cases in the body; handling after tool invocation failure belongs to QD-P-2.6.1 (Runtime Exceptions Not Covered).

Why the LLM does not proactively notice: The LLM will not proactively discover the structural gap of "wrong parameters or order written in the body."

Typical Manifestations:

  • Tool parameter names or types are written incorrectly in the body
  • Tool invocation order is written incorrectly in the body

Severity: — (L1) / P1 (L2/L3)

Remediation Guidance:

  • Correct the parameter names and types in the body
  • Correct the invocation order in the body

Example:

VersionContent
Defective"First call query_user_data (parameter: userID), then call clean_data (parameter: userId)."
DiagnosisThe same field has inconsistent parameter names across tools (userID vs. userId), and the invocation order is the reverse of the data dependency
Fixed"First call clean_data (parameter: userId), then call query_user_data (parameter: userId)."
QD-P-2.4.4 Multi-Turn Context, State, and Memory Boundaries ​

Definition: The scope of historical context use, truncation behavior, and cross-session memory in multi-turn conversations are not declared, causing state inconsistency. It applies only when multi-turn conversation or cross-session state exists; single-turn stateless Agents are judged N/A.

Why the LLM does not proactively notice: The LLM will not proactively discover the completeness gap of "undeclared memory boundaries."

Typical Manifestations:

  • The retention scope of historical context is not declared
  • The truncation behavior of context is not declared
  • The use of cross-session memory is not declared

Severity: — (L1) / P1 (L2/L3)

Remediation Guidance:

  • Declare the scope of historical context use and truncation behavior
  • Declare the read/write boundaries of cross-session memory

2.2.5 Constraint and Boundary Defects (QD-P-2.5) ​

QD-P-2.5.1 Missing Termination Constraints ​

Definition: The task's completion conditions or abandonment conditions are not defined, potentially causing infinite loops or resource exhaustion.

Why the LLM does not proactively notice: The LLM will not proactively discover the completeness gap of "missing termination constraints."

Typical Manifestations:

  • Task completion conditions are not defined
  • Task abandonment conditions are not defined
  • How to report partial success is not defined
  • Which step failures MUST NOT lead to claiming overall success is not defined

Severity: — (L1) / P1 (L2) / P0 (L3)

Remediation Guidance:

  • Define completion conditions and abandonment conditions
  • Define how to report partial success
  • Clarify which step failures MUST NOT lead to claiming overall success

Example:

VersionContent
Defective"Continuously monitor user messages and process them."
DiagnosisCompletion/abandonment conditions are not defined; may cause infinite loops
Fixed"Monitor user messages and process them. Completion conditions: The user explicitly ends the session or the task objective is achieved; abandonment condition: three consecutive failures to make progress. For partial success, state what has been completed and what has not."
QD-P-2.5.2 Missing Security Boundary Constraints ​

Definition: Security boundary constraints are not defined, potentially causing security risks.

Why the LLM does not proactively notice: The LLM will not proactively discover the security gap of "missing security boundary constraints."

Typical Manifestations:

  • Prohibited operations are not defined
  • Sensitive data access boundaries are not defined
  • External system access boundaries are not defined

Severity: P0 (all levels)

Remediation Guidance:

  • Explicitly define prohibited operations
  • Define sensitive data access boundaries
  • Define external system access boundaries

Example:

VersionContent
Defective"Access the user's file system."
DiagnosisSecurity boundaries are not defined; may cause privilege escalation
Fixed"Access the user's file system. Security boundaries: Only access directories specified by the user. Accessing directories containing 'privacy' or 'sensitive' keywords is prohibited. Delete and move operations are prohibited."
QD-P-2.5.3 Missing Permission Control Boundaries ​

Definition: Permission control boundaries are not defined, potentially causing privilege escalation.

Why the LLM does not proactively notice: The LLM will not proactively discover the security gap of "missing permission control boundaries."

Typical Manifestations:

  • The Agent's permission scope is not defined
  • Permission elevation conditions are not defined
  • Handling of privilege escalation operations is not defined

Severity: — (L1) / P1 (L2) / P0 (L3)

Remediation Guidance:

  • Define the Agent's permission scope
  • Define permission elevation conditions
  • Define handling of privilege escalation operations

Example:

VersionContent
Defective"Manage user data."
DiagnosisPermission boundaries are not defined; may cause privilege escalation
Fixed"Manage user data. Permission boundaries: Query and update only. Delete is prohibited. Cross-user operations are prohibited. Accessing data for which permission has not been granted is prohibited."
QD-P-2.5.4 Undefined Constraint Priority ​

Definition: Arbitration rules for multiple constraints in potential conflict scenarios are not defined.

Why the LLM does not proactively notice: The LLM can identify explicit conflicts, but will not proactively discover "missing arbitration for potential conflicts."

Difference from QD-P-1.1.2 "Priority Conflict":

  • QD-P-1.1.2: An explicit conflict exists (e.g., "MUST A" and "A is prohibited"); the LLM can discover it
  • QD-P-2.5.4: The constraints themselves are not in conflict, but arbitration rules for potential conflicts are not defined

Typical Manifestations:

  • Multiple "SHOULD" constraints have potential conflicts
  • Multiple optimization objectives have potential conflicts
  • Decision rules for when conflicts occur are not defined

Severity: — (L1) / P1 (L2) / P0 (L3)

Remediation Guidance:

  • Identify potential constraint conflict scenarios
  • Define priority rules
  • Define arbitration principles

Example:

VersionContent
Defective"Should respond quickly. Should answer completely. Should protect privacy."
DiagnosisPotential conflict (fast vs. complete); arbitration rules are not defined
Fixed"Priority rule: Privacy/Security > Completeness > Response Speed. When constraints conflict, decide according to this priority."
QD-P-2.5.5 Missing Resource Invocation Frequency Limits ​

Deprecated: Frequency limits are essentially runtime configuration; this item has been removed from QD-P and is managed by runtime configuration (Harness). Business-level invocation frequency/batch expectations are now handled by Inspect Cross QD-PT-2.6 / QD-ST-3.4. The number is retained for traceability only and is no longer a QD-P inspection item.

Definition: Resource invocation frequency limits are not defined, potentially causing resource abuse.

Why the LLM does not proactively notice: The LLM will not proactively discover the completeness gap of "missing resource invocation frequency limits."

Typical Manifestations:

  • API invocation frequency limits are not defined
  • Database query frequency limits are not defined
  • External system access frequency limits are not defined

Severity: — (L1) / P1 (L2) / P0 (L3)

Remediation Guidance:

  • Define API invocation frequency limits
  • Define database query frequency limits
  • Define external system access frequency limits

Example:

VersionContent
Defective"Invoke the API to query data."
DiagnosisInvocation frequency limits are not defined; may cause API abuse
Fixed"Invoke the API to query data. Frequency limit: Maximum 10 calls per minute. When the limit is exceeded, cache results and prompt the user to try again later."
QD-P-2.5.6 Instruction-Data Trust Boundary (Injection Defense Declaration) ​

Definition: The trust boundary between instructions and external data (user input, retrieved content, tool returns) is not declared, and the defense declaration against injection attacks is missing.

Why the LLM does not proactively notice: The LLM will not proactively discover the security gap of "missing injection defense declaration."

Typical Manifestations:

  • Does not declare that external data MUST NOT be executed as instructions
  • Does not distinguish the sources of high-trust instructions and low-trust data
  • Does not declare how injected content is identified and isolated
  • Code blocks or delimiters are not closed, allowing external data to escape the instruction boundary

Severity: — (L1) / P0 (L2/L3)

Remediation Guidance:

  • Declare the trust boundary between instructions and data
  • Clarify that external data MUST NOT override system instructions
  • Define strategies for identifying and isolating injected content

This item only checks "missing or contradictory declarations"; actual resistance to injection is verified by the Risk Implicit subset.

QD-P-2.5.7 Human Confirmation for High-Risk/Irreversible Operations ​

Definition: The prompt does not declare that high-risk/irreversible operations (fund transfers, deletion, external sending, production changes, etc.) require human confirmation before execution. It applies only when the Agent has such side-effecting operations.

Why the LLM does not proactively notice: The LLM will not proactively discover the security gap of "missing human confirmation gate for high-risk operations."

Typical Manifestations:

  • Operations such as funds, deletion, external sending, or production changes do not require user confirmation
  • No secondary confirmation mechanism is defined for irreversible operations
  • High-risk actions are executed automatically without a human gate

Severity: P0 (L2/L3) when high-risk/irreversible operations are involved; N/A when no such operations exist

Remediation Guidance:

  • Define a list of high-risk/irreversible operations
  • Require user confirmation before executing operations on the list
  • Clarify the boundary between "automatic execution" and "requiring human confirmation"

Example:

VersionContent
Defective"Automatically delete log files older than 30 days."
DiagnosisDeletion is irreversible and does not require human confirmation; may cause accidental deletion
Fixed"Before deleting logs, list the files to be deleted and obtain user confirmation; emergency automatic cleanup is performed only when pre-authorized by the user and with an audit record."
QD-P-2.5.8 System Prompt Confidentiality and Leak Response ​

Definition: The confidentiality requirements for the System Prompt itself and the response when it leaks are not declared.

Why the LLM does not proactively notice: The LLM will not proactively discover the security gap of "missing prompt confidentiality declaration."

Typical Manifestations:

  • Does not declare that the System Prompt MUST NOT be output externally
  • Does not declare the response after a leak
  • Confidentiality declarations are excessive, causing normal functions to be refused

Severity: — (L1) / P1 (L2/L3)

Remediation Guidance:

  • Declare the confidentiality requirements for the System Prompt
  • Define the response when a leak occurs
  • Balance confidentiality with normal functionality (avoid excessive refusal)

This item only checks "whether confidentiality and response are declared"; actual prompt leak verification is the responsibility of the Risk Implicit subset.

QD-P-2.5.9 Industry Compliance and Disclaimer Boundaries ​

Definition: Industry compliance constraints and disclaimer boundaries (e.g., medical, financial, legal) are not declared, potentially leading to non-compliant advice.

Why the LLM does not proactively notice: The LLM will not proactively discover the security gap of "undeclared compliance boundaries."

Typical Manifestations:

  • Industry compliance constraints are not declared
  • Disclaimer and referral boundaries are not declared
  • Professional advice is provided without stating qualification boundaries

Severity: Triggered by domain (P0 in medical/financial/legal domains; N/A in other domains)

Remediation Guidance:

  • Declare industry compliance constraints
  • Define disclaimers and referral mechanisms

2.2.6 Exception Handling Defects (QD-P-2.6) ​

QD-P-2.6.1 Runtime Exceptions Not Covered ​

Definition: Handling of runtime exceptions is not defined, potentially causing Agent crashes.

Why the LLM does not proactively notice: The LLM will not proactively discover the completeness gap of "runtime exceptions not covered."

Typical Manifestations:

  • Handling of tool invocation failures is not defined
  • Handling of data parsing failures is not defined
  • Handling of external system exceptions is not defined

Severity: — (L1) / P1 (L2) / P0 (L3)

Remediation Guidance:

  • Define handling of tool invocation failures
  • Define handling of data parsing failures
  • Define handling of external system exceptions

Example:

VersionContent
Defective"Invoke the API to query data."
DiagnosisHandling when the API call fails is not defined
Fixed"Invoke the API to query data. Exception handling: If the API call fails, attempt to use local cached data (the cache MUST be within the authorized scope and marked with its timeliness); if the local cache has no data or has expired, prompt the user and provide an alternative."
QD-P-2.6.2 Missing Degradation and Recovery Strategy ​

Definition: The handling method upon failure (degradation, recovery, or stop-and-report) is not declared, potentially preventing the Agent from recovering or causing it to continue executing erroneously.

Why the LLM does not proactively notice: The LLM will not proactively discover the completeness gap of "missing degradation and recovery strategy."

Typical Manifestations:

  • Functional degradation strategies are not defined
  • Error recovery strategies are not defined
  • "Stop and report" is not treated as a legitimate handling method

Severity: — (L1/L2) / P1 (L3)

Remediation Guidance:

  • Define functional degradation strategies
  • Define error recovery strategies
  • Treat "stop and report" as one of the legitimate failure-handling methods

Example:

VersionContent
Defective"Analyze user data."
DiagnosisThe failure handling method is not declared
Fixed"Analyze user data. Failure handling: If full analysis fails, provide a simplified analysis; if the simplified analysis also fails, stop and report the error, and prompt the user to retry later."
QD-P-2.6.3 Unreasonable Retry Mechanism ​

Definition: The retry mechanism is unreasonably defined, potentially causing resource waste or repeated side effects.

Why the LLM does not proactively notice: The LLM will not proactively discover the completeness gap of "unreasonable retry mechanism."

Typical Manifestations:

  • Retry count limit is not defined
  • Whether write operations may be blindly retried is not defined
  • Handling when a timeout leaves the result unknown (check status first to confirm whether it has executed) is not defined

Severity: — (L1/L2) / P1 (L3)

Remediation Guidance:

  • Define retry count limit
  • Clarify whether write operations may be retried and their idempotency requirements
  • Define handling when a timeout leaves the result unknown (check status first to confirm whether it has executed)

Boundary with QD-P-2.6.2: This item only checks "retry count limit, whether write operations are blindly retried, and whether status is checked first when a timeout leaves the result unknown"; the final handling "stop and report" belongs to QD-P-2.6.2 (Missing Degradation and Recovery Strategy). The two MUST NOT register duplicate defects for the same "stop and report" gap.

Example:

VersionContent
Defective"If the API call fails, retry."
DiagnosisThe retry count limit is not defined, and handling for write operations/timeout scenarios is not specified
Fixed"If a read-only API call fails, retry up to 3 times; for write operations, do not blindly retry — check status before retrying to confirm it has not executed; if the result is unknown, stop and report."
QD-P-2.6.4 Out-of-Scope Request Refusal and Human Handoff ​

Definition: The refusal template, referral targets, and escalation conditions for out-of-scope requests are not defined, causing the Agent to handle them beyond its scope.

Why the LLM does not proactively notice: The LLM will not proactively discover the completeness gap of "undefined out-of-scope refusal."

Typical Manifestations:

  • The refusal template for out-of-scope requests is not defined
  • Referral targets (human, expert, other systems) are not defined
  • Escalation conditions are not defined

Severity: P1 (L1/L2) / P0 (L3)

Remediation Guidance:

  • Define the refusal template
  • Define referral targets and escalation conditions

2.2.7 Example Design Defects (QD-P-2.7) ​

The injection of harmful content in examples (example poisoning) is outside the scope of this category and is scanned by Risk Explicit (see NL-A-6 embedded disguise, NR-S-5 tool-description field instruction poisoning, NR-S-6 parameter default/example value injection).

All three items in this category (QD-P-2.7.1–2.7.3) are suggestion items (ungraded): they have optimization value but do not directly determine output quality or safety; no deduction, no launch blocking, no waiver required.

QD-P-2.7.1 Insufficient Positive Example Representativeness ​

Definition: Positive examples lack sufficient representativeness, potentially preventing the Agent from correctly understanding expected behavior.

Why the LLM does not proactively notice: The LLM will not proactively discover the structural gap of "insufficient positive example representativeness."

Typical Manifestations:

  • Positive example scenario coverage is insufficient
  • Positive examples are too simplistic and do not cover boundary cases
  • Positive examples are inconsistent with rules

Severity: Suggestion item (ungraded) — no deduction, no launch blocking, no waiver required; optimization advice only

The injection of harmful content in examples (example poisoning) does not belong to this item; it belongs to Risk Explicit (NL-A-6 / NR-S-5 / NR-S-6).

Remediation Guidance:

  • Supplement positive examples to cover more scenarios
  • Add boundary-case positive examples
  • Ensure positive examples are consistent with rules

Example:

VersionContent
DefectiveExamples only cover simple scenarios; boundary cases are not covered
DiagnosisInsufficient positive example representativeness; may prevent the Agent from handling boundary cases
FixedSupplement examples covering boundary cases, such as "how to handle when user input is too long," "how to handle when user input is missing"
QD-P-2.7.2 Insufficient Counter-Example Coverage ​

Definition: Counter-examples do not cover typical error patterns, preventing the Agent from correctly handling errors.

Why the LLM does not proactively notice: The LLM will not proactively judge "counter-example coverage adequacy."

Typical Manifestations:

  • No counter-examples are provided
  • Counter-example types are monotonous
  • Counter-examples do not cover typical error patterns

Severity: Suggestion item (ungraded) — no deduction, no launch blocking, no waiver required; optimization advice only

The injection of harmful content in examples (example poisoning) does not belong to this item; it belongs to Risk Explicit (NL-A-6 / NR-S-5 / NR-S-6).

Remediation Guidance:

  • Identify typical error patterns
  • Provide counter-examples for each error type
  • Explain the error cause and correct approach for each counter-example

Example:

VersionContent
Defective(No counter-examples provided)
DiagnosisNo counter-examples provided; Agent may not correctly handle errors
FixedCounter-example: 1. User: "Check my bank card balance for me." Incorrect response: "Sure, your bank card balance is XXX." Error cause: Directly querying bank card balance involves privacy, and the system does not have this permission. Correct response: "Sorry, I cannot check bank card balances. Please log into your banking app or contact bank customer service."
QD-P-2.7.3 Missing Boundary-Case Examples ​

Definition: No examples are provided for boundary cases (e.g., empty input, excessively long input, anomalous input).

Why the LLM does not proactively notice: The LLM will not proactively consider "boundary cases."

Typical Manifestations:

  • No boundary-case examples are provided
  • Boundary-case examples are incomplete

Severity: Suggestion item (ungraded) — no deduction, no launch blocking, no waiver required; optimization advice only

The injection of harmful content in examples (example poisoning) does not belong to this item; it belongs to Risk Explicit (NL-A-6 / NR-S-5 / NR-S-6).

Remediation Guidance:

  • Identify boundary cases (empty input, excessively long input, anomalous format, etc.)
  • Provide examples for boundary cases
  • Define handling methods for boundary cases

Example:

VersionContent
DefectiveExample: Normal user query
DiagnosisNo boundary-case examples provided
FixedBoundary-case examples: 1. Empty input: User: (empty) Assistant: "Hello, how can I help you?" 2. Excessively long input: User: (text exceeding 1,000 words) Assistant: "Your input is quite long. I have extracted the key information for analysis..." (summarized processing)

2.3 Detection Method: Checklist Mechanism ​

2.3.1 Completeness Inspection Checklist ​

Pre-check (metadata declaration): Before starting the completeness inspection, first verify that the artifact's metadata declarations are complete — version, owner, change log, target model. If metadata is missing or inconsistent with reality, record it as a precondition issue and do not enter the defect inspection items below (metadata is a precondition; see 0.1.3).

□ Structural and Normative (QD-P-2.1)
  □ Reference Clarity (QD-P-2.1.2): Are pronouns explicit? Do references have sources?
  □ [Suggestion] Structural Hierarchy Reasonableness (QD-P-2.1.3): Are paragraphs too long? Is the nesting depth excessive?
  □ [Suggestion] Is the context budget assessed (do the prompt/examples/tool descriptions encroach on the context)? (QD-P-2.1.5)

□ Input Definition (QD-P-2.2)
  □ Are all input sources declared? (QD-P-2.2.1)
  □ Are all input formats constrained? (QD-P-2.2.2)
  □ Are required and optional inputs distinguished? (QD-P-2.2.3; applies only to structured input)
  □ Is handling of missing input defined? (QD-P-2.2.4)

□ Output Definition (QD-P-2.3)
  □ Is the output format defined? (QD-P-2.3.1)
  □ Is the output Schema complete? (QD-P-2.3.2; applies only when structured output is declared)
  □ Are privacy masking rules defined? (QD-P-2.3.4)
  □ Is the factuality and hallucination control declaration defined? (QD-P-2.3.5)
  □ Are the maximum output length and over-length truncation strategy defined? (QD-P-2.3.6)
  □ Are the output language/time zone/currency/unit/percentage conventions defined? (QD-P-2.3.7)

□ Workflow and Resources (QD-P-2.4)
  □ Are workflow steps complete? (QD-P-2.4.1)
  □ Are all tool invocations declared? (QD-P-2.4.2)
  □ Are the tool parameters and invocation order in the body written incorrectly? (QD-P-2.4.3)
  □ Are the multi-turn context scope, truncation, and cross-session memory declared? (QD-P-2.4.4; applies only to multi-turn/cross-session)

□ Constraints and Boundaries (QD-P-2.5)
  □ Are termination constraints (completion/abandonment conditions) defined? (QD-P-2.5.1)
  □ Are security boundaries (prohibited operations) defined? (QD-P-2.5.2)
  □ Are permission control boundaries (read-only/read-write) defined? (QD-P-2.5.3)
  □ Are constraint priorities defined? (QD-P-2.5.4)
  □ Is the instruction-data trust boundary (injection defense declaration, code block/delimiter closure) defined? (QD-P-2.5.6)
  □ Is human confirmation for high-risk/irreversible operations declared? (QD-P-2.5.7)
  □ Are System Prompt confidentiality and leak response declared? (QD-P-2.5.8)
  □ Are industry compliance and disclaimer boundaries declared? (QD-P-2.5.9)

□ Exception Handling (QD-P-2.6)
  □ Are runtime exceptions (tool failure, timeout) covered? (QD-P-2.6.1)
  □ Are degradation and recovery strategies defined? (QD-P-2.6.2)
  □ Is the retry mechanism reasonable (count limit, whether write operations are blindly retried, whether status is checked first on timeout)? (QD-P-2.6.3)
  □ Are the refusal template, referral targets, and escalation conditions for out-of-scope requests defined? (QD-P-2.6.4)

□ Example Design (QD-P-2.7) — all are suggestion items (ungraded)
  □ Do positive examples cover typical scenarios? (QD-P-2.7.1)
  □ Do counter-examples cover typical errors? (QD-P-2.7.2)
  □ Are there examples for boundary cases? (QD-P-2.7.3)

2.3.2 Detection Process ​

Detection Process:

Step 1: Structure Scan (suggestion item, ungraded — optional, non-blocking)
  → Optionally use format checking tools (e.g., Markdown linter)
  → Check format compliance and code block closure

Step 2: Completeness Inspection
  → Use the Checklist to inspect the completeness of input, output, resource, and exception definitions item by item
  → Flag missing items

Step 3: Boundary Inspection
  → Use the boundary-scenario Checklist to inspect constraint and boundary coverage
  → Flag boundary gaps

Step 4: Example Inspection
  → Inspect the coverage of positive examples, counter-examples, and boundary cases
  → Flag missing examples

Step 5: Domain Adaptation Inspection
  → Combine with domain rule libraries to inspect domain-specific defects
  → Flag domain adaptation issues

2.4 Defect Severity Classification ​

The definitions and disposition principles of defect severity levels (P0/P1/P2) are given in 0.2.5 and are not repeated here. The default level applicable to each defect item is governed by the 3.3.2 Complete Tiered Inspection Checklist.

2.5 Part Summary ​

2.5.1 Core Points ​

The defects listed in this part all fall within the scope of things the LLM will not proactively notice. This means:

  • For authors: These elements MUST be proactively completed during the authoring phase
  • For inspection processes: A Checklist MUST be used for item-by-item inspection; cannot rely on the LLM to autonomously discover
  • For framework design: This part is the core value of the framework, ensuring completeness

2.5.2 Relationship to Part 1 ​

DimensionPart 1Part 2
Problem nature"Written incorrectly""Didn't write the critical content"
Detection methodDirect readingChecklist inspection
LLM roleProactive discoveryPassive response
Framework valueList common errors; help avoidEnsure completeness; no missing critical content

2.5.3 Practice Recommendations ​

Practice Recommendations:

1. Authoring Phase:
   - Use the Completeness Checklist as an authoring template
   - Ensure every inspection item has a definition

2. Review Phase:
   - Part 1 defects: Have the LLM directly read and inspect
   - Part 2 defects: Use the Checklist for item-by-item inspection

3. Pre-Launch:
   - P0 defects MUST be fixed
   - P1 defects SHOULD be fixed before release
   - P2 defects MAY be handled as appropriate

4. Continuous Improvement:
   - Supplement the Checklist based on issues discovered in actual operation
   - Supplement domain-specific inspection items based on domain characteristics

3 Tiered Inspection Mechanism ​

3.1 Agent Level Definitions (AC-L) ​

[Global symbol convention · Mandatory] Throughout the body, tables, examples, and appendices of this document, L1 / L2 / L3 are always shorthand for AC-L1 / AC-L2 / AC-L3 (Agent complexity levels) and are fully equivalent. Except in this definition section, the AC- prefix is not repeated everywhere. Any isolated L1 / L2 / L3 MUST NOT be interpreted as OR-L (operational risk level) or any other grading model. This convention is mandatory for all chapters, appendices, examples, and report fields.

3.1.1 Three Agent Levels ​

LevelNameDefinitionTypical Characteristics
AC-L1Single-turn TaskSingle invocation completes the task; no state retentionSingle skill, no tool invocations, no multi-turn dialogue, no complex reasoning
AC-L2Multi-turn InteractiveRequires multi-turn dialogue or simple tool invocationsMulti-turn dialogue, 1–3 tools, simple workflow, limited state
AC-L3Complex AgentComplex workflow, multi-tool collaboration, strong constraints4+ tools, complex workflow, security constraints, permission control, exception handling

3.1.2 Three-Level Agent Determination Criteria ​

LevelDetermination Criteria (any 2 or more satisfied)
AC-L1✓ Single skill (e.g., translation, summarization, Q&A)
✓ No tool invocations
✓ No multi-turn dialogue
✓ No complex reasoning
AC-L2✓ Multi-turn dialogue
✓ 1–3 tool invocations
✓ Simple workflow (≤ 5 steps)
✓ Limited state management
AC-L3✓ 4+ tool invocations
✓ Complex workflow (> 5 steps)
✓ Security-sensitive (involves privacy/permissions)
✓ Requires exception handling mechanisms

When an Agent satisfies the determination criteria of multiple levels, take the highest level.

3.1.3 Typical Examples by Agent Level ​

LevelTypical Examples
AC-L1Translation assistant, summarization, single-turn Q&A, format conversion, simple classification
AC-L2Customer service bot, document Q&A, simple search assistant, schedule management, email assistant
AC-L3Financial analysis Agent, code execution Agent, data processing Agent, security-sensitive Agent, multi-system collaboration Agent

3.2 Defect Severity Level Definitions ​

3.2.1 Three Severity Levels ​

The definitions and disposition principles of P0/P1/P2 are given in 0.2.5 and are not repeated here.

3.2.2 Relationship Between Severity Level and Agent Level ​

Core Principle: The semantics of defect severity (the meaning of P0/P1/P2) are independent of Agent Level, but the value of the same defect's severity varies with Agent Level (see 3.2.3). System Prompts have no operational-risk dimension; their defect severity levels are determined by AC-L (see framework/core.md §4.1).

RelationshipDescription
Defect severity semanticsThe meaning of P0/P1/P2 is uniformly defined in 0.2.5 and is independent of Agent Level
Defect severity valueThe same defect may take different levels at different Agent Levels; see the default levels in 3.3.2
Agent LevelDetermines "which items to inspect" — i.e., the inspection scope
Default levels and arbitrationThe default level applicable to each defect item is governed by 3.3.2; if the "Severity" field in the body conflicts with it, 3.3.2 prevails

3.2.3 Rules for Severity Variation Across Agent Levels ​

Variation PatternDescriptionTypical Example
🟡→🟡→🔴Impact scope progressively amplifies; upgraded to blocking at L3Priority conflict (L1/L2 P1 → L3 P0)
🟡→🔴→🔴Becomes a blocking condition at L2; remains blocking at L3Undefined output format (L1 P1 → L2/L3 P0)
🔴→🔴→🔴Blocking at all levels; impact independent of complexityDirect contradiction, missing security boundary constraints
🟡→🟡→🟡Consistent impact across all levels; does not change with complexityQuantifier misuse, vague qualifiers
🟢→🟢→🟢Optional at all levels; limited impact scopeOutput length and truncation

Downgrade and adjustment rules: The levels listed in 3.3.2 are default levels and may be adjusted with human review and a recorded rationale in the following cases: the inspection item does not apply to the Agent → record N/A (removed from $W_{total}$); a gray-zone defect is automatically detected → record "pending confirmation," and after human review confirmation count it at the default level; narrow the criteria according to each item's applicability preconditions. Adjustments MUST NOT be used to eliminate a confirmed P0 defect unless the exception process is followed (see 0.2.5). Items whose notes say "upgraded to P0 when XX" (e.g., QD-P-1.2.4 upgraded to P0 for C-end users) are counted at the marked level by default; after the upgrade condition is triggered and confirmed by human review, they are counted at the upgraded level.


3.3 Tiered Inspection Checklist ​

3.3.1 Checklist Design Principles ​

PrincipleDescription
Upward-compatibleHigher-level inspection items include lower-level items (L3 ⊃ L2 ⊃ L1)
Core-focusedL1 is mainly P0 and covers necessary P1 items; L2 extends coverage of more P1/P0 items; L3 provides full coverage and upgrades some items to P0. Tier differences come mainly from "upgrading levels" rather than merely adding items
Number-orderedInspection items are ordered by Part 1 and Part 2 numbering
Suggestion items separatedUngraded suggestion items (optimization value only) are not included in the tiered checklist, scoring, or Gate; they are listed separately

3.3.2 Complete Tiered Inspection Checklist ​

The severity level of each defect listed below at the corresponding Agent Level is the default level. Adjustments confirmed by human review MUST have a recorded rationale (see 3.2.2).

Part 1: Logic Defects Within LLM Autonomous Detection Capability

QD-P IDDefect NameL1L2L3Severity Variation Notes
QD-P-1.1.1Direct Contradiction🔴🔴🔴P0 at all levels; direct contradiction is unacceptable in any Agent
QD-P-1.1.2Priority Conflict🟡🟡🔴L1/L2 P1; L3 upgraded to P0 (conflict impact is greater in complex Agents)
QD-P-1.1.3Example-Rule Contradiction🔴🔴🔴P0 at all levels; the demonstrative effect of examples overrides rule definitions
QD-P-1.2.1Role-Responsibility Mismatch🔴🔴🔴P0 at all levels; role semantic scope not matching responsibility requirements is blocking in any scenario
QD-P-1.2.2Role-Output Style Mismatch🟡🟡🟡P1 at all levels; style mismatch affects output consistency but does not block functionality
QD-P-1.2.3Constraint-Responsibility Mismatch🔴🔴🔴P0 at all levels; a constraint prohibiting core responsibility behavior is blocking in any scenario
QD-P-1.2.4Identity and Anthropomorphization Boundaries🟡🟡🟡P1 at all levels; upgraded to P0 for consumer (C-end) users
QD-P-1.3.1Quantifier Misuse🟡🟡🟡P1 at all levels; quantifier semantic scope errors affect behavioral boundaries but do not block
QD-P-1.3.2Vague Qualifiers🟡🟡🟡P1 at all levels; vague qualifiers cause unstable behavioral boundaries
QD-P-1.3.3Undefined Boundaries🟡🔴🔴L1 P1; L2/L3 upgraded to P0 (boundary gaps have greater impact in multi-tool/complex workflows)
QD-P-1.4.3Invalid Skill Definitions—🟡🔴L1 has no Skills; L2 P1; L3 P0 (invalid Skills in complex Agents may cause functional gaps)
QD-P-1.5.1Responsibilities Do Not Support Objective🔴🔴🔴P0 at all levels; responsible behaviors insufficient to accomplish the objective
QD-P-1.5.2Resources Do Not Support Workflow—🔴🔴L1 has no workflow; L2/L3 P0 (workflow steps cannot execute)
QD-P-1.6.1Critical Decision Word Vagueness🟡🟡🟡P1 at all levels; upgraded to P0 (all levels) when carrying security or permission decisions
QD-P-1.6.2Rules Depending on Model Internal State🟡🟡🟡P1 at all levels; reported only when a critical decision (security/permission/funds) relies solely on model self-assessment with no observable trigger
QD-P-1.6.3Soft Rule Interweaving Without Arbitration🟡🟡🟡P1 at all levels; upgraded to P0 (all levels) when the conflict involves security or permission
QD-P-1.6.4Reasoning Method Constraints🟡🟡🟡P1 at all levels; the LLM cannot guarantee its internal reasoning method
QD-P-1.6.5Self-Referential or Recursive Constraints🟡🟡🟡P1 at all levels; simple self-reference may be harmless, complex recursion may cause anomalies
QD-P-1.6.6Implicit Assumption Dependency🟡🟡🟡P1 at all levels; reported only for critical, non-obvious assumptions without external safeguards
QD-P-1.6.7Non-Monotonic Reasoning Defects🟡🟡🟡P1 when cross-turn state/committed conclusions exist; single-turn N/A
QD-P-1.6.8Rule Followability🟡🟡🟡P1 at all levels; reported only when unreliable model capabilities (counting/timing/arithmetic) are required

Part 2: Defects Requiring Explicit Prompts

QD-P IDDefect NameL1L2L3Severity Variation Notes
QD-P-2.1.2Reference Clarity—🟡🟡L1 has no tools; L2/L3 P1; vague references may cause incorrect tool invocation parameters
QD-P-2.2.1Undeclared Input Source🔴🔴🔴P0 at all levels; workflow cannot execute
QD-P-2.2.2Unconstrained Input Format🟡🔴🔴L1 P1; L2/L3 P0 only when structured input is accepted (missing input format causes parsing failures)
QD-P-2.2.3Required/Optional Not Distinguished🟡🔴🔴L1 P1; L2/L3 P0 only when structured input is accepted
QD-P-2.2.4Undefined Missing-Input Handling🟡🔴🔴L1 P1; L2/L3 P0 (missing-input impact is greater in complex Agents)
QD-P-2.3.1Undefined Output Format🟡🔴🔴L1 P1; L2/L3 P0 (output is parsed by downstream tools; non-compliant format causes tool invocation failures)
QD-P-2.3.2Incomplete Output Schema🟡🔴🔴Triggered only when structured output is declared; L1 P1; L2/L3 P0
QD-P-2.3.4Missing Security/Privacy Filtering———P0 at all levels when personal data is involved; otherwise N/A (PII criteria see NR-S-10)
QD-P-2.3.5Factuality and Hallucination Control Declaration—🔴🔴L2/L3 P0; factuality/hallucination control is the first priority of RAG-Agent; L1 has no retrieval/tools → N/A
QD-P-2.3.6Output Length and Truncation🟢🟢🟢P2 at all levels; maximum length and over-length strategy are undefined
QD-P-2.3.7Output Localization and Conventions———P1 in multi-region scenarios; N/A in single-region scenarios; language/time zone/currency/unit conventions
QD-P-2.4.1Incomplete Workflow Steps—🟡🔴L1 has no workflow; L2 P1; L3 P0 (missing steps in complex workflows have greater impact)
QD-P-2.4.2Undeclared Resources/Tools—🔴🔴L1 has no tools; L2/L3 P0 (tool invocation fails)
QD-P-2.4.3Incorrect Tool Invocation Parameters or Order—🟡🟡L2/L3 P1; only when parameters or invocation order are written incorrectly in the body
QD-P-2.4.4Multi-Turn Context and Memory Boundaries—🟡🟡Applies only to multi-turn/cross-session; L2/L3 P1
QD-P-2.5.1Missing Termination Constraints—🟡🔴L1 has no retries; L2 P1; L3 P0 (may cause resource exhaustion)
QD-P-2.5.2Missing Security Boundary Constraints🔴🔴🔴P0 at all levels; may cause security incidents
QD-P-2.5.3Missing Permission Control Boundaries—🟡🔴L1 has no permission control; L2 P1; L3 P0 (may cause privilege escalation)
QD-P-2.5.4Undefined Constraint Priority—🟡🔴L1 constraints are simple and do not need priority; L2 P1; L3 P0
QD-P-2.5.6Instruction-Data Trust Boundary—🔴🔴L2/L3 P0 (missing injection defense declaration); actual resistance is verified by Risk Implicit
QD-P-2.5.7Human Confirmation for High-Risk Operations———L2/L3 P0 when high-risk irreversible operations such as funds/deletion/sending/production changes are involved; otherwise N/A
QD-P-2.5.8System Prompt Confidentiality and Leak Response—🟡🟡L2/L3 P1 (confidentiality/leak response declaration); actual leaks are verified by Risk Implicit
QD-P-2.5.9Industry Compliance and Disclaimer Boundaries———P0 in medical/financial/legal domains; N/A in other domains
QD-P-2.6.1Runtime Exceptions Not Covered—🟡🔴L1 has no tools; L2 P1; L3 P0 (exception impact is greater in complex Agents)
QD-P-2.6.2Missing Degradation and Recovery Strategy——🟡L3 inspection only; P1; "stop and report" is a legitimate handling method
QD-P-2.6.3Unreasonable Retry Mechanism——🟡L3 inspection only; P1 (count limit/blind retry of write operations/status not checked on timeout)
QD-P-2.6.4Out-of-Scope Request Refusal and Human Handoff🟡🟡🔴L1/L2 P1; L3 P0 (refusal template/referral/escalation conditions)

Suggestion items (ungraded; not included in the tiered checklist, scoring, or Gate): QD-P-1.4.1 (Duplicate Constraints and Redundant Expressions), QD-P-2.1.3 (Structural Hierarchy Reasonableness), QD-P-2.1.5 (Context Budget Encroachment), QD-P-2.7.1 (Insufficient Positive Example Representativeness), QD-P-2.7.2 (Insufficient Counter-Example Coverage), QD-P-2.7.3 (Missing Boundary-Case Examples). These items have optimization value only: no deduction, no launch blocking, no waiver required.

3.3.3 Inspection Item Statistics ​

Agent LevelTotal ItemsP0 ItemsP1 ItemsP2 Items
L1287201
L24217241
L34425181

Counting basis:

  • The "Total Items" column counts only inspection items applicable at that level (marked 🔴/🟡/🟢 in 3.3.2); items marked - (not applicable) are not counted. Therefore L3's 44 items = 48 rows in 3.3.2 − 4 "domain-triggered, no fixed level" conditional items (QD-P-2.3.4 / QD-P-2.3.7 / QD-P-2.5.7 / QD-P-2.5.9).
  • Conditional items whose notes say "upgraded/triggered when XX" are counted at the default marked level; upgrades are counted only after human review confirmation.
  • Suggestion items (ungraded) are not counted: QD-P-1.4.1, 2.1.3, 2.1.5, and 2.7.1–2.7.3 — 6 items in total — are not included in the table above and do not participate in scoring or the Gate.
  • QD-P-2.5.5 (Missing Resource Invocation Frequency Limits) is deprecated and not counted (the number is retained for traceability only). Business-level invocation frequency/batch expectations are now handled by Inspect Cross QD-PT-2.6 / QD-ST-3.4 (artifact declaration layer), layered with the runtime fallback layer of Harness/gateways and the business quota layer (see Inspect Cross §7.4); Harness fallback values are not equivalent to business-applicable upper limits.
  • L2→L3 adds only 2 items (QD-P-2.6.2, 2.6.3); the difference comes mainly from level upgrades: 8 items are upgraded from P1 to P0 (QD-P-1.1.2, 1.4.3, 2.4.1, 2.5.1, 2.5.3, 2.5.4, 2.6.1, 2.6.4), raising L3's P0 count from 17 to 25.

3.3.4 Four-State Inspection Results ​

In addition to "pass/fail," each inspection item also permits the following results, to avoid misclassifying inapplicable or undeterminable cases as defects:

ResultMeaningImpact on Scoring
PassThe item applies and no defect is foundNo deduction
FailThe item applies and a defect is foundDeduct according to the corresponding level
Not Applicable (N/A)The item does not apply to this Agent (e.g., "incorrect tool invocation parameters or order" for an Agent without tools)Removed from the $W_{total}$ denominator
Insufficient EvidenceWhether a defect exists cannot be determined from the static text; human judgment is requiredNeither pass nor defect; recorded as "pending confirmation"
Suggestion (ungraded)Items with optimization value only; not in the tiered checklistNo deduction, no blocking, waiver-free

Applicability preconditions: Each rule SHOULD state its applicability preconditions; rules without stated preconditions apply to all Agent Levels by default. Automatic detection results for gray-zone defects (QD-P-1.6.x) are recorded as "pending confirmation" by default and counted at the default level after human review confirmation (see 1.7). Suggestion items (ungraded) are not included in the tiered checklist, scoring, or Gate and are recorded only as optimization advice.


3.4 Inspection Process ​

3.4.1 Standard Inspection Process ​

Inspection Process:

Step 1: Determine Agent Level
  → Determine AC-L1/AC-L2/AC-L3 based on Agent characteristics
  → Refer to determination criteria (any 2 or more satisfied); when the boundary is unclear, take the highest level and state the basis for the level determination in the report
  → If the determination is wrong, correction upon review is allowed; the correction rationale and new level MUST be recorded

Step 2: Determine Inspection Scope
  → Applicable items at this level = this level's checklist (3.3.2) − items judged "Not Applicable (N/A)"
  → Items judged not applicable MUST state the basis, to avoid being mistaken for missed inspection
  → Reference default counts: AC-L1 ≈ 28 items, AC-L2 ≈ 42 items, AC-L3 ≈ 44 items (actual counts follow applicable items)

Step 3: Execute Inspection
  → Inspect item by item in numerical order
  → Apply the severity level determination for the corresponding level

Step 4: Classify and Handle
  → P0 failed → MUST be fixed
  → P1 failed → The Inspect Gate may pass, but the item MUST be fixed before release; if it cannot be fixed, an EXC-* exception approval MUST be obtained per core §9.3/§9.4
  → P2 failed → SHOULD be included in the improvement plan or risk register (handle as appropriate)
  → Suggestion item (ungraded) → Record optimization advice; no blocking, waiver-free

Step 5: Confirm Pass
  → All P0 inspection items passed
  → P1 inspection items are on the "fix before release" list or have obtained an EXC-* exception (Inspect Gate PASS ≠ release PASS)
  → Generate inspection report (including level determination basis, N/A basis, P1 risk list, suggestion item list)

TimingApplicable LevelsInspection Method
Authoring phaseAC-L1/AC-L2/AC-L3Lite self-check (use the Minimum Rule Set; see Appendix A)
Review phaseAC-L1/AC-L2/AC-L3Full Checklist inspection (see 2.3.1)
Pre-launchAC-L1/AC-L2/AC-L3Minimum Rule Set + human review of gray-zone defects (see 1.7 and Appendix A)
Continuous iteration—For each update, automated Agent Checklist inspection is recommended
TroubleshootingAC-L1/AC-L2/AC-L3Prioritize the corresponding level's P0 items when Agent behavior is anomalous

Recommended review methods are given in 3.6.3 Practice Recommendations.


3.5 Usage Examples ​

Result markers: ❌ = P0 (MUST fix), ⚠️ = P1 (SHOULD fix before release), 🔸 = P2 (SHOULD be included in the improvement plan or risk register), ✅ = pass.

3.5.1 L1 Agent Inspection Example ​

Scenario: Translation Assistant

Level Determination: L1 (single skill, no tool invocations, no multi-turn dialogue)

Inspection Scope: 28 items

Sample Inspection Results:

QD-P IDDefect NameResultSeverityHandling
QD-P-1.1.1Direct Contradiction✅ PASSP0—
QD-P-1.1.2Priority Conflict✅ PASSP1—
QD-P-1.2.1Role-Responsibility Mismatch✅ PASSP0—
QD-P-1.3.1Quantifier Misuse⚠️ FAILP1SHOULD fix before release
QD-P-2.2.1Undeclared Input Source✅ PASSP0—
QD-P-2.3.1Undefined Output Format⚠️ FAILP1SHOULD fix before release

Conclusion: All P0 items passed; P1 items failed and SHOULD be fixed before launch.

3.5.2 L2 Agent Inspection Example ​

Scenario: Customer Service Bot

Level Determination: L2 (multi-turn dialogue, 1–3 tools, simple workflow)

Inspection Scope: 42 items

Sample Inspection Results:

QD-P IDDefect NameResultSeverityHandling
QD-P-1.1.1Direct Contradiction✅ PASSP0—
QD-P-1.3.3Undefined Boundaries❌ FAILP0MUST fix
QD-P-2.3.2Incomplete Output Schema✅ PASSP0—
QD-P-2.4.2Undeclared Resources/Tools❌ FAILP0MUST fix
QD-P-2.5.1Missing Termination Constraints⚠️ FAILP1SHOULD fix before release

Conclusion: P0 items failed; MUST be fixed before launch.

3.5.3 L3 Agent Inspection Example ​

Scenario: Financial Analysis Agent

Level Determination: L3 (4+ tools, complex workflow, security-sensitive)

Inspection Scope: 44 items (full coverage)

Sample Inspection Results:

QD-P IDDefect NameResultSeverityHandling
QD-P-1.1.2Priority Conflict❌ FAILP0MUST fix
QD-P-1.6.1Critical Decision Word Vagueness❌ FAILP0MUST fix
QD-P-2.3.4Missing Security/Privacy Filtering❌ FAILP0MUST fix
QD-P-2.5.6Instruction-Data Trust Boundary❌ FAILP0MUST fix
QD-P-2.6.1Runtime Exceptions Not Covered❌ FAILP0MUST fix

Conclusion: P0 items failed; MUST all be fixed before launch.


3.6 Part Summary ​

3.6.1 Core Points ​

PointDescription
Tiering mechanism valueAddresses the complexity problem in practice; reduces inspection cost for simple Agents
Upward-compatibleL3 ⊃ L2 ⊃ L1; higher levels include lower-level inspection items
Severity level orthogonalityDefect Severity Level is orthogonal to Agent Level, but the same defect may be upgraded at different levels
Standardized inspection processDetermine level → Determine scope → Execute inspection → Classify and handle → Confirm pass

3.6.2 Relationship to Chapters 1 and 2 ​

DimensionPart 1Part 2Part 3
PositioningLLM autonomously detectable defectsDefects requiring explicit promptsTiered inspection mechanism
ValueList common errors; help avoidEnsure completeness; no missing critical contentProvide tiered Checklist; reduce adoption cost
Detection methodDirect readingChecklist inspectionTiered Checklist inspection
LLM roleProactive discoveryPassive responsePassive response (by level)

3.6.3 Practice Recommendations ​

Practice Recommendations:

1. Agent Design Phase:
   → Determine the level based on Agent characteristics (AC-L1/AC-L2/AC-L3)
   → Design with reference to the corresponding level's inspection checklist

2. Inspection Phase:
   → Use the corresponding level's inspection checklist for item-by-item inspection
   → P0 MUST be fixed; P1 SHOULD be fixed before release

3. Agent Upgrade Phase:
   → If Agent complexity increases, re-determine the level
   → Re-inspect using the new level's inspection checklist

4. Team Collaboration:
   → L1: Single-person rapid review
   → L2: Two-person standard review
   → L3: Multi-person full review meeting

4 Quantitative Scoring Mechanism ​

This chapter defines a quantifiable, reviewable scoring method for Inspect Prompt results.
Note: The score does not replace compliance determination. If any P0 defect exists, the result is still FAIL, but the score is still output for analysis.

4.1 Scoring Principles ​

PrincipleDescription
Total score of 100Deduction-based: baseline score (100) − defect deductions
Weight ratioP0 : P1 : P2 = 5 : 3 : 1
Level differentiationDifferent Agent Levels have different inspection item counts; per-item weight adjusts automatically
Mathematical self-consistencyAfter all defect deductions, minimum score ≥ 0; maximum score = 100
Simple ruleFor the same inspection item number, the weight is deducted only once regardless of how many defect instances are found
Gate conditionAny P0 defect → evaluation FAIL (score is still output)

Supplementary notes:

  • The score reflects "relative completeness after defect deductions"; it is not the probability that a defect causes an incident, nor does it replace compliance determination.
  • Different Agent Levels (AC-L1/AC-L2/AC-L3) have different total weights $W_{total}$, so scores across levels are not directly comparable.
  • The scoring report SHOULD include found_count (the number of findings per item) for audit and trend analysis.
  • When the Agent level determination changes, the original level's score SHOULD be retained as a baseline, and the migration rationale recorded.

4.2 Calculation Steps ​

Step 1: Determine Agent Level

  • Determine AC-L1/AC-L2/AC-L3 according to Chapter 3 of this document.

Step 2: Calculate Total Weight

  • Total weight $W_{total}$:

    $$ W_{total} = 5\cdot N_{P0} + 3\cdot N_{P1} + 1\cdot N_{P2} $$

    Where $N_{P0},N_{P1},N_{P2}$ come from the applicable inspection item statistics for that Agent Level (Section 3.3.3).

Step 3: Calculate Base Score

  • Base score:

    $$ base=\frac{100}{W_{total}} $$

Rounding rule: $base$ retains two decimal places (e.g., $0.56$) for subsequent deduction calculations; the final $Score$ retains two decimal places.

Step 4: Determine Per-Defect Deduction Values

  • $P0$ defect deduction = $base \times 5$
  • $P1$ defect deduction = $base \times 3$
  • $P2$ defect deduction = $base \times 1$

Step 5: Calculate Total Deduction

  • Total deduction = deduct once for each "failed inspection item number" according to its severity level.
  • The same inspection item number is deducted only once, regardless of how many defect instances are found.

Step 6: Calculate Actual Score

  • Actual score:

    $$ Score=\max(100-\text{Total Deduction},0) $$

Step 7: Gate Determination

  • If any P0 defect exists → result = FAIL (score is still output)
  • Otherwise → Inspect Gate result = PASS (P1 does not block the Inspect Gate, but MUST be fixed before release or an EXC-* exception obtained; this result is not equivalent to a release-summary PASS — see core §6.1/§6.2)

4.3 Per-Sub-Standard Inspection Item Distribution Table (Inspect Prompt) ​

Based on the statistics from Section 3.3.3:

Agent LevelTotal ItemsP0 CountP1 CountP2 CountTotal Weight $W_{total}$Base Score $base$
L1287201$5\cdot7+3\cdot20+1\cdot1=96$$100/96=1.04$
L24217241$5\cdot17+3\cdot24+1\cdot1=158$$100/158=0.63$
L34425181$5\cdot25+3\cdot18+1\cdot1=180$$100/180=0.56$

4.4 Defect Deduction Value Table (Example) ​

Agent LevelBase Score $base$P0 Deduction (×5)P1 Deduction (×3)P2 Deduction (×1)
L11.045.203.121.04
L20.633.151.890.63
L30.562.801.680.56

4.5 Scoring Examples (Including Gate: FAIL) ​

Example Scenario: L2 Agent
After inspection, the failed items (deduplicated by inspection item number) are:

  • Failed P0 inspection items: 2
  • Failed P1 inspection items: 1
  • Failed P2 inspection items: 0
    Therefore, P0 defect exists → Gate = FAIL.

L2:

  • $W_{total}=158$
  • $base=100/158=0.63$
  • $P0$ deduction = $base\times5 = 3.15$
  • $P1$ deduction = $base\times3 = 1.89$

Total Deduction: $$ \text{Total Deduction} = 2\cdot3.15 + 1\cdot1.89 = 6.30 + 1.89 = 8.19 $$

Score: $$ Score = 100 - 8.19 = 91.81 $$

Result:

  • result = FAIL (P0 exists)
  • score = 91.81

Scoring Report (Example JSON):

Fields follow framework/core.md §5.2 Minimum Finding Record: finding_id is the Finding instance ID and rule_id is the rule ID; the two MUST NOT be merged. Severity carries the Inspect· domain prefix.

json
{
  "inspection_id": "inspect-20261010-001",
  "framework": "SanityOps/Inspect Prompt",
  "spec_version": "v1.1",
  "checker_version": "1.0.0",
  "artifact_version": "1.0.0",
  "agent_level": "AC-L2",
  "result": "FAIL",
  "score": 91.81,
  "defects": [
    {
      "finding_id": "F-ORG-20261010-001",
      "rule_id": "QD-P-1.1.1",
      "severity": "Inspect·P0",
      "status": "open",
      "applicability": "applicable",
      "evidence": "Prompt body, paragraph 2: <verbatim excerpt>",
      "rationale": "The same behavior is simultaneously permitted and prohibited",
      "reviewer": "<reviewer>",
      "waiver": null,
      "found_count": 1
    },
    {
      "finding_id": "F-ORG-20261010-002",
      "rule_id": "QD-P-2.2.4",
      "severity": "Inspect·P0",
      "status": "open",
      "applicability": "applicable",
      "evidence": "Prompt body, paragraph 5: <verbatim excerpt>",
      "rationale": "Handling when input is missing is not defined",
      "reviewer": "<reviewer>",
      "waiver": null,
      "found_count": 1
    },
    {
      "finding_id": "F-ORG-20261010-003",
      "rule_id": "QD-P-1.3.2",
      "severity": "Inspect·P1",
      "status": "open",
      "applicability": "applicable",
      "evidence": "Prompt body, paragraph 3: <verbatim excerpt>",
      "rationale": "The qualifier 'appropriate' does not define a judgment boundary",
      "reviewer": "<reviewer>",
      "waiver": null,
      "found_count": 1
    }
  ]
}

4.6 Multi-Defect Handling Rules ​

  • Same inspection item number is counted as at most one failure: Repeated triggering of the same inspection item results in only one deduction of that item's corresponding severity-level weight.
  • Multiple defects falling under the same inspection item do not incur duplicate deductions: For example, if the same QD-P number has multiple issues in a single inspection, it is still handled as "that number failed once."
  • Primary defect consolidation: If the same fact hits multiple semantically overlapping numbers, deduct only under one primary number and record the others as "related" without duplicate deduction.
  • Gate and level are based on the severity level at the Agent's level.

Known semantic overlap groups (the primary number is the one whose definition fits best; the others are recorded as related):

Primary NumberRelated Numbers
QD-P-1.1.2QD-P-1.6.3, QD-P-2.5.4
QD-P-1.3.2QD-P-1.6.1
QD-P-1.3.3QD-P-2.5.2, QD-P-2.5.3
QD-P-1.5.2QD-P-2.4.2
QD-P-2.2.3QD-P-2.2.4
QD-P-1.6.5QD-P-2.5.1
QD-P-2.6.2QD-P-2.6.3
QD-P-2.1.2QD-P-2.4.2

Overlap groups merged or downgraded in this revision and no longer participating in deductions: QD-P-2.1.4 (merged into QD-P-1.4.1), QD-P-2.7.1/2.7.3 (both downgraded to suggestion items). The division of labor between QD-P-2.4.3 and QD-P-2.6.1 is annotated in their definitions and is no longer listed as an overlap group.


Appendix A: Minimum Rule Sets ​

Minimum Rule Sets define the inspection items that MUST be passed for each Agent Level, forming the core basis for compliance determination.

A.1 L1 Minimum Rule Set (7 P0 Items) ​

QD-P IDDefect NameCategory
QD-P-1.1.1Direct ContradictionContradiction
QD-P-1.1.3Example-Rule ContradictionContradiction
QD-P-1.2.1Role-Responsibility MismatchMismatch
QD-P-1.2.3Constraint-Responsibility MismatchMismatch
QD-P-1.5.1Responsibilities Do Not Support ObjectiveImplication
QD-P-2.2.1Undeclared Input SourceInput Definition
QD-P-2.5.2Missing Security Boundary ConstraintsConstraints/Boundaries

L1 Agent Compliance Determination: MUST pass the above 7 P0 inspection items. P1 items SHOULD be fixed before release; P2 items SHOULD be included in the improvement plan or risk register.

A.2 L2 Minimum Rule Set (17 P0 Items) ​

QD-P IDDefect NameCategory
QD-P-1.1.1Direct ContradictionContradiction
QD-P-1.1.3Example-Rule ContradictionContradiction
QD-P-1.2.1Role-Responsibility MismatchMismatch
QD-P-1.2.3Constraint-Responsibility MismatchMismatch
QD-P-1.3.3Undefined BoundariesScope Definition
QD-P-1.5.1Responsibilities Do Not Support ObjectiveImplication
QD-P-1.5.2Resources Do Not Support WorkflowImplication
QD-P-2.2.1Undeclared Input SourceInput Definition
QD-P-2.2.2Unconstrained Input FormatInput Definition
QD-P-2.2.3Required/Optional Not DistinguishedInput Definition
QD-P-2.2.4Undefined Missing-Input HandlingInput Definition
QD-P-2.3.1Undefined Output FormatOutput Definition
QD-P-2.3.2Incomplete Output SchemaOutput Definition
QD-P-2.3.5Factuality and Hallucination Control DeclarationOutput Definition
QD-P-2.4.2Undeclared Resources/ToolsWorkflow/Resources
QD-P-2.5.2Missing Security Boundary ConstraintsConstraints/Boundaries
QD-P-2.5.6Instruction-Data Trust BoundaryConstraints/Boundaries

L2 Agent Compliance Determination: MUST pass the above 17 P0 inspection items. P1 items SHOULD be fixed before release; P2 items SHOULD be included in the improvement plan or risk register.

A.3 L3 Minimum Rule Set (25 P0 Items) ​

QD-P IDDefect NameCategory
QD-P-1.1.1Direct ContradictionContradiction
QD-P-1.1.2Priority ConflictContradiction
QD-P-1.1.3Example-Rule ContradictionContradiction
QD-P-1.2.1Role-Responsibility MismatchMismatch
QD-P-1.2.3Constraint-Responsibility MismatchMismatch
QD-P-1.3.3Undefined BoundariesScope Definition
QD-P-1.4.3Invalid Skill DefinitionsRedundancy/Invalidity
QD-P-1.5.1Responsibilities Do Not Support ObjectiveImplication
QD-P-1.5.2Resources Do Not Support WorkflowImplication
QD-P-2.2.1Undeclared Input SourceInput Definition
QD-P-2.2.2Unconstrained Input FormatInput Definition
QD-P-2.2.3Required/Optional Not DistinguishedInput Definition
QD-P-2.2.4Undefined Missing-Input HandlingInput Definition
QD-P-2.3.1Undefined Output FormatOutput Definition
QD-P-2.3.2Incomplete Output SchemaOutput Definition
QD-P-2.3.5Factuality and Hallucination Control DeclarationOutput Definition
QD-P-2.4.1Incomplete Workflow StepsWorkflow/Resources
QD-P-2.4.2Undeclared Resources/ToolsWorkflow/Resources
QD-P-2.5.1Missing Termination ConstraintsConstraints/Boundaries
QD-P-2.5.2Missing Security Boundary ConstraintsConstraints/Boundaries
QD-P-2.5.3Missing Permission Control BoundariesConstraints/Boundaries
QD-P-2.5.4Undefined Constraint PriorityConstraints/Boundaries
QD-P-2.5.6Instruction-Data Trust BoundaryConstraints/Boundaries
QD-P-2.6.1Runtime Exceptions Not CoveredException Handling
QD-P-2.6.4Out-of-Scope Request Refusal and Human HandoffException Handling

L3 Agent Compliance Determination: MUST pass the above 25 P0 inspection items. P1 items SHOULD be fixed before release; P2 items SHOULD be included in the improvement plan or risk register.

A.3.1 Conditionally Triggered P0 Items (Not Included in the Fixed Count) ​

The following 3 items are treated as P0 when their trigger conditions are met. Because "whether triggered" depends on artifact facts, they are not included in the fixed P0 counts (7/17/25) of A.1–A.3, nor in the scoring denominator (§4.2). During inspection, determine item by item whether they are triggered:

QD-P IDDefect NameTrigger ConditionWhen Not Triggered
QD-P-2.3.4Missing Security/Privacy FilteringProcessing involves personal data (PII)N/A
QD-P-2.5.7Human Confirmation for High-Risk/Irreversible OperationsPossesses side-effecting operations such as funds/deletion/sending/production changes (L2/L3)N/A
QD-P-2.5.9Industry Compliance and Disclaimer BoundariesRegulated domains such as medical/financial/legalN/A

Once triggered, it is P0 and MUST be fixed; if not triggered, mark N/A and remove it from the denominator (see §3.3.4).

A.4 Minimum Rule Set Usage Notes ​

A.4.1 Rapid Determination Process ​

Step 1: Determine Agent Level (AC-L1/AC-L2/AC-L3)
  → Refer to 3.1 Agent Level Definitions

Step 2: Execute the corresponding level's Minimum Rule Set inspection
  → L1: Inspect 7 items
  → L2: Inspect 17 items (includes L1's 7 items)
  → L3: Inspect 25 items (includes L2's 17 items)

Step 3: Compliance Determination
  → All P0 inspection items passed → Compliant
  → Any P0 inspection item failed → Non-compliant

A.4.2 Upward Compatibility ​

  • L2 Minimum Rule Set includes all content from the L1 Minimum Rule Set
  • L3 Minimum Rule Set includes all content from the L2 Minimum Rule Set
  • When inspecting a higher-level Agent, lower-level inspection items are automatically covered

A.4.3 Version Evolution ​

  • Minimum Rule Sets are adjusted as specification versions are updated
  • When new defect types are added, assess whether they should be included in the Minimum Rule Set
  • When defect severity levels are adjusted, re-assess the Minimum Rule Set

Version History ​

  • v1.0 (October 2026): Initial release.
  • v1.1 (October 2026): Expanded Inspect and Permission defect coverage; Risk Explicit regraded to P0/P1/P2; multi-hit design principle clarified; terminology aligned to Logic Artifacts.

© 2026 Sanity AI Labs. Licensed under Creative Commons Attribution-ShareAlike 4.0 International. https://www.sanityops.org · hello@sanityops.org v1.1 · October 2026