Silent-failure coverage

A silent failure is a wrong outcome while every call reports success: an empty answer, an answer cut off at the length limit, an agent going round in circles. Nothing throws, so nothing shows in your logs. This page lists what Zespan looks for and what each check needs from your traces to run.

What Zespan looks for

Failure	What it looks like
Empty answer with status ok	Calls that return no answer and no tool call while reporting success, rising against the hour before
Truncated output	Answers cut off at the output length limit, rising against the hour before
Provider block or refusal	Answers the provider blocked or the model refused, rising against the hour before
Dropped or cut stream	Streamed answers that ended without a finish signal
Swallowed provider error or timeout	A failed child call inside a trace that otherwise reports success
Retries hiding failures	Tool calls that only succeeded after retries
Agent loop with no progress	The same tool call and result repeating within a run
Repeated error pattern	The same error recurring across traces
Tool failure rate regression	A rise in a tool’s failure rate
Context growth until calls fail	Context that grows run after run until model calls fail
Latency regression	A rise in latency against the agent’s recent baseline
Runaway cost per run	A rise in cost per run, with the cause where Zespan can name it
Change followed by errors	A rise in errors that starts at a change on your timeline
Evaluator pass-rate drop	A drop in an evaluator’s pass rate, with the factor that changed where Zespan can name it
User asks again	A user repeating the same question in a session

Each finding says how sure Zespan is (see Findings → Tiers), and shows the evidence it rests on.

Check families

Findings on a trace are grouped into check families. These are the names you filter by on the Findings page:

Family	What it checks
D1 Grounding and answer quality	Whether the answer is supported by what the agent retrieved or was given
D2 Instructions and policy	Whether the answer follows the system instructions and the output contract
D3 Claims against effects	Whether what the agent says it did matches the tool calls it made: an action claimed with no call, a call to a tool that is not defined, a tool error or result the answer ignores, the same call repeated
D4 Requests against tool calls	Whether the answer and the calls meet what was asked and required: totals that do not add up, numbers that contradict each other, required items missing, length limits
D5 Figures and format	Whether structured output (such as JSON) is valid and keeps its expected format
D6 Conversation signals	What the user does next: asking again, correcting the agent, giving up
D7 Drift against the agent’s own baseline	Changes against the agent’s own recent traces: answer length and structure, refusal rate, language, tool mix, topics and inputs
Domain packs	Checks for a kind of product: customer support, data analysis, marketing and analytics, and AI products. They need a confirmed tool mapping

What the checks need

Inputs must be recorded. Truncation, block and stream-cut detection read the finish outcome from a current SDK or OpenTelemetry mapping; empty-answer detection also works from captured answer text. See the finish outcome reference.
Trace checks read what was recorded. The answer, the conversation, the system instructions, the tool definitions and the tool results must have been captured in a form the check can read. A check that lacks an input reports that it could not run and names the input; it never passes. See Capture fields.
Windows, not single calls. Rate checks compare the last hour with the hour before and report a rise. One isolated bad call, or a rise concentrated in one or two stuck runs, is not flagged.
Intended fallbacks look like swallowed errors. A failed call followed by a fallback answer marks the trace failed whether the fallback was intended or not.
Retries inside a model client are not seen. Retry detection covers tool calls that report a retry count.
Repeated tool calls. The repeated-tool-call signal does not compare arguments, so the same tool called for different items can look like a loop. The loop check that compares arguments and results is the stronger one.
Evaluator drops need volume. An evaluator drop gets a named factor only with at least 30 evaluated samples in each window.

Next steps

Findings — the findings on a trace and the project’s Findings page
Finish outcome reference — the fields behind truncation, block, empty and stream-cut detection
Capture fields — what each check reads and why a check could not run
Alerts — alert on the rate of cut-off, blocked and empty answers
Traces — how each call ended

Tracing & Observability

Evaluations

Datasets & Experiments

Guardrails

Prompt Management

Alerts & Incidents

Compliance

Cost & Analytics

ZespanPilot (AI Copilot)

Silent-failure coverage

What Zespan looks for

Check families

What the checks need

Next steps

​What Zespan looks for

​Check families

​What the checks need

​Next steps

What Zespan looks for

Check families

What the checks need

Next steps