Requests log
The requests log shows all spans recorded for your project. Each row represents a single operation and shows:
Above the table, a summary line reads “In the 30 loaded: p50 … · p95 … · $… cost”. Latency percentiles and cost are measured on the traces currently loaded in the table (the 30 shown, not the whole range), while the failure percentage covers the whole selected range.
A trace sent with Send a sample trace during onboarding shows a Sample badge, so you can tell it apart from your application’s own traces. See Sign-up and onboarding.
Filtering
The filter bar lets you narrow the list by model, status, outcome, date range, environment, and user ID. Filters combine freely and the table updates immediately. The Outcome filter (theoutcome query parameter on the API) keeps only traces that contain a call that ended a particular way: truncated, blocked, refusal, empty_response, or finish_missing (shown as Stream cut). Any other value is rejected with a 400. See How a call ended. The search box (the search query parameter on the API) does a case-insensitive substring match across trace ID, event ID, model, provider, operation, error message, prompt text, completion text, user ID, and environment.
Each row also shows the outcome chips of its calls next to the status, and the traces list API returns them as outcome_kinds (a list of the kinds above, empty when none).
Pagination
The table uses cursor-based pagination returning up to 200 events per page. Cursor pagination ensures consistent results even if new agent events arrive while you are browsing.Opening a trace
Click any row to open the trace detail view — the redesigned layout described below. If the operation is part of a multi-span agent run, every span is available in the span navigator and the flow/flame visualizations: agent turns, tool calls, retrieval steps, and every LLM interaction.How a call ended (finish outcomes)
A call can finish with a success status and still not give the user a good answer. Zespan reads how each call ended from the finish information your SDK or OpenTelemetry instrumentation records, and marks it with a chip. The chips appear on the trace row, on the span row in the span navigator, and in the span detail, where each one explains itself and links back here. Nothing about the answer’s text is read: only the finish reason and a few yes/no flags are used, so the chips work with content capture off. See the finish outcome reference for the fields.Truncated
The answer hit the output length limit and was cut off. The call still reports success, so error alerts stay quiet while users get unfinished answers (broken JSON, half a summary). Fix: raise the maximum output length for those calls or ask for a shorter answer, and treat a cut-off answer as incomplete in your code (retry with a higher limit or continue the generation). Prevent: alert on thetruncated_rate metric.
Blocked
The provider blocked the answer with its content or safety filter. A block by your own Zespan guardrail is not shown as Blocked: it appears as a guardrail hit. Fix: compare the blocked input with the inputs that pass, adjust the prompt or pre-filter the input, and show a clear message instead of an empty answer. Prevent: alert onblocked_rate.
Refusal
The model declined to answer. The call reports success, so a refusal reads like any other answer. Fix: check the prompt and the input that triggered it, and handle a refusal in your code instead of passing it on as the answer. Prevent: alert onblocked_rate (it counts blocks and refusals together).
Empty response
The call returned no answer and no tool call while reporting success. A common cause is a model account that is out of credit or quota. Fix: retry once, then fail over to another model or provider, and check the account has credit and quota left. Prevent: alert onempty_response_rate.
Stream cut
A streamed answer ended without a finish signal, so it is likely incomplete: a dropped connection, a proxy or gateway timeout, or the client closing the stream early. Fix: check proxy, gateway and load-balancer timeouts on the streaming route, read the stream to its end, and retry a stream that ended without a finish signal.Causes and alerts
When these outcomes rise sharply against the hour before, an incident or issue can name the cause (Truncated answers, Blocked answers, Streams cut short, or Silent failure for empty responses) with the example calls it was measured from. These are shown as Possible cause, not confirmed unless verified causes are enabled for your project, and need enough calls with a recorded outcome in both windows before they appear.Trace detail
The trace detail view opens top-down from a one-line judgment to the exact span that caused it. From the top, it shows the verdict, Where it broke (only when something did), the key facts, a one-paragraph what happened, the failing span (when one failed) and the three lenses, then the span navigator and inspector. Where it broke sits directly under the verdict, so the place a run went wrong is on the first screen; the span waterfall follows below it. Keyboard shortcuts are behind ? (see Keyboard shortcuts).Verdict
At the top, a verdict banner states in plain English whether the run was Healthy, Degraded, Failed, or Running — with a headline that names what happened (“Trace failed —check_refund_eligibility timed out on all 3 attempts”) and a short summary. It’s computed instantly and deterministically from the run’s spans, statuses, latencies, and eval results — the same classifier that powers verdict-based Issue clustering, so a trace and its Issue always agree.
A run can finish with no span error and still go wrong: a total that is not the sum of its lines, a tool that was never defined, a refund time the policy contradicts. When a functional check found that on the run, the verdict never says Healthy or “Completed cleanly”. It says Completed, but a silent failure was found: followed by the finding’s title, and the header tag reads Silent failure. A check that Zespan could not prove (a “likely” or “signal” finding) says Completed, but a possible issue was found: and the tag reads Possible issue; it never claims proof. A proven, high-severity finding is shown in the failure tone; every other finding is shown in the warning tone. When the finding was raised into an issue, View issue links to it. The trace list and the MCP zespan_trace_verdict_get tool read the same findings, so they agree with the page (the MCP tool has no member role, so it only counts findings shown to everyone). A finding appears only for the people who may see it: while your functional detectors run in the default shadow mode, only owners and admins see them and everyone else sees the verdict the spans alone give.
- Jump to broken step (Jump to flagged step when a finding, not an error, set the verdict) takes you straight to the span the verdict blames (the deepest failing span, or the slowest tool on a degraded run).
- Explain generates an LLM narrative on demand — see Root cause below. The instant verdict is free; the LLM narrative runs only when you ask.
Where it broke
When a span failed or a finding was raised, a Where it broke panel sits right under the verdict, above the waterfall. A healthy run has no panel. For each place the run went wrong, first failure first, it shows:
More than three places fold behind Show N more. A failing parent span whose child also failed is left out: it only passed the error up. Under the points the panel repeats Where the time went (time by span kind, as on the Overview tab) and has Layers this run touched, the layer strip with the origin Zespan found when it has one.
On a run that errored, a Cause and Fix block sits under the points. It uses the first of these that exists and says which:
- The issue this trace belongs to. Its cause (Verified from spans when Zespan proved it, otherwise Auto-detected), numbered Fix and Prevent steps in the same style as the issue page, the issue’s saved remediation when one exists, and a link such as Seen 12 times · open issue.
- This trace’s own root-cause analysis, when one is already saved or has been proven from its spans (opening the page never starts an AI analysis).
- The automatic classifier over the failing span’s error code, message and finish outcome, marked Auto-detected.
Issues on this trace
Whatever the verdict, an Issues on this trace block lists the problems Zespan found on the run, so a successful trace still shows what is wrong with it. Each row has a type (Functional, Technical, or a pack name such as Support), a one-line cause, the first fix step and, when an issue exists, a link with how often it was seen. It lists:- the open issues that list this trace among their runs (an issue keeps only its most recent runs, so an older run of a busy issue may not appear here),
- functional findings you may see (see Verdict for who sees them) that no issue covers yet,
- technical signals of the run itself: a tool that needed retries, or a tool call far slower than the others. A run with only such a signal stays Completed (it reads “Completed, but degraded”) and lists the signal, with the numbers it was found from and no invented fix.
What happened
Under the verdict, one short paragraph says what happened in this run. It is written from the trace’s own data only (the verdict summary, the failing span and the error it recorded, the number of agents and tool calls, and the duration). No AI model is called, so it appears instantly and costs nothing, and it never states a cause the data doesn’t show: for a failed run it names where the run failed and quotes the error that span recorded rather than guessing why. Only a run whose verdict is Failed is said to have failed; on any other run, a span that errored (for example one that was retried and then succeeded) is described as having errored or timed out. A cost or duration that was not recorded is left out rather than shown as$0 or 0ms. For the AI’s analysis of the cause, open Root cause. A trace with very little recorded data gets one short plain sentence (for example, how long it ran), and a trace with none shows no paragraph. On a run with a finding it says “No span errored, but a silent failure was found: ” (or “a possible issue was flagged” for a finding Zespan could not prove) instead of describing the run as clean.
When a span failed, a row below the paragraph names it (its name, kind, and an Error or Timed out tag) with an Open span button that selects it in the inspector.
Where it broke
Under the verdict, Where it broke lays the trace’s stack layers in a row: Instructions, Retrieval, Memory, Tool, MCP connection, Model, Guardrail, Agent logic. Each cell says how Zespan knows about the layer:
Below the row, the attribution reads Failed at: Tool step, the reason, the evidence, and for an ambiguous case the step that would settle it. Each span in the navigator also carries the layer it belongs to, and when passages in the model input were tagged (retrieved text, memory, tool results), the input shows them with a label so you can see what the model was given and from where. Passages are placed inside the one message they were found in; a tagged passage that cannot be placed on the stored text is counted apart instead of guessed. While stack attribution is in shadow mode, the row and the span layers are shown to owners and admins only. See Stack layers.
Convert to scenario
The trace detail header includes a Convert to scenario button alongside Replay and Add to dataset. It turns this one trace into a regression test: a dedicated one-item dataset plus a ready-to-run Simulations scenario, without leaving the trace. See Convert to scenario for what it creates and how to run it.The three lenses
A trace can pass its final-answer check and still be unhealthy underneath. The lenses panel grades the same run at three altitudes, from your evaluation scores:- Final response (black-box) — was the output itself good?
- Trajectory (glass-box) — was the sequence of steps right?
- Per step (white-box) — was each individual call correct?
Verdicts and evidence
Not everything on this panel is a graded check. Two kinds of row appear:- Verdicts — an LLM judge or a deterministic check assessed the output. These carry a pass/fail and the threshold they were scored against.
- Evidence — a behavioural signal observed how the run went without looking at what the agent said.
user_retrymeans the same question was asked more than once in a session;agent_loopmeans one tool was called three or more times in a trace.
degraded on the strength of evidence alone; that means worth reviewing, not the agent was wrong.
Span navigator
The left pane lists every span in the run. Two layouts, toggled at the top:- Waterfall — spans on a shared timeline with duration bars, so you see where wall-clock time went.
- Tree — the call hierarchy, so you see who called whom.
+1.24s). Above the tree:
- Kind pills — one per span kind present (LLM, Tool, Agent, Plan, RAG, …), each with a count; click to filter to that kind.
- All / Errors / Critical path — the critical-path filter shows only the ancestors and descendants of the failing (or slowest) span.
- Search — filter by name, agent, or kind.
Inspector
The right pane is a resizable inspector. With no span selected it shows trace-overview tabs; click a span (or a navigator row) and it becomes that span’s detail. The strip has four tabs. Everything else is under More, and a tab there appears only when this run has data for it (a run with no retrieval has no Retrieval tab).
Under More:
The tab you have open stays visible in the strip, even when it normally lives under More. The flow and flame graphs are views of the span navigator on the left — see Flow diagram and Flame graph.
Keyboard shortcuts
The page no longer shows a permanent shortcut bar; it shows a one-line hint, Press ? for shortcuts. Press ? (not while typing in a field) or choose Keyboard shortcuts in the account menu to open the list; this page’s shortcuts come first, under This page, and F and R are listed only on runs where they do something. The shortcuts work whether or not the list is open:
Per-span detail shows the span’s identity (its own ID and parent ID, with copy buttons and a copy-link-to-span permalink), an I/O tab (prompt, system prompt, completion, tool args/result, decision rationale, RAG chunks, and image/video generation fields), a Metadata tab, and an Evaluators tab when scores target that span.
Flow diagram
The Flow tab draws the run as a node graph — one node per span, connected by its call structure — with two layouts you toggle: Hierarchical (who called whom) and Chronological (positioned by wall-clock start). Edges are labeled by kind (delegation, tool call, infers) and colored, failed steps stand out, and each node carries an execution-order badge. Pan and zoom with the mouse, fit-to-width, lock panning, or open full screen. Click a node to inspect that span.Flame graph
The Flame tab shows span durations as stacked bars, one row per depth level, in two modes: Chronological (bars positioned by wall-clock time — gaps are real idle time) and Hierarchical (each parent’s width divided among its children by duration share — fully packed, so structure reads cleanly on a mostly-idle run). Bars are colored by span kind, with a legend that reflects the kinds actually present. Open it full screen for a taller view. Click a bar to inspect that span.Retrieval
For a RAG run, the Retrieval tab surfaces the retrieved documents — chunk text, document/chunk IDs, source, and relevance score — and a per-retrieval-span context count. Retrieval is captured automatically by the LangChain and LlamaIndex integrations, or explicitly for manual RAG viarecordRetrieval() / record_retrieval(). This is the same context the RAG evaluators score.
Session context
When a trace belongs to a session (an SDK-suppliedsession_id), the Session tab shows the whole conversation: a rollup (user, turn count, total duration and cost) and every sibling trace in the session — the current one highlighted — each linking to its own trace. Traces without a session don’t show the tab.
Guardrail violations in the trace
When a guardrail check blocks, warns, or redacts something during a run, the violation appears as its own span in the navigator and the flow/flame views — positioned as a child of the span it evaluated, colored to match the guardrail outcome, right alongside the tool call and LLM spans instead of in a separate log you have to cross-reference. Select the guardrail span to see which policy fired, the check phase (pre or post), and the exact value that triggered it. From there, click Promote to Policy to turn that one violation into a permanent guardrail rule — the tool, field, and value are pre-filled from the violation itself. See Promoting a violation to a policy.Delegation compliance
When a span showsDelegated from <agent name>, a compact badge next to the agent’s name (in the Agents tab and on the flow graph) summarizes that agent’s guardrail-compliance history: pass rate, violation count, and its most recent violation. This is pulled from anywhere in your organization — every project, not just the current one — so you can see at the moment of a handoff whether the agent you’re about to trust has a clean record elsewhere in your org.
The same badge appears on that agent’s own page in the Agent Registry.
Root cause
Click Explain on the verdict banner, or open the Root cause tab, to run an AI root-cause analysis on the full agent trace. Zespan examines all spans — timing, errors, token counts, tool call results, and agent delegation chains — and returns:- A plain-English summary of the entire agent run
- The identified root cause type (e.g.
tool_failure,token_limit_exceeded,agent_loop) - The specific problem span, and — for a multi-agent run — the causal agent: which upstream agent’s output poisoned the downstream failure, and at which handoff step
- Contributing factors across the trace
- Step-by-step fix recommendations with implementation notes
- A prevention tip
Next steps
- Stack layers — what the Where it broke row and its tiers mean, and when an origin is shown
- Instrumentation — which layers Zespan sees through real spans, and what to add
- Issues — recurring failures grouped from traces like this one
- Span kinds — the kinds each span is stored under