Stack layers — where a failure started

When a trace fails, the useful question is not only which step errored but which part of your stack caused it. Zespan answers with two separate statements and a level of confidence for each, and it says plainly when it cannot tell.

​

The layers

​

Failed at, and started in

Every attribution makes two claims, and they are kept apart:

• Failed at: Tool step is where the symptom showed. It is usually a recompute: the span that errored, the stop reason, the loop, and it links to the span. When the symptom itself is less certain the line says so: Likely failed at: Tool step or Possibly failed at: Tool step.
• The origin, for example Likely Retrieval, is where it most likely began. This is the harder claim. A tool error on a tool span is a fact about the tool step; whether the tool, the arguments the model wrote, or the server behind it started the failure is a separate question.

A rule that only observes a symptom (an error on a span, a step limit, a loop, a flagged outcome) can say where it showed, but it can only say where it started at Possibly unless a named check settles it, for example that the arguments the model wrote were invalid, or that the same call and result kept repeating.

​

How sure

Next to the layer you may also see how the evidence was gathered: An ambiguous claim always comes with What would settle it, one step you can take:

• We will compare after your next instructions deploy: Zespan waits for a change and compares before and after
• Run the same input on a second model: a check you can do yourself
• Add retrieval spans to tell these apart: an instrumentation step

​

Needs data

When the data is missing, the card says exactly which input is missing instead of guessing. It never says it “couldn’t analyze” a trace that has spans.

​

When a claim is shown

Not every claim is shown in every setup. Zespan shows a weaker kind of claim only for the layers and setups where it has validated that kind of claim, and holds it back otherwise. Where a claim is held back you see the Failed at line and Origin not determined yet, and the deciding step where there is one. While stack attribution runs in shadow mode (the default for a new deployment), nothing in this page is shown to editors, viewers or billing members: no chip, no Where it broke row, no span layer, no Failures by layer card. Owners and admins see the claims marked Shadow, which means computed but not yet shown to the rest of the team.

​

Setups

What can be shown depends on how much your traces record, so each trace is read in the setup its data supports. Instrumentation shows, per layer, how much of your traffic is seen through real spans and the one step that would turn guesses into observations.

​

AI gateways as evidence

When your model calls go through an AI gateway that exports OpenTelemetry traces to Zespan (AI gateways), the gateway’s spans are in the Infra / platform layer, and its retry and fallback attempts are evidence for findings about provider errors: A fallback that succeeded does not hide the failed provider: it stays in the evidence. These are symptom-only claims, so the origin stays at Possibly, and a gateway never produces a Tool, Retrieval or Agent logic candidate.

​

Where you see it

• Issues and Incidents: a chip under the root cause, with the layer, how sure, why, and who owns the fix. See Issues and Incidents
• Traces: the Where it broke row. See Traces
• Overview and Agents: Failures by layer
• ZespanPilot: replies word a layer claim the way the attribution allows. See ZespanPilot
• API: a stack object on findings, proven causes, issues, incidents and the trace. See the API reference

​

Next steps

• Instrumentation: see what is observed, who gets each layer, and confirm or correct an attribution
• Issues: filter recurring failures by the layer they started in
• Traces: the Where it broke row on a trace
• Span kinds: the span kinds each layer is read from
• Silent-failure coverage: the silent failures Zespan looks for