> ## Documentation Index > Fetch the complete documentation index at: https://docs.zespan.com/llms.txt > Use this file to discover all available pages before exploring further. # Stack layers — where a failure started > How Zespan says where in your stack a failure showed and where it most likely started: the layers, the difference between failing at a step and starting in one, how sure each claim is, what the 'Needs data' messages ask for, and when a claim is shown. When a trace fails, the useful question is not only *which step errored* but *which part of your stack caused it*. Zespan answers with two separate statements and a level of confidence for each, and it says plainly when it cannot tell. > [!NOTE] > > This is a recompute from the data your traces already carry. It never calls a model, and it never changes a finding's identity: the same failure keeps the same id whether or not an origin is shown. ## The layers | Layer | What it covers | | - | - | | **Instructions** | The system prompt and prompt versions: what the model was told | | **Agent logic** | The loop around the model: steps, limits, retries, hand-offs, when to stop | | **Tool** | A tool your agent calls, including its arguments and its result | | **MCP connection** | The connection to an MCP server: the server, the tool list, the schema | | **Retrieval** | What the retriever returned: missing, empty or wrong documents | | **Memory** | What the agent remembered and put into the prompt: missing, stale, wrong or overflowing | | **Model** | The model call itself: ignoring context, inventing a value, a refusal, a cut-off | | **Guardrail** | A check that blocked or changed a reply | | **Source data** | The system your tool or retriever reads from | | **Infra / platform** | Rate limits, timeouts, outages, authentication | | **Not determined** | Zespan could not tell, or the change is not a fault in your stack (a measurement change, a traffic change) | ## Failed at, and started in Every attribution makes two claims, and they are kept apart: * **Failed at: Tool step** is *where the symptom showed*. It is usually a recompute: the span that errored, the stop reason, the loop, and it links to the span. When the symptom itself is less certain the line says so: *Likely failed at: Tool step* or *Possibly failed at: Tool step*. * The origin, for example *Likely Retrieval*, is *where it most likely began*. This is the harder claim. A tool error on a tool span is a fact about the tool step; whether the tool, the arguments the model wrote, or the server behind it started the failure is a separate question. A rule that only observes a symptom (an error on a span, a step limit, a loop, a flagged outcome) can say where it showed, but it can only say where it started at **Possibly** unless a named check settles it, for example that the arguments the model wrote were invalid, or that the same call and result kept repeating. ## How sure | What you see | What it means | | - | - | | **Caused by** | An exact recompute from recorded data, with no competing explanation left | | **Likely** | One layer clearly leads and a named check supports it | | **Possibly** | A weaker claim, shown muted with a dashed outline. It always shows the evidence it rests on and why, and it never reads as a diagnosis | | **Either … or …** | Two layers fit equally well. Zespan does not split a count between them and does not pick one | Next to the layer you may also see how the evidence was gathered: | Label | Meaning | | - | - | | `OBSERVED` | The layer was recorded as real spans | | `INFERRED` | The layer was guessed, for example a tool whose name looks like a retriever. Inferred claims are held to a stricter bar | | `2 LAYERS AGREE` | Two independent sources point at the same layer | An ambiguous claim always comes with **What would settle it**, one step you can take: * *We will compare after your next instructions deploy*: Zespan waits for a change and compares before and after * *Run the same input on a second model*: a check you can do yourself * *Add retrieval spans to tell these apart*: an instrumentation step ## Needs data When the data is missing, the card says exactly which input is missing instead of guessing. It never says it "couldn't analyze" a trace that has spans. | Message | What to do | | - | - | | *Needs data: add retrieval spans to tell a retrieval miss from the model ignoring context.* | Record retrieval as spans, see [Integrations](https://docs.zespan.com/sdk/integrations/overview.md) | | *Needs data: chunk hashes (content capture is off).* | Turn on prompt storage for the project, or accept that retrieval claims stay weaker | | *Needs data: the prompt and answer text (content capture is off), so the answer cannot be checked against what was retrieved.* | Turn on prompt storage in the project's privacy settings | | *Needs data: send the requested max\_tokens.* | Pass the token limit you request on each model call | | *Needs data: add MCP client spans.* | Use the [MCP integration](https://docs.zespan.com/sdk/integrations/mcp.md) | | *Needs data: link prompt versions to spans.* | Use [prompt management](https://docs.zespan.com/sdk/prompt-management.md) so a call records which version it used | | *Needs data: tool results (capture is off).* | Turn on tool input and output storage for the project | | *Needs data: tag memory in the prompt to tell memory from retrieval.* | Mark memory passages when you build the prompt | | *Needs data: the stored context was cut at the storage limit, so a missing fact cannot be ruled out.* | Nothing to change in your code; very long contexts are stored up to a limit | ## When a claim is shown Not every claim is shown in every setup. Zespan shows a weaker kind of claim only for the layers and setups where it has validated that kind of claim, and holds it back otherwise. | Claim | Shown | | - | - | | **Failed at** line | Always. It is a recompute | | **Caused by** | Always, whatever your setup. It is an exact recompute | | **Likely**, **Either … or …**, inferred and **Possibly** claims | Only for a layer and setup where that kind of claim has been validated | Where a claim is held back you see the **Failed at** line and *Origin not determined yet*, and the deciding step where there is one. While stack attribution runs in shadow mode (the default for a new deployment), nothing in this page is shown to editors, viewers or billing members: no chip, no **Where it broke** row, no span layer, no **Failures by layer** card. Owners and admins see the claims marked **Shadow**, which means computed but not yet shown to the rest of the team. ### Setups What can be shown depends on how much your traces record, so each trace is read in the setup its data supports. | Setup | What your traces have | | - | - | | Full stack | Spans for the layers your agent uses | | No retrieval spans | Retrieved text in the prompt but no retriever span | | No MCP spans | MCP-looking tools but no MCP client spans | | Model calls only | Only model spans | | Content capture off | Prompt storage is off for the project, or the stored text has expired | [Instrumentation](https://docs.zespan.com/dashboard/instrumentation.md) shows, per layer, how much of your traffic is seen through real spans and the one step that would turn guesses into observations. ## AI gateways as evidence When your model calls go through an AI gateway that exports OpenTelemetry traces to Zespan ([AI gateways](https://docs.zespan.com/sdk/otel-integration.md#ai-gateways)), the gateway's spans are in the **Infra / platform** layer, and its retry and fallback attempts are evidence for findings about provider errors: | Failed attempt | Candidates | | - | - | | Rate limit (429) | **Model** (provider error) first, **Infra / platform** (capacity) second | | Rejected credentials (401, 403) | **Infra / platform** (credentials or configuration) first, **Model** (provider error) a distant second | | Server error (5xx) or timeout | **Model** (provider error) first, **Infra / platform** (network) second | A fallback that succeeded does not hide the failed provider: it stays in the evidence. These are symptom-only claims, so the origin stays at **Possibly**, and a gateway never produces a Tool, Retrieval or Agent logic candidate. ## Where you see it * **Issues** and **Incidents**: a chip under the root cause, with the layer, how sure, why, and who owns the fix. See [Issues](https://docs.zespan.com/dashboard/issues.md#where-it-started) and [Incidents](https://docs.zespan.com/dashboard/incidents.md#where-it-started) * **Traces**: the **Where it broke** row. See [Traces](https://docs.zespan.com/dashboard/traces.md#where-it-broke) * **Overview** and **Agents**: **Failures by layer** * **ZespanPilot**: replies word a layer claim the way the attribution allows. See [ZespanPilot](https://docs.zespan.com/dashboard/zespanpilot.md#where-it-started) * **API**: a `stack` object on findings, proven causes, issues, incidents and the trace. See the [API reference](https://docs.zespan.com/api-reference/introduction.md) ## Next steps * [Instrumentation](https://docs.zespan.com/dashboard/instrumentation.md): see what is observed, who gets each layer, and confirm or correct an attribution * [Issues](https://docs.zespan.com/dashboard/issues.md): filter recurring failures by the layer they started in * [Traces](https://docs.zespan.com/dashboard/traces.md): the **Where it broke** row on a trace * [Span kinds](https://docs.zespan.com/reference/span-kinds.md): the span kinds each layer is read from * [Silent-failure coverage](https://docs.zespan.com/reference/silent-failure-coverage.md): the silent failures Zespan looks for This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.