> ## Documentation Index
> Fetch the complete documentation index at: https://docs.zespan.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Canary — scheduled test runs

> Zespan runs a small set of golden tasks against an agent on a schedule and checks each answer with plain code checks. It reports checks passed, never accuracy.

A canary is a small set of tasks you wrote, run against one agent on a schedule. After each run I check the answers with the code checks you chose and say how many checks passed. That tells you that your agent still answers and still answers in the shape you expect, before a real user finds out.

> [!NOTE]
>
> A canary result is "checks passed" on "canary runs". It is never a score for accuracy or quality, and it is never a percentage. A check Zespan could not judge is counted as **not checked**, never as passed.

The canary page needs the new agent layout. Open an agent and choose **Canary**.

## Set up a canary

**1. Pick the golden tasks**

Build a [dataset](https://docs.zespan.com/dashboard/datasets.md) of up to 10 tasks. A task that came from a real run carries its source run, which Zespan uses to check that the run did not call a tool that can move money or delete data. To use real runs, create the dataset from traces (**Datasets**, then **New dataset**, then **From traces**), so each task keeps its source run. On a run of an agent that has a canary, the button beside the run reads **Open the canary's dataset** and opens that dataset page. It does not add the run for you: Zespan has no way yet to add a single run to an existing dataset, so a task you paste in by hand has no source run and waits for the safety box.

**2. Pick what Zespan calls**

Most agents are called at their own endpoint: register it once under [HTTP targets](https://docs.zespan.com/dashboard/http-targets.md) and pick it here. Your agent then uses its own model keys and tools, so it pays for its own calls. An agent that is only a managed prompt can instead run on your own model connection at your own price.

**3. Choose how often**

Every N minutes, from 10 to 1,440. The default is 60.

**4. Add checks**

Under each task, add the checks that say what a good answer holds. Every task is always checked for an answer first. Saving replaces the checks of this canary.

**5. Let it start**

Owners and admins can save a canary. It starts on its own once it has a place to call and at least one task that can run on its own. There is no estimate to click first, but the page always shows what to expect: "I will send 3 tasks to `refund_agent` every 60 minutes: up to 72 calls a day."

Everyone who can read the dashboard can see the page. Only owners and admins see the controls. A canary whose safety box was already ticked starts again without a new tick, except after a pause for a risky tool.

## The checks

Checks are plain code. No model judges an answer, and no model writes a task.

| Check | Passes when |
| - | - |
| **Answered** | The call returned a success status with a body. Always checked first. |
| **Equals** | The answer, trimmed, equals your text. |
| **Contains all of** | Every text you listed occurs in the answer, ignoring case (up to 10). |
| **Contains none of** | None of the texts occurs. |
| **Matches pattern** | The answer matches your pattern. Patterns that could run for a very long time are refused. |
| **JSON has paths** | The answer is JSON and each path you listed has the type you gave. |
| **JSON value equals** | The value at your path equals the text, number, true, false or null you gave. |
| **Tool was called** | A tool with that name appears in your agent's own spans for the call. Without those spans this check is **not checked**. |
| **Answers within** | The call took less than your number of milliseconds. |

A result reads "6 of 7 checks passed, 1 not checked". A run with failures reads, in separate parts, "4 of 8 checks passed", "1 of 3 tasks failed" and "1 not checked".

## What a canary reads and what it never touches

A canary only asks your agent questions you wrote. It does not read your real runs to build tasks, and it does not pool anything across customers.

* **Safe tasks only.** Before a canary runs, I check the tools the source run of each task called. A task whose source run called a tool that can move money or delete data is never run: "I won't run this task on my own: its source run called `issue_refund`, which can move money or delete data." A task that did not come from a run, or whose tools I could not read, waits for you to tick the safety box: "These tasks are safe to run again and again: no money moves and no records change."
* **Paused for a risky tool.** If your agent's own spans show that a canary call used such a tool, I pause the canary and ask you in [Needs you](https://docs.zespan.com/dashboard/needs-you.md) to change the task or point it at a test endpoint.
* **Apart from your real numbers.** Canary calls are kept out of your run counts, costs and issues. They are stored on their own, and the page shows them under each run's events.
* **No retries of a call.** A failed call is not retried. A run that fails is confirmed once, 10 minutes later.

> [!WARNING]
>
> Some agents do not carry Zespan's mark on their own spans for a canary call. For these agents the spans may count as real runs. The page says so ("Your agent's own spans for my canary calls do not carry my mark, so they may count as real runs.") and shows how many spans I could not check today.

## What the canary costs

I make no model call of my own for a canary. When your agent is called at its own endpoint, your agent pays for its model and tool calls, and the page says "I can't see what each call costs you yet" until your agent's own spans come back with a cost. Owners and admins then see the measured cost per run and per day. When the canary runs on your model connection, owners and admins see the estimated cost per day at your price, and I refuse to run a model whose price I do not know.

Caps always apply, and Zespan never raises its own cap.

| Limit | Value |
| - | - |
| Tasks per run | 10 (at most 25) |
| Time per run | 120 seconds |
| Shortest interval | 10 minutes |
| Runs per canary per day | 48 |
| Calls per project per day | 500 |
| Your model connection, per project per day | $2 by default, at most $20 |
| Zespan's own limit, all projects | 20,000 calls a day |

A run can fail and also be cut short by a limit; the page and Home say both. The canary has a shadow mode in which I count what I would run and make no calls: then Home shows no canary heartbeat and no next canary, and nothing promises one.

Limits reset at 00:00 UTC, shown in your own time zone. When a limit is reached the page says "Paused: today's canary limit is reached. Resets at 02:00." and Home says the same. A canary is never silently skipped.

## What Home shows

[Home](https://docs.zespan.com/dashboard/home.md) says what the canary found, in first person, and each fact once.

| Where | What it says |
| - | - |
| **Status pill** | "canary passed 14:12" under **Watching**, only while the last passing run is younger than two intervals. |
| **Headline** | When your agent has sent no real run for a week but the canary passed after the last one: "No real runs since Nov 14. My canary for `refund_agent` passed 14:12, so it answers. Is the SDK still sending?" It names only the agent the canary reached. |
| **My day** | **Running**: "`refund_agent`: canary run, task 2 of 5." **Next**: "Canary for `refund_agent` at 15:12". |
| **Critical now** | "I see this now: `refund_agent` did not answer my canary at 14:12. All 3 tasks failed (timeout)." If it fails to answer twice in a row, it ranks higher. A canary that fails the same check on a second run: "My canary for `refund_agent` failed 2 of 3 tasks twice, at 14:02 and 14:12." Each has **Open the canary run**. |
| **Done for you** | "My canary ran 6 times on Nov 21 (UTC) on `refund_agent`; all passed." The day is the UTC day, because that is the day I count runs in. A day with a failure has no roll-up, and a first failure says "I'll run it again at 14:22 to be sure." A passing run whose answers changed shape says so. A stopped canary says when its limit resets. |
| **Needs you** | A canary I paused for a risky tool, with **Open canary settings**. |
| **How I checked** | "My canary checks 2 agents; the last run was at 14:12." and which agents have no canary. |

Home never shows a dollar figure for the canary.

## Next steps

* [Home](https://docs.zespan.com/dashboard/home.md) — where the canary's results appear
* [Datasets](https://docs.zespan.com/dashboard/datasets.md) — build the golden tasks
* [HTTP targets](https://docs.zespan.com/dashboard/http-targets.md) — register the endpoint a canary calls
* [Agents](https://docs.zespan.com/dashboard/agents.md) — open an agent and choose Canary


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.