> ## Documentation Index
> Fetch the complete documentation index at: https://docs.zespan.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Annotation Queues

> Route a filtered set of traces to a human reviewer for manual pass/fail or score annotation, stored alongside automated judge scores.

## What it is

An **Annotation Queue** is a filtered set of traces routed to a human for manual review and scoring — not an automated LLM-as-judge run. Use one when you need:

* **Ground truth for evaluator calibration** — a human-scored sample to check whether an LLM-judge evaluator's verdicts actually agree with a person's judgment.
* **Human review of ambiguous cases** — traces where an automated judge score doesn't get you sufficient confidence and you want a person to look directly at the input/output before it counts as pass or fail.

<Note>
  Annotation Queues require the **Team** plan or above — the same tier as Datasets and Simulations.
</Note>

## Where to find it

Go to **Annotations** in the left sidebar (in the **Develop** group, alongside Prompts, Evaluations, Guardrails, and Datasets).

## Creating a queue

<Steps>
  <Step title="Click New queue">
    Opens the queue-creation dialog.
  </Step>

  <Step title="Name and describe it">
    Give the queue a name (unique within the project) and an optional description.
  </Step>

  <Step title="Set the time range">
    Pick a **From** and **To** date/time. This is the window of already-ingested traces the queue will be populated from.
  </Step>

  <Step title="Create, then Populate">
    Click **Create**. The queue starts empty — click **Populate** on its card to actually pull matching traces into it.
  </Step>
</Steps>

<Note>
  A queue's filter criteria also support narrowing by operation, model, session, and verdict level (pass/fail/warning) via the API, beyond the time-range fields the dashboard's creation dialog currently exposes.
</Note>

### Populating is a snapshot, not a live feed

Clicking **Populate** runs a bounded scan (up to 500 traces) against everything matching the queue's stored filter criteria at that moment, and adds any traces not already in the queue. It is **not** a live subscription — traces ingested after you populate don't appear automatically. Click **Populate** again later to top the queue up with newly-matching traces; already-added traces are never duplicated.

## Working a queue

Click **Work queue →** on a queue's card to open its review view: an item list on the left, a review panel on the right.

### The item list

Traces are grouped into three tabs — **Pending**, **Annotated**, **Skipped** — so you always know what's left to review and can revisit anything already decided.

### The review panel

Selecting an item shows:

* A **lens** badge — **Response (final output)** for a whole-trace review, or **Step (one call)** when the item is scoped to one specific span within the trace
* The trace's **input** and **output**
* **Retrieved context**, if the trace has a retrieval step, with its chunk count

### Submitting an annotation

For a pending item, you can:

* **Skip** — no score, verdict, or label required. Always available.
* **Submit annotation** — record your review. You can supply:
  * A **verdict** — explicit **Pass** or **Fail** buttons
  * A **label** (optional free text, e.g. `good`, `bad`, `needs_fix`)
  * A **score** (optional, 0–1)
  * **Notes** (optional free text)

<Warning>
  A categorical (label-only) annotation **always requires an explicit Pass/Fail verdict** — there is no way to submit one without picking one. A label like `bad` or `needs_fix` is arbitrary free text with no inherent pass/fail meaning to Zespan, so it can never stand in for a verdict on its own. The **Submit annotation** button stays disabled until you've either entered a numeric score or explicitly clicked Pass or Fail. (A numeric score alone is sufficient — it implies its own verdict at the ≥0.5 threshold — but an explicit verdict you set alongside a score always overrides that default.)
</Warning>

## How human annotations are stored

Submitting an annotation writes one row into the same `evaluation_scores` data automated evaluators write to, tagged `human_annotated`. This is a deliberate design choice — not a separate "annotations" table — so a human annotation shows up everywhere an automated score already does: the evaluator's trend chart, the [Cost-Quality Frontier](/dashboard/costs#cost-quality-frontier), and per-trace score views. There's no separate dashboard or query path to check for human-reviewed results; they're already there, distinguished from judge-model scores only by the `human_annotated` tag and the reviewing user's identity.

A numeric annotation is stored as a numeric score; a label-only annotation is stored as a categorical score carrying your label and the verdict you explicitly chose.

## Next steps

* [Evaluations](/dashboard/evaluations#retroactive-evaluation) — run an automated evaluator against the same historical traces instead of reviewing them by hand
* [Evaluations — reading results](/dashboard/evaluations#reading-evaluation-results) — how categorical/boolean scores (including human annotations) render across the dashboard