Skip to main content
In the sidebar, this page is the Annotations tab of the Test lab row. See Finding your way around the dashboard.

​
What it is

An Annotation Queue is a filtered set of traces routed to a human for manual review and scoring — not an automated LLM-as-judge run. Use one when you need:
  • Ground truth for evaluator calibration — a human-scored sample to check whether an LLM-judge evaluator’s verdicts actually agree with a person’s judgment.
  • Human review of ambiguous cases — traces where an automated judge score doesn’t get you sufficient confidence and you want a person to look directly at the input/output before it counts as pass or fail.
Annotation Queues require the Pro plan or above — the same tier as Datasets and Evaluations. On a lower plan the page shows what it is for and an Upgrade to Pro button instead of the queues (see What a locked page shows). Through the API, creating, renaming, deleting and populating a queue need Pro (403 PLAN_REQUIRED below it); reading queues you already have does not.

​
Where to find it

Go to Annotations in the left sidebar (in the Develop group, alongside Prompts, Evaluations, Guardrails, and Datasets).

​
Creating a queue

1

Click New queue

Opens the queue-creation dialog.
2

Name and describe it

Give the queue a name (unique within the project) and an optional description.
3

Set the time range

Pick a From and To date/time. This is the window of already-ingested traces the queue will be populated from.
4

Create, then Populate

Click Create. The queue starts empty — click Populate on its card to actually pull matching traces into it.
A queue’s filter criteria also support narrowing by operation, model, session, and verdict level (pass/fail/warning) via the API, beyond the time-range fields the dashboard’s creation dialog currently exposes.

​
Populating is a snapshot, not a live feed

Clicking Populate runs a bounded scan (up to 500 traces) against everything matching the queue’s stored filter criteria at that moment, and adds any traces not already in the queue. It is not a live subscription — traces ingested after you populate don’t appear automatically. Click Populate again later to top the queue up with newly-matching traces; already-added traces are never duplicated.

​
Renaming and deleting a queue

Both calls need the evaluations:manage permission and the Team plan or above, and authenticate like the other queue routes.
  • PATCH /v1/projects/{id}/annotation-queues/{queueId} renames or re-describes a queue. Send name (1 to 120 characters, trimmed) and/or description (up to 500 characters, or null to clear it); at least one is required. The queue’s items, annotations, and filter criteria are untouched. It returns { queue } with the item count. A name already used by another queue in the project returns 409, and an unknown queue returns 404.
  • DELETE /v1/projects/{id}/annotation-queues/{queueId} deletes the queue and its items and returns 204. Scores that were already written from annotations stay in your evaluation data, because they are review history rather than queue state.
Creating, renaming, and deleting a queue are recorded in the audit log as annotation_queue.created, annotation_queue.updated, and annotation_queue.deleted.

​
Working a queue

Click Work queue → on a queue’s card to open its review view: an item list on the left, a review panel on the right.

​
The item list

The tiles and tab counts describe the items that have loaded, and say “the loaded N” when the queue holds more than has been fetched. Traces are grouped into three tabs — Pending, Annotated, Skipped — so you always know what’s left to review and can revisit anything already decided.

​
The review panel

Selecting an item shows:
  • A lens badge — Response (final output) for a whole-trace review, or Step (one call) when the item is scoped to one specific span within the trace
  • The trace’s input and output
  • Retrieved context, if the trace has a retrieval step, with its chunk count

​
Submitting an annotation

For a pending item, you can:
  • Skip — no score, verdict, or label required. Always available.
  • Submit annotation — record your review. You can supply:
    • A verdict — explicit Pass or Fail buttons
    • A label (optional free text, e.g. good, bad, needs_fix)
    • A score (optional, 0–1)
    • Notes (optional free text)
A categorical (label-only) annotation always requires an explicit Pass/Fail verdict — there is no way to submit one without picking one. A label like bad or needs_fix is arbitrary free text with no inherent pass/fail meaning to Zespan, so it can never stand in for a verdict on its own. The Submit annotation button stays disabled until you’ve either entered a numeric score or explicitly clicked Pass or Fail. (A numeric score alone is sufficient — it implies its own verdict at the ≥0.5 threshold — but an explicit verdict you set alongside a score always overrides that default.)

​
How human annotations are stored

Submitting an annotation writes one row into the same evaluation_scores data automated evaluators write to, tagged human_annotated. This is a deliberate design choice — not a separate “annotations” table — so a human annotation shows up everywhere an automated score already does: the evaluator’s trend chart, the Cost-Quality Frontier, and per-trace score views. There’s no separate dashboard or query path to check for human-reviewed results; they’re already there, distinguished from judge-model scores only by the human_annotated tag and the reviewing user’s identity. A numeric annotation is stored as a numeric score; a label-only annotation is stored as a categorical score carrying your label and the verdict you explicitly chose.

​
Next steps

  • Evaluations — run an automated evaluator against the same historical traces instead of reviewing them by hand
  • Evaluations — reading results — how categorical/boolean scores (including human annotations) render across the dashboard