Skip to main content

​
What it is

An Annotation Queue is a filtered set of traces routed to a human for manual review and scoring — not an automated LLM-as-judge run. Use one when you need:
  • Ground truth for evaluator calibration — a human-scored sample to check whether an LLM-judge evaluator’s verdicts actually agree with a person’s judgment.
  • Human review of ambiguous cases — traces where an automated judge score doesn’t get you sufficient confidence and you want a person to look directly at the input/output before it counts as pass or fail.
Annotation Queues require the Team plan or above — the same tier as Datasets and Simulations.

​
Where to find it

Go to Annotations in the left sidebar (in the Develop group, alongside Prompts, Evaluations, Guardrails, and Datasets).

​
Creating a queue

1

Click New queue

Opens the queue-creation dialog.
2

Name and describe it

Give the queue a name (unique within the project) and an optional description.
3

Set the time range

Pick a From and To date/time. This is the window of already-ingested traces the queue will be populated from.
4

Create, then Populate

Click Create. The queue starts empty — click Populate on its card to actually pull matching traces into it.
A queue’s filter criteria also support narrowing by operation, model, session, and verdict level (pass/fail/warning) via the API, beyond the time-range fields the dashboard’s creation dialog currently exposes.

​
Populating is a snapshot, not a live feed

Clicking Populate runs a bounded scan (up to 500 traces) against everything matching the queue’s stored filter criteria at that moment, and adds any traces not already in the queue. It is not a live subscription — traces ingested after you populate don’t appear automatically. Click Populate again later to top the queue up with newly-matching traces; already-added traces are never duplicated.

​
Working a queue

Click Work queue → on a queue’s card to open its review view: an item list on the left, a review panel on the right.

​
The item list

Traces are grouped into three tabs — Pending, Annotated, Skipped — so you always know what’s left to review and can revisit anything already decided.

​
The review panel

Selecting an item shows:
  • A lens badge — Response (final output) for a whole-trace review, or Step (one call) when the item is scoped to one specific span within the trace
  • The trace’s input and output
  • Retrieved context, if the trace has a retrieval step, with its chunk count

​
Submitting an annotation

For a pending item, you can:
  • Skip — no score, verdict, or label required. Always available.
  • Submit annotation — record your review. You can supply:
    • A verdict — explicit Pass or Fail buttons
    • A label (optional free text, e.g. good, bad, needs_fix)
    • A score (optional, 0–1)
    • Notes (optional free text)
A categorical (label-only) annotation always requires an explicit Pass/Fail verdict — there is no way to submit one without picking one. A label like bad or needs_fix is arbitrary free text with no inherent pass/fail meaning to Zespan, so it can never stand in for a verdict on its own. The Submit annotation button stays disabled until you’ve either entered a numeric score or explicitly clicked Pass or Fail. (A numeric score alone is sufficient — it implies its own verdict at the ≥0.5 threshold — but an explicit verdict you set alongside a score always overrides that default.)

​
How human annotations are stored

Submitting an annotation writes one row into the same evaluation_scores data automated evaluators write to, tagged human_annotated. This is a deliberate design choice — not a separate “annotations” table — so a human annotation shows up everywhere an automated score already does: the evaluator’s trend chart, the Cost-Quality Frontier, and per-trace score views. There’s no separate dashboard or query path to check for human-reviewed results; they’re already there, distinguished from judge-model scores only by the human_annotated tag and the reviewing user’s identity. A numeric annotation is stored as a numeric score; a label-only annotation is stored as a categorical score carrying your label and the verdict you explicitly chose.

​
Next steps

  • Evaluations — run an automated evaluator against the same historical traces instead of reviewing them by hand
  • Evaluations — reading results — how categorical/boolean scores (including human annotations) render across the dashboard