> ## Documentation Index
> Fetch the complete documentation index at: https://docs.zespan.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Simulations — test LLM behavior before shipping
> Run your LLM application against a dataset of test cases to catch regressions before they reach production. Compare outputs, scores, and latency across prompt versions.
Simulations let you run your LLM configuration against a prepared dataset of test cases before deploying a change. Instead of finding out a prompt change broke something in production, you run a simulation, review the outputs and evaluation scores, and only promote the change when you're confident it behaves correctly.
Simulations require the **Team** or **Scale** plan.
## Core concepts
A collection of test cases. Each case has an input (the user message or prompt variables) and optionally an expected output or reference answer.
A test configuration: which dataset to use, which target to test (a prompt/model, an HTTP endpoint, or a multi-turn conversation), and which evaluators to run.
One execution of a scenario against a single input — the platform sends it to your target, collects the output, and scores it. Use a batch run to execute a scenario across every item in a dataset.
Multiple simulation runs executed in parallel, typically used to compare different prompt versions side by side.
## Preparing a dataset
Before running simulations, you need a dataset of test cases. Simulations use the same datasets as [Datasets](/dashboard/datasets) — there's no separate dataset-creation flow on the Simulations page, just a picker that lists your project's existing datasets. Build one from real traces, upload a CSV, or add rows manually as described on the [Datasets](/dashboard/datasets) page, then select it when creating a scenario or starting a run below.
## Creating a scenario
On the Simulations page, click **New scenario**.
Both can be picked here as defaults, or chosen per-run when you start a run.
A scenario's target is what actually receives each test case's input. Pick one:
* **Prompt Target** — Zespan calls a model directly. Pick a project LLM connection/provider/model, then write a prompt template using `{{input}}` for the test case's input.
* **HTTP Target** — Zespan sends each test case to a URL you control instead of calling a model itself. Configure the URL, HTTP method, and a JSON request body template (also using `{{input}}`). Optionally add custom request headers (e.g. an `Authorization` token your endpoint expects) and a request timeout in milliseconds (defaults to 30000). Use this to simulate against your own deployed agent/service rather than a bare prompt.
* **Conversation (multi-turn)** — Zespan runs a simulated multi-turn conversation: a "simulated user" model drives the conversation against an inner prompt target for up to 20 turns, stopping on max turns, a regex match against the agent's reply, or an evaluator pass/fail verdict. Use this for testing agents that need several exchanges to complete a task, not just a single-shot response. Configure the extra stop conditions under **Stop Conditions** in the scenario form: a regex pattern to match against the agent's reply, and a **Stop on pass** / **Stop on fail** select tied to the scenario's attached evaluator (disabled until an evaluator is attached). The simulated user's own provider is independently configurable — it doesn't have to match the target agent's provider. The inner target's **prompt template** is applied to *every* turn, with `{{input}}` standing in for that turn's message — the initial test-case input on the first turn, and the simulated user's reply on each turn after. Leave it as `{{input}}` to send each turn through untouched. The run's **Conversation Transcript** always shows what the simulated user actually said, not the template-wrapped text the target received.
Attach an evaluator to score the outputs — any evaluator defined on the Evaluations page. You can also add lightweight assertions (contains / not-contains / regex match) and a max-latency check directly on the scenario, independent of the evaluator.
If the attached evaluator's template has a system prompt (a real LLM-judge rubric), Zespan scores runs with a real judge call: one call for a single-turn target, or for a **Conversation (multi-turn)** target, one call per turn plus a final holistic call that judges the whole transcript — each turn's judge score/verdict show up in the **Conversation Transcript**, alongside the run's total judge cost and latency (see [Reading run results](#reading-run-results)). An evaluator created via the quick **New Evaluator** dialog has no system prompt, so it falls back to the existing metric-key/assertion scoring instead — no judge call, no LLM cost.
**Reference Match** — an optional Select (**None** / **Exact match** / **Fuzzy match ≥80% similarity**) compares the target's output against the dataset item's `expectedOutput`. It's skipped automatically when the run's dataset item has no `expectedOutput` set, and its result appears as a `reference_match` entry alongside the scenario's other assertion results.
Click **Save**. The scenario is ready to run.
Scenarios can be edited after creation — click **Edit** on any row in the Scenarios list to reopen this same form pre-filled with the scenario's current config. The form's title and submit button switch to **Edit Scenario** / **Save Changes**, with a **Cancel** button to back out without saving.
Prompt and conversation targets call an LLM through the project's own **LLM connection** (HTTP targets never call an LLM — they call your URL instead). Without a connection configured, starting a run on a prompt or conversation scenario fails with "No LLM connection configured — add one in Settings → LLM Connections." See [LLM Connections](/platform/llm-connections).
## Convert to Scenario
For a fast way to turn one specific trace or conversation into a regression test, click **Convert to Scenario** on a trace's detail page (next to **Replay** and **Add to dataset**) or on a session's detail page — see [Trace detail](/dashboard/traces#trace-detail) and [Session detail](/dashboard/sessions#session-detail). Zespan:
1. Creates a new, dedicated one-item dataset named after the source trace or session, with a timestamp, so it doesn't get mixed into a dataset you're already building
2. Adds the trace's or session's captured input and output as that dataset's single item
3. Creates a new scenario pointing at that dataset, with **Reference Match** set to **Fuzzy match** by default so the scenario has a pass/fail signal against the captured output right away
A scenario converted from a trace uses a **Prompt Target**; one converted from a session — inherently multi-turn — uses a **Conversation (multi-turn)** target. Either way, the new scenario's provider and model default to the project's baseline LLM connection rather than being pinned to the original trace's model — the original provider/model are noted in the new scenario's description instead, so the regression test stays current with whatever your project runs today.
Convert to Scenario doesn't auto-run. It lands you on the Simulations page with a success toast — review the new scenario and click **Run** when you're ready.
## Persona configuration for conversation scenarios
When a scenario's target is **Conversation (multi-turn)**, the simulated user's behavior is driven by a **Simulated User Prompt** — a system prompt for the model playing your end user. You can write this prompt as free text, or use the structured **Persona** fields to compose it instead:
| Field | Description |
| --------------- | ---------------------------------------------------------------------------------------------------------------------- |
| Name | The persona's name (e.g. "Alex") |
| Goal | What the simulated user is trying to accomplish (e.g. "get a refund") |
| Expertise level | `novice`, `intermediate`, or `expert` — shapes how much domain knowledge the simulated user brings to the conversation |
| Tone | The persona's manner (e.g. "polite but increasingly impatient") |
All persona fields are optional and independent of each other — set only the ones you care about. If at least one persona field is set, it composes the simulated user's system prompt; if none are set, the free-text **Simulated User Prompt** is used as-is. Persona and free-text prompt are not combined — a scenario uses one or the other.
### Adversarial persona presets
Above the persona fields, an **Adversarial preset** picker offers four built-in presets for adversarial testing: **Prompt Injection Attacker**, **Escalating Angry Customer**, **Social Engineer**, and **Confused Novice**. Picking one populates the Name, Goal, Tone, and Expertise level fields above with a ready-made adversarial persona — the fields stay fully editable afterward, so a preset is a starting point, not a lock.
A preset also bakes a tactic-ladder instruction into the simulated user's system prompt, which is the primary adaptive mechanism: it pushes the simulated user to vary its approach across turns rather than repeating the same angle. This works on every run with no evaluator required, because the simulated user model already sees the full conversation history each turn and can tell when it's being resisted or deflected.
If the scenario also has an evaluator attached whose template has a real judge rubric (see [Add evaluators](#creating-a-scenario)), a second reinforcement layer kicks in automatically: each turn's already-computed pass/fail judge verdict is prepended to the target's reply before the simulated user's next turn, sharpening the adaptive signal for free — no extra LLM call.
## Running a simulation
Click **Run** on any scenario to execute it once. Zespan:
1. Sends the scenario's input — either the value you typed, or (if you leave input blank) the first item in the scenario's default dataset — to the target: a model call, an HTTP request, or a simulated multi-turn conversation, depending on the target type
2. Records the response, latency, and (for conversation targets) per-turn token count and cost
3. Runs the scenario's assertions and, if one is attached, its evaluator against the output
A single **Run** only ever scores one item. To run a scenario — or several — across every item in a dataset, use a **batch run** (below).
Simulation runs appear in the **Runs** tab. Click any run to see its full output, evaluation results, and (for conversation targets) turn-by-turn transcript.
## Batch runs — comparing versions
A batch run executes the same dataset against multiple configurations simultaneously, making it easy to compare prompt versions head-to-head.
Select two or more scenarios (or one scenario with multiple prompt version variants).
Click **Run batch**. All variants run in parallel.
When all runs complete, the batch run's detail page lists every run with its score, latency, and cost side by side.
Use batch runs before promoting a prompt from `staging` to `production`. Run the current production version and the candidate version through the same dataset and only promote if the candidate scores better on your key evaluators.
## CI gate for batch runs
Batch runs support the same kind of CI-blocking quality gate as prompt versions — see [The quality gate](/dashboard/prompts#the-quality-gate) for the prompt-version workflow this mirrors.
`POST /projects/{id}/simulations/batch-runs/{batchRunId}/gate` scores a candidate batch run against an explicit baseline: pass a `baselineRunId` in the request body. There's no auto-inferred baseline (unlike the prompt gate, which defaults to whatever holds the `production` label) — you choose which prior batch run to compare against.
The gate checks three signals, all with defaults:
| Signal | Body field | Default threshold |
| ------------------------------- | ------------------------------ | --------------------- |
| Average score drop vs. baseline | `thresholds.avgScoreDropMax` | ≤ 3 percentage points |
| Items regressed ≥10pp | `thresholds.regressedItemsMax` | 0 |
| Pass-rate drop vs. baseline | `thresholds.passRateDropMax` | ≤ 5 percentage points |
`thresholds` and each of its fields are optional — omit any of them to use the default. All three signals must pass for the gate to pass.
If the candidate batch run is still `queued` or `running`, the route returns `202 { "pending": true }` instead of a verdict — poll until the batch run finishes, then call the gate route again. This is the same polling pattern the [`zespan-gate` CLI](/sdk/cli) uses for the prompt-version gate.
Once the candidate batch run has completed, the gate route returns the verdict directly: `passed`, a human-readable `exitReason`, the three deltas (`avgDelta`, `regressedItems`, `passRateDelta`), the `baselineRunId` used, and `evaluatedAt`. It also persists that same result into the candidate batch run's `experimentMetadata.gateResult` — a non-destructive merge that leaves any other metadata already stored there untouched — so the verdict can be read back from the batch run's detail later without re-running the gate.
The gate is also runnable straight from the dashboard, not just the API/CI. Once a batch run reaches `completed`, its detail page shows a **Gate Check** card: pick a **Baseline** from a dropdown of other completed batch runs on the same dataset (defaulting to the most recent), optionally open the collapsed **Advanced thresholds** section to override any of the three thresholds above, then click **Run Gate Check**. The result renders inline — pass/fail, the `exitReason`, and three stat chips (avg score delta, regressed items, pass-rate delta), each highlighted when that specific signal breached its threshold — plus a collapsible per-item regression table. Thanks to the persisted `experimentMetadata.gateResult`, the card shows the last checked result automatically on page load, with a **Re-run** button to check again.
## Reading run results
Each simulation run's detail view shows:
* **Summary** — scenario name, target type, status, verdict, score, and latency
* **Output** — the target's raw response
* **Evaluation results** — each assertion's pass/fail with expected vs. actual, plus the attached evaluator's score if one is configured
* **Run timeline** — every step the run went through (queued, target executed, assertions evaluated, evaluator applied, completed/failed), useful for debugging a failed or unexpectedly-scored run
For a run on a **Conversation (multi-turn)** scenario, the detail view also shows a **Conversation Transcript** — every turn in order, labeled **Simulated User** or **Target Agent**, with that turn's latency and cost, plus a total conversation cost computed from real per-model pricing across all turns.
A run in progress streams its status live — no need to reload the page. Use **Retry** to re-run a failed or completed run with the same input, or **Cancel** to stop one that's still running.
## Plan limits
| Plan | Datasets | Scenarios | Monthly simulation runs |
| ----------------- | --------- | --------- | ----------------------- |
| Free / Solo / Pro | None | — | — |
| Team | Unlimited | Unlimited | 10,000 items/month |
| Scale | Unlimited | Unlimited | Unlimited |
Simulations are a Team/Scale-only feature — every mutating simulations route (creating or editing a scenario, and starting, retrying, batching, or cancelling a run) requires the **Team** plan or above server-side. There is no intermediate Pro-tier access.