Skip to main content
Simulations let you run your LLM configuration against a prepared dataset of test cases before deploying a change. Instead of finding out a prompt change broke something in production, you run a simulation, review the outputs and evaluation scores, and only promote the change when you’re confident it behaves correctly.
Zespan simulations dashboard showing dataset runs and evaluation results
Simulations require the Team or Scale plan.

​
Core concepts

Dataset

A collection of test cases. Each case has an input (the user message or prompt variables) and optionally an expected output or reference answer.

Scenario

A test configuration: which dataset to use, which target to test (a prompt/model, an HTTP endpoint, or a multi-turn conversation), and which evaluators to run.

Simulation run

One execution of a scenario — the platform runs each dataset item through your LLM configuration and collects outputs and scores.

Batch run

Multiple simulation runs executed in parallel, typically used to compare different prompt versions side by side.

​
Preparing a dataset

Before running simulations, you need a dataset of test cases. Simulations use the same datasets as Datasets — there’s no separate dataset-creation flow on the Simulations page, just a picker that lists your project’s existing datasets. Build one from real traces, upload a CSV, or add rows manually as described on the Datasets page, then select it when creating a scenario or starting a run below.

​
Creating a scenario

1

Click New scenario

On the Simulations page, click New scenario.
2

Choose a default dataset and evaluator (optional)

Both can be picked here as defaults, or chosen per-run when you start a run.
3

Choose a target type

A scenario’s target is what actually receives each test case’s input. Pick one:
  • Prompt Target — Zespan calls a model directly. Pick a project LLM connection/provider/model, then write a prompt template using {{input}} for the test case’s input.
  • HTTP Target — Zespan sends each test case to a URL you control instead of calling a model itself. Configure the URL, HTTP method, and a JSON request body template (also using {{input}}). Use this to simulate against your own deployed agent/service rather than a bare prompt.
  • Conversation (multi-turn) — Zespan runs a simulated multi-turn conversation: a “simulated user” model drives the conversation against an inner prompt or HTTP target for up to 20 turns, stopping on max turns, a keyword match, or an evaluator verdict. Use this for testing agents that need several exchanges to complete a task, not just a single-shot response. The inner target’s prompt template is applied to every turn, with {{input}} standing in for that turn’s message — the initial test-case input on the first turn, and the simulated user’s reply on each turn after. Leave it as {{input}} to send each turn through untouched. The run’s Conversation Transcript always shows what the simulated user actually said, not the template-wrapped text the target received.
4

Add evaluators

Attach an evaluator to score the outputs — any evaluator defined on the Evaluations page. You can also add lightweight assertions (contains / not-contains / regex match) and a max-latency check directly on the scenario, independent of the evaluator.
5

Save the scenario

Click Save. The scenario is ready to run.
Prompt and conversation targets call an LLM through the project’s own LLM connection (HTTP targets never call an LLM — they call your URL instead). Without a connection configured, starting a run on a prompt or conversation scenario fails with “No LLM connection configured — add one in Settings → LLM Connections.” See LLM Connections.

​
Persona configuration for conversation scenarios

When a scenario’s target is Conversation (multi-turn), the simulated user’s behavior is driven by a Simulated User Prompt — a system prompt for the model playing your end user. You can write this prompt as free text, or use the structured Persona fields to compose it instead: All persona fields are optional and independent of each other — set only the ones you care about. If at least one persona field is set, it composes the simulated user’s system prompt; if none are set, the free-text Simulated User Prompt is used as-is. Persona and free-text prompt are not combined — a scenario uses one or the other.

​
Running a simulation

Click Run now on any scenario to start a simulation run. Zespan:
  1. Iterates over every item in the dataset
  2. Sends each item’s input to the scenario’s target — a model call, an HTTP request, or a simulated multi-turn conversation, depending on the target type
  3. Records the response, latency, token count, and cost
  4. Runs each configured evaluator on the output
  5. Aggregates results into a run summary
Simulation runs appear in the Run history table. Click any run to see per-item results.

​
Batch runs — comparing versions

A batch run executes the same dataset against multiple configurations simultaneously, making it easy to compare prompt versions head-to-head.
1

Click New batch run

Select two or more scenarios (or one scenario with multiple prompt version variants).
2

Start the batch

Click Run batch. All variants run in parallel.
3

Compare results

When all runs complete, the comparison view shows side-by-side scores, latency, and cost for each variant. Rows with significant differences are highlighted.
Use batch runs before promoting a prompt from staging to production. Run the current production version and the candidate version through the same dataset and only promote if the candidate scores better on your key evaluators.

​
Reading run results

Each simulation run’s detail view shows:
  • Summary cards — average score per evaluator, total cost, average latency
  • Per-item table — each dataset item with its output, scores, and a link to the full trace
  • Score distribution — histogram of score spread across items
  • Failed items — items where the model returned an error or a score below threshold
Click any item row to see the full output text and all evaluator scores for that item.
For a run on a Conversation (multi-turn) scenario, the item’s detail view also shows a Conversation Transcript — every turn in order, labeled Simulated User or Target Agent, with that turn’s latency and cost, plus a total conversation cost computed from real per-model pricing across all turns.

​
Plan limits

Simulations are a Team/Scale-only feature — every mutating simulations route (creating or editing a scenario, and starting, retrying, batching, or cancelling a run) requires the Team plan or above server-side. There is no intermediate Pro-tier access.