Skip to main content
LLM Connections are separate from your Zespan API key. An API key authenticates the SDK sending traces to Zespan. An LLM Connection is a provider credential you supply so Zespan can call an LLM on your behalf, billed to you.

​
What it is

Some Zespan features need to call an LLM to do their job — running a prompt in the Playground, scoring an evaluation with an LLM judge, or executing a simulation scenario. These features are bring-your-own-key (BYOK): you connect a provider credential in Project Settings, and Zespan uses it to run those calls. You pay your provider directly for the inference — Zespan doesn’t mark it up or meter it. Credentials are encrypted at rest with AES-256-GCM and are write-only — once saved, the raw key is never returned by the API or shown in the UI again. Every connection in the list only ever displays a masked placeholder for the key.
ZespanPilot and Zespan’s built-in AI features (cost optimization, anomaly detection, root-cause analysis, natural-language queries, overview summaries) don’t need a connection — they work out of the box. LLM Connections only affect the BYOK features listed below.

​
Where to connect

Go to Project Settings → LLM Connections.
Some newer Claude models, including claude-sonnet-5, claude-opus-5, and the 5.5 models, don’t accept a temperature setting. When you use one of them through your own connection, Zespan leaves temperature unset automatically so calls don’t fail.

​
Connection types

Connect a key directly from OpenAI, Anthropic, or Google. Give the connection a name and paste the provider’s API key. Use this when you already have a standing account with one of these providers and want Zespan to call it directly.
Bedrock and Vertex don’t authenticate with a single API key, so they take structured credentials instead. Everything is encrypted at rest, and no secret field is ever returned to the browser. For Bedrock, prefer Assume role so Zespan never stores long-lived AWS keys.

​
Bedrock: assume role

In role mode, Zespan sends a per-project External ID every time it assumes your role. Adding it to the role’s trust policy means only your project can use that role, even if someone else learns the role ARN.
1

Copy your External ID

Go to Settings → LLM Connections → Add connection → Cloud provider → Amazon Bedrock → Assume role. Zespan shows the project’s External ID (zespan- followed by 32 hex characters) and a ready-made trust policy, each with a copy button.
2

Add it to the role's trust policy

Use the principal and External ID shown in the dashboard; the values above are placeholders.
3

Save the connection

Paste the role ARN and save. Zespan sends the External ID on every assume-role call for this project.
A role whose trust policy requires a different External ID can’t be assumed until you update the condition to the ID shown for this project. A role whose trust policy has no sts:ExternalId condition still accepts requests from anyone who can name it — add the condition to every role you use with Zespan.

​
Private and self-hosted endpoints

By default, an LLM connection can’t point at a private address, such as an in-VPC vLLM or Ollama gateway — see Outbound request policy. On a self-hosted deployment, operators can allow specific private endpoints with the LLM_CONNECTIONS_ALLOW_PRIVATE environment variable on the API and background-job processes:
An invalid value stops the API from starting, and the error names the variable and the bad entry. Saving a connection whose base URL is still refused fails with baseUrl host is not allowed.
Only list endpoints you operate. Any project that can create an LLM connection can reach every listed endpoint.

​
Default connection and multiple connections

The first connection you add for a project is automatically set as its default. You can change the default at any time from the connections list — each connection has a Set default action, and only one connection per project can be default. When a run needs credentials and doesn’t specify a connection explicitly, Zespan resolves them in this order:
  1. The connection you explicitly picked for that run (if any)
  2. The project’s default connection
  3. If no default is set, the most recently created connection
If a project has two or more connections, a compact LLM connection picker appears on the run surface (Playground window, quality gate, run-over-dataset, simulations, eval dry-run) so you can choose which one runs — and pays — for that execution. With zero or exactly one connection, the picker stays hidden: with one connection there’s nothing to choose, so it’s used automatically and silently.

​
Features that require a connection

The following surfaces are fail-closed — without at least one LLM connection configured for the project, they show a “Connect an LLM provider” state instead of running:
  • Playground
  • Evaluations — the LLM-judge dry-run, auto-eval scoring, and Run Eval
  • Simulations
  • Run-over-dataset
If you delete or run out of a connection mid-use, the next run attempt returns a clear error rather than falling back to any shared account.
Prompt Enhance is a Pro feature and does not need an LLM connection — it works out of the box. ZespanPilot and Zespan’s other built-in AI features are likewise exempt — see the callout above.

​
Metered on Zespan’s key, then a connection

Semantic Judge guardrails (custom_llm) work differently from the fail-closed surfaces above: they run on Zespan’s own key out of the box, up to a monthly quota that scales with your plan (100–5,000 checks, unlimited on Scale) — no connection required to get started. Once that quota is used up, checks fail closed according to the guardrail’s configured action instead of running unmetered on Zespan’s key. Connecting your own LLM removes the monthly cap entirely for Semantic Judge checks on that project — they run on your connection instead, at your own provider’s rates. See Guardrails → AI usage limits.

​
Testing a connection

Each connection in the list has a Test button. For key-based connections (OpenAI, Anthropic, Google, gateway, OpenRouter) it sends a small, cheap authenticated request to the provider — for example, listing available models — to confirm the credential is valid and reachable, without running a real completion. For Bedrock and Vertex there’s no provider-wide models-list to hit with just credentials, and which models are enabled varies by account and region — so the test runs a one-token generation against the model you name. That’s the truest check: it exercises the real auth path (assume-role / access keys / service-account) and confirms the model is enabled where you’re pointing. If you test without naming a model, Zespan confirms the credentials were stored but tells you plainly that it didn’t run a live check. Either way the result (success, or an error) is shown inline, and no secret field is ever echoed back.

​
Deleting a connection

Deleting a connection removes it immediately — any run relying on it will no longer resolve credentials. If the connection you delete was the project’s default, Zespan automatically promotes the most recently created remaining connection to default (if any are left).

​
Self-hosting

Self-hosted deployments must set LLM_CONNECTION_SECRET — a 64-character hex string (32 bytes) used to encrypt connection keys and HTTP target credentials at rest with AES-256-GCM. It is required in production, and new credentials are never encrypted with SSO_SECRET. Credentials saved earlier under SSO_SECRET can still be read, so keep SSO_SECRET set. For Bedrock role mode, set ZESPAN_AWS_PRINCIPAL_ARN to the AWS principal your deployment uses, so the trust policy shown in the dashboard names it. See Environment variables and Self-hosting.

​
Next steps