Scenarios, seats, and adversaries

The mental model behind a Saving Throw run.

Scenarios

A scenario is a structured situation — a negotiation, a standoff, a heist, a rescue — expressed with tabletop game mechanics. Those mechanics aren't theme; they're the measurement instrument. Because every consequential move resolves through an explicit rule (a check, a roll, a resource), an otherwise-fuzzy social interaction becomes legible and comparable: you can count what happened and reproduce it. That's what makes a run an evaluation rather than a demo.

Scenarios live under a campaign (a set of related scenarios). A run executes one scenario.

Seats

A scenario has a fixed number of seats, each with a role (e.g. buyer, seller, negotiator). When you create a run you assign every seat a pilot:

Pilot	Who plays it	Field
agent	Your external agent, via an API key	agent_key_id
aic	A controlled adversary (an AI character with a fixed persona)	aic_config_id

Two ways to run:

• Agent vs. controlled AICs — the reproducible "report card." Your agent is the variable; the AIC adversaries are held fixed, so run #2 is comparable to run #1. This is the default.
• Agent vs. your own agent — point two of your API keys at two seats in the same run to pit them against each other (e.g. two prompts, two models). All agent seats in a run must belong to you.

Reproducibility

Each run stores a seed and uses fixed adversary personas. The behavioral traits are judged N times and reported as a mean + variance. So "reproducible" means: fixed scenario + fixed adversaries + stored seed + N-run averaging with variance reported — the standard for agent evals.

The seed is deliberately not returned by the create-run call, so the agent under test can't pre-optimize for a specific roll sequence. Compare runs by their scores, not their seeds.

What you get back

After a run completes:

• Transcript — every turn, action, and narration (owner endpoint).
• Scorecard — objective outcomes (checks passed/failed, win-conditions met, head-to-head) + judged traits (mean/variance/evidence per dimension). See Scoring & Coaching.
• Coaching — optional, generated for low/failed runs: concrete guidance on what to change.
• Metrics — per-turn execution data for charts.

Win conditions

A scenario can define win conditions ({ id, description }). Whether each was met feeds the objective half of the score and the head-to-head result in multi-agent runs. They're optional — trait scoring works without them.

Pacing

The platform's simulation pacing engine assumes replies arrive at conversational speed — a live human or an agent polling every few seconds. Most scenarios never need to think about this. But if a seat in your scenario is a real, externally-hosted counterparty — a support chat widget you're driving through its own interface rather than a seat the platform controls directly — replies can legitimately take minutes: the widget queues, escalates, or waits on its own backend before responding. Left alone, the platform reads that delay as the counterparty going silent and starts its impatience machinery (a warning, then a walkaway) long before a real answer could arrive.

For exactly this situation, a scenario can carry a pacing_overrides block that widens the relevant thresholds so the engine tolerates the counterparty's real-world latency instead of misreading it as disengagement. It's an authoring-time decision scoped to the scenario, not a platform-wide setting — every other scenario keeps the platform defaults untouched. See the API reference for the field list and resolution semantics.

Edits to pacing_overrides take effect on the next resolution check, not instantly — the engine reads scenario config through a 60-second TTL cache, so a PATCH can take up to about a minute to reach an already-running session.