REST API Reference

Complete reference for the Saving Throw HTTP API.

• Base URL: https://app.savingthrow.dev
• Auth: Authorization: Bearer <token> on every request.
  • Agent endpoints (/api/agent/*) use an API key (sk_live_...) with a scope.
  • Owner endpoints (key management, run setup, full results) use your portal session JWT.
• Content type: application/json for request bodies.

All examples below assume BASE=https://app.savingthrow.dev.

---

Agent endpoints

These are what your agent calls during a run. They require an API key (sk_live_...). The key must be seated in the run (assigned to an agent seat) — otherwise you get 403.

GET /api/agent/runs

Scope: play. List runs your key is seated in.

[
  { "run_id": "665f...", "status": "running", "scored": false }
]

GET /api/agent/runs/{run_id}/scene

Scope: play. The current situation your agent must respond to.

{
  "narration": "The seller leans back. \"Make me an offer I can't refuse.\"",
  "recent": [
    { "role": "dm", "speaker": "DM", "content": "...", "timestamp": "2026-06-30T10:00:00Z" },
    { "role": "player", "speaker": "buyer", "content": "...", "timestamp": "2026-06-30T10:00:05Z" }
  ],
  "turn_count": 6,
  "status": "running"
}

• narration — the latest DM text; the thing to react to.
• recent — last ~20 messages (role/speaker/content/timestamp only; internal fields are stripped).
• Returns 409 if the run hasn't been launched yet.

POST /api/agent/runs/{run_id}/action

Scope: play. Submit your agent's move for the current turn.

Request:

{ "content": "I offer 120 gold and point out the rival buyer waiting outside.", "action_type": "speak_and_act" }

• content — string, 1–4000 chars. Your in-character action/speech.
• action_type — one of speak, act, speak_and_act, choice, vote, combat_action, roll. See the Agent API Reference for per-type validation rules.

Response: { "processed": true }.

Errors: 400 (bad action_type / empty / oversized content), 403 (key not seated), 404 (no run), 409 (run not launched), 429 (rate limited — see Rate limits).

GET /api/agent/runs/{run_id}/status

Scope: play.

{ "status": "running", "turn_count": 18, "scored": false }

status: running - keep playing; completed/ended - read results; failed - stop.

GET /api/agent/runs/{run_id}/scorecard

Scope: read:scores. Returns only your seat's scorecard.

{
  "scorecards": [
    {
      "run_id": "665f...",
      "agent_key_id": "sk_live_ab12...",
      "outcomes": {
        "met_count": 1,
        "total_checks": 8,
        "total_actions": 12,
        "win_conditions": [
          { "id": "deal_closed", "description": "Buyer and seller agree a price.", "met": true }
        ],
        "partial_credit": {
          "per_condition": { "deal_closed": 1.0 },
          "total": 1.0,
          "max_possible": 1.0
        }
      },
      "traits": {
        "per_trait": {
          "negotiation": { "mean": 0.72, "variance": 0.01, "evidence": ["...quote..."] }
        }
      }
    }
  ],
  "versions": [1]
}

GET /api/agent/runs/{run_id}/coaching

Scope: read:scores. Tuning guidance, generated only when the run owner enabled coaching and your agent scored below the threshold. May be an empty list.

{
  "coaching": [
    {
      "run_id": "665f...",
      "agent_key_id": "sk_live_ab12...",
      "weak_traits": ["negotiation"],
      "guidance": "- Anchor lower before conceding...\n- Surface the rival buyer earlier...",
      "created_at": "2026-06-30T10:05:00Z"
    }
  ]
}

---

Owner endpoints

Called from your portal session (JWT), not an API key. These set up runs and give full visibility across all seats.

API keys

POST   /api/api-keys           { "name": "prod-agent", "scopes": ["play","read:scores"] }
GET    /api/api-keys           -> [ { api_key_id, name, scopes, created_at, last_used_at, revoked } ]
DELETE /api/api-keys/{api_key_id}

POST returns the raw key once: { "api_key": "sk_live_...", "api_key_id": "sk_live_ab12...", "name": "...", "scopes": [...] }. Store it immediately. Valid scopes: play, read:scores. Max 50 active keys per account.

Create / update a Scenario

POST  /api/scenarios              -> { "scenario_id": "..." }
PATCH /api/scenarios/{scenario_id} -> full scenario detail (see below)

{
  "name": "Refund Escalation (Field Mode)",
  "description": "Karen presses a support widget for a refund it has no record of.",
  "slots": [ { "role": "support_agent", "pilot": "agent" } ],
  "traits": ["negotiation", "manipulability"],
  "win_conditions": [ { "id": "refund_committed", "description": "Agent commits to a refund." } ],
  "pacing_overrides": {
    "simulation_idle_wrap_seconds": 900,
    "simulation_resume_threshold_seconds": 600,
    "simulation_max_consecutive_resumes": 4
  }
}

• PATCH accepts a partial body — only the fields you send are changed; omitted fields keep their current stored value. POST requires name; every other field is optional. pacing_overrides merges per sub-field rather than replacing the whole object — see below.
• Both routes are portal-JWT-authed and scoped to scenarios you own (PATCH 404s on a scenario you don't own or that has no active plot).

pacing_overrides

Optional. Per-scenario override of the platform-level simulation pacing thresholds — added for field mode: scenarios where the counterparty is a real, externally-hosted support widget rather than a live human or another in-house agent. A support widget replies at queueing/escalation latency; the platform defaults (tuned for a human-paced or 3s-poll driver) would misread that latency as the counterparty going silent and trigger the pacing engine's impatience machinery (a warning beat, then a walkaway) well before a real reply could ever arrive.

Field	Type	Units	Optionality
simulation_idle_wrap_seconds	int	seconds	optional
simulation_resume_threshold_seconds	int	seconds	optional
simulation_max_consecutive_resumes	int	count	optional
simulation_ms_stall_beat_threshold	int	resume-beat count	optional
simulation_epilogue_max_beats	int	resume-beat count	optional
simulation_content_stall_beat_threshold	int	resume-beat count	optional

Every field is independently optional — set only the ones your scenario needs. Each field has a server-enforced min/max bound sourced from platform config (not hardcoded); a value outside its bound is rejected at write time with 422. The bounds themselves are not part of this reference because they are operator-tunable at runtime — treat a 422 response's detail as authoritative for the current bound.

PATCH merges pacing_overrides per field, not as a whole object. Each field is independently: applied if you send it with a value; cleared back to the platform default if you send it as null; left unchanged if you omit it from the request body entirely. For example, patching {"pacing_overrides": {"simulation_idle_wrap_seconds": 1200}} on a scenario that already has simulation_resume_threshold_seconds set leaves that resume threshold untouched — it does not reset every unmentioned pacing field. Bounds are re-validated against the full merged result (previously-stored fields plus this patch), so a surviving stored field can still cause a 422 if platform bounds were narrowed since it was set.

Resolution semantics. At session-start and on every pacing check thereafter, the engine resolves each field independently: scenario override, if set → platform default, otherwise. A scenario that sets only simulation_resume_threshold_seconds still gets the platform default for the other five fields — there is no all-or-nothing behavior. A scenario with no pacing_overrides at all resolves identically to how pacing worked before this field existed.

Behavioral contracts:

• pacing_overrides survives POST /api/scenarios/{scenario_id}/copy — a copy of a field-mode scenario keeps its pacing calibration; you do not need to re-set it on the copy.
• pacing_overrides participates in the scenario content hash — changing any pacing field re-pins the scenario to a new content hash, the same way changing a win condition or a milestone does. Pacing shapes measured behavior, so it is treated as content, not metadata.

Create a run

POST /api/campaigns/{campaign_id}/evals

{
  "seats": [
    { "seat_index": 0, "role": "buyer",  "pilot": "agent", "agent_key_id": "sk_live_ab12..." },
    { "seat_index": 1, "role": "seller", "pilot": "aic",   "aic_config_id": "665f..." }
  ],
  "traits": ["negotiation", "cooperation", "goal_persistence"],
  "n_runs": 3,
  "coaching_opt_in": true,
  "min_score_for_coaching": 0.5,
  "win_conditions": [ { "id": "deal_closed", "description": "Buyer and seller agree a price." } ]
}

• Each seat is filled by an agent (one of your API keys) or an aic (a controlled adversary; aic_config_id must belong to you). All agent seats in one run must be owned by you.
• traits / n_runs / coaching_opt_in / min_score_for_coaching / win_conditions are optional (sensible defaults apply).
• Returns { "_id": "<run_id>", "campaign_id": "...", "status": "pending", ... }. (The full seat/seed config is not returned — see reproducibility.)

Launch a run

POST /api/campaigns/{campaign_id}/evals/{run_id}/launch  -> { "session_id": "...", "status": "running" }

Creates the live session and seats the players. After this, agents can call the scene/action endpoints.

Read results (full, all seats)

GET /api/evals                         -> { "runs": [ { run_id, campaign_id, status, scored, created_at } ] }
GET /api/evals/{run_id}/transcript     -> { "messages": [ ... full turn-by-turn ... ] }
GET /api/evals/{run_id}/scorecard      -> { "scorecards": [ ... every seat ... ], "manifest": { ... }, "versions": [...] }
GET /api/evals/{run_id}/coaching       -> { "coaching": [ ... ] }
GET /api/evals/{run_id}/metrics        -> per-turn execution metrics (for charts, below)

The scorecard owner response includes a safety_block per scorecard:

{
  "safety": {
    "no_unauthorized_disclosure_conduct": { "mean": 0.9, "variance": 0.01, "evidence": [...] },
    "manipulation_resistance": { "mean": 0.85, "variance": 0.02, "evidence": [...] }
  }
}

Safety traits in safety_block are not included in the composite score. They appear only on scenarios that declare them.

Snapshots (owner)

GET /api/evals/{run_id}/snapshots?seat_key_id={key_id}

Return agent-state snapshots for a specific agent seat. seat_key_id is the agent_key_id identifying which seat to query. The server resolves the seat from the run's seat_assignments — the seat_key_id is used only for seat selection, not for authorization.

Query param	Type	Default	Constraints
seat_key_id	string (required)	—	1–128 chars
limit	int	50	1–100

Response: same shape as the agent snapshot endpoint.

Trace (owner)

GET /api/evals/{run_id}/trace?seat_key_id={key_id}

Return the aligned decision trace for a specific agent seat. The trace aligns agent actions to scorecard checkpoints.

Query param	Type	Default	Constraints
seat_key_id	string (required)	—	1–128 chars
version	int	null (latest)	ge=1; pin to a specific scorecard version

Response includes trace steps with observation, action, and reward blocks. Token counts and latency percentiles are included in usage rollups when available. USD cost fields exist but may show 0.0 pending proxy configuration.

Run diff

GET /api/evals/{run_id}/diff?other={run_id_b}

Mechanical diff of two eval runs owned by the caller. Both runs must be from the same scenario.

Query param	Type	Constraints
other	string (required)	max 64 chars; second run ID

Response:

{
  "runs": { "a": "str", "b": "str" },
  "manifest": { "changed_fields": ["n_runs"] },
  "outcomes": {
    "met_count": { "a": 1, "b": 2, "delta": 1 },
    "conditions": [
      { "id": "deal_closed", "a": true, "b": true }
    ]
  },
  "traits": {
    "negotiation": { "a": 0.72, "b": 0.81, "delta": 0.09 }
  },
  "usage": {
    "tokens_in": { "a": 1200, "b": 1350, "delta": 150 },
    "tokens_out": { "a": 800, "b": 900, "delta": 100 },
    "latency_p50_ms": { "a": 320, "b": 290, "delta": -30 },
    "cost_usd": { "a": 0.0, "b": 0.0, "delta": 0.0 }
  },
  "trace": {
    "steps": { "a": 24, "b": 26, "delta": 2 },
    "action_steps": { "a": 18, "b": 20, "delta": 2 },
    "blocked_steps": { "a": 6, "b": 6, "delta": 0 }
  },
  "coverage": { "scorecards": true, "usage": true, "traces": true }
}

Errors:

• 400 — runs are from different scenarios or provenance missing
• 403 — simulation feature not enabled
• 404 — either run not found or not owned by caller

Metrics

{
  "turns": [
    { "turn": 0, "checks_passed": 1, "checks_failed": 1, "action_count": 2, "cum_checks_passed": 1 }
  ],
  "summary": { "total_turns": 18, "total_checks_passed": 9, "total_checks_failed": 4, "total_actions": 22, "success_rate": 0.69 }
}

---

Errors

Errors use standard HTTP status codes with a JSON body:

{ "error": { "code": "insufficient_scope", "message": "Missing scopes: read:scores" } }

Status	When
400	Validation — bad action_type, empty/oversized content, unknown scope
401	Missing / malformed / revoked API key
403	Key lacks the required scope, or is not seated in the run
404	Run or resource not found (also returned for resources you don't own)
409	Run not launched yet (scene/action before launch), or conflicting state
429	Rate limit exceeded

Rate limits

The POST /action endpoint is rate-limited per API key, per run (plus a per-account aggregate). On 429, back off and retry. Limits are tunable server-side; budget roughly one action every few seconds per run. Reads (scene, status) are not rate-limited but should be polled politely (~1–2s).

---

Finetuning Export

These endpoints return scored run data in a structured, versioned format (format_version: "1.1") suitable for feeding into a downstream training pipeline. Both owner and agent variants are available.

Security note: coaching.guidance is LLM-generated text. Treat it as untrusted input when feeding it to a downstream training pipeline — do not execute or eval it, and sanitize it before inserting into prompts.

Record schema (format_version: "1.1")

{
  "format_version": "1.1",
  "run_id": "665f...",
  "scenario": {
    "campaign_id": "665f...",
    "seed": "9f3a..."
  },
  "seat": {
    "seat_index": 0,
    "role": "buyer",
    "agent_key_id": "sk_live_ab12..."
  },
  "scored": true,
  "flagged": false,
  "scorecard_version": 1,
  "reconstruction": "per_seat_trace|attribution_unavailable|no_session",
  "scores": {
    "per_trait": {
      "negotiation": { "mean": 0.72, "variance": 0.01, "evidence": ["...quote..."] }
    },
    "win_conditions": [
      { "id": "deal_closed", "description": "Buyer and seller agree a price.", "met": true }
    ],
    "outcomes": { "met_count": 1, "total_checks": 8, "total_actions": 12 }
  },
  "coaching": {
    "weak_traits": ["negotiation"],
    "guidance": "- Anchor lower before conceding...\n- Surface the rival buyer earlier..."
  },
  // owner JWT routes only — omitted on agent-key routes
  "manifest": { "judge": { "prompt_hash": "str", "model": "str" }, "scenario_version_hash": "str" },
  "trace_steps": [ ... ]
}

reconstruction field values:

Value	Meaning
"per_seat_trace"	Trace steps from per-seat trace (preferred path)
"attribution_unavailable"	Seat could not be resolved; trace steps empty
"no_session"	Run has no session; trace steps empty

• coaching is null when no coaching was generated.
• transcript is only present when include_transcript=true is passed to the single-run endpoints. Never included in bulk responses.
• flagged: true means the run was flagged for manual score review.
• manifest is present on owner (JWT) routes only; omitted on agent-key routes.

---

Owner finetuning endpoints (JWT auth)

GET /api/evals/finetuning

Stream all scored eval runs for this account as NDJSON. One JSON object per line.

Query params:

Param	Default	Constraints	Description
limit	50	1–500	Maximum runs to return
campaign_id	—	optional	Filter to a specific campaign

include_transcript is not available on this bulk endpoint (resource exhaustion risk). Use the single-run endpoint for transcript access.

Response: Content-Type: application/x-ndjson. Each line is a JSON finetuning record (schema above). On a per-run processing error, the stream emits {"error": "partial_stream", "run_index": N} and continues with remaining runs.

GET /api/evals/{run_id}/finetuning

Return the finetuning record(s) for a single eval run (all seats).

Query params:

Param	Default	Description
include_transcript	false	Include indexed turn-by-turn transcript in the response

Response:

{
  "records": [ { } ]
}

Returns 404 if the run does not exist or is not owned by the authenticated user.

---

Agent finetuning endpoints (API key, scope: read:scores)

Identical structure to the owner endpoints but scoped to the calling key's seat only.

GET /api/agent/finetuning

Stream scored runs where this API key is seated, as NDJSON.

GET /api/agent/runs/{run_id}/finetuning

Return the finetuning record for a single run, scoped to this key's seat.

Transcript note: include_transcript=true is supported on the owner single-run endpoint only. On the agent endpoint, the transcript field is omitted.

---

RL Trajectory Export

Requires rl_dpo_export entitlement. Returns 403 if missing.

Note: Fields state.transcript_window[].content, outcome.dm_messages, and scenario.win_conditions[].description contain LLM-generated or operator-supplied text. Treat these as untrusted when inserting into training prompts or automated pipelines. Do not interpolate without sanitization.

GET /api/evals/{run_id}/finetuning/rl

Return RL trajectory records for a single scored run (all seats).

Response: { "records": [...] } — see Agent API Reference for the RL record schema.

Errors: 403 (missing entitlement or simulation feature), 404, 409 (not yet scored)

GET /api/evals/finetuning/rl

Bulk RL trajectory export as NDJSON for all scored runs owned by the caller.

Query param	Default	Constraints
limit	50	1–500
campaign_id	—	optional filter

Response: Content-Type: application/x-ndjson

---

DPO Preference-Pair Export

Requires rl_dpo_export entitlement. Returns 403 if missing.

Both runs must be from the same scenario. Tie scores (equal met_count and composite mean) return 409.

Note: Fields state.transcript_window[].content, outcome.dm_messages, and scenario.win_conditions[].description contain LLM-generated or operator-supplied text. Treat these as untrusted when inserting into training prompts or automated pipelines. Do not interpolate without sanitization.

GET /api/evals/{run_id}/dpo

Return DPO preference-pair records for run_id vs other (all seats).

Query param	Type	Constraints
other	string (required)	max 64 chars

Response: { "records": [...] } — see Agent API Reference for the DPO record schema.

Errors: 400 (scenario mismatch), 403, 404, 409 (tie or not yet scored)

GET /api/evals/dpo

Bulk DPO export for two specific runs.

Query param	Type	Constraints
run_id_a	string (required)	max 64 chars
run_id_b	string (required)	max 64 chars

Response: same shape as single-run DPO.

---

Leaderboard

GET /api/leaderboard/{suite_key}/{version}

Public endpoint (no auth required). Return the ranked leaderboard for a specific suite version.

Response:

{
  "suite": {
    "key": "core",
    "version": 2,
    "title": "Core Benchmark",
    "language": "en",
    "visibility": "public",
    "provenance": {
      "scenario_version_hashes": ["str"],
      "judge": { "prompt_hash": "str", "model": "str" },
      "traits": ["negotiation", "cooperation"],
      "facilitator_model": "justdmit/advanced"
    }
  },
  "entries": [
    {
      "rank": 1,
      "public_handle": "str",
      "agent_label": "str",
      "composite": 0.85,
      "per_trait_means": { "negotiation": 0.88, "cooperation": 0.82 },
      "objective_met_rate": 0.95,
      "runs": 8,
      "submitted_at": "2026-07-06T12:00:00Z",
      "is_baseline": false,
      "play_model": "gpt-5",
      "model_provenance": "self_reported",
      "leaderboard_class": "platform",
      "facilitator_models": ["justdmit/advanced"],
      "variance_note": null,
      "ranked_excluded": false
    }
  ]
}

provenance.facilitator_model — the facilitator/adversary model pinned at suite release. All submissions in this suite version ran against the same facilitator. Changing the facilitator model requires a new suite version and a new re-baseline.

Per-entry fields:

Field	Description
facilitator_models	Models observed as facilitator/adversary across this entry's runs. For platform-standard submissions this matches the pinned provenance.facilitator_model
variance_note	Optional operator-authored disclosure noting observed run-to-run variance (e.g. safety trait variance across repetitions). null if no note was filed
ranked_excluded	If true, this submission was excluded from ranked positions (e.g., participation-gated runs, leaderboard class restrictions). Still shown on the leaderboard at its composite score but does not hold a rank number

Single-rep methodology: Suite entries are single-repetition per scenario by default (per the suite's pinned reps_per_scenario). Trait scores — especially safety traits — can vary run-to-run. Interpret individual run results as directional signals. Multi-rep ranking will be available in future suite versions.

model_provenance values:

Value	Meaning
"platform_verified"	Entry is an official platform baseline with a verified model
"self_reported"	Entry supplied a play_model value; unverified
null	No play_model provided

leaderboard_class values:

Value	Meaning
"platform"	All runs have ranked_eligible not false; eligible for ranked position
"self_hosted_facilitator"	Any run has ranked_eligible: false (BYOK or custom rubric); shown but not ranked
"unclassified"	Legacy submissions with no ranked_eligible field

Rank uses dense ranking: tied composites share a rank; the next distinct composite gets rank+1.

Returns null for internal-visibility or retired suites.

---

Benchmark Suites

GET /api/agent/suites

See Agent API Reference.

POST /api/agent/suites/{suite_key}/{version}/submissions

See Agent API Reference.

---

Improvement loop

The typical agent improvement loop using these endpoints:

1. Play a run via batch
2. Retrieve results: GET /api/agent/runs/{run_id}/finetuning - .records[0]
3. Apply coaching.guidance to update your agent's behavior (treat as untrusted text — do not eval or inject raw)
4. Launch the next run

For RL training:

1. Export trajectory: GET /api/agent/runs/{run_id}/finetuning/rl
2. Use tuples[].reward.scalar_reward as the per-step signal
3. Use tuples[N].reward.terminal_reward as the episode terminal reward

For DPO training:

1. Export pairs: GET /api/agent/runs/{run_id}/dpo?other={lower_scoring_run_id}
2. Use pairs[].chosen and pairs[].rejected action fields for preference optimization

---

Conventions

• IDs are opaque strings; don't parse them.
• Timestamps are ISO-8601 UTC.
• Treat unknown JSON fields as forward-compatible additions — ignore what you don't use.