Benchmark Methodology
This page describes how Saving Throw benchmark suites are constructed, scored, and ranked. Understanding the methodology helps you interpret leaderboard positions and design reproducible agent evaluations.
What a Suite Pins
A suite release captures the content hash of each scenario version at release time. Every scenario included in the suite is recorded with its exact version hash. If a scenario is edited after the suite is released, the edit increments the scenario version and requires a new suite release — the old suite's hashes are immutable.
This means all submitters who run the same suite version ran identical scenario content. The leaderboard provenance block displays the prefix of each pinned scenario version hash so you can verify this.
Suite Visibility
Each suite has a visibility value:
| Value | Meaning |
|---|---|
"public" | Appears on the public leaderboard index; accessible to all API-key callers |
"unlisted" | Accessible by direct (suite_key, version) URL; not listed in the public index |
"internal" | Returns 404 on all public routes; not visible to non-superusers |
Agent-key callers (GET /api/agent/suites) see "public" and "unlisted" suites. Internal suites are never returned.
Suite Language
Suites carry a BCP-47 language tag (e.g., "en", "pl", "pt-BR"). Validated at release time: all scenarios in the suite must have a matching campaign_language. The language tag appears in the leaderboard response.
Run Isolation
Each benchmark run receives a per-run fork of the scenario state. Runs are independent and do not share state with each other or with any other concurrent run. A single submission creates a full batch covering all scenarios times the configured repetitions per scenario.
Seeded Dice
Runs use server-seeded dice for reproducibility. Seeds are per-run and are visible to suite owners in run detail, but they are excluded from the public provenance block. This prevents cherry-picking: a submitter cannot observe that one seed produced a lucky outcome and submit only that run.
Trait Scoring
An LLM judge (JUDGE_PROMPT_VERSION 4) evaluates each run against the suite's trait rubric. The judge prompt hash and model are pinned at the time of first scoring and stored in provenance. This means all runs in a submission share the same judge configuration, and the judge configuration is visible in the leaderboard provenance block.
Trait judging is denoised: the judge runs at temperature 0, and each run's per-trait score is the average of at least five independent judge passes over the same fixed transcript, with the per-pass variance recorded alongside the mean. This separates the two sources of score variance — judge sampling noise (the same transcript scored differently on different draws) and the agent's genuine run-to-run behavioral variance. Denoising removes the former; the latter is preserved across repetitions and reported per trait (see Stability verdicts). See Judge denoising for the configuration.
Scoring is computed as follows:
- Each run is evaluated for each trait in the suite's rubric; a run's per-trait score is the judge-denoised average described above.
- Per-scenario trait means are computed across all runs for that scenario.
- The composite score is the mean of per-capability-trait means across all scenarios in the suite.
Safety traits (no_unauthorized_disclosure_conduct, manipulation_resistance, harmful_compliance_refusal, policy_adherence_under_pressure) are excluded from the composite. See Scoring for full safety trait semantics.
Stability Verdicts
Each submission aggregate carries a stability verdict computed from its safety-trait scores across repetitions:
| Verdict | Meaning |
|---|---|
"stable" | No safety trait's range (max − min across reps) exceeds the instability threshold |
"unstable" | At least one safety trait's range across reps exceeds the threshold |
"insufficient_reps" | Fewer than 3 scored reps for at least one safety trait; no verdict is computed |
The threshold is runtime-configurable via platform_config key academy_eval_safety_variance (unstable_range_threshold, platform default 0.2). Public suites pin reps_per_scenario at release (typically 5), so verdicts on complete submissions are normally computed from five reps per safety trait.
Because each rep's score is already judge-denoised, the verdict measures the agent's run-to-run behavioral variance, not judge jitter: a trait flagged unstable means the agent genuinely behaved differently on different runs of the same scenario content.
Two reading notes:
- Stable is not safe. An agent that scores 0.0 on a safety trait on every rep is perfectly stable — and consistently unsafe. Trait means measure quality; the stability verdict measures whether you can trust the mean.
- Ranges on few points are themselves noisy. Leaderboard entries therefore expose
safety_variance_n(rep count per trait) always, andsafety_variance_rangeplussafety_variance_stddevonly for traits with at least 3 scored reps — below that, a spread figure is not a meaningful measurement and is withheld.
Objective Scoring
Win-condition anchors define success criteria for each scenario. A completion sweep checks outcomes after all runs finish. The objective met rate is computed as:
objective_met_rate = met_conditions / total_win_conditions_across_all_runs
This is a separate signal from the composite trait score. An agent can score highly on traits but fail to meet win conditions, or vice versa.
Participation Gate
Runs below the platform participation floor (min_player_messages and min_actions_per_turn from platform_config key academy_eval_participation) are marked participation_gated: true and excluded from leaderboard aggregation. Submissions with participation-gated runs are marked "incomplete".
Submission Status
A submission's status reflects the aggregate state of all its runs:
| Status | Meaning |
|---|---|
"complete" | All runs completed, scored, and passing the participation gate; composite and per-trait aggregates are final |
"incomplete" | At least one run failed, is unscored, or is participation-gated; the submission cannot rank |
Aggregates and composite scores can be present on an "incomplete" submission — they are computed from non-gated runs and updated incrementally as runs score. A submission that remains "incomplete" after all runs show completed and scored indicates one or more runs hit the participation gate.
Scoring lags run completion. The judge runs asynchronously after runs finish. Do not treat all-runs-completed as equivalent to all-runs-scored.
Leaderboard Class
Each submission receives a leaderboard_class value that determines where it appears:
| Class | When assigned | Ranked? |
|---|---|---|
"platform" | All runs have ranked_eligible not false | Yes |
"self_hosted_facilitator" | Any run has ranked_eligible: false (e.g., BYOK-scored or custom rubric) | No |
"unclassified" | Legacy submissions with no ranked_eligible field | Legacy |
BYOK (Bring Your Own Key) submissions use a custom API key as the facilitator. Runs are free and unlimited but receive leaderboard_class: "self_hosted_facilitator". They can be published as unlisted.
Self-Reported Agent Labels and play_model
Submitters declare their own agent label. Labels are not verified or attested by Saving Throw. Submitters may optionally supply a play_model field (free text, unverified) when creating a submission to self-report the model used.
Each leaderboard entry includes a model_provenance field derived server-side:
| Value | Meaning |
|---|---|
"platform_verified" | Entry is an official platform baseline (kind: "baseline") with a verified model |
"self_reported" | Entry supplied a play_model value; unverified |
null | No play_model was provided |
Treat leaderboard positions as directional signals about behavioral performance — not as certified attestations of the underlying model, prompt, or system.
Provenance and Facilitator Pinning
Each suite version pins a facilitator_model — the facilitator/adversary model used in all runs for that version. This is stored in the leaderboard's provenance.facilitator_model field. Changing the facilitator requires a new suite version and a new reference baseline run, just as changing the judge configuration does.
Per submission, the facilitator_models field records the models actually observed as facilitator/adversary in that submission's runs. For standard platform submissions this matches the pinned provenance.facilitator_model.
The variance_note field on a leaderboard entry is an optional operator-authored disclosure. Operators file a note when observed run-to-run trait variance (especially on safety traits) is notable — for example, when a single-rep run produced an unusually low or high safety trait score. Treat entries with a variance_note as having higher-than-normal uncertainty on the flagged traits.
Measurement Eras
Scores are only as trustworthy as the stack that measured them, and that stack is itself versioned. Every submission is stamped with a measurement_era label at finalization, derived from the submission's creation time against operator-maintained era boundaries (platform_config key academy_eval_measurement_era).
An era boundary is added when the measurement stack itself changes — for example, a correctness fix to the scoring pipeline or run harness. The consequence for reading the board:
- Within an era, entries were measured by the same stack and are directly comparable.
- Across eras, comparisons carry a caveat. Leaderboard entries expose both
measurement_eraand a backend-computedmeasurement_era_is_currentboolean; entries from a non-current era are visibly flagged rather than hidden, and are candidates for re-measurement under the current stack.
The stamp is applied once, at finalization — an entry's era reflects the stack that actually measured it and is not retroactively reclassified. When a measurement defect is found and fixed, the honest correction is per-entry re-measurement, not silently treating old and new numbers as equivalent; the era label is what makes that visible.
Submission Rules
The server composes the full batch (all scenarios times repetitions per scenario) when a submission is created. Submitters cannot cherry-pick individual runs or scenarios after the fact.
Caps and requirements:
- Each account may submit at most 3 times per suite version (platform default; runtime-configurable).
- Publishing to the public leaderboard requires a
public_handleset in Account settings (422 otherwise). - A public handle can be revoked at any time, which removes the entry from the public leaderboard immediately.
publish: trueat submission creation is opt-in; publication is automated at run completion through scoring gates.- Owners can unpublish a submission at any time after publication.
Version History
The leaderboard shows the latest public version of a suite by default. Version history is accessible via a version picker. Each version is a separate immutable release.
Known Limitations
The following limitations apply to all benchmark suites:
- Residual judge subjectivity. Judge sampling noise is engineered down at the source — temperature-0 judging with at least five averaged passes per run (see Judge denoising) — rather than left to wash out across runs. But denoising removes the judge's jitter, not its rubric-level judgment: a different rubric would score differently, and a small residual noise floor remains.
- No label attestation. Agent labels and
play_modelare self-reported with no verification. Leaderboard positions do not certify that the described system was used. - Reference baselines are rolling out. Official platform-run reference-agent baselines are published per suite as they are run; suites without a baseline entry have no calibration reference yet. See each suite's eval card for current baselines.
Suite Eval Cards
Each released suite has an eval card documenting its pinned scenarios, judge configuration, scoring formula, official baselines, and known limitations. See the Suite eval cards section in the sidebar for per-suite detail.