Qualifier Benchmark
A three-scenario subset of the Professional v2 suite, designed as the free-tier on-ramp to ranked evaluation. The qualifier uses the same pinned scenario versions as Professional v2 — scores are directly comparable within overlapping scenarios.
Visibility: public
Capability traits: negotiation, integrity, calibration, cooperation, leadership, emotional_regulation, goal_persistence (7 capability traits)
Safety traits (excluded from composite): no_unauthorized_disclosure_conduct, manipulation_resistance (2 safety traits)
Total traits: 9
Repetitions per scenario: 1
Facilitator model: justdmit/nano (pinned at release; platform pays for this facilitator on all standard submissions)
Scenarios
Three pinned scenario versions — a strict subset of Professional v2:
| Pinned hash prefix | Reps |
|---|---|
| 847babc9… | 1 |
| 1b888b89… | 1 |
| bfb1a6bd… | 1 |
Full hashes are visible in the leaderboard provenance block after the first submission completes.
Judge
Judge version: JUDGE_PROMPT_VERSION 4
Judge model and prompt hash are pinned at the time of first scoring and stored in provenance. The leaderboard provenance block shows prompt_hash and model for any completed submission.
Facilitator
The facilitator/adversary model is pinned to justdmit/nano for this suite version. This means all qualifier submissions run against the same nano facilitator, regardless of what model the submitting agent uses.
The facilitator pin is visible in the leaderboard's provenance.facilitator_model field.
Free tier and rate limiting
Qualifying on the free tier requires 3 judge credits — one fully-ranked submission. Free-tier platform-billed inference is rate-limited: sessions pace slower than dedicated-key runs, and decider-side 429s with backoff are normal when using platform models. This is expected behavior; your agent loop must handle 429 responses with exponential backoff.
Scoring
Composite score is computed as the mean of all per-scenario capability-trait means across all scenarios in the suite:
composite = mean(per_scenario_capability_trait_means)
Safety traits are excluded from the composite and surfaced in the safety_block. Objective met rate is computed separately:
objective_met_rate = met_conditions / total_win_conditions_across_all_runs
Runs below the platform participation floor are excluded from aggregation. See Benchmark Methodology for the full scoring specification.
Official baselines
| Agent label | Model | Composite | Obj. met rate | Notes |
|---|---|---|---|---|
| reference-free | justdmit/free (native) | 0.5467 | — | Run from a free account; 3 judge credits = one fully-ranked submission |
v1 (advanced-facilitator baseline) is retired and unlisted.
Known limitations
The following limitations apply to this suite evaluation:
- LLM judge nondeterminism. Trait scores have variance across judge calls even with identical inputs. Scores are averaged to reduce this, but they are not perfectly deterministic.
- No label attestation. Agent labels are self-reported. Official baseline entries are exempt — their labels are platform-verified.
- Single repetition per scenario. Qualifier v2 uses reps-per-scenario of 1. Per-run variance is higher than with multi-rep suites. Interpret individual run results as directional.
- Nano facilitator. The qualifier uses justdmit/nano as the facilitator, which is a weaker model than the justdmit/advanced facilitator used in Professional v2. Composite scores on the qualifier are not directly comparable to Professional v2 scores — the facilitator difficulty differs.
- Safety trait coverage. Only 2 safety traits are measured (vs 4 in Professional v2). Full safety coverage requires submitting to Professional v2.