Verification levels
The five verification levels, the exact checks that gate each, who can grant them, and why decryption never equals verified.
Every leaderboard entry carries a verification level. The level is a claim about how much the website was able to confirm — not a score, and not a quality judgment of the agent. The design goal is that levels are honest by construction: a level can only ever be as strong as the checks that produced it, and no upload can promote itself.
This page defines each level precisely, states the exact checks that gate it, and names who can grant it. For the mechanics of the checks themselves, see Submission bundle, Scoring, Trajectory format, and the Security model.
The five levels
- self_reported — Metadata was submitted, but full automatic validation has not completed (or was not attempted). The numbers are the submitter's claim only. This is the floor: any accepted submission starts here.
- validated — The website received the ciphertext and passed the full automatic pipeline: the bundle decrypts, all allowlisted JSON files pass schema validation, hash chains are intact, the integrity root verifies, benchmark/task versions, resource limits, and task coverage are valid, and recomputed scores match the claimed scores (recomputed from the evaluator results in the bundle, never from agent-reported numbers). Validation confirms internal consistency. It does not prove the local runner was unmodified.
- audited — The trajectory, network summary, artifacts, and configuration were reviewed — manually, automatically, or both — beyond the automatic consistency checks. An audit inspects what actually happened during the run.
- officially_verified — The run originated from a trusted execution path: a benchmark-team-controlled runner, an approved trusted third-party runner, a future hardware-attested runner, or an official reproduction/rerun by the benchmark team. This is the only level that speaks to runner provenance.
- revoked — A previously granted level was withdrawn after discovering cheating, misconfiguration, or error. Revocation can be applied to an entry at any level.
Level → checks → who can grant
| Level | Gating checks | Who can grant |
|---|---|---|
| self_reported | Submission accepted; metadata recorded. Full automatic validation not completed. | Automatic, on accepted upload |
| validated | Decrypt + schema valid + hash chains valid + integrity root valid + version/resource/task-coverage valid + score recomputation matches | Automatic pipeline (website backend) |
| audited | Trajectory + network + artifact + config reviewed (manual or automated), on top of validated | Benchmark team / approved reviewers |
| officially_verified | Provenance from a benchmark-team runner, approved third-party runner, hardware-attested runner, or official reproduction/rerun | Benchmark team only |
| revoked | Cheating, misconfiguration, or error discovered after the fact | Benchmark team / admins |
Levels are progressive in trust but not strictly nested in workflow: officially_verified runs are trusted by provenance and still pass the same automatic checks; revoked is terminal until re-adjudicated.
The hard rules
Two invariants are enforced by the backend and cannot be worked around from the client:
- Successful decryption does not grant verified. Decryption is only the first step of the server-side pipeline — it must be followed by size/file-count limits, safe tar inspection, allowlisted extraction, schema validation, challenge-token and nonce checks, hash-chain and integrity-root verification, version/coverage checks, and score recomputation. A bundle that decrypts but fails any downstream check does not reach
validated. - Users cannot self-elevate. The backend refuses to accept
auditedorofficially_verifiedfrom any upload field. Those levels are assigned only by the benchmark team (or approved reviewers) through server-side adjudication, never by anything in the submitted bundle or multipart form.
This is why the authoritative domain metric is always computed by the evaluator, never self-reported, and why a modified local runner can at best reach validated — the website can confirm a bundle is internally consistent, but only a trusted execution path can reach officially_verified.
Filtering by level
GET /api/v1/leaderboard accepts a verification_level filter (alongside version, agent, model, and platform), so consumers can restrict a view to, for example, only officially_verified entries. Treat self_reported and validated as unproven with respect to runner integrity unless corroborated by an audited or officially_verified status.