Scoring & trust

Submission bundle

The <run-id>.ale-submission.tar.zst.age format, its allowlisted files, how to obtain the encryption key, and the exact server-side validation order.

A submission bundle is the single, encrypted artifact a local runner produces after a run. It carries everything the website needs to independently re-check a result — and nothing it should not. This page specifies the wire format, the allowlisted file set, and the exact order in which the website processes an upload.

Format

The file name is <run-id>.ale-submission.tar.zst.age. The extensions read outside-in and describe how the runner builds it, inside-out:

age( zstd( tar( allowlisted files ) ) )

tar — the allowlisted files (below) are packed into an uncompressed tar archive.
zstd — the tar is compressed with Zstandard.
age — the compressed archive is encrypted with age (X25519) to the website's

public key, using the audited pyrage library.

Encryption is never hand-rolled. The runner holds only the website's public key; the matching private key lives on the website and is never committed to a repo, baked into an image, written to logs, or sent to a browser. See the Security model.

Allowlisted files

The bundle contains exactly these files. Anything else in the archive is ignored on extraction. Each JSON file is validated against a schema in packages/contracts/schemas.

File	Contents
`manifest.json`	Run identity and binding: `submission_id`, `challenge_token`, `nonce`, `benchmark_version`, `required_task_set_hash`, `agent`, `model`, `container_images` (name + digest), `resource_limits`, `started_at`/`finished_at`, and the `integrity_root`.
`events.jsonl.zst`	The zstd-compressed, hash-chained trajectory event stream (schema `ale-trajectory-event/v1`). Validated via its hash chain, not as JSON. See Trajectory format.
`task_results.json`	Per-task evaluator results: `task_slug`, `measured_metric`, `claimed_normalized_score`, `num_trials`, `stderr`, and an `evaluator_result_hash`.
`score_breakdown.json`	The claimed aggregate: `claimed_aggregate_score`, `aggregation_policy_type`, and a `per_task` breakdown. See Scoring.
`faithfulness.json`	Reproduction-genre judge report: `applicable`, `passed`, and per-check detail. See the Faithfulness gate.
`network_summary.json`	Offline-sandbox attestation: `sandbox_offline` plus any recorded `anomalies`.
`artifacts_manifest.json`	A list of run artifacts by `kind`, `path`, `sha256`, and `size_bytes` (a manifest, not the artifact bytes themselves).
`integrity.json`	Integrity anchors: `integrity_root`, `event_chain_head`, `event_count`, and the hashing `algorithm`.

All JSON files carry schema_version: "ale-robotics-submission/v1" (the trajectory stream carries ale-trajectory-event/v1).

What is excluded

Bundles never contain hidden seeds (stored only as a commitment; see Trajectory format), hidden reference solutions, API keys, private prompts, hidden model reasoning / chain-of-thought, or unredacted full trajectories. The domain metric is always the evaluator's, never self-reported.

Obtaining the public key

Fetch the current key from the API. This is public and requires no authentication:

GET /api/v1/submission-key

{
  "algorithm": "age-x25519",
  "public_key": "age1...",
  "key_id": "...",
  "schema_versions": ["ale-robotics-submission/v1"]
}

The runner encrypts to public_key. Before uploading you must also obtain a one-time challenge via POST /api/v1/challenges (auth required) and record its challenge_token and nonce in manifest.json. Upload the ciphertext to POST /api/v1/submissions.

Server-side processing order

The website is not a benchmark runner — users run locally, then upload. On receiving a bundle the server runs a fixed, fail-closed pipeline. Each step must pass before the next:

Receive ciphertext, compute its SHA-256.
Decrypt with the private key.
Enforce limits on decompressed size and file count (zstd bomb-safe cap).
Safe tar inspect — reject path traversal, symlinks, hardlinks, devices, FIFOs, and

absolute paths.

Extract allowlisted files only.
Validate JSON schemas for every present JSON file.
Verify the challenge token (server-signed, one-time, unexpired) and its nonce,

bound to this submission and user.

Verify hash chains over events.jsonl.zst against integrity.json.
Verify the integrity root — manifest.integrity_root must match integrity.json.
Verify benchmark/task versions + coverage against the release manifest.
Recompute scores from the evaluator results and compare to the claimed values.
Audit summary, then persist metadata.
Delete the plaintext and ciphertext temp files.

Successful decryption does not grant a verified status. The recomputation and gate results determine the verification level; the backend refuses any attempt to self-elevate via an upload field. Bundle retention defaults to KEEP_SUBMISSION_BUNDLES=false, so plaintext and ciphertext are removed after processing.