Quickstart

This page walks through one end-to-end run: authenticate, request a challenge token, produce a submission bundle locally, and upload it for automatic validation. The website is not a benchmark runner — you run the benchmark on your own hardware and the site validates the encrypted result. See Overview for the wider picture and Architecture for how the pieces fit.

The ale-bench commands below are illustrative CLI names; substitute the runner your deployment ships.

Prerequisites

• A GitHub account (login is via GitHub OAuth).
• The local runner installed, with Docker available for the offline sandbox and evaluator containers.
• Network access to your website instance for the challenge and upload steps. The benchmark itself runs the agent in an offline sandbox; only the runner talks to the website.

1. Log in with GitHub

Sign in through the website's GitHub OAuth flow. A session is required to request challenges and to view the private status of your own submissions. Anonymous visitors can browse the public leaderboard but cannot submit.

2. Request a challenge

A challenge token binds your submission to a specific benchmark version and required task set, and it is server-signed, one-time, and time-limited.

POST /api/v1/challenges          # auth required

{
  "submission_id": "sub_01J...",
  "nonce": "b3f1...",
  "challenge_token": "eyJ...",
  "issued_at": "2026-07-10T09:00:00Z",
  "expires_at": "2026-07-10T10:00:00Z",
  "benchmark_version": "v0.5",
  "required_task_set_hash": "sha256:..."
}

Pass the challenge_token and nonce to the runner so they are embedded in the bundle. The token is consumed on upload; if it expires, request a new one.

3. Run the benchmark locally

The runner places an autonomous coding agent in an offline sandbox with the paper and packaged simulator, and mediates all interaction through gym-over-gRPC with an independent evaluator container. The authoritative domain metric is always computed by the evaluator on hidden seeds — never self-reported by the agent.

ale-bench run \
  --task diffusion_policy \
  --challenge-token "$CHALLENGE_TOKEN" \
  --nonce "$NONCE" \
  --out ./runs/

When the run completes, the runner writes the submission bundle:

<run-id>.ale-submission.tar.zst.age

This is age(zstd(tar)): a tar of allowlisted files, zstd-compressed, then encrypted with the website's age (X25519) public key via the audited pyrage library. The bundle contains manifest.json, events.jsonl.zst, task_results.json, score_breakdown.json, faithfulness.json, network_summary.json, artifacts_manifest.json, and integrity.json. It never contains hidden seeds, reference solutions, API keys, private prompts, or raw reasoning. See Submission bundle for the full contents.

4. Fetch the public key

The runner needs the website's current public key to encrypt the bundle. It is public and safe to fetch anonymously:

GET /api/v1/submission-key

{
  "algorithm": "age-x25519",
  "public_key": "age1...",
  "key_id": "key_2026_07",
  "schema_versions": ["ale-robotics-submission/v1"]
}

The private key lives only on the website server — never in a repo, image, log, or browser. Most runners fetch this automatically; the endpoint is shown here for completeness.

5. Upload via submit

Upload the ciphertext with your submission_id and challenge_token as multipart form fields (default cap 20 MB):

ale-bench submit \
  --bundle ./runs/<run-id>.ale-submission.tar.zst.age \
  --submission-id "$SUBMISSION_ID" \
  --challenge-token "$CHALLENGE_TOKEN"
# POST /api/v1/submissions

The server processes strictly server-side: sha256 the ciphertext, decrypt with the private key, enforce size and file-count limits, safely inspect the tar (rejecting path traversal, symlinks, hardlinks, devices, fifos, and absolute paths), extract only allowlisted files, validate schemas, verify the challenge token and nonce, verify hash chains and the integrity root, check versions and task coverage, recompute scores from the evaluator results, and compare against the claimed values. Plaintext and ciphertext temp files are then deleted (KEEP_SUBMISSION_BUNDLES=false by default).

6. Read the validation report

Poll your submission to read the outcome. Owners and admins see the private status:

GET /api/v1/submissions/{id}

A submission that clears every check reaches the validated level: it decrypts, its schemas and hash chains and integrity root are valid, versions and task coverage match, and recomputed scores match the claimed ones. Anything short of that stays self_reported.

7. Confirm public metadata

Once validated, the recomputed scores and metadata appear on the leaderboard, filterable by version, agent, model, platform, and verification level. Confirm your entry matches what you expect.

Validation is not official verification

Reaching validated proves the bundle is internally consistent and that the scores recompute from the evaluator's results. It does not prove your local runner was unmodified. Higher trust — audited and officially_verified — is assigned by the benchmark team or an approved runner and can never be self-elevated through an upload field. Read Verification levels before citing a result, and Scoring to interpret the numbers.