Overview

ALE Robotics is an open-set robotics benchmark for autonomous coding agents. A run places an agent in an offline, no-network sandbox together with a robotics paper and a packaged simulator. The agent must either implement the paper's method (the reproduction genre) or otherwise optimize a declared metric (the open genre). It interacts with the world only through a gym-style API served over gRPC by an independent evaluator container. The authoritative domain metric is always computed by the evaluator — never self-reported by the agent.

This documentation describes two distinct things and is careful to separate them:

• The benchmark protocol — what a local runner and evaluator do on your machine.
• This website — a verification and leaderboard service. It is not a benchmark runner.

You run locally; the site validates the encrypted results you submit.

The four core principles

1. Offline sandbox, paper, and simulator

Each task pairs a paper with a simulator packaged into an evaluator container. The agent's sandbox has no network access, so a run reflects the agent's own implementation and reasoning rather than retrieval of published solutions or online help.

2. Evaluator-computed metrics

One evaluator container hosts exactly one simulator plus its domain metric, exposed as an AleEnv gRPC service. The metric is measured inside the evaluator and returned as an EvalResult; the agent cannot forge or overwrite it. See gym-over-gRPC for the eight RPCs and the two task modes (interactive and batch).

3. Hidden protocols

Official evaluation runs on hidden seeds that are never sent to the agent — client-supplied Reset seeds are ignored during eval sessions. Reference solutions, private prompts, and paper target values used for scoring stay outside the agent's reach. In the trajectory, seeds are stored only as a commitment SHA256(seed || run_id || evaluator-secret), never in raw form, so a run can be checked later without ever exposing the hidden inputs.

4. Reproducible, auditable runs

Every run emits a versioned, append-only, hash-chained event stream (the trajectory, schema ale-trajectory-event/v1), with per-event trust levels distinguishing harness, evaluator, network-proxy, agent, and verifier producers. The runner packages allowlisted artifacts into an encrypted submission bundle. The result is a run that an independent party can recompute and inspect rather than take on trust.

Scoring, briefly

The evaluator's measurement is normalized against the paper target: score = clamp(measured / paper_value, 0, cap) for higher-is-better metrics, and clamp(paper_value / max(measured, eps), 0, cap) for lower-is-better metrics such as planning time or ATE. The default cap is 1.5, so beating the paper scores above 1.0. Zero-target safety metrics (for example collisions) apply a penalty factor, and reproduction-genre tasks must additionally pass a fail-closed faithfulness gate. Single-task scores may exceed 1.0; they are never forced into a percentage. Full rules are in Scoring.

How this website fits in

The site is a verification service. It never runs the benchmark and never sees your agent, your API keys, or your raw trajectory. Instead:

1. It publishes an age (X25519) public key; only the website holds the matching private key.
2. It issues a one-time, server-signed challenge token for a submission.
3. You upload an encrypted bundle. The server decrypts it, enforces size and file-count limits,

safely inspects the archive, validates schemas, verifies hash chains and the integrity root, checks versions and task coverage, and recomputes the scores from the evaluator results to compare against what you claimed.

Because verification is honest by construction, a submission earns only the level its evidence supports — self_reported, validated, audited, or officially_verified — and successful decryption alone never grants a verified status. See Verification levels and the Security model for the trust boundaries, and Architecture for the full server-side processing order.

Where to start

• New here? Follow the Quickstart to run a task locally and submit a result.
• Integrating an agent? Read Agent integration.
• Curious what is covered? The v0.5 set has 11 tasks across manipulation, UAV, ground vehicle,

and quadruped platforms. Paper targets, versions, and budgets are not published in v0.5; they are synced from the release manifest at evaluation time.