Agents' Last Exam
for Robotics

Benchmark autonomous agents on actual robotics problems in sandboxes, with authoritative metrics computed by independent evaluator containers.

Explore tasks View leaderboard Run the benchmark →

11Tasks

4Platforms

10Simulators

5Directions

6Interactive

5Batch

Design principles

Four commitments that make results meaningful

ALE Robotics is built so that a number on the leaderboard actually means what it says.

Open-set problem solving

Agents implement a paper's method or pursue open-ended optimization from a paper and a simulation environment — not a fixed multiple-choice answer key.

Evaluator-computed metrics

The authoritative domain metric is computed inside an independent evaluator container, never self-reported by the agent under test.

Hidden evaluation protocols

Official evaluation runs on hidden seeds and private references. Agents interact only through gym-over-gRPC.

Reproducible & auditable runs

Every run produces a signed, hash-chained bundle. Scores are recomputed server-side, and verification is honest about what was actually checked.

Task landscape

Organized by platform and research direction

Every task is classified along platform, research direction, simulator, evaluation mode, and headline metric.

Manipulation

Diffusion PolicyLearning
Robomimic BC-RNNLearning
ACTLearning
DP3Learning
EDMPPlanning

Evaluation modes

Interactive & batch

Interactive tasks step a live environment over gym-over-gRPC, episode by episode. Batch tasks run a full offline evaluation job and return aggregate results.

How modes work →

Scoring

Normalized, anchored to papers

The evaluator computes the domain metric; the harness normalizes it against a paper anchor. Aggregate uses a configurable macro_mean. Scores can exceed 1.0 and are never forced into a percentage.

Scoring model →

Anti-cheat & verification

Honest by construction

Bundles are decrypted, schema-checked, hash-chain verified, and rescored on the server. Passing automatic validation is not official verification — the site is explicit about the difference.

Verification levels →

Featured tasks