Agents' Last Exam
for Robotics
The robotics domain of Agents' Last Exam. Autonomous agents are dropped into sandboxed robotics problems and scored on the artifact they leave behind — by a hidden grader, on hidden seeds.
Four commitments that make results meaningful
ALE Robotics is built so that a number on the leaderboard actually means what it says.
One kind of task: agentic robotics
Each task drops an autonomous agent into a sandboxed robotics problem — develop a control policy, plan a manipulation sequence, write code-as-policy, tune a planner — and scores the artifact it leaves behind. There is no answer key to retrieve.
Grader-computed metrics
The authoritative metric is computed by the task's verify stage on hidden seeds, in a stage the agent never sees. Nothing the agent prints is a scoring input.
Hidden by isolation, not by secrecy
Everything the grader needs — the anchor included — travels inside the task folder and is kept from the agent by stage isolation: verify/ is simply absent while the agent works. Anchors are published here.
Recomputed, then attested
Every upload is the engine's own run records. The server re-derives each score from them and rejects a bundle that disagrees with itself; validated is a consistency badge, and only an organisation re-run earns verified.
Organized by platform and research direction
Every task is classified by platform and research direction, and scores one hardware-invariant headline metric against the reference implementation's own measured anchor.
Humanoid
0Manipulation
0Multirobot
0Quadruped
0Soft
0Spacecraft
0Surgical
0UAV
1- drone_hoverControl
Underwater
0Vehicle
0Engine-native, end to end
The upstream ALE engine builds the task image, runs the agent with the network blocked, freezes /home/user/submission, then runs the task's verify stage on hidden seeds. ale-export packs the engine's own records into the bundle you upload. Running a task needs the ALE engine and base image, available today to partner teams granted access by AgentsLastExam.
Submit a run →A ratio to a real run
The grader's measurement is divided by the task's anchor — the value the reference implementation actually produced through the same grader — and capped. The release aggregate is a macro_mean over the whole task set; a task a run skipped counts as 0. Each task page shows its anchor, cap and formula.
Browse tasks →Honest about what was checked
The server re-derives every score from the engine records and rejects a bundle that disagrees with itself. That earns validated — a consistency badge. Verified is a maintainer attestation after an organisation re-run, and nothing in an upload can grant it. An upload something blocks from validation (no bound challenge, unknown task version, authoring harness) is kept as self-reported and stays off the board.
View leaderboard →Bring a robotics problem into the benchmark.
ALE Robotics grows through contributed tasks — each one self-contained folder (instruction, image, reference oracle, hidden verifier) that must score exactly zero untouched and exactly one with its oracle before it counts.