drone_hover

Task goal

The ALE engine drops an autonomous agent into a sandboxed robotics problem built from the task's own image, with the network blocked. The agent reads instruction.md — the only thing it is told — develops its solution, and leaves the artifact under /home/user/submission. Scoring happens afterwards: the task's verify stage, absent from the sandbox while the agent works, runs the artifact on hidden seeds and computes the authoritative Mean episode reward. Nothing the agent prints is ever a scoring input.

Scoring

The verifier compares the measured Mean episode reward (higher is better) against the task's anchor — the value the reference implementation in oracle/ actually produced through the same grader, never a figure copied from a paper. The capped ratio is the leaderboard number; the saturating reward is the engine's training signal and is only checked for consistency.

ratio  = clamp(measured / anchor, 0, 1.5)
reward = clamp(ratio / 0.825, 0, 1)

A ratio of 1.0 means the agent matched the reference; beating it scores above 1.0, capped at 1.5. The release aggregate is a macro_mean over the whole task set, and a task the submission did not run counts as 0.

Anchor policy. Everything the grader needs, the anchor included, travels inside the task folder under verify/ and is hidden from the agent by stage isolation — the verify stage is simply absent while the agent works — not by keeping the number secret. That is why the anchor is published here.

Run & submit

Runs are made by the upstream ALE engine from the task folder, exported with ale-export, and uploaded here. The server re-derives every score from the engine's own records and rejects a bundle that disagrees with itself. Running a task needs the ALE engine (upstream repository private, not on PyPI) and the ALE base image; today both are available to partner teams granted access by AgentsLastExam — the task mirror is reference material for everyone else.

ale run tasks/control/drone_hover --agent claude-code --runs-dir runs
ale-export runs/<run_id> --out drone_hover.ale-engine-run.tar.zst.age \
  --challenge-file challenge.json --public-key "$SUBMISSION_PUBLIC_KEY"

A run whose every engine-record check passes under a bound challenge, and whose every score recomputes from the registry anchor, is validated — a consistency badge, not proof of authenticity. Verified is a maintainer attestation after an organisation re-run. Anything a check blocks (no bound challenge, unknown task version, authoring harness) is kept as self-reported and stays off the leaderboard.