Contributing

ALE Robotics is an open-set robotics benchmark for autonomous coding agents. There are two contribution paths, both submitted through the guided form on Contribute: add a task (on an existing evaluator) and add an evaluator (a new scored environment — an interactive simulator OR a dataset). An approved contribution is opened as a pull request into the benchmark repository for you — you never need repository access. Read this alongside Task authoring and Evaluator authoring.

The website is not a benchmark runner. Task metadata, paper targets, versions, and budgets are not hand-entered here — they are synced from a benchmark release manifest. In v0.5 those paper-derived fields are not published and render as "Not published" / "TBD".

The two-image model (applies to both paths)

Every task ships two Dockerfiles, and the split is what keeps the benchmark honest:

• Base env-only image (sandbox/Dockerfile → ale-task-<task>) — the image the agent

runs in. It carries the simulator/environment + language deps + the gRPC client, and zero of the paper's method. This is agent-visible.

• Example-implementation image (sandbox/impl.Dockerfile → ale-task-<task>-impl,

built FROM the base) — adds the official/example method. It is hidden from the agent and used only by the reference self-test. An evaluator contribution ships the same pair for its accompanying task.

The method must never enter a shipped layer of the base image (multi-stage whitelist COPY, or purge .git in the same RUN as the clone). The form collects both Dockerfiles; the server pre-check enforces that the impl's final FROM is the base, not an evaluator.

Path 1 — Add a task

A task pairs a robotics paper (or an open-ended goal) with an existing evaluator and its evaluator-computed domain metric. The agent works OFFLINE (no network) and either reproduces the paper's method (reproduction genre) or optimizes the headline metric (open genre).

• Add the task to the registry and make it validate against

packages/contracts/schemas/task-registry.schema.json.

• Declare platform, direction, evaluator, mode

(interactive / batch), metric_name, and metric_direction (higher_is_better / lower_is_better).

• Provide the base + example-impl Dockerfile pair, a hidden reference solution + eval

protocol (kept out of the sandbox and the submission bundle), and — for reproduction tasks — a paper PDF link (a URL, e.g. arXiv / OpenReview / PMLR; fetched on each runner's machine so we don't redistribute the PDF).

• Do not fabricate paper targets. They arrive from the release manifest.
• Reproduction-genre tasks must pass the fail-closed Faithfulness gate

(unfaithful => final score 0), so state the method claim precisely.

Path 2 — Add an evaluator

An evaluator is a new scored environment — an interactive simulator OR a dataset — computing an authoritative domain metric, exposing an AleEnv gRPC service with the 8 unary RPCs (CreateEnv, GetSpec, Reset, Step, StartEvalSession, GetEvalResult, SubmitBatchEval, GetBatchEvalStatus). See gym-over-gRPC. (We say "evaluator", not "simulator", precisely because a dataset-backed evaluator is a first-class kind.)

• The authoritative metric is always computed by the evaluator, never self-reported by

the agent.

• Official eval runs on hidden seeds that are never sent to the agent; client-supplied

Reset seeds are ignored during eval sessions. Store seeds only as the commitment SHA256(seed || run_id || evaluator-secret).

• Support the mode: interactive (live Reset/Step, then StartEvalSession) or

batch/dataset (agent writes to the shared /submission volume, SubmitBatchEval runs the native pipeline). See Interactive tasks and Batch tasks.

• Ship the evaluator alongside an accompanying task's env + example-impl Dockerfile pair

(and, for a dataset evaluator, the dataset data-image Dockerfile).

• Emit results your metric implements per Scoring: higher/lower-is-better

direction, and any zero-target safety metric (e.g. collisions) with its limit.

Adding a new agent integration (a coding-agent deployer) is a benchmark-repo change, not a website contribution path — see Agent integration.

Quality standards

• JSON artifacts validate against the schemas in packages/contracts/schemas/.
• Runs are deterministic given a seed and fully reproducible offline.
• Trajectory hash chains and the integrity root verify.
• Never force a score into a percentage; single-task scores may exceed 1.0 (cap 1.5).

Safety checks

• No network in the agent sandbox.
• A declared safety metric that is missing is treated as worst-case (penalty factor 0),

so wire safety metrics through end to end.

• Bundles must never contain hidden seeds, reference solutions, API keys, private prompts,

or unredacted trajectories. See Security model.

• Successful decryption does not grant a verified status; contributors cannot

self-elevate to audited / officially_verified (see Verification levels).

PR checklist

• [ ] Scoped to one path (task or evaluator).
• [ ] Both Dockerfiles provided (env-only base + example-impl); the method never enters

a shipped layer of the agent-visible base image.

• [ ] Reproduction tasks include a paper PDF link (URL, not the file).
• [ ] Schemas validate for registry changes.
• [ ] Metric is evaluator-computed; direction and safety limits declared.
• [ ] Hidden seeds stored only as commitments; no secrets in code, images, logs, or bundles.
• [ ] Offline determinism and reproduction verified.
• [ ] Docs cross-links updated; no fabricated paper targets/versions/budgets.

Templates & Code of Conduct

Use the JSON Schemas in packages/contracts/schemas/ and the registry seed at content/tasks/registry.json as templates for task and artifact structure. All participants must follow the project's Code of Conduct — see CODE_OF_CONDUCT.md in the repository root. For running and self-hosting the service, see Deployment.