Contribute
Grow the benchmark
ALE Robotics grows through two contribution paths, each with clear quality and safety gates. Everything is submitted through a guided form. Start with the path that matches what you want to add.
Add a task
Add a task on an existing evaluator
Bring a robotics paper (or an open-ended goal) into the benchmark on an evaluator that already exists.
- 01Define task data (task.yaml): metric name, direction, paper target, scoring cap and penalties, evaluation mode.
- 02Add paper.md / brief.md, a paper PDF link, and the base env-only sandbox Dockerfile the agent runs in.
- 03Add the example-implementation Dockerfile (built FROM the base) — the official method, kept out of the agent's image.
- 04Provide a hidden reference solution + eval protocol kept out of the sandbox and the submission bundle.
Add an evaluator
Add a new evaluator (simulator or dataset)
Add a new scored environment that computes an authoritative domain metric — an interactive simulator or a dataset-backed evaluator.
- 01Implement an EnvAdapter: reset/step/episode_success/metrics for interactive, or run_batch_eval for batch/dataset.
- 02Provide the paired env + example-impl Dockerfiles and a metrics README documenting the authoritative metric.
- 03Support interactive and/or batch modes over the gym-over-gRPC AleEnv service.
- 04Keep the shared gym library py3.8-compatible and dependency-minimal.
Quality & safety standards
- Authoritative metrics are always evaluator-computed, never agent-reported.
- Hidden seeds and reference solutions never leave the trusted boundary.
- The agent sandbox stays offline; the deny-all network policy is never weakened.
- Reproduction-genre tasks keep the fail-closed faithfulness gate.
- Every run produces a signed, hash-chained, auditable trajectory.
PR checklist
- The task/evaluator follows the relevant authoring doc and templates.
- Both Dockerfiles are provided (env-only base + example-impl); no method leaks into the agent's base image.
- No secrets, credentials, hidden seeds, or reference solutions are committed.
- Interactive and/or batch behavior is documented and tested.
- Metric direction, paper anchor, and scoring config are explicit.
- Safety review: sandbox stays offline; no new egress hosts introduced.
Ready to contribute?
Submit through the guided form above — an approved contribution is opened as a pull request against the benchmark repository following the checklist. All contributors agree to the Code of Conduct.