Task authoring

This page covers the "add a paper" contribution path: attaching a new task to a simulator that already has a working evaluator. If you need to package a new simulator or domain metric, start with Evaluator authoring first — a task is only meaningful once its evaluator can compute the authoritative metric on hidden seeds.

A task pins one paper to one evaluator, declares how the evaluator's metric maps to a normalized score, and ships the two things the agent reads inside the offline sandbox: the paper and a short brief. Everything the agent must not see — hidden seeds and the reference solution — lives outside the sandbox.

Task layout

A task is a directory keyed by its slug. On an existing simulator you author these files; the evaluator image and its AleEnv gRPC service are reused unchanged.

tasks/<slug>/
  task.yaml          # metric contract, mode, scoring cap/penalties, genre
  paper.md           # the paper (or a faithful, citable rendering of it)
  brief.md           # task-specific instructions shown to the agent
  sandbox/
    Dockerfile       # offline agent workspace built on the shared base image
  reference/         # HIDDEN — never mounted into the sandbox, never bundled
    solution/ ...

task.yaml

task.yaml is the machine-readable contract. It maps directly onto the tasks table (slug, platform, direction, simulator, mode, metric_name, metric_direction, paper_value, genre) and is what the website ingests from the release manifest.

slug: my_paper_task
title: "My Paper (RSS 2024)"
platform: manipulation          # manipulation | uav | vehicle | quadruped
direction: learning             # learning | planning | control | navigation | locomotion
simulator: robosuite+robomimic  # must match an existing evaluator
genre: reproduction             # reproduction | open

mode: interactive               # interactive | batch | read_from_task_config

metric:
  name: success_rate
  direction: higher_is_better    # higher_is_better | lower_is_better
  paper_value: <from paper>      # the metric anchor (see below)
  score_cap: 1.5                 # default 1.5; beating the paper scores >1.0, capped

penalties:                       # optional; zero-target safety metrics
  - metric: collision_rate
    limit: 0.05                  # factor = 1 - min(1, rate/limit)

mode selects the protocol surface. interactive drives episodes live via Reset/Step, then StartEvalSession runs hidden-seed scoring (Interactive tasks); batch writes a submission to the shared /submission volume and calls SubmitBatchEval (Batch tasks); read_from_task_config fixes the eval procedure entirely in config. The mode never changes who computes the metric — the evaluator always does.

Scoring fields feed Scoring. For higher-is-better, score = clamp(measured/paper_value, 0, cap); for lower-is-better (e.g. planning_time, ATE), score = clamp(paper_value/max(measured, eps), 0, cap). The default cap is 1.5. Each penalties entry declares a zero-target safety metric and multiplies the score by factor = 1 - min(1, rate/limit). Declare every safety metric you care about: a missing declared safety metric is treated as worst-case (factor 0), not skipped.

paper.md and brief.md

paper.md is the paper the agent must reproduce (reproduction genre) or the reference context for an open-genre optimization. brief.md is the task-specific instruction sheet — the metric being optimized, the interaction mode, budget constraints, and any allowed/ disallowed assets. Both are mounted read-only into the offline sandbox; keep them free of targets you do not want the agent to see. In v0.5, paper targets, versions, and budgets are not published — they are synced from the release manifest, so do not hard-code numbers into brief.md that would leak the anchor.

Sandbox Dockerfile

sandbox/Dockerfile builds the agent's offline workspace on the shared base image. It installs the simulator client stubs and paper dependencies, but has no network at run time and no evaluator exec access — the agent reaches the evaluator only through gym-over-gRPC (gym-over-gRPC protocol). Never COPY the reference/ tree or any seed material into this image.

The metric anchor

paper_value is the anchor the normalized score divides against. It is the single number most likely to distort results, so treat it as a reviewed artifact: cite the exact table/ figure in paper.md, note the reported setting, and record its provenance in the task metadata. The website reads the authoritative value from the release manifest; the tasks column is nullable and renders as "Not published" when a release withholds it (as in v0.5). The agent never receives the anchor.

Hidden reference

The reference/ directory holds a working reference implementation used offline by maintainers to validate the anchor and calibrate the Faithfulness gate rubric for reproduction-genre tasks. It is never mounted into the sandbox and never appears in a submission bundle — bundles carry no hidden reference solutions or hidden seeds. Keep it in a maintainer-only location outside the buildable task context.

Human review gates

Adding a paper passes through maintainer review before it reaches a release:

• Contract review — task.yaml parses, mode/metric/genre are consistent, and the

simulator matches an existing evaluator.

• Anchor sign-off — paper_value is traced to a cited source and reproduced against the

hidden reference within tolerance.

• Leak audit — paper.md, brief.md, and sandbox/Dockerfile disclose no targets,

seeds, or reference paths.

• Faithfulness calibration — for reproduction genre, the rubric is authored and dry-run

so the fail-closed judge (unfaithful => final = 0) behaves as intended.

Only after these gates does the task enter a release manifest and become discoverable on the site. See Contributing for the submission workflow.