Contributing & ops

Evaluator config manifest

How to author an evaluator's env_config_schema — the extensible capability manifest that declares which platforms, robots, scenarios, and metrics a task may select on your shared evaluator. Covers the field-spec grammar, when-gating, per-parent value maps, the metric menu, and the live syntax checker.

An evaluator is one image that serves many scenarios. A single robomimic image ships every RoboSuite arm; a single MuJoCo Playground image ships Go1, Spot, G1, H1, and more. The env_config manifest (env_config_schema) is how you declare, once, the full menu of things a task may select on your evaluator — and it is deliberately extensible: you freely add new keys and branches that wire to your simulator's features, and they flow straight into the task-submission form.

This is the "define once, select per task" contract. The task never rebuilds your image; it picks a point in your menu at runtime via CreateEnv(env_config).

The manifest you write here is mirrored by the benchmark adapter's env_config_schema class attribute (ale_gym.adapter). The Contribute form's manifest field, the intake pre-check, and the benchmark CI all validate it with the same static validator, so a manifest the form shows green is exactly what merge accepts.

The shape

env_config_schema is a JSON object mapping each selectable key → a field-spec:

{
  "robot":    {"type": "enum", "values": ["Panda", "Sawyer", "IIWA"], "default": "Panda", "desc": "RoboSuite arm"},
  "obs_mode": {"type": "enum", "values": ["low_dim", "image"], "default": "low_dim", "desc": "observation set sent over the wire"},
  "dataset_path": {"type": "str", "default": "/data/square/ph/low_dim.hdf5", "desc": "dataset hdf5 whose env_meta defines the scene"}
}

A field-spec has exactly these attributes (any other key is a typo and is rejected):

Attribute	Required	Meaning
`desc`	yes	Human-readable one-liner shown next to the control in the task form.
`type`	no (default `enum`)	`enum` \	`multi` \	`str` \	`int` \	`float`.
`values`	for `enum`/`multi`	The legal menu — a list, or a per-parent map (see below). Must be absent/null for `str`/`int`/`float`.
`range`	numeric only	`[min, max]` for `int`/`float`; `min <= max`; the default must fall inside it.
`default`	recommended	The value used when a task omits the key. Must be legal (a member of `values`; a list for `multi`; inside `range`).
`when`	no	Gates this field on a parent's value — see when-gating.

Field types

enum — pick exactly one of values (e.g. which arm, which scene).
multi — pick a subset of values; the default must be a list (e.g. a sensor suite

["rgb", "depth"], a set of scenario worlds).

str / int / float — a free/opaque value with no membership check (e.g. a dataset

path, a horizon, a randomization intensity). Do not give these a values list; use range to bound a numeric.

Conditional fields with `when`

Heterogeneous simulators (Isaac Gym, AirSim, CARLA) have a choice tree: pick an embodiment, then options that only make sense for it. Express that with when, which makes a field appear only when a parent key holds a given value (or one of a list):

{
  "embodiment": {"type": "enum", "values": ["arm", "quadruped", "humanoid"], "default": "arm", "desc": "robot class"},
  "arm_model":  {"type": "enum", "values": ["Franka", "UR5"], "default": "Franka", "desc": "arm model", "when": {"embodiment": "arm"}},
  "gait":       {"type": "enum", "values": ["trot", "walk"], "default": "trot", "desc": "gait", "when": {"embodiment": ["quadruped", "humanoid"]}}
}

The task form reveals arm_model only when embodiment=arm, and prunes it if the author later switches to quadruped — so an illegal cross-branch combination can never be submitted. Rules the validator enforces:

the parent key must exist and be an enum (you cannot gate on a multi or a free

field — there is no single value to match);

each when value must be a legal value of that parent (otherwise the field would

never appear — a silent dead control);

the when graph must be acyclic (no A when B, B when A).

Per-parent value maps

When a child's menu depends on the parent's value, give values as a map keyed by the parent value instead of a flat list:

{
  "suite":   {"type": "enum", "values": ["libero_10", "libero_90"], "default": "libero_10", "desc": "benchmark suite"},
  "task_id": {"type": "enum",
              "values": {"libero_10": [0, 1, 2], "libero_90": [0, 1, 2, 3, 4]},
              "default": 0, "desc": "task within the suite",
              "when": {"suite": ["libero_10", "libero_90"]}}
}

A per-parent map requires a when naming that same parent, and each branch key must be a legal value of the parent (a branch the parent can never take is a dead branch).

Papers report different headline numbers on the same simulator (success rate, SPL, collision rate, ATE…). Declare all the metrics your evaluator can compute in a metric_menu, so a task can select which one it is scored on via env_config.metric:

{
  "metric_name": "success_rate",
  "metric_direction": "higher",
  "metric_menu": {
    "success_rate":   {"direction": "higher", "desc": "fraction of eval episodes solved"},
    "spl":            {"direction": "higher", "desc": "success weighted by path length"},
    "collision_rate": {"direction": "lower",  "desc": "collisions per episode (safety)"}
  }
}

metric_name is your default headline metric (always selectable even if you omit it from the menu). Every menu entry needs a direction (higher or lower) and a desc.

Extensibility — add whatever your sim needs

The manifest is open by design. If your simulator exposes a knob none of the examples cover — a weather/sensor randomization range, a friction coefficient, a curriculum stage — just add a new key with the right type (and range if numeric). It appears in the task form automatically; no website change is needed. The only hard constraints:

keys must not start with _ and must not be the literal metric — those are reserved

for the harness (_eval_seeds, the metric-selection channel);

an empty manifest {} is legal (your evaluator then accepts any env_config — useful for

a single-scenario sim), but declaring the menu is strongly preferred so task authors and maintainers can see what is legal.

Later, if a task author finds your evaluator is missing a metric or a scenario they need, they can open an Edit-Evaluator PR that extends this manifest — the same review flow, reusing your evaluator's files.

Validate before you submit

The Contribute form runs a live syntax check on the manifest as you type: it calls the same static validator used by the intake pre-check and the benchmark CI lint (scripts/lint_env_config_schemas.py), and shows either the exact errors or a green "valid" line. Common errors it catches:

an attribute typo (defualt, valeus) — rejected as an unknown attribute;
an enum/multi field missing values, or a free field that wrongly declares them;
a range with min > max, or a default outside it;
a when that references a non-existent parent, an illegal parent value, a non-enum

parent, or forms a cycle;

a metric_menu entry with a bad direction or missing desc;
a reserved key collision.

Because the browser check, the server pre-check, and CI share one validator, there are no surprises at merge time.

Checklist

[ ] Every selectable platform/robot/scenario your image supports has a manifest key.
[ ] Each field-spec has a desc; enum/multi have values; numerics use range.
[ ] Heterogeneous choices use when (and per-parent values maps where the menu varies).
[ ] metric_menu lists every metric a task may be scored on, each with a direction.
[ ] The live syntax checker shows the manifest as valid before you submit.

See Evaluator authoring for the adapter/Dockerfile side and Task authoring for how a task selects from this menu.