Evaluator authoring

An evaluator is the independent container that owns one simulator and computes one task's authoritative domain metric. The agent never scores itself: it interacts only through gym-over-gRPC, and the harness reads the result over its own connection. This page describes how to add a new simulator/evaluator. For wiring a paper onto an existing evaluator, see Task authoring.

What an evaluator is responsible for

One evaluator container = one simulator + one metric, exposing an AleEnv gRPC service with eight unary RPCs: CreateEnv, GetSpec, Reset, Step, StartEvalSession, GetEvalResult, SubmitBatchEval, GetBatchEvalStatus. You do not implement the gRPC plumbing by hand — ale_gym provides the service scaffold, message types, and adapter base classes. Your job is to implement an EnvAdapter and declare the metric.

Non-negotiable properties:

• The metric is computed inside the evaluator, never accepted from the agent.
• Official scoring runs on hidden seeds. Seeds are never sent to the agent, and

client-supplied Reset seeds are ignored during an eval session.

• Only the harness's env is official=true; agents may create their own practice envs.

Implement the EnvAdapter

Subclass the ale_gym adapter and implement the methods for the mode(s) you support. The signatures below are illustrative — they mirror the gRPC messages, so keep them serialization-friendly (plain dicts / arrays, no live handles).

from ale_gym.adapter import EnvAdapter, EvalResult

class MySimAdapter(EnvAdapter):
    # --- interactive mode ---
    def reset(self, seed: int, options: dict) -> dict:
        """Return the initial observation for one episode."""

    def step(self, action) -> tuple:
        """Return (observation, reward, terminated, truncated, info)."""

    def episode_success(self, info: dict) -> bool:
        """Authoritative per-episode success predicate."""

    def metrics(self, episodes: list) -> EvalResult:
        """Aggregate hidden-seed episodes into the authoritative metric(s)."""

    # --- batch mode ---
    def run_batch_eval(self, submission_dir: str) -> EvalResult:
        """Run the simulator's native pipeline over /submission at full speed."""

EvalResult carries the named metric values plus any declared safety metrics (see below). Everything the harness scores comes from this object.

Declare what a task may select — the config manifest

One evaluator image serves many scenarios, robots, and metrics. Declare that menu once in the adapter's env_config_schema (and a metric_menu), and every task selects a point in it at runtime via CreateEnv(env_config) — no per-task image rebuild. This manifest is extensible: add whatever keys your simulator's features need and they flow into the task form automatically. It has its own page: Evaluator config manifest.

Choose interactive, batch, or both

Interactive suits per-step control loops (implement reset/step/episode_success/ metrics); batch suits simulators with a heavyweight native evaluation harness where the agent writes a submission to the shared /submission volume (implement run_batch_eval). An evaluator may support both. See Interactive tasks and Batch tasks for the session lifecycles.

Declare the metric and safety limits

The metric declaration feeds scoring, so it must state:

• The authoritative metric name and its direction. Higher-is-better (e.g. success

rate, episode reward) and lower-is-better (e.g. planning time, ATE) are scored with different formulas.

• Any zero-target safety metrics (e.g. collision rate) as penalty factors. A safety

metric that is declared but missing from EvalResult is treated as worst-case (factor 0) — always emit every declared metric.

Do not hard-code paper targets, versions, or budgets in the evaluator; those are synced from the release manifest and are not published in v0.5.

Package with a Dockerfile and metrics README

Ship two files alongside the adapter:

• Dockerfile — pins the simulator and its dependencies, installs ale_gym, and runs

the AleEnv service as the container entrypoint. The container must run fully offline (no network) and deterministically given a seed.

• metrics/README.md — documents the authoritative metric: definition, unit,

direction, aggregation over episodes, and every declared safety metric with its limit. This is the human-readable source of truth reviewers check against the code.

Keep ale_gym lean

ale_gym is imported by both the agent sandbox and every evaluator, so keep it Python 3.8-compatible and dependency-minimal — no simulator-specific or heavy transitive dependencies. Simulator dependencies belong in the evaluator's Dockerfile, not in ale_gym.

Checklist

• [ ] EnvAdapter implements the methods for the declared mode(s).
• [ ] Metric computed in-evaluator; hidden seeds never leak to the agent.
• [ ] Metric name, direction, and safety limits declared; all declared metrics emitted.
• [ ] Dockerfile runs offline and deterministically.
• [ ] metrics/README.md matches the code.
• [ ] ale_gym changes stay py3.8-compatible and dependency-free.

See Contributing for the PR checklist and review gates.