Batch tasks

Batch mode is the second of the two task modes in ALE Robotics. Where an interactive task has the agent drive episodes live over gym-over-gRPC with Reset/Step, a batch task hands scoring to the evaluator's own pipeline. The agent produces an artifact — a trained policy, a planner configuration, a set of solved trajectories — writes it to a shared volume, and asks the evaluator to run its native evaluation loop at full speed on hidden seeds.

Batch mode exists because some simulators score far more efficiently in their native harness than through per-step gRPC round-trips: vectorized GPU rollouts (Isaac, MJX), ROS/Gazebo launch graphs, or planners that need their own process topology. The trade-off is the same in both modes: the authoritative domain metric is always computed by the evaluator, never self-reported by the agent, and always on HIDDEN seeds the agent never sees.

How a batch task runs

1. The agent reads the task's submission_contract (via GetSpec, alongside the paper and

task config) to learn exactly what file(s) to produce, in what format, and where.

1. The agent trains/plans/solves inside its OFFLINE sandbox and writes the result to the

shared /submission volume at the path the contract specifies.

1. The agent calls SubmitBatchEval, which tells the evaluator the submission is ready.
2. The evaluator runs its native pipeline — its own rollout or planning loop — on hidden

seeds, at full speed, in its own container.

1. The harness polls GetBatchEvalStatus until the run reaches a terminal state, then reads

the EvalResult over its own gRPC connection via the docker-exec loopback client. The agent has no evaluator exec access and cannot read the result channel.

The /submission directory is a shared mount between the agent container and the evaluator container; it is the only data path from agent to native pipeline. Nothing the agent writes there is treated as a score — it is input to the evaluator, which recomputes the metric itself.

The submission contract

Each batch task declares a submission_contract in its task config. It is the authoritative spec for the artifact: file layout, format, and any shape/dtype constraints. Because contracts are per-task, read the contract from GetSpec rather than assuming a shape. Conceptually:

{
  "submission_path": "/submission",
  "entrypoint": "policy.pt",
  "format": "torch_state_dict",
  "notes": "Files, formats, and required keys are defined per task; read them from GetSpec."
}

The batch RPCs

Two of the eight AleEnv unary RPCs are specific to batch mode:

GetBatchEvalStatus is a poll: the agent may call it to observe progress, but only the harness's env is official=true, and the harness reads the final EvalResult over its own connection. Client-supplied Reset seeds are ignored in eval sessions; batch runs use hidden seeds committed as SHA256(seed || run_id || evaluator-secret) and never exposed raw.

Scoring

Batch scoring is identical to interactive scoring — see Scoring. The evaluator's measured metric is normalized against the paper target: score = clamp(measured/paper_value, 0, cap) for higher-is-better, or clamp(paper_value/max(measured, eps), 0, cap) for lower-is-better metrics (e.g. planning_time). The default cap is 1.5. Zero-target safety metrics apply a penalty factor, and reproduction-genre tasks additionally pass the fail-closed faithfulness gate. Every batch run appends to the hash-chained trajectory with trusted_evaluator-level events.

Batch tasks in v0.5

Four of the eleven v0.5 tasks run in batch mode:

A fifth task, edmp (MPiNets / PyBullet), is also non-interactive but uses a distinct read-from-task-config mode rather than a /submission handoff. The remaining six v0.5 tasks are interactive.

Paper targets, task versions, and compute budgets are not published in v0.5; they are synced from the release manifest rather than fabricated here. For the full set and per-task metadata see the task registry, and for the live drive-loop alternative see Interactive tasks.