Scoring

ALE Robotics scores an agent by comparing the evaluator-measured domain metric against a paper target. The authoritative metric is always produced by the evaluator container on hidden seeds — never self-reported by the agent (see Tasks and Verification). Scoring lives in the runner's harness/scoring.py; the website independently recomputes scores from the decrypted evaluator results and refuses to trust the agent's claimed numbers.

Gap score

Each task has a headline metric and a direction. The per-task gap score normalizes the measured value against the paper target, clamped to a cap.

CAP = 1.5  # default; synced from the release manifest
EPS = 1e-9

# higher-is-better (e.g. success rate, episode reward)
score = clamp(measured / paper_value, 0, CAP)

# lower-is-better (e.g. planning_time, ATE)
score = clamp(paper_value / max(measured, EPS), 0, CAP)

A score of 1.0 means the agent matched the paper. Beating the paper produces a score above 1.0, capped at 1.5. Scores are dimensionless ratios: they may exceed 1.0 and are never rendered as percentages.

Paper targets, benchmark versions, and budgets are not published in v0.5 — they are synced from the release manifest at scoring time, not embedded in this document.

Zero-target safety penalties

Some metrics have a target of zero (e.g. collisions). These are not scored as ratios; instead they multiply the gap score by a penalty factor derived from a declared limit.

# rate = observed rate of the zero-target metric; limit = declared budget
factor = 1 - min(1, rate / limit)   # 1.0 = clean, 0.0 = at/over the limit
score  = gap_score * factor

A declared safety metric that is missing from the results is treated as worst-case and receives factor = 0 (fail-closed). Multiple penalties multiply together.

Faithfulness gate (reproduction genre)

Reproduction-genre tasks must actually implement the paper's method, not just hit the metric. After the gap score and penalties, they pass through a fail-closed faithfulness gate — an LLM judge over the trajectory and artifacts.

# reproduction genre only
final = score * (1 if faithful else 0)   # unfaithful => final = 0

Open-genre tasks (optimize a metric by any means) skip this gate.

Aggregation: macro_mean

The release aggregate is the unweighted mean of per-task normalized scores over the required task set.

# missing tasks are EXCLUDED, not coerced to zero,
# unless the release policy states otherwise.
present = [t.score for t in required_tasks if t.completed]
aggregate = sum(present) / len(present)

Excluding missing tasks (rather than zeroing them) keeps a partial run honest: a run that covers 4 of 11 tasks reports an average over those 4, and its coverage is recorded separately. Coverage against the required_task_set_hash is checked during verification; a run is not comparable to a full run unless coverage is complete.

Worked example

An illustrative reproduction task with a higher-is-better headline metric plus a zero-target safety metric (numbers are illustrative, not published targets):

gap    = clamp(0.63 / 0.70, 0, 1.5) = 0.90
factor = 1 - min(1, 0.02 / 0.05)    = 0.60
score  = 0.90 * 0.60                = 0.54
final  = 0.54 * 1 (faithful)        = 0.54

Now aggregate across a 3-task required set where one task was not completed:

task A final = 0.54
task B final = 1.12   # beat the paper, above 1.0
task C       = missing -> excluded

macro_mean = (0.54 + 1.12) / 2 = 0.83   # over 2 of 3 tasks

Pass rate (task completion, orthogonal to score)

Alongside the gap score the leaderboard shows a pass rate: did the agent complete a task's official evaluation, independent of how well it scored. It is computed from the evaluator's authoritative per-episode stats — an episode that ran to its end without being aborted (reset mid-episode / session torn down) is a pass — never from the agent's self-report. A run can pass (finish every episode) yet score poorly against the paper, and vice-versa, which is why both columns exist. (Mirrors the pass column on the ALE agents-last-exam board.)

Leaderboard categories: Overall / Reproduction / Open-ended

The board has three tabs. Each submission carries its task's genre (reproduction | open) in the run manifest, so the split needs no extra data:

• Reproduction — reproduction-genre runs (faithfulness-gated).
• Open-ended — open-genre runs (beat the baseline by any faithful means; no gate).
• Overall — both, blended.

A row's category score is the mean of its covered tasks in that genre; Overall blends all covered tasks. The top-of-page chart plots the Top-N agents by Pass Rate, Score, or Both. Runtime and token counts shown on the board are harness-recorded (from the LLM gateway's usage log — authoritative), while the effort tier is submitter-declared. See the benchmark repo's docs/dev/leaderboard-genre-and-pass-rate.md for the data flow.

What the runner vs the website does

• Runner (local): computes gap scores, penalties, and the faithfulness gate during the

run and writes score_breakdown.json / faithfulness.json into the submission bundle.

• Website: recomputes scores from the evaluator results in the decrypted bundle and

compares them to the claimed values. A match is required for the validated verification level; it does not by itself prove the local runner was unmodified.