Faithfulness gate

What the gate is

The faithfulness gate is a fail-closed check applied to reproduction-genre tasks. After the evaluator has computed the authoritative domain metric and the harness has produced a normalized, penalty-adjusted score (see Scoring), an LLM judge reviews whether the submission actually implements the method described in the paper. If the judge does not return a clean "faithful" verdict — because the submission is unfaithful, or because the judge run itself failed — the task's final score is forced to 0, regardless of how well the measured metric scored.

The gate exists because the domain metric alone cannot distinguish "reproduced the paper's method" from "reached the same number some other way." The evaluator measures performance; the gate judges method faithfulness. Both must hold for a reproduction task to score.

When it applies

Open-genre tasks are graded purely on the evaluator's metric and the standard penalty factors; there is no "correct method" to be faithful to, so the gate does not run. The task's genre is fixed by the task definition, not chosen by the agent. See Overview for the genre split.

Fail-closed semantics

"Fail-closed" means the gate defaults to the safe outcome — a zero — whenever it cannot affirmatively confirm faithfulness:

• Unfaithful verdict → final score 0.
• Judge error, timeout, or an inconclusive/ambiguous verdict → treated as a failure → final

score 0.

• Faithful verdict → the normalized, penalty-adjusted score stands unchanged.

There is no "benefit of the doubt." A missing or errored gate result is never coerced into a pass. This mirrors how a missing declared safety metric is treated as worst-case in Scoring: the trust-preserving default is the low score, not the high one.

What the judge checks

The judge reasons over the paper, the task specification, and the agent's system-visible run record — the submission's code and files, model messages, tool calls, shell commands, and stdout/stderr recorded in the trajectory. Its question is qualitative and singular: does this submission faithfully implement the paper's method? That includes catching submissions that substitute a different method, hardcode or memorize outputs, target the evaluator's scoring path instead of the task, or otherwise reach the metric without implementing the paper.

What the judge does not check

The gate is deliberately narrow. It does not:

• Compute, adjust, or override the domain metric — that is always the evaluator's job, on hidden

seeds, never self-reported by the agent.

• Enforce a numeric performance threshold. A faithful implementation that scores poorly still

passes the gate (its low normalized score stands); an unfaithful one that scores well still gets 0.

• Run on open-genre tasks.
• See hidden seeds, hidden reference solutions, or private prompts — these are never in the run

record or the submission bundle.

• Read chain-of-thought. CoT is never recorded, so the judge works only from system-visible

messages, tool calls, and command output.

The gate is a faithfulness check, not a proof of correctness or a substitute for the evaluator.

How it is recorded

The gate runs as part of harness scoring, outside the agent's offline sandbox. Its outcome is written to the trajectory at trust level derived_verifier and summarized in the allowlisted faithfulness.json file inside the submission bundle. An illustrative shape (the authoritative schema is validated server-side on upload):

{
  "task": "diffusion_policy",
  "genre": "reproduction",
  "gate_applied": true,
  "verdict": "faithful",
  "judge_completed": true,
  "final_score_multiplier": 1
}

verdict is one of faithful, unfaithful, or error; final_score_multiplier is 1 only for a completed faithful verdict and 0 in every other case. The file never contains hidden reasoning, hidden reference material, or the judge's chain-of-thought.

How the website enforces it

The website is not a benchmark runner — runs happen locally. During server-side processing it reads faithfulness.json and re-applies the fail-closed rule while it recomputes scores from the evaluator results and compares them against the claimed values. For a reproduction task, a final score above 0 must be backed by a completed faithful verdict; anything else recomputes to 0 and surfaces as a mismatch. Passing the gate is part of reaching the validated verification level; the gate result is also among the artifacts a reviewer inspects for audited. Successful decryption alone never grants a verified status.