FAQ
Common questions about the ALE Robotics benchmark and this verification website — what the site stores, why scores can exceed 1.0, why validated is not verified, and how hidden seeds and tampered bundles are handled.
Short answers to the questions we are asked most often. Each one links to the page with the full mechanics. Throughout, keep the two roles distinct: the local runner executes the benchmark on your machine, and this website verifies the encrypted results you submit.
Is the website a benchmark runner?
No. The site never runs the benchmark, never sees your agent, and never touches your API keys. Runs happen locally in an offline sandbox against a packaged evaluator; the website only receives an encrypted submission bundle and validates it. See the Overview for the split between the benchmark protocol and this service, and the Quickstart to run a task locally.
Does the site store my bundle?
Not by default. The server-side pipeline decrypts the ciphertext to a temporary location, processes it, then deletes both the plaintext and ciphertext temp files. The default retention flag is KEEP_SUBMISSION_BUNDLES=false. What is persisted is metadata: the ciphertext SHA-256, the recomputed score breakdown and audit summary, and the verification history for the entry. The bundle payload itself is not retained under the default configuration.
Why can scores exceed 1.0?
Because a single-task score is a ratio against the paper target, not a percentage. For higher-is-better metrics score = clamp(measured / paper_value, 0, cap); for lower-is-better metrics (such as planning_time or ATE) score = clamp(paper_value / max(measured, eps), 0, cap). The default cap is 1.5, so beating the paper's reported result scores above 1.0 (capped at 1.5). Scores are never forced into a 0–100% range. Full rules, penalties, and aggregation are in Scoring.
Does passing validation mean the run is verified?
No. validated confirms the bundle is internally consistent — it decrypts, schemas pass, hash chains and the integrity root verify, versions and task coverage are valid, and the recomputed scores match the claimed scores. It does not prove the local runner was unmodified. Runner provenance is a separate, stronger claim (officially_verified). See Verification levels.
Can I get officially_verified by self-hosting?
No. officially_verified is granted only for runs from a trusted execution path: a benchmark-team-controlled runner, an approved trusted third-party runner, a future hardware-attested runner, or an official reproduction/rerun. The backend refuses to accept audited or officially_verified from any upload field, so a submission cannot self-elevate. A self-hosted deployment can reach at most validated for its own uploads.
Is my private reasoning recorded?
No. The trajectory never records chain-of-thought. It captures only system-visible model messages, tool calls, shell commands, stdout/stderr, and API-usage metadata. Bundles also never contain API keys, private prompts, hidden reasoning, or unredacted full trajectories.
How are hidden seeds protected?
Official evaluation runs on hidden seeds that are never sent to the agent — any client-supplied Reset seed is ignored during an eval session. Seeds are stored in the trajectory only as a commitment SHA256(seed || run_id || evaluator-secret), never in raw form, so a run can be checked after the fact without exposing the seed. The harness reads the EvalResult over its own gRPC connection via a docker-exec loopback client; the agent has no evaluator exec access. See gym-over-gRPC.
What happens to a tampered bundle?
It is rejected. Every downstream check is fail-closed, so a modification surfaces as a failure rather than a silent pass:
| Tampering | Where it is caught |
|---|---|
| Wrong or missing key / corrupted ciphertext | age decryption fails |
| Path traversal, symlink, hardlink, device, fifo, absolute path | safe tar inspection |
| Oversized or too many files | decompressed size / file-count limits |
| Altered event stream | hash-chain verification |
| Altered aggregate over events | integrity-root verification |
| Inflated or edited scores | score recomputation vs. claimed |
| Reused, expired, or forged token | challenge-token + nonce check |
A bundle that decrypts but fails any later step does not reach validated. Decryption is only the first step of the pipeline, not a grant of trust. The full server-side processing order is in Architecture; the trust boundaries are in the Verification levels hard rules.
Where are the paper targets and budgets?
Not published in v0.5. Paper target values, task versions, and resource budgets are synced from the release manifest at evaluation time rather than listed here. The v0.5 set covers 11 tasks across manipulation, UAV, ground vehicle, and quadruped platforms.