Architecture

ALE Robotics is two systems that meet at a single artifact: a benchmark run that executes on your own machine, and a verification website that inspects the encrypted bundle a run produces. The site is not a benchmark runner — you run tasks locally, and the site validates what you upload. This page describes both, and the trust boundaries that make results honest by construction.

Benchmark run architecture

A run places an autonomous coding agent in an offline, no-network sandbox together with a robotics paper and a packaged simulator. The agent must implement the paper's method (reproduction genre) or optimize a metric (open genre). It never talks to the simulator directly: it interacts only through gym-over-gRPC with an independent evaluator container.

+----------------------+        gym-over-gRPC         +-----------------------+
|  agent sandbox       |  (8 unary RPCs, client env)  |  evaluator container  |
|  OFFLINE, no network |----------------------------->|  1 simulator + 1      |
|  untrusted_agent     |                              |  domain metric        |
+----------+-----------+                              |  trusted_evaluator    |
           |                                          +-----------+-----------+
           | trajectory events                                    ^
           v                                    docker-exec        | official=true env
+----------------------+   loopback gRPC client   +----------------+ reads EvalResult
|  harness             |------------------------->|  (harness-only path)
|  trusted_harness     |
|  scoring + trajectory|---> submission bundle ---> (upload) ---> website
+----------------------+

One evaluator container = one simulator + one domain metric, exposing an AleEnv gRPC service with eight unary RPCs: CreateEnv, GetSpec, Reset, Step, StartEvalSession, GetEvalResult, SubmitBatchEval, GetBatchEvalStatus. Only the harness's environment is marked official=true. The authoritative domain metric is always computed by the evaluator, never self-reported by the agent.

Tasks run in one of two modes:

Trust boundaries in a run

• Hidden seeds never reach the agent. Official evaluation runs on hidden seeds; client-supplied Reset seeds are ignored during eval sessions. Seeds are stored only as a commitment SHA256(seed || run_id || evaluator-secret), never raw.
• The result path is the harness's, not the agent's. The harness reads EvalResult over its own gRPC connection via a docker-exec loopback client. The agent has no evaluator exec access.
• Trajectory events are labeled by producer. The versioned, append-only, hash-chained JSONL stream (ale-trajectory-event/v1) tags each event trusted_harness, trusted_evaluator, trusted_network_proxy, untrusted_agent, or derived_verifier. Cross-producer ordering uses wall_time + producer_sequence; monotonic clocks are valid only within a single producer. Chain-of-thought is never recorded — only system-visible messages, tool calls, shell commands, stdout/stderr, and API-usage metadata.

Scores are computed in harness/scoring.py from evaluator results (see Scoring); reproduction-genre tasks additionally pass a fail-closed faithfulness gate. The run's output is an encrypted submission bundle — never the raw simulator, seeds, or reference solutions.

Website architecture

The self-hostable site is a small stack behind Caddy, brought up with docker compose up --build:

Caddy (auto HTTPS)
   |-> web  (Next.js)   -- leaderboard, docs, submission UI
   `-> api  (FastAPI)   -- challenges, submissions, verification
             |-> SQLite (WAL, DATABASE_URL-swappable to Postgres)
             `-> ephemeral /tmp submission volume (plaintext + ciphertext deleted after processing)

Login is GitHub OAuth. The versioned API exposes GET /api/v1/submission-key, POST /api/v1/challenges, POST /api/v1/submissions (20 MB default cap), GET /api/v1/submissions/{id}, and GET /api/v1/leaderboard.

Key custody defines the primary trust boundary. The local runner holds only the website's age (X25519) public key; the website holds the private key, which never appears in the repo, image, logs, or browser. Bundles are age(zstd(tar)) produced via the audited pyrage library — never hand-rolled crypto.

On receipt, the server processes each upload in a fixed order:

receive ciphertext -> sha256 -> decrypt with private key
  -> enforce decompressed size / file-count limits
  -> safe tar inspect (reject traversal / symlink / hardlink / device / fifo / absolute paths)
  -> extract allowlisted files only -> validate JSON schemas
  -> verify challenge token (server-signed, one-time, unexpired) + nonce
  -> verify hash chains -> verify integrity root
  -> verify benchmark/task versions + coverage
  -> recompute scores from evaluator results -> compare recomputed vs claimed
  -> audit summary -> persist metadata
  -> delete plaintext + ciphertext temp files

Default retention is KEEP_SUBMISSION_BUNDLES=false.

Trust boundaries on the website

• Successful decryption does not grant verified. Decryption only proves the bundle was encrypted to the site's key. Verification levels — self_reported, validated, audited, officially_verified, revoked — are assigned by the backend from what it can prove (see Verification levels).
• Users cannot self-elevate. The backend refuses any upload field that would set audited or officially_verified; those come only from a benchmark-team-controlled, approved third-party, hardware-attested, or official-rerun runner.
• Bundles carry no secrets. Allowlisted files (manifest.json, events.jsonl.zst, task_results.json, score_breakdown.json, faithfulness.json, network_summary.json, artifacts_manifest.json, integrity.json) never contain hidden seeds, reference solutions, API keys, private prompts, hidden reasoning, or unredacted full trajectories.

For the deeper rationale behind these boundaries, see the Security model; to stand the site up yourself, see Deployment.