Skip to content
Agents' Last Exam for Robotics
Scoring & trust

Trajectory format

The versioned, append-only, hash-chained JSONL event stream (ale-trajectory-event/v1) — trust levels, ordering, seed commitments, and what is deliberately not recorded.

A run produces a trajectory: a single append-only stream of events describing everything the harness, evaluator, network proxy, and agent did. It is the evidence the website replays to move a submission from self_reported toward validated. This page specifies the wire format; see Submission bundle for how it is packaged and Verification levels for how it is checked.

Format

The trajectory is newline-delimited JSON (JSONL). Each line is one event object conforming to the schema ale-trajectory-event/v1. The stream is:

  • Versioned — every event carries schema: "ale-trajectory-event/v1", so readers can

reject or migrate unknown versions rather than guessing.

  • Append-only — events are only ever appended; existing lines are never edited or

reordered on disk.

  • Hash-chained — each event records the hash of its predecessor, so any deletion,

reordering, or in-place edit breaks the chain and is detectable.

In the bundle the file appears as events.jsonl.zst. The website verifies the hash chain end to end and folds the final link into the integrity root before recomputing scores.

Hash chaining

Each event carries a hash over its own canonicalized content plus the prev_hash of the event before it, forming a tamper-evident chain:

event.hash = SHA256(prev_hash || canonical_json(event_without_hash))

The first event uses an all-zero prev_hash. Verification recomputes every link; a single altered byte anywhere in the stream invalidates every subsequent hash. This proves the stream is internally consistent — it does not prove the local runner was unmodified, which is why decryption and a valid chain alone only yield validated, never officially_verified.

Trust levels

Every event is attributed to a producer with an explicit trust level. This is the core of the "honest by construction" design: the agent's own claims are recorded, but marked as untrusted, and the authoritative metric always comes from the evaluator.

Trust levelProducerAuthority
trusted_harnessThe run harnessOrchestration, timing, session boundaries
trusted_evaluatorThe evaluator containerAuthoritative domain metric and eval results
trusted_network_proxyThe offline network proxyRecords blocked/allowed egress
untrusted_agentThe agent under testSelf-reported; never authoritative
derived_verifierPost-hoc verifierRecomputed values (e.g. scores, hashes)

The evaluator's EvalResult is read by the harness over its own gRPC connection, so evaluator-trust events are not relayed through the agent. See gym-over-gRPC and Scoring.

Ordering

Producers run concurrently, so a single global clock does not exist. Cross-producer ordering uses the pair wall_time + producer_sequence. producer_sequence is a monotonic counter that is only meaningful within one producer; wall_time is the UTC timestamp used to interleave events across producers. Do not assume any single monotonic clock spans producers.

Seed commitments

Official evaluation runs on hidden seeds that are never sent to the agent. To keep the trajectory auditable without leaking them, seeds are stored only as a commitment:

seed_commitment = SHA256(seed || run_id || evaluator_secret)

The raw seed never appears in the stream or the bundle. A holder of the seed and secret can later verify the commitment; an agent replaying the trajectory cannot recover the seed.

What is NOT recorded

The trajectory captures system-visible behavior, not model internals:

  • No chain-of-thought. Only system-visible model messages, tool calls, shell commands,

stdout/stderr, and API-usage metadata are recorded.

  • No raw hidden seeds — commitments only.
  • No secrets — no API keys, private prompts, hidden reference solutions, or the evaluator

secret.

Example event

{
  "schema": "ale-trajectory-event/v1",
  "producer_sequence": 42,
  "wall_time": "2026-07-10T14:03:22.481Z",
  "run_id": "run_7Q3f...",
  "trust_level": "untrusted_agent",
  "producer": "agent",
  "type": "tool_call",
  "payload": {
    "tool": "shell",
    "command": "python train.py --config configs/dp.yaml",
    "cwd": "/workspace"
  },
  "prev_hash": "3b1e9c...af02",
  "hash": "9d47a2...ee18"
}

An evaluator result event instead carries trust_level: "trusted_evaluator" and, for eval sessions, references the relevant seed_commitment rather than any seed value.