Dataset · CLE
Continual Learning Exam
102 human-authored persona ladders across 14 occupational domains. Every ladder gates its levels behind the one before it — so an agent reaches the last level only by holding on to the artefacts, terminology and constraints it established in the first.
Overview
What CLE measures
Most agent benchmarks ask for one artefact and score it. A working professional never gets that: the second request assumes the first one's output, the fourth assumes a vocabulary nobody restated. CLE is built around that dependency. Each ladder is a single persona's workday, cut into gated levels — the level's inputs are the previous level's outputs, and a run stops at the first level that fails. Reaching level 4 is therefore evidence that the agent carried context forward, not that it got a lucky draw.
Ladders were authored by domain practitioners, and the rubric is hidden from the solver: the agent sees the task brief, while grading runs off a checklist the agent never reads — file existence, structural correctness, exact key fields, source and anchor verification, and explicit negative checks. Domains were assigned by reading each ladder end to end rather than from author-declared labels, and clustered by the competence and artefact type they exercise, so ladders that share an industry but not a skill set — RTL verification and grid load forecasting, say — stay separate.
Domain composition
Family sets the hue; every domain is named in the legend with its own count and share, so identity never rests on colour alone. Shares are of all 102 ladders.
What one graded level contains
1 · Task brief
What the solver sees: Inputs, Output,
Format Requirements, Constraints. Written as a colleague
would hand it over — the deliverable path is exact, the reasoning is not spelled out.
2 · Hidden rubric
The full checklist, withheld from the solver: file existence, structure, key-field exactness, source & anchor verification, negative checks, plus grading notes on what does not earn credit.
3 · Item-level verdict
Every rubric line scored individually with the evidence quoted, then summed to a
level score and a PASS/FAIL gate. Attempt count is kept.
4 · Full trajectory
The complete actor run for the graded attempt — reasoning, tool calls, tool output, messages — typically 40–130 segments per level. Nothing is summarised away.
The dataset is model-agnostic: task briefs, linked workspaces and hidden rubrics are the core artefacts. The previews below focus on the case design and verification pressure points rather than a particular model run.
Data samples
Three curated cases from the public dataset, selected for narrative clarity, cross-domain breadth and visual demo value. Each preview shows the five linked tasks, representative evidence, machine-checkable outputs and evaluation pressure points.
Citation
@misc{cle2026,
title = {{CLE}: A Continual Learning Exam for Agents},
author = {Memento},
year = {2026},
note = {https://memento.run/products/data/cle/}
}
TODO — swap in the real author list, venue and arXiv id before release.