ledger

A ledger of experiment runs, read from artifacts already on disk.

Deliberately NOT an experiment tracker. It does not wrap training, does not ask for log_metric() calls, and requires no change to how anything is run. Most research projects already leave structured artifacts behind -- config files, per-eval JSON dumps, benchmark results -- and the numbers that end up in a README were transcribed from them by hand. This reads those artifacts so the numbers become derivable, and re-derivable when they change.

Adoption cost is the design constraint. A tool that requires new discipline gets used for a week; one that reads what is already there keeps working while you forget it exists.

Two rules keep it honest:

Record what is unambiguous, refuse to guess the rest. A config file is a specification with no result attached, so it is stored with no metrics rather than an invented one. An unrecognised file shape yields no run rather than a wrong one.

Never rank a metric whose direction is undeclared. WER 0.043 -> 0.053 is a regression; accuracy 0.90 -> 0.94 is an improvement. A comparison that guesses will confidently order an ablation backwards, which is worse than refusing.

Metric dataclass

One final value read from a run's artifacts -- not a logged curve; see the module docstring's per-tracker conventions for what "final" means per adapter. step/split disambiguate a metric a run recorded more than once (a checkpoint, a train vs. eval split).

RunRecord dataclass

One experiment run, as discovered on disk.

adapter_caveats(adapters)

Caveats for the readers that produced a set of runs, deduplicated.

Takes adapter names rather than rows so it is callable from anywhere and testable without a database. Unknown and NULL adapters contribute nothing: NULL means "recorded before this was tracked", and inventing a caveat for a reader we cannot name would be worse than saying nothing.

collapse_to_last(metrics)

A step series collapsed to its last row per metric NAME, first-appearance order kept. Keyed on the name alone: on the live worst case split carried the sweep coordinate, so keying on (metric, split) collapsed nothing and 4 of 33 names survived a row cap. Truncation is the caller's budget, not this function's.

compare(conn, family, metric=None, project=None)

Rank every arm of an ablation family by a metric, within ONE project.

The question this exists for: a sweep of N named config variants is a designed experiment, and which arm won usually lives only in filenames and memory. Arms with no value for the metric are listed in without_metric rather than dropped -- an arm that was never evaluated is a finding.

A family name is unique per PROJECT, not globally: families() and runs.list both present (project, family) as the unit. This selected WHERE family = ? alone, so an English-ASR sweep and a Mandarin-ASR sweep that both called their arms asr_baseline/asr_biglm were pooled into one ranking with a named winner -- and, worse, _corpus_agreement keys on run name, so the collision ERASED the disagreement and the tool reported "all arms on aishell-1" for arms half of which ran on librispeech.

count_runs(conn, project=None, family=None)

How many runs match, ignoring any limit.

list_runs truncates and the caller could not tell: 212 of 222 runs vanished from a runs.list response whose message read "10 run(s)", while the families truncation in the same response was reported exactly.

detail(conn, project, name)

One run in full: its row, decoded config, every recorded metric, and the caveat (if any) for the adapter that read it -- None if no such run exists rather than an empty dict, so a caller can tell "not found" from "found, nothing to report".

families(conn, project=None)

Every named run family and its run count, largest first -- what attest runs list prints below the individual runs so a reader knows what is comparable with runs compare.

family_named_in(conn, text)

The one recorded family text names as a whole word, else None.

"Compare the kdsweep runs" names its family in prose; a small model calling runs.ask passes the question and leaves family empty, and the router then asked "which family?" about the one it had just been told. Two families named at once is ambiguous and returns None rather than picking one.

list_runs(conn, project=None, family=None, limit=20)

Runs in the ledger, optionally filtered by project/family. Ordered project, family, name -- grouped for display so related runs sit together, not started DESC, so this is not a "most recent runs" query; a caller wanting recency order does its own sort on the result.

metric_direction_path()

Public wrapper for _metric_direction_path, so a non-domain caller can find the same file unknown_direction_message names -- mcp/ provenance.py's runs.record merges declared directions into it and may not import the private name directly (see tests/ test_architecture.py::test_no_mcp_module_imports_a_private_domain_name).

metric_directions()

Built-in METRIC_DIRECTION, overlaid with a user's TOML file if present.

Read fresh on every call (not cached at import time) so tests and a user editing the file are both respected, matching db.embed_dims()'s pattern.

nested_arms(values)

Group a run's metric rows by nested split key, when it has several.

Real artifacts record the arms of an experiment INSIDE one file, under keys like arms.Baseline / arms.Treatment_Eigen, or lm-eval-harness's results.mmlu_marketing. The ledger's unit is the file, so all of those collapse into one run -- and _split_rank returns the same rank for every one of them, so _best_step takes the max across siblings.

Measured on a real corpus: a run reporting Baseline 0.000, Control_RAG 0.644, Oracle_Post 0.988 and Treatment_Eigen 0.655 scored 0.988. Every arm was credited with its own oracle upper bound, and on ~/nota's lm-eval output each model was ranked on its own easiest subtask, which reordered the bottom half of the table.

Returns {} when there is nothing to fan out -- one nested group, or none.

run_named_in(conn, text)

(project, name) for the one recorded run text names, else None.

Run names are identifiers (kdsweep_t4, layer_importance), so only tokens that look like one are tried; a name shared by two projects is ambiguous and returns None rather than picking one.

runs_named(conn, names)

(project, name) for every recorded run whose name is one of names.

For resolving a run a question mentions by name ("show me kdsweep_t4"): runs.detail needs the project too, and a reader does not say it. More than one row means the name is ambiguous across projects, which the caller must not resolve by picking one.

sample_runs(conn, project=None, family=None, limit=10)

Runs to show for a listing: spread across projects when not filtered.

list_runs orders by (project, family, name), so a LIMIT let the alphabetically-first project fill the whole answer -- the live ledger has 18 projects and 858 runs, and a default listing returned 10, all ablation. The count said "10 of 858" and not that they were one project, so an honest count made a partial answer look complete.

A filtered call is already narrowed by the caller and passes straight through.

scan(conn, root, project=None)

Read artifacts under a workspace root into the ledger. Idempotent.

Every subdirectory of root is treated as a project and read by the convention-based adapter -- no project needs to be known in advance. A directory yielding no recognisable runs is reported in empty rather than silently omitted, so "found nothing" is never mistaken for "nothing there".

Replaces a project's rows wholesale rather than merging: the artifacts on disk are the source of truth, so a run that vanished there vanishes here.

unknown_direction_message(metric)

The refusal compare() raises for one named metric with no declared direction -- factored out so attest runs record can print the exact same sentence rather than a paraphrase when a --direction was owed and not given. _no_direction_message is the sibling refusal for "nothing in this family has a known direction"; this one is for a single named metric.

workspace_root(explicit=None)

Where the projects live. Explicit argument, then RESEARCH_ROOT, else None.

Deliberately no default. Guessing a directory would either scan something the user did not mean or silently find nothing; returning None lets the caller say "set RESEARCH_ROOT" instead of reporting an empty success.