ledger

A ledger of experiment runs, read from artifacts already on disk.

Deliberately NOT an experiment tracker. It does not wrap training, does not ask for log_metric() calls, and requires no change to how anything is run. Most research projects already leave structured artifacts behind -- config files, per-eval JSON dumps, benchmark results -- and the numbers that end up in a README were transcribed from them by hand. This reads those artifacts so the numbers become derivable, and re-derivable when they change.

Adoption cost is the design constraint. A tool that requires new discipline gets used for a week; one that reads what is already there keeps working while you forget it exists.

Two rules keep it honest:

Record what is unambiguous, refuse to guess the rest. A config file is a specification with no result attached, so it is stored with no metrics rather than an invented one. An unrecognised file shape yields no run rather than a wrong one.

Never rank a metric whose direction is undeclared. WER 0.043 -> 0.053 is a regression; accuracy 0.90 -> 0.94 is an improvement. A comparison that guesses will confidently order an ablation backwards, which is worse than refusing.

Metric dataclass

One final value read from a run's artifacts -- not a logged curve; see the module docstring's per-tracker conventions for what "final" means per adapter. step/split disambiguate a metric a run recorded more than once (a checkpoint, a train vs. eval split).

RunRecord dataclass

One experiment run, as discovered on disk.

adapter_caveats(adapters)

Caveats for the readers that produced a set of runs, deduplicated.

Takes adapter names rather than rows so it is callable from anywhere and testable without a database. Unknown and NULL adapters contribute nothing: NULL means "recorded before this was tracked", and inventing a caveat for a reader we cannot name would be worse than saying nothing.

collapse_to_last(metrics)

A step series collapsed to its last row per metric NAME, first-appearance order kept. Keyed on the name alone: on the live worst case split carried the sweep coordinate, so keying on (metric, split) collapsed nothing and 4 of 33 names survived a row cap. Truncation is the caller's budget, not this function's.

compare(conn, family, metric=None, project=None)

Rank every arm of an ablation family by a metric, within ONE project.

The question this exists for: a sweep of N named config variants is a designed experiment, and which arm won usually lives only in filenames and memory. Arms with no value for the metric are listed in without_metric rather than dropped -- an arm that was never evaluated is a finding.

A family name is unique per PROJECT, not globally: families() and runs.list both present (project, family) as the unit. This selected WHERE family = ? alone, so an English-ASR sweep and a Mandarin-ASR sweep that both called their arms asr_baseline/asr_biglm were pooled into one ranking with a named winner -- and, worse, _corpus_agreement keys on run name, so the collision ERASED the disagreement and the tool reported "all arms on aishell-1" for arms half of which ran on librispeech.

count_runs(conn, project=None, family=None)

How many runs match, ignoring any limit.

list_runs truncates and the caller could not tell: 212 of 222 runs vanished from a runs.list response whose message read "10 run(s)", while the families truncation in the same response was reported exactly.

detail(conn, project, name)

One run in full: its row, decoded config, every recorded metric, and the caveat (if any) for the adapter that read it -- None if no such run exists rather than an empty dict, so a caller can tell "not found" from "found, nothing to report".

families(conn, project=None)

Every named run family and its run count, largest first -- what attest runs list prints below the individual runs so a reader knows what is comparable with runs compare.

list_runs(conn, project=None, family=None, limit=20)

Runs in the ledger, optionally filtered by project/family. Ordered project, family, name -- grouped for display so related runs sit together, not started DESC, so this is not a "most recent runs" query; a caller wanting recency order does its own sort on the result.

metric_direction_path()

Public wrapper for _metric_direction_path, so a non-domain caller can find the same file unknown_direction_message names -- mcp/ provenance.py's runs.record merges declared directions into it and may not import the private name directly (see tests/ test_architecture.py::test_no_mcp_module_imports_a_private_domain_name).

metric_directions()

Built-in METRIC_DIRECTION, overlaid with a user's TOML file if present.

Read fresh on every call (not cached at import time) so tests and a user editing the file are both respected, matching db.embed_dims()'s pattern.

nested_arms(values)

Group a run's metric rows by nested split key, when it has several.

Real artifacts record the arms of an experiment INSIDE one file, under keys like arms.Baseline / arms.Treatment_Eigen, or lm-eval-harness's results.mmlu_marketing. The ledger's unit is the file, so all of those collapse into one run -- and _split_rank returns the same rank for every one of them, so _best_step takes the max across siblings.

Measured on a real corpus: a run reporting Baseline 0.000, Control_RAG 0.644, Oracle_Post 0.988 and Treatment_Eigen 0.655 scored 0.988. Every arm was credited with its own oracle upper bound, and on ~/nota's lm-eval output each model was ranked on its own easiest subtask, which reordered the bottom half of the table.

Returns {} when there is nothing to fan out -- one nested group, or none.

sample_runs(conn, project=None, family=None, limit=10)

Runs to show for a listing: spread across projects when not filtered.

list_runs orders by (project, family, name), so a LIMIT let the alphabetically-first project fill the whole answer -- the live ledger has 18 projects and 858 runs, and a default listing returned 10, all ablation. The count said "10 of 858" and not that they were one project, so an honest count made a partial answer look complete.

A filtered call is already narrowed by the caller and passes straight through.

scan(conn, root, project=None)

Read artifacts under a workspace root into the ledger. Idempotent.

Every subdirectory of root is treated as a project and read by the convention-based adapter -- no project needs to be known in advance. A directory yielding no recognisable runs is reported in empty rather than silently omitted, so "found nothing" is never mistaken for "nothing there".

Replaces a project's rows wholesale rather than merging: the artifacts on disk are the source of truth, so a run that vanished there vanishes here.

unknown_direction_message(metric)

The refusal compare() raises for one named metric with no declared direction -- factored out so attest runs record can print the exact same sentence rather than a paraphrase when a --direction was owed and not given. _no_direction_message is the sibling refusal for "nothing in this family has a known direction"; this one is for a single named metric.

workspace_root(explicit=None)

Where the projects live. Explicit argument, then RESEARCH_ROOT, else None.

Deliberately no default. Guessing a directory would either scan something the user did not mean or silently find nothing; returning None lets the caller say "set RESEARCH_ROOT" instead of reporting an empty success.