claims

Check assertions written in Markdown against runs in the ledger.

A README says "MAE 0.353 eV vs experiment @ aDZ". That number was transcribed by hand from an artifact, and nothing checks it: re-run the benchmark, and the document keeps asserting 0.353 forever. This closes that loop.

The format is an HTML comment beside the prose it describes:

The cut leaves WER essentially unchanged (**0.053 vs. 0.043** baseline).
<!-- claim: ablation/whisper-small-ablated metric=wer value=0.053 tol=0.001 -->

An HTML comment because it is the only annotation that is invisible in every Markdown renderer, plain text so grep and git diff work on it, and adjacent to the claim rather than in a separate file that drifts. The prose is never touched; a claim is added beside an assertion that already exists, so annotating is incremental -- one claim is useful on its own.

Five verdicts, and the distinctions between them are the point:

supported     a run matches, within tolerance
contradicted  a run matches the reference but DISAGREES on the value
unsupported   no run matches -- the claim may be true, nothing backs it
ambiguous     the reference matched several runs; which is meant is unknown
stale         the artifact changed after the claim was last verified

unsupported and contradicted must never collapse together: one means "I never recorded this", the other means "this is wrong", and the response differs. ambiguous exists because silently taking the first of several matches is how a checker reports a confident wrong answer.

Claim dataclass

One <!-- claim: ... --> annotation, parsed but not yet checked.

raw keeps the annotation's original body so a malformed field can be reported with the text that failed to parse, not a reconstruction of it.

Verdict dataclass

One claim's outcome: which VerdictKind, why, and what it was checked against -- source_path names the run the ledger read, so an auditor's "from which file?" has an answer without opening the ledger separately.

matched_split/matched_step are the MATCHED RUN ROW's split/step -- what was actually found -- distinct from claim.split/claim.step, which are what the document asked for. The two are different quantities that happen to share a name one level down; the matched_ prefix here keeps a reader from having to guess which is which.

VerdictKind

Bases: StrEnum

The closed set of claim-check outcomes.

A StrEnum member IS a str -- every existing verdict.verdict == "supported" comparison, Counter tally, and dict(...) == {"supported": 1} assertion keeps working untouched -- but a misspelled verdict at a construction site is now a ty error instead of a silently-tallied new category in counts.

check(conn, root, *, resolver=None)

Verify every claim under root. Read-only; never edits a document.

resolver is optional and INJECTED. When one is passed, the citation lint runs too, so a claim naming a key no configured source has is reported here rather than only from cite.check -- a claim whose number agrees but whose citation does not resolve was reading as plain supported from the tool whose name says it checks claims.

Injected rather than imported: this module verifies numbers against the ledger and must not grow an import-time dependency on the bibliographic readers, which are an optional surface. No resolver means no lint, which is exactly the behaviour every caller had before.

The two concerns stay separate verdicts. A claim can be contradicted on its number AND uncited on its key, and folding one into the other would hide something the author has to fix.

check_citations(claims_, resolver)

Claims whose citation key no configured source can resolve.

A lint, not a verification: it says the key is unknown here, never that the cited work fails to support the claim. Claims with no cite are skipped -- every claim written before this field existed has none, and reporting them all would turn one new field into a document-wide false alarm.

check_claim(conn, claim)

Check one claim against the ledger and return its Verdict.

Deterministic throughout: no run, no matching metric, or more than one candidate row all become distinct VerdictKinds rather than a guess -- disambiguating by distance to the claimed value would let the claim select its own evidence, so "more than one row, no split=/step=" is AMBIGUOUS, never a silent pick of the closest.

coverage(root)

Numbers in prose that no claim annotation covers.

The inverse of check: that verifies claims that exist, this finds assertions that were never made checkable. A document with zero contradicted claims looks healthy while asserting a dozen unverifiable numbers, and nothing surfaces the difference.

Reports a decimal as uncovered when no claim in the same file states that value (within its own tolerance). Deliberately line-agnostic: a claim annotation usually sits a line or two from the prose it describes, and requiring adjacency would produce false alarms for correctly-annotated documents.

find_claims(root)

Every claim in every Markdown file under root.

parse_file(path)

Claims in one Markdown file, plus complaints about malformed ones.

A malformed annotation is reported, never skipped: silent skipping is how a claim disappears from review without anyone deciding to remove it.