Skip to content

attestation

attestation makes research provenance auditable and fully local, for any scientist who reads papers, keeps results files, and writes claims. It keeps a personalised science feed and a knowledge graph built from what you read; traces a number in your draft to the results file it came from and a citation key to your bibliography; and does symbolic derivations — all exposed to agents as MCP tools, with nothing leaving the machine. For computational work it adds an experiment ledger: it reads the runs you already have on disk (results files and W&B/MLflow/Sacred/DVC/Hydra directories) and ranks the arms of a sweep with caveats. The feed, provenance and symbolic tools assume nothing about your field; only the ledger assumes you run experiments.

Try it in 60 seconds

No model server, no ingest, no config — the run ledger and claim checker are pure local computation over files that already exist. This uses the sweep in examples/workspace:

git clone https://github.com/mgoldey/attestation ~/attestation && cd ~/attestation && uv sync
export ATTEST_DB=/tmp/attest-demo.db

uv run attest runs scan --root examples/workspace
uv run attest runs compare kdsweep
uv run attest claims examples/workspace/speech-distill/FINDINGS.md

Measured at 0.94s total with no LLM backend reachable at all. What it says:

kdsweep — ranked by val_loss (lower_is_better), all arms on librispeech-100h
winner: kdsweep_t4
  caveat: the top two arms differ by 0.03 (1.4%) -- too close to call from
          these numbers alone
  caveat: each arm is a single run; no seed replication, so this ranking
          cannot separate configuration from run-to-run variance

7 claim(s): 1 contradicted, 5 supported, 1 unsupported
1 malformed

attest claims exits 1 on a contradiction (so it can gate a commit), which is why a && chain stops there.

Every tracker ranks arms. The second caveat — this ranking cannot separate configuration from noise — is the part that earns its keep, and the contradicted verdict is a number in a document that no longer matches the artifact it came from.

What it does

Three things for every scientist, one for computational experiments, and one way to use all of them:

For every scientist

  • The feed and knowledge graph rank a personalised reading list from cosine similarity plus click-trained terms, and derive a concept graph from the same tagging pass — no separate content pipeline. Any RSS/Atom source; arXiv is a default, not a requirement. See docs/guides/feed.md.
  • Data and citation provenance check a number in your prose against the results file that produced it — a JSON or CSV under results/ from any instrument, script or spreadsheet export, not only a training run — with one of five verdicts (supported, contradicted, unsupported, ambiguous, stale), and lint a citation key against your BibTeX or Zotero library. See docs/guides/claims-and-citations.md.
  • Symbolic derivations run in a sandboxed subprocess with a timeout and memory cap, never eval-ing input. See sym.* in docs/guides/agents.md.

For computational experiments

  • The experiment ledger reads runs from artifacts already on disk — no instrumentation, no log_metric() calls — and ranks the arms of a sweep with caveats rather than a silent verdict. Five tracker layouts (W&B, MLflow, Sacred, DVC, Hydra) are read as conventions of their own. This is the one part that assumes you run experiments — though a plain results/ file is read the same way, so the claim checker traces every number through it. See docs/guides/ledger.md.

From an agent

  • All of the above are exposed as 47 MCP tools, restricted per session into feed/provenance/knowledge/symbolic surfaces. See docs/guides/agents.md and the repo's own src/attestation/skills/ (seven skills: one per agent surface plus setup, plus two write-side skills, attestation-record and attestation-annotate).

Install

Two tiers: Python 3.12+ and uv alone runs the ledger and claim checker; add Ollama for the feed, tagging, and knowledge graph. Models are optional and pulled for you.

uv run attest install          # idempotent setup: .env, models, first ingest
uv run attest install --check  # diagnose only, exits 1 on gaps

See docs/guides/install.md for prerequisites and the manual steps attest install automates.

Chat with it from Discord

Once attest install has run and hermes-agent is installed, the feed answers from a phone. Four steps, all outside this repo:

  1. In the Discord Developer Portal create an application and a Bot, enable the Server Members and Message Content intents on the Bot page, and copy the token. Then hermes gateway setup, pick Discord, and paste the token plus your own Discord user ID (not the bot's).
  2. Allowlist the feed surface for chat in ~/.hermes/config.yaml, so each turn carries 2 tool schemas instead of 131:

yaml mcp_servers: attestation-feed: {enabled: true} # attest install wrote the entry, disabled platform_toolsets: discord: [attestation-feed] # an MCP server here is an allowlist

  1. systemctl --user restart hermes-gateway, then invite the bot with https://discord.com/oauth2/authorize?client_id=<APP_ID>&scope=bot&permissions=274878286912 and DM it "What are your recommendations today?"

Measured on gemma4:e2b over a GTX 1080: a turn dropped from 54-249s at 15-29k prompt tokens to 23-28s at ~5k. The other half of that saving is keeping the model resident, since a cold load alone was 30s. Full recipe, Telegram variant, and the keep-alive timer in docs/guides/agents.md.

Golden paths

A golden path is a directory under examples/ with a README in seven fixed sections, a run.sh that runs those README commands verbatim, and its own inputs on disk — tests/test_golden_paths.py runs every one whose prerequisite is none and pins a line of its output, so the docs are what the suite asserts. examples/README.md is the full catalogue, with a runtime for each. Thirteen paths, grouped by prerequisite:

None — pure local computation:

  • agents/ — the install doctor, attest emit's configs, one surface over stdio
  • citations/ — a BibTeX library, a draft, one citation key that resolves nowhere
  • dvc/ — a real dvc repro pipeline, four foreach arms ranked
  • flows/ — forty items scored, every MCP tool over stdio, four MLflow arms
  • hydra/ — a real Hydra --multirun sweep, four arms ranked
  • mlflow/ — a real MLflow directory, four arms, one contradicted claim
  • model-servers/ — attest ingest/tag against an in-process stub server
  • ranking/ — twenty hand-built rows, rank.rank_rows's classifier-only AUC beside the blended-order AUC
  • sacred/ — a real Sacred FileStorageObserver directory, four arms ranked
  • tensorflow/ — a real Keras/CSVLogger run, four learning-rate arms ranked
  • wandb/ — a real offline W&B run directory, four arms ranked
  • workspace/ — the ledger and claim checker, three claims wrong on purpose

A model server at LLM_BASE_URL: every path above needs no model; this one does, to score prompts against a running LLM.

  • prompt-evals/ — the tagging prompt's dev score and the transfer gate

See examples/README.md for what each demonstrates and how long it takes. The most thorough is flows/:

uv run --group examples python examples/flows/run_all.py --offline

examples/flows/README.md explains each flow; examples/flows/RESULTS.md records what the live run measured.

Documentation

uv run --group docs mkdocs serve runs a browsable site over everything below. The guides, one per question:

Plus docs/concepts.md for first-ten-minutes vocabulary, the CLI reference (docs/reference/cli.md) and API reference under docs/, design records under docs/superpowers/specs/, measurement lessons (docs/measurement-lessons.md), CONTRIBUTING.md for the gates and conventions, and CHANGELOG.md for what's landed.

Licence

MIT — see LICENSE.