Run records: a signed, portable account of one research run
Date: 2026-10-17
Status: roadmap step 5, built on feat/run-record. The default key (a
local ed25519 key per install) is implemented; whether records should instead
be signed by a per-researcher or per-organisation key is a pending decision,
recorded under "Open decisions".
Depends on: claims checking (claims.py), the citation lint
(cite.check, citations.py), the paper-research clients
(2026-09-10-paper-research-design.md), and AgentMarkit's brief scripts
(gather_sources.py, check_brief.py), which call the generic API below.
What this is for
Every check Attestation runs today answers in a terminal or a tool reply and then evaporates. A reviewer handed a draft cannot tell which papers the agent actually read, what it searched for, which version of which tool produced "12 supported, 1 contradicted", or whether the draft they hold is the one that was checked. A run record is one file that says all of that, that anyone can check without Attestation's database, and that breaks visibly if a verdict, an input or the signer is changed afterwards.
It records what happened. It does not prove that what happened was right: a
supported verdict in a valid record means "this tool, on these inputs, said
supported", never "the claim is true". The verifier says so in its output.
What a record holds
One in-toto Statement v1 (_type: https://in-toto.io/Statement/v1) with
predicateType: https://github.com/mgoldey/attestation/run-record/v1:
| Field | Holds |
|---|---|
subject[] |
The artifacts the record is about, as in-toto ResourceDescriptors (name, digest.sha256): the run's outputs, or, for a check with no output, the documents it checked. Never empty (in-toto requires one). |
predicate.run |
id (uuid4), started_at, finished_at (UTC ISO 8601, the signer's clock). |
predicate.tool |
name (attest claims, cite.check, research-brief...), attestation_version, git_sha and git_dirty when Attestation runs from a checkout, else null. |
predicate.models[] |
{name, role, endpoint?} for every model the run called. Empty for the deterministic checks, which call none. |
predicate.queries[] |
{source, query, hits}: each search, per source (openalex, pubmed, library, feed...), with the number of hits it returned. |
predicate.papers[] |
{id, title?, depth} for each paper actually read, depth one of full, abstract, title. |
predicate.inputs[] |
ResourceDescriptors with annotations.role (draft, results, bib, sources, other) and annotations.bytes: a sha256 of every input file. |
predicate.checks[] |
Generic check results (below). |
predicate.outputs[] |
ResourceDescriptors for the files the run produced. |
predicate.signer |
{alg: "ed25519", keyid, public_key}. Inside the signed payload, so a swapped key breaks the signature. |
name is a path relative to the run's root (the working directory unless the
caller passes another), so a reviewer who unpacks the same tree anywhere can
recompute every digest.
The generic check result
Every checker reports the same shape, so a new one (cite.support, a LaTeX check, an agent's own) plugs in without a schema change:
{"check": "claims", "target": "draft.md",
"summary": {"supported": 11, "contradicted": 1},
"results": [{"subject": "draft.md:42", "verdict": "contradicted",
"message": "run says 0.061", "data": {"metric": "wer", "value": 0.053}}]}
check names the checker, target the input name it judged (optional),
summary is derived from results when the record is built (never supplied by
the caller, so it cannot disagree with them), and data is free-form JSON for
whatever the checker wants a reader to see. verdict is a free string: the
claims checker's six verdicts, uncited for cite.check, and whatever
cite.support defines, without this module knowing any of them.
Format: DSSE envelope over a JCS-canonical in-toto Statement
Chosen: the record file is a DSSE envelope (payloadType:
application/vnd.in-toto+json, payload = base64 of the Statement,
signatures: [{keyid, sig}]), the payload being the Statement serialized as
RFC 8785 JCS. Signatures are ed25519 over DSSE's PAE encoding.
Why this and not the alternatives:
- DSSE over a detached
.sig. The ask was "a JSON file plus a detached signature". DSSE is a detached signature over exact payload bytes -- it just travels in the same file. One file is what gets emailed, attached to a manuscript or uploaded with a dataset; a pair gets separated, and the half that survives is unverifiable. DSSE also binds the payload type into what is signed (PAE), so a signature over a record cannot be replayed as a signature over some other JSON. It is the envelope in-toto, SLSA, Sigstore andcosign attestuse, so standard tooling reads it. - in-toto Statement over a home-grown wrapper. The product is called Attestation; the industry's word for "signed statement about artifacts, identified by digest" is an in-toto attestation. Using its Statement layer (subject + predicateType + predicate) costs nothing and means a policy engine that understands in-toto can at least route these records.
- A custom predicate over SLSA Provenance v1 or in-toto test-result.
SLSA Provenance describes a build (
buildDefinition,builder); a research run's verdicts, queries and reading depth would all be stuffed intobyproducts, and a SLSA verifier would then make claims about a build that never happened.test-resulthas only passed/warned/failed test names, which loses the verdict vocabulary that is this product's point. The predicate reuses in-toto's ResourceDescriptor for every file, so the part a generic tool cares about (what files, which digests) is standard. - JCS over "sorted keys, no whitespace". DSSE signs bytes, so
verification never needs canonical form. Canonical form buys stability:
the same record content always produces the same bytes, hence the same
digest, hence a record can be named by its hash and two builds compared
byte for byte.
json.dumps(sort_keys=True, separators=(",", ":"))is almost JCS but not quite: it sorts by code point where JCS sorts by UTF-16 code unit, and it prints1e-07,1.0and1e+16where JCS (ES6 number formatting) prints1e-7,1and10000000000000000. A verifier written in another language would then disagree on the bytes. JCS is ~60 lines of stdlib Python here, tested against the RFC's number vectors. Integers outside +/-2^53 and non-finite floats are refused rather than rounded.
Keys
Default: an ed25519 key per install, generated on first use at
$HERMES_HOME/attestation/keys/run-record-ed25519.pem (PKCS#8 PEM,
unencrypted), file mode 0600 in a 0700 directory, created with
O_EXCL so two processes racing on first use cannot both write one. A key
file readable by group or others is refused, the way ssh refuses one.
keyid is the hex sha256 of the raw 32-byte public key.
cryptography is added as a direct dependency. It was already in uv.lock
at 50.0.0 as a transitive dependency of the mcp stack, so every install
already has it: making it direct pins nothing new and adds no download.
Ed25519 over RSA/ECDSA: deterministic signatures (no nonce to get wrong),
32-byte keys that fit in the record, and the default of minisign, OpenSSH
and Sigstore's ed25519 path.
Pluggable: a KeySource is anything with signer() -> Signer. Two ship:
LocalKeySource(path)-- the default; generates on first use.ATTEST_RECORD_KEY=/path/key.pem-- an existing PKCS#8 ed25519 key, never generated (a typo must not silently mint a new identity).
attest record key prints the public key and keyid; --export pub.pem
writes the public key for a collaborator.
What a signature proves: the public key is embedded in the record, so by
default verification proves integrity (nothing changed since signing) and
consistency (it was signed by the key it names), not identity. Identity
needs the verifier to already trust a key: attest record verify --key
pub.pem or --keyid <hex> pins it, and a record signed by any other key
fails.
Verifying
attest record verify <file> [--root DIR] [--key pub.pem | --keyid HEX]
(also python -m attestation.run_record.verify, and the runs.verify_record
tool). verify.py imports only the standard library, cryptography, and
canonical.py -- no database, no model, no attestation.db -- so a reviewer
can run it from a bare install.
It reports, per item, one of ok, FAILED, or not checked (why):
- The envelope parses and its
payloadTypeis in-toto's. - The signature verifies against the embedded key (and the pinned key, if given).
- The payload is in canonical (JCS) form -- a non-canonical payload is a warning, not a failure: the signature still covers its exact bytes.
- The Statement has in-toto's
_type, ourpredicateType, and a non-empty subject. - Each
checks[].summaryequals the tally of its ownresults. - Every input, output and subject found under
--root(default: the current directory) hashes to its recorded digest; a missing file is "not checked", never a pass.
And it says what it cannot check: that verdicts are correct (re-running needs the ledger and library), that the listed papers were really read and the queries really run (the signer's word), and that the timestamps are true (no trusted timestamp authority). Exit 0 only when nothing FAILED.
Surface
- Python API (
attestation.run_record):RunRecorderaccumulates a run (add_input,add_output,add_query,add_paper,add_model,add_check) andsign()s it into an envelope;from_spec(dict)builds one from plain JSON, which is what other tools and agents call. - CLI:
attest claims ... --record out.json(checksclaimsandcite.check, sinceattest claimsruns the citation lint too; inputs are the Markdown files, the run artifacts each verdict read, and the.bibfiles the resolver consulted).attest record create --spec run.json --out rec.jsonfor any other tool (AgentMarkit's brief scripts emit the spec).attest record verify,attest record key. - MCP:
cite.check(path, record=...)writes a record of that check.runs.attest(spec, out)creates one from a spec;runs.verify_record(path, root)verifies. Both in a newmcp/run_record.pymodule, registered under therunsnamespace so the provenance surface serves them.
Writing refuses to overwrite an existing record (--force to replace),
matching runs.record's new-files-only rule.
Open decisions (for the user)
- Whose key signs. Default implemented: one key per install. Alternatives,
each a new
KeySource: a per-researcher key the researcher holds (portable across machines, needs a distribution story); an org key on a hosted AgentMarkit machine (records attest "AgentMarkit ran this", not "this person"); Sigstore keyless signing (identity from an OIDC login, public transparency log -- conflicts with the local-only guarantee unless opt-in). - Trusted time. Timestamps are the signer's clock. An RFC 3161 timestamp authority would make them evidence, at the cost of a network call.
- Where records live. Today they are written where the caller says.
An index of records in the DB (so
runs.detailcould list them) is deferred until something reads it. - Encrypting the key at rest. Unencrypted with 0600 perms, like an SSH key without a passphrase; a passphrase would break unattended agent runs.
Testing
tests/test_run_record.py: round trip (build, sign, write, verify);
tampering -- an edited verdict, an edited input file, a swapped embedded key,
a pinned wrong key, an edited summary -- each FAILS and names what failed;
JCS stability (RFC 8785 number vectors, UTF-16 key order, the same record
built twice is byte-identical except the run id and time, and key order in the
input does not change the bytes); key file permissions and refusal of a
world-readable key; the verifier module imports no sqlite3 and no
attestation.db; attest claims --record and the two MCP tools end to end on
a fixture workspace.