ingest

Deterministic feed ingest: fetch -> dedup -> clean -> embed -> store. No LLM.

content_hash(title, summary)

A dedup key for one item: title+summary, SHA-256. Two feeds syndicating the same paper hash identically without any cross-feed ID to rely on.

extract_ids(guid, url)

(doi, arxiv_id) from an entry's guid and url, or Nones.

Pure and network-free; used by migration 007's backfill and by every new item. Formats are the ones measured on the live database 2026-09-05: arXiv guids oai:arXiv.org:1003.0563v2, Nature URLs carrying the DOI suffix under the 10.1038 prefix. Anything else stays NULL.

run_ingest(conn, embedder, feeds_path, parse=feedparser.parse)

Fetch every registered feed, dedup, embed, and store -- deterministic throughout, per the module docstring; no LLM runs here.

Embedding happens in a pass separate from the dedup/store transaction (see the comment below _new_entries): the embed call is the slow HTTP round trip to the model server, and holding a DB lock across it would block every other reader and writer for that long. One feed's failure is counted and does not stop the others -- _ingest_outcome makes that call over the outcomes this loop collects.

strip_boilerplate(text)

Drop HTML tags and arXiv's own "Announce Type / Abstract:" preamble, collapsing whitespace -- so a stored summary is the actual abstract, not the feed entry's markup and boilerplate around it.

sync_feeds(conn, feeds_path)

Seed the feeds table from feeds.toml.

INSERT OR IGNORE, so this is a no-op for feeds already present: the database -- not the TOML file -- is the source of truth once seeded. Use attestation.feeds.add_source / remove_source to change the feed set.